Contents of Article


Working with File Transforms and XFORM Macros

Basic concepts of Files Transforms

Enabling and disabling the use of XFORM macros

Maintaining the integrity of the XFORM session

The XFORM READ action

The XFORM WRITE action

Impact of XFORM on STATE information

Using XFORM for validation of file changes

Importing and exporting external data

Using XFORM with View, Browse or Read-Only files

Using external programs to process XFORM data

Finding strings in an XFORM session using XFIND

A cautionary note on using XFORM

Sample XFORM macros



Introduction


SPFLite introduces an advanced editing concept: File Transforms and XFORM Macros. This section will explain the concepts of File Transforms, and introduces XFORM macros, which implement the needed support.


A full explanation of XFORM macros and how to write and use them can be found in  A Tutorial on writing XFORM Macros.


This is a new concept, and while it is believed to satisfy the needed requirements, not all possible use cases have been explored or tested. If you feel changes or improvements would enhance it,  be sure to report them on the SPFLite user forum at https://spflite.freeforums.net/ so we can try to resolve the issue.


As this is new technology, it must be stressed that, until you are comfortable with this technique, and you have fully tested your XFORM macro(s), you should ensure you have adequate backups, or operate in "test mode" with copies of your original data. That sounds a lot like being in "beta test mode", and at first, since the XFORM macros are your creation, that's just what it will be. 


A file transform is the process, by which the format of a file is copied and temporarily changed during the loading of a file, to making viewing and editing of the file easier, or to make it possible to view and edit a file that otherwise could not be edited at all. If a SAVE is issued, the file transform process is reversed, and the data is written back to the original file.


The primary reasons to use a file transform are:

    • to access files in ways you ordinarily wouldn't or couldn't do
    • to access files that cannot be directly edited by SPFLite


There are many reasons why editing some files with SPFLite may be inconvenient:


    • Some files may be highly structured and technical. 
    • The file may not be ‘readable’ by normal means.
    • The data may be a complex format, or need conversion to be editable.
    • The file may include check sums or has encrypted or password-protected data.
    • You might also need to look at, and possibly modify, files that are not "text" at all, such as EXE and DLL files.


SPFLite alone cannot handle such editing requirements. To address these complex types of edits, something more is needed. That is what a File Transform is for.


Basic concepts of File Transforms


To understand where File Transform and XFORM ‘fit in’ to augment SPFLite normal processing, it is helpful to consider what happens during an ordinary edit session.


During File OPEN


    • User selects a file for EDIT, using an EDIT primary command or from a File Manager screen
    • SPFLite starts up an edit session
    • SPFLite reads the file
    • SPFLite stores the file into memory
    • The lines of data are made available for editing


During File SAVE


    • SPFLite writes the lines in the edit session to the external file
    • If STATE ON is in effect, STATE data, such as labels, tags, excluded-line markers, color highlighting, and other "attributes" of the file are written to the STATE data for this file. 


A File Transform alters the Standard READ and Standard WRITE actions of an edit session, as discussed above.  The items highlighted above will be performed by your XFORM macro.


An XFORM macro, like all macros in SPFLite, is written in thinBasic, and is stored in the SPFLite MACROS directory. It may have any name you wish. 


During  File Transforms, your XFORM macro will be replacing those standard READ and WRITE actions, SPFLite itself will perform NO file I/O operations.  As such, your XFORM macro must utilize several of the file-based functions provided by thinBasic. To find the documentation you need for this, do the following:


    • On an SPFLite command line, type HELP BASIC. This will bring up the Help file for thinBasic
    • Scroll down the index on the left side of the document, until you see a major heading of ThinBasic Modules. (Don't go to ThinBasic Functions / File functions. That's the wrong part of the document.)
    • Expand the ThinBasic Modules heading, and scroll down to the topic File. On the File topic, click on [+] if present to expand it.
    • ThinBasic is very modular, before using the functions in any of the optional modules, you must tell thinBasic which you will be using. This is done with a USES command.  So you need to insert a USES "File" statement in your XFORM macro.  Then, you can use whatever  functions you require that are provided in the FILE module, such as OPEN, CLOSE, READ, WRITE, GET ATTRIBUTES, and so on. 


A good naming convention for XFORM macros is to prefix the name with something unique, such as X or X_ so they are easy to find in the MACROS directory when you create and edit them. The sample XFORM macros supplied with SPFLite have a prefix of X, such as XHEX.MACRO.


Enabling and disabling the use of XFORM macros


XFORM processing is optional, and is quite flexible in how you may or may not want it invoked. You can, for a particular file-type:

    • Specify all XFORM activity as disabled.  This is the default.
    • Have a particular XFORM macro used for all I/O activity (ON mode).
    • Specify an XFORM macro name, but leave it in dormant (or OFF mode).
    • Override the normal XFORM setting for a single Edit. The override can:
    • Override the specified XFORM name with an alternate.
    • De-activate an XFORM which would normally be invoked. (Force OFF)
    • Activate an XFORM which was established in ‘dormant (OFF) mode. (Force ON)
    • Activate a specific XFORM for a file which normally doesn’t use XFORM.

To specify the use of an XFORM macro by changing a file's PROFILE setting, you can do one of the following:

    • Issue a PROFILE EDIT  while editing the type of file to which you wish to add XFORM support. 
    • Issue PROFILE EDIT extension-name from a command line to add XFORM support to a specific file type.
    • In File Manager, click on Configs, which brings up a list of all profiles. Click on the profile name of interest, which will bring up the Profile Editor dialog.
    • In an edit session, use the XFORM primary command to set, modify or display the XFORM settings for the file type of the file you are editing.


The XFORM command has the following syntax:


XFORM  [ xname | NONE ]   [ ON | OFF ]


If  XFORM is entered with no parameters, it will simply display the current XFORM settings. The xname is the name of the macro to be used, where the macro is in the standard MACROS directory, with a file name of xname.MACRO. 


The default for all file types is initially XFORM NONE.  When you specify an xname, SPFLite will check to ensure the xname.MACRO exists, and if it doesn't, it will issue a warning. You can still make the setting; this is just a reminder for you to eventually define the macro when you have a chance. 


The ON and OFF settings allow you to specify whether XFORM processing is to be used all or most of the time (when you'd use ON) or when XFORM will be used seldom if ever (when you'd use OFF).  The ON and OFF settings can always be overridden for specific Edits (see below).


To override the normal XFORM settings on an EDIT (or similar) primary command,  use an overriding XFORM parameter. This parameter may be one of the following, and must begin with a / slash:


    • /xname - used to enable an XFORM macro named xname. This will override any xname that might be defined in the profile. Use the /xname option for a file that rarely uses XFORM processing, or when you want to override an existing xname that might be in effect. 

    • /NONE or /OFF - used to disable XFORM processing, regardless of how the XFORM Profile option is set. 

    • /ON - used to enable XFORM processing. For this to work, the XFORM Profile option must already have an xname defined. It is an error to use /ON if the XFORM profile has an xname of NONE, since SPFLite will not know what macro to use. 



Maintaining the integrity of the XFORM session


An XFORM edit is an unusual process, SPFLite is yielding control of the READ and WRITE processes to you, something that has not been supported before now. This requires you to cooperate by using the support appropriately.  i.e. Follow the recommended procedures:


    • If you are in the middle of editing a file using XFORM, don't change the XFORM settings prior to saving the file. Wait until you have saved it, before changing its XFORM settings.


    • If you issue a SAVEAS, SPFLite changes the active filename of the current session.  If this change also altered the file type, then the Profile for the new type will be loaded.  If that Profile contains different XFORM settings. This opens the possibility for a variety of conflicts. In simple terms – don’t do this!


    • Likewise, if you issue a RENAME primary command, don't change the file extension on the file name, for the same reasons as for SAVEAS.


    • Eventually, SPFLite might be able to carefully inspect the effects of a SAVEAS or RENAME and figure out the appropriate action to take.  That is a long-term development effort, and still may not be able to resolve all issues.  So for now , please understand that you have to "work within the system" for everything to function properly.


The XFORM READ action


The "XFORM READ" action operates as follows:


    • User selects a file for EDIT, using an EDIT command or from a File Manager screen, ensuring that an XFORM xname is defined in the Profile, or supplied for the edit by an overriding XFORM option
    • SPFLite starts up an edit session
    • SPFLite launches the XFORM macro, providing it with the user-selected file name, and a parameter of "READ" to identify this as an "XFORM READ" action
    • The XFORM macro opens and reads the file, using thinBasic FILE operations
    • The XFORM macro re-formats the data however it wishes
    • The XFORM macro stores the reformatted data as text lines in the edit session
    • The XFORM macro closes the file, using thinBasic FILE operations
    • The XFORM macro passes back a return code indicating completion of the READ action
    • The lines of data are made available for editing


Keep in mind that when the description above says things like, "the XFORM macro reads the file", it means that is what SPFLite expects it to do.  What the macro actually does is entirely up to the macro writer, so long as it load records into the Edit session. 


A clever user might use the XFORM protocols to use XFORM macros for something totally unrelated to File Transforms. As long as you obey the protocols, what you actually do in the macro is up to you. It's hard to imagine what that might be, but SPFLite has some pretty smart users out there. One of these days, those users just might surprise us.


The XFORM WRITE action


The "XFORM WRITE" action operates as follows:


    • The XFORM macro opens the external file for output, using thinBasic FILE operations
    • The XFORM macro scans the data lines in the edit session, assembles and re-formats them as required, and the data is written to the external file on disk
    • STATE data for the file is NOT written to the STATE data for this file, whether STATE ON is in effect or not. (See Impact of XFORM on STATE information, below.)
    • The XFORM macro closes the file, using thinBasic FILE operations
    • The XFORM macro passes back a return code indicating completion of the WRITE action
    • If the "XFORM WRITE action" was initiated by an END command, the edit session is closed and the tab it was located in is dismissed.



Impact of XFORM on STATE information


An XFORM macro does its own Read and Write operations, using functions built into the thinBasic script language, rather than having SPFLite do it. That is the whole point of a File Transform. So when an XFORM WRITE action occurs, SPFLite has no opportunity to "jump in" and save the file's STATE data. Thus  any STATE information that might have existed beforehand is considered to be invalidated. There is no other way this can be handled. The old STATE data can no longer be relied upon.


If you are going to edit files using XFORM, and the macro saves the data, you are going to lose any STATE data that might have existed beforehand. Because of this limitation, most users are probably going to operate with STATE OFF for files processed via XFORM.


SPFLite decides how to handle this situation by examining the return code provided by the XFORM macro in it’s HALT statement. The return code must be one of the following:


    • 0:  XFORM WRITE action was successful, the file has been written to disk. SPFLite will delete any existing STATE information for the file. If this is a simple completion, you can set the message text of the HALT statement to a "" null string, or blanks. to prevent a message from getting displayed on the screen. 


    • 8:  XFORM WRITE action failed for some reason. The exact reason should be contained in the message text of the HALT statement.  SPFLite will not delete any existing STATE information, and if the SAVE was triggered by END, it will not close the session.



Using XFORM for validation of file changes


Suppose you are using XFORM to edit some file that has a complex, strict data format which must be maintained, yet you want to be able to edit it. As the data in the Edit session has to be re-processed into the original data format, the XFORM WRITE function must obviously validate the data during the WRITE process.  If an error is detected, the WRITE function should format an appropriate message and issue it via a HALT(8, error-message)


This will signal SPFLite that the file has not been written, and it will return to normal Edit mode and display the error message.  The user can correct the condition, and re-issue the END/SAVE command.


The extent of this validation is entirely up to the XFORM macro writer.  It should be thorough to prevent saving bad data back to a file and possibly destroying it.


Importing and exporting external data


Various other SPFLite commands move data in and out of a session to external files.  Commands such as COPY, CUT, PASTE, CREATE, REPLACE.   For all such commands, an active XFORM macro is NOT INVOKED to perform the WRITE, all I/O is handled normally by SPFLite.  


XFORM READ is only invoked for EDIT, BROWSE, VIEW and RELOAD.  


XFORM WRITE is only invoked for SAVEAS and SAVE/END.


If you attempt to do one of these other WRITE actions, SPFLite will issue a message notifying you that XFORM is active, and the copy will be done without invoking  the XFORM macro. It does that just as a reminder, and does not constitute an "error" condition. The message will give you an opportunity to cancel the command in case it was not what you intended.


Because COPY, PASTE, CREATE and REPLACE work this way, you can think of the text data you are working on as "the" file. The text becomes your "interface" to the real data. Being able to export and import data in this way allows you to conveniently save "pieces" of the original file in an easy and much more manageable, text-oriented way, rather than the possible-binary format of the original file. You may use this fact to devise clever and innovative ways to manage your data.


Using XFORM with View, Browse or Read-Only files


While an XFORM macro can be used to read a file into a VIEW or BROWSE session (or the file could simply be Read-Only), the XFORM WRITE function will never be invoked since SPFLite does not allow SAVE commands to be issued in that type of session.


Using external programs to process XFORM data


thinBasic, the script engine used by SPFLite, is remarkably fast and efficient. If your XFORM-processed files are not excessively large, and the transformation processing not overly complicated, the performance of XFORM macros should be more than sufficient for most users.


However, you may choose to, or need to process the data with an external utility application.


To do that, in your XFORM macro, you would use the SPFLite supplied functions SPF_Exec and/or SPF_Shell to launch the program(s) you needed. This would probably involve using temporary files to pass data between the external programs and your XFORM macro.  But your XFORM macro will still ultimately end up passing text records into the Edit session.


Finding strings in an XFORM session using XFIND


As a development separate from XFORM support itself, SPFLite will supply a macro called XFIND.MACRO. The purpose of this macro is to find strings in data that was originally in binary form, such as in EXE and DLL files, even when such data "straddles" one or more "lines" in an XFORM edit session.


Development of XFIND is in progress. Details about the XFIND macro will be released as soon as it is available.


A cautionary note on using XFORM


Advanced users might try to use the XFORM technology to access files they normally can't access, such as EXE and DLL files. If you do this merely to look at those files, that is fine. However, if you create an XFORM procedure to modify such binary files, there is a saying for that: "Be careful what you wish for; you might get it."


Modifying such files requires detailed technical knowledge of their structure. The best advice we can give you is, be sure you know what you're doing. If you don't, you could destroy those binary files. Even if the change you intended is seemingly saved "successfully", you may still run into problems. 


Sample XFORM macros


SPFLite supplies several sample XFORM macros, to help give you a guide as to writing your own, and you might wish to use some of them as-is if you find them useful. As samples, they illustrate techniques, useful functions you might need, and so on. There is no claim that these are the only way or most optimal way to accomplish the task at hand.


These macros represent the known "best practices" at the time they were written. Since XFORM is a new technology, these macros may be revised over time as XFORM support evolves.  To ensure you are up to date with the latest techniques, come back frequently to the SPFLite user forum, ask questions and offer suggestions if you have them, and keep current with new SPFLite releases. 


Here are the currently defined sample XFORM macros:


XCFG.MACRO         Shows access to a non-standard file type (SQL). Intended to read the Profile Names from the SPFLite CFG file


Xhex.MACRO        Provides a basic Hex file editor


Xlines.MACRO        Produces a dump format display of text lines. Demonstrates how to deblock RECFM U F and V format files


XRW.MACRO        Provides Read / Write access to a normal ANSI Text file


Xzip.MACRO        Shows access to a non-standard file type (ZIP). Produces a list of ZIP file contents

Created with the Personal Edition of HelpNDoc: Produce Kindle eBooks easily