ScriptBasic Developers Manual

by Peter Verhas

Table of Contents

[Contents]

1. Introduction

ScriptBasic is an open source scripting implementation of the programming language BASIC. The language and the implementation make the software a considerable choice in many cases when there is a need for some scripting tool. This software can be used where traditionally Perl, Python, TCL or some other scripting interpreter is used.

The language itself is BASIC with powerful command set and large number of extension modules. The language was designed so that this is familiar for all BASIC programmers no matter which dialect one has experience with. Thus the learning curve of the language is very steep: you can start to write your first programs just in few minutes.

The interpreter is built modular, well documented, and easy to read source code and has well defined and documented interfaces. It is easy to embed the interpreter into an application and it is also easy to write external modules extending the language. Applications embedding ScriptBasic can build virtual machines in multiple or in single process running interpreters simultaneously independent of each other, but they can also execute multi-thread applications running interpreter threads that provide features for the BASIC programs running parallel to communicate to each other.

This means that ScriptBasic can be the language of choice in situations when the application programmer wants to implement some programmability feature into the application. It will attract the users of the application because there is no need to learn a new programming language at the time when they have to learn the functions of the application anyway. The programmer on the other hand has an easy job to integrate ScriptBasic into the application because it was designed for the purpose.

To sum up these in a listing the language and the implementation has the following features:

[Contents]

1.1. Chapters

This documentation has four main chapters. These are

I advise you to read the architecture chapter first to get the overall picture. After that you can read one of the three middle chapters based on your plans. If you plan to embed the interpreter, read the second chapter. If you want to write an extension module, read the third chapter. If you intend to write a preprocessor, read the fourth chapter. You may want to do more than one even.

By now you may not need what the differences are. This will change after reading the architecture chapter.

The last chapter is to be read just before you start to experiment with ScriptBasic. This will give you guide lines how to compile the program from source. This may not be needed for extension module and preprocessor developers, but it does not do any harm either. It may help.

[Contents]

2. Interpreter Architecture

This chapter tells you the architecture of the interpreter. It is not a must to read this chapter, and you may find that some topic is irrelevant or not needed to learn to embed or to extend ScriptBasic. However understanding the internal working order of ScriptBasic should help you understand why some of the extending or embedding interfaces work the way they actually do. So I recommend that you read on and do not skip this chapter.

To read this chapter and to understand the internal working of the interpreter is vital when you decide to write an internal preprocessor. Internal preprocessors interact with not only the execution system but also the reader, lexer, syntaxer and builder modules along the way one after the other as they do their unique job processing a BASIC program, thus internal preprocessor writers have to understand how these modules work.

ScriptBasic is not a real scripting language. This is a mix of a scripting language and compiled languages. The language is scripting in the sense that it is very easy to write small programs, there is no need for long variable and function declarations. On the other hand the language is compiled into an internal code that is executed afterwards. This is the same or similar technique, which is used in the implementations of the language Java, Perl, Python and many other languages.

ScriptBasic as a language is a BASIC dialect that implements most BASIC features that BASIC implementations usually do. However ScriptBasic variables are not typed, and dynamic storage, like arrays are automatically allocated and released. A ScriptBasic variable can store a string, an integer, a real value or an array. Naturally a variable can not store more than one of any of these types of values. But you need not declare a variable to be INTEGER or REAL, or STRING. A variable may store a string at a time and the next assignment command may release the original value and store a different value in the variable.

When a program is executed it goes through several steps. The individual steps are implemented in different modules each being coded in a separate C language source file. These modules were developed so that they provide clear interface and thus could be replaced. The lexical analyzer uses the functions provided by the reader, the syntax analyzer uses the functions provided by the lexical analyzer and so on. The modules never dig into each others private area.

The modules are listed here with some explanation.

The following sections detail these modules and also some other modules that help these modules to perform their actual tasks.

[Contents]

2.1. External Preprocessor

The module that implements the external preprocessor handling is coded in the C source file `epreproc.c' (Note that there is a file `ipreproc.c' that handles internal preprocessors.) The sole function in this file is named epreproc. This is quite a complex function that gets a lot of arguments listing all the preprocessors that have to be executed one after the other on the source file or on the file created by the previous preprocessor in case there are more than one preprocessors to be applied in chains. The function gets a series of preprocessors to be executed in the command line but in case there is no preprocessor defined in the arguments it executes the preprocessors that have to be executed according to the BASIC program file name extension and the configuration file.

Only the preprocessors that are configured can be executed. Each preprocessor has a symbolic name, which is used to name it in the configuration file or on the command line using the option `-p'.

An external preprocessor is a program that reads a text file and produces a text file. ScriptBasic executes the external preprocessor in a separate process and supplies it with the input and the output file name on the command line. For example the `scriba.conf.lsp' configuration file may contain the following lines

preproc (
  extensions (
     heb "heb"
     )
  external (
    heb (
      executable "/usr/bin/scriba /usr/share/scriba/source/heber.bas"
      directory "/var/cache/scriba/hebtemp/"
      )    
    )
  )

The key executable defines the executable image of the preprocessor. In this case this is an executable image and an argument because the HEB (HTML Embedded Basic) preprocessor is written in BASIC. The other arguments, namely the input file name and the output file name are appended after this string separated by space. When the preprocessor is executed the actual command line is

/usr/bin/scriba /usr/share/scriba/source/heber.bas myprog.heb /var/cache/scriba/hebtemp/ADBKPLNBDELMMNKGNBBLHAHGBKKJFIMN

(The above sample command line may be split into two lines by the printing/displaying system, but this is essentially a single command line with the executable name and the two arguments.)

The final argument is a file name automatically generated by ScriptBasic using MD5 calculation of the input file name full path (may and presumably should not be correct in this example above as I just typed some random 32 characters). The preprocessor should read the file myprog.heb and should generate /var/ca@dots{MN} (forgive me for not typing that long file name again).

If there are more preprocessor to be executed on the generated result then the next is executed on the generated file as input file and another output file is generated.

The preprocessor program should gracefully exit the process it runs in and exit with the code 0. Any other process exit code is treated as error and the further processing of the BASIC program is aborted.

The sample code for the HEB preprocessor can be found in the ScriptBasic source distribution. Note that this preprocessor implementation is a sample and is not meant to be a professional, commercial grade preprocessor.

[Contents]

2.1.1. External preprocessor handling

=abstract This module starts the external preprocessors. =end

=toc

[Contents]

2.1.2. Execute external preprocessors

This function executes the external preprocessors that are needed to be executed either by the command line options or driven by the extensions.

The command line option preprocessors are executed as listed in the character array ppszArgPreprocessor. These preprocessors are checked to be run first.

If there is no preprocessors defined on the command line then the preprocessors defined in the config file for the extensions are executed. The input file name is also modified by the code. The input file name is modified so that it will contain the source code file name after the preprocessing.

The return value of the function is the error code. This is PREPROC_ERROR_SUCCESS if the preprocessing was successful. This value is zero. If the return value is positive this is one of the error codes defined in the file errcodes.def prefixed by PREPROC_.

int epreproc(ptConfigTree pCONF,
             char *pszInputFileName,
             char **pszOutputFileName,
             char **ppszArgPreprocessor,
             void *(*thismalloc)(unsigned int),
             void (*thisfree)(void *)
  )@{

The first argument pCONF is the configuration data pointer which is passed to the configuration handling routines.

The second argument pszInputFileName is the pointer to the pointer to the input file name.

The third argument is an output variable. This will point to the output file name upon success or to NULL. If this variable is NULL then an error has occured or the file needed no preprocessing. The two cases can be separated based on the return value of the function. If the file needed preprocessing and the preprocessing was successfully executed then this variable will point to a ZCHAR string allocated via the function thismalloc. This is the responsibility of the caller to deallocate this memory space after use calling the function pointed by thisfree.

The fourth argument ppszArgPreprocessor is an array of preprocessors to be used on the input file. This array contains pointers that point to ZCHAR strings. The ZCHAR strings contain the symbolic names of the external preprocessors that are defined in the configuration file. The configuration file defines the actual executable for the preprocessor and the temporary directory where the preprocessed file is stored. The final element of this pointer array should be NULL. If the pointer ppszArgPreprocessor is NULL or the pointer array pointed by this contains only the terminating NULL pointer then the extensions of the file name are used to determine what preprocessors are to be applied. Preprocessors are applied from left to right order of the file extensions.

The arguments thismalloc and thisfree should point to malloc and free or to a similar functioning function pair. These functions will be used via the myalloc.c module and also to allocate the new pszOutputFileName string in case of success. This means that the caller should use the function pointed by thisfree to release the string pointed by pszOutputFileName after the function has returned.

[Contents]

2.2. Reader

The module reader is implemented in the C source file `reader.c' This is a very simple module it just reads the lines from the source code files and stores the actual text in memory using linked lists. There is not too much possibility to configure this module except that the memory handling functions and the file opening, closing and reading functions are used via function pointers that can be altered by the caller.

Like any other module in ScriptBasic the reader module uses a module object. This is like a class definition except that the interpreter is coded in C and thus there is nothing like VTABLE or inheritance. Otherwise the code is object oriented. Here we list the actual definition of the reader object. Note however that this is actually a copy of the actual definition from the file `reader.c' and it may have been changed since I wrote this manual. So the reader object by the time I wrote this manual was:

#define BUFFER_INITIAL_SIZE 1024
#define BUFFER_INCREMENT 1024
 typedef struct _ReadObject {
  void * (*fpOpenFile)(char *, void *);
  int (*fpGetCharacter)(void *, void *);
  void (*fpCloseFile)(void *, void *);
  void *pFileHandleClass;

void *(*memory_allocating_function)(size_t, void *); void (*memory_releasing_function)(void *, void *); void *pMemorySegment;

ptConfigTree pConfig;

char *Buffer; long dwBuffer; long cBuffer;

pSourceLine Result; pSourceLine CurrentLine; long NextCharacterPosition; char fForceFinalNL;

pReportFunction report; void *reportptr; int iErrorCounter; unsigned long fErrorFlags;

pImportedFileList pImportList;

char *FirstUNIXline; struct _PreprocObject *pPREP; } ReadObject, *pReadObject ;

The pointers fpOpenFile, fpGetCharacter and fpCloseFile point to functions that are used to open the input file. The pointer pFileHandleClass set by the higher code using the module reader is passed to these functions without caring its meaning. This is not used by the standard file input/output functions that are used by the command line version of the program, but can be useful for program environment when the source file is stored in some other forms and not in a file. An example of such use can be seen in the function scriba_LoadProgramString implemented in the file `scriba.c'.

The linked list of source lines is stored in the structure named SourceLine The definition of this structure is

typedef struct _SourceLine {
  char *line;
  long lLineNumber;
  long LineLength;
  char *szFileName;
  struct _SourceLine *next;
  } SourceLine, *pSourceLine;

You can see that each source line is pointed by the field line and the length of the line is also stored. The reason for this extra field is that the line itself may contain zero character although this is rare for a program source file to contain zero character inside.

Before the file is read the function reader_InitStructure should be called. This is usual for the ScriptBasic modules. This function initializes the reader object to the usual values that actually ScriptBasic needs.

The reader provides function reader_ReadLines that actually reads the lines and also processes all lines that contain an include or import directive to include a line.

The reader has some extra functions that are specific to ScriptBasic or generally saying are specific to program source reading.

Source programs under UNIX usually start with a line

#! /usr/bin/scriba

that tells the operating system how to start the code.

((((Some very old and totally outdated version of some UNIX systems check the first four characters to look for #! /. There is a space between the ! and the /. So if you want to be look a real code geek put this extra space before the executable path. To be honest I have never encountered this issue since 1987 when I met my first UNIX at TU Delft, Hollandia. OK that's for the story, get back to the reader!))) Always close the parentheses you open!)

The reader recognizes this line if this is the very first line of the program and unlinks it from the list. Instead you can reach this line via the reader object variable FirstUNIXline. This is specific to program source reading and not general file reading. But the reader module is a program source reader or more specific: a BASIC, even ScriptBasic program source reader, though it was coded to be as general as possible.

Another specific issue is the new line at the end of the last line. Lines are usually terminated by new-line. This line terminating character is included in the string at the end of each element of the linked list the reader creates. However when the last line is terminated by the pure EOF some syntax analyzers may fail (ScriptBasic syntax analyzer is also an example, but the reader is kind to care about this). For the reason if the variable fForceFinalNL is set to be TRUE this line gets the extra new-line when read.

This module contains the functions that read a source file.

Script basic has several passes until it can start to execute the code. The very first pass is to read the source lines from the files. The routines in this module perform this task and build up a linked list that contains the ascii values of the lines.

The input functions are parametrized, and the caller should support. If you have different system dependent file reading functions, or if you have the input file in some format in memory or in any other data holding space you can support these routines with character fetch functions.

[Contents]

2.2.1. reader_IncreaseBuffer()

When the reader encounters a line which is longer than the currently allocated input buffer it calls this function to increase the size of the input buffer. The input buffer is linearly increased by BUFFER_INCREMENT size (defined in the header section of reader.c

When a new buffer is allocated the bytes from the old buffer are copied to the new and the old buffer is released. It is vital that the buffer is always referenced via the pRo->buffer pointer because resizing buffer does change the location of the buffer.

If the memory allocation fails the function return READER_ERROR_MEMORY_LOW error. Otherwise it returns zero.

int reader_IncreaseBuffer(pReadObject pRo
  )@{

[Contents]

2.2.2. reader_gets()

This function reads a newline terminated line from the file. The file is identified by function pRo->fpGetCharacter and the pointer fp.

When the input buffer is too small it automatically increases the buffer. The terminating new line is included in the buffer. If the last line of the file is not terminated by newline an extra newline character is added to this last line.

The addition of this extra newline character can be switched off setting pRo->fForceFinalNL to false. Even if this variable is false the normal newline characters which are present in the file are included in the buffer.

int reader_gets(pReadObject pRo,
                 void *fp
  )@{

[Contents]

2.2.3. reader_ReadLines()

This function calls reader_ReadLines_r() to read the lines of the file given by the file name szFileName into pRo->Result. For further information see reader_ReadLines_r().
int reader_ReadLines(pReadObject pRo,
                     char *szFileName
  )@{
The function returns zero or the error code.

[Contents]

2.2.4. reader_ReadLines_r()

This function reads the lines of a file and creates a linked list of the read lines.
int reader_ReadLines_r(pReadObject pRo,
                       char *szFileName,
                       pSourceLine *pLine
  )@{
The file is identified by its name given in the string variable szFileName. The file is opened by the function pointed by pRo->fpOpenFile This function should return a void pointer and this void pointer is passed to reader_gets() (reader_gets) to get a single character.

The argument pLine is a pointer to a SourceLine pointer. The linked list lines read will be chained into this pointer. The last read line will be followed by the line pointed by *pLine and *pLine will point to the first line.

This design makes it easy to use and elegant to perform file inclusions. The caller has to pass the address of the pointer field next of the source line after which the file is to be inserted.

See also ReadLines that calls this function.

[Contents]

2.2.5. reader_ProcessIncludeFiles()

This function is called from reader_ReadLines() after calling reader_ReadLines_r().

This function goes through all the lines and checks if there is any line containing an include directive.

An include directive is a line starting with a word INCLUDE (case insensitive) and is followed by the file name. The file name can be enclodes between double quotes.

Note that the processing of the include directives are done on the characters on the line, because they are processed before any tokenization of the lexer module. This can cause some problem only when there is an include like line inside a multiline string. For example:

a = """Hey this is a multiline string
include "subfile.txt"
"""

This will include the file subfile.txt and its content will become part of the string. This becomes more complicated when the file subfile.txt contains strings.

The file name may not be enclosed between double quotes. In this case the file is tried to be found in predefined system directories.

If the programmer uses the command IMPORT instead of INCLUDE the file will only be included if it was not included yet into the current program.

void reader_ProcessIncludeFiles(pReadObject pRo,
                                pSourceLine *pLine
  )@{

The file read is inserted into the plce where the include statement was.

[Contents]

2.2.6. reader_LoadPreprocessors()

Preprocessors are not part of ScriptBasic. They can be implemented as external DLLs and should be configured in the configuration file.

When a line contains

USE preprocessorname

this reader module loads the preprocessor DLL or SO (dll under unix) file.

void reader_LoadPreprocessors(pReadObject pRo,
                                 pSourceLine *pLine
  )@{

[Contents]

2.2.7. reader_StartIteration()

The package supports functions that help upper layer modules to iterate through the lines read. This function should be called to start the iteration and to set the internal iteration pointer to the first line.
void reader_StartIteration(pReadObject pRo
  )@{

[Contents]

2.2.8. reader_NextLine()

This function returns a string which is the next line during iteration. This function does NOT read anything from any file, only returns a pointer to a string that was already read.

This function can be used together with reader_NextCharacter(). When a line was partially passed to an upper layer that uses reader_NextCharacter this function will only return the rest of the line.

char *reader_NextLine(pReadObject pRo
  )@{

[Contents]

2.2.9. reader_NextCharacter()

This function gets the next character from the actual line, or gets the first character of the next line.

This function does NOT read anything from any file, only returns a character from a string that was already read.

When the last character of the last line was passed it return EOF

int reader_NextCharacter(void *p
  )@{

[Contents]

2.2.10. reader_FileName()

This function returns the file name of the actual line. This is the string that was used to name the file when it was opened. This can be different for different lines when the reader is called several times to resolve the "include" statements.
char *reader_FileName(void *p
  )@{

[Contents]

2.2.11. reader_LineNumber()

This function returns the line number of the current line durig iteration. This number identifies the line in the file where it was read from.

long reader_LineNumber(void *p
  )@{

[Contents]

2.2.12. reader_InitStructure()

This function should be called to initialize the reader structure. It sets the file handling routines to the standard fopen, fclose and getc functions, and also sets the function pointers so that the module uses malloc and free.

void reader_InitStructure(pReadObject pRo
  )@{

[Contents]

2.2.13. reader_RelateFile()

This function gets a file name, which is either absolute or relative to the current working directory and another file name which is absolute or relative to the first one.

The return value of the function is a file name which is either absolute or relative to the current woring directory.

The return value is dynamically allocated and is to be release by the caller. The allocation function is taken from the class function and the segment is pMemorySegment.

char *reader_RelateFile(pReadObject pRo,
                       char *pszBaseFile,
                       char *pszRelativeFile
  )@{

[Contents]

2.2.14. reader_DumpLines()

This is a debug function that prints the lines into a debug file.
void reader_DumpLines(pReadObject pRo,
                      FILE *fp
  )@{

[Contents]

2.3. Lexer

The module lexer is implemented in the C source file `lexer.c' This is a module that converts the read characters to a list of tokens. The lexer recognizes the basic lexical elements, like numbers, strings or keywords. It starts to read the characters provided by the reader and group it p into lexical elements. For example whenever the lexical analyzer sees a " character it starts to process a string until it finds the closing ". When it does the module creates a new token, links it to the end of the list and goes on.

To do this the lexical analyzer has to know what is a keyword, string or number.

Because general purpose, table driven lexical analyzers are usually rather slow ScriptBasic uses a proprietary lexical analyzer that is partially table driven, but not so general purpose as one created using the program LEX.

There are some rules that are coded into the C code of the lexical analyzer, while other are defined in tables. Even the rules coded into the C program are usually parameterized in the module object.

Lets see the module object definition from the file `lexer.c' (Note that the C .h header files are extracted from the .c files thus there is no need to double maintain function prototypes.)

Note however that this is actually a copy of the actual definition from the file `lexer.c' and it may have been changed since I wrote this manual. So the lexer object by the time I wrote this manual was:

typedef struct _LexObject {
  int (*pfGetCharacter)(void *);
  char * (*pfFileName)(void *);
  long (*pfLineNumber)(void *);
  void *pvInput;
  void *(*memory_allocating_function)(size_t, void *);
  void (*memory_releasing_function)(void *, void *);
  void *pMemorySegment;

char *SSC; char *SCC;

char *SFC; char *SStC; char *SKIP;

char *ESCS; long fFlag;

pReportFunction report; void *reportptr; int iErrorCounter; unsigned long fErrorFlags;

char *buffer; long cbBuffer;

pLexNASymbol pNASymbols; int cbNASymbolLength;

pLexNASymbol pASymbols;

pLexNASymbol pCSymbols; pLexeme pLexResult; pLexeme pLexCurrentLexeme; struct _PreprocObject *pPREP; }LexObject, *pLexObject;

This struct contains the global variables of the lexer module. In the first "section" of the structure you can see the variables that may already sound familiar from the module reader. These parameterize the memory allocation and the input source for the module. The input functions are usually set so that the characters come from the module reader, but there is no principal objection to use other character source for the purpose.

The variable pvInput is not altered by the module. It is only passed to the input functions. The function pointer name pfGetCharacter speaks for itself. It is like getc returns the next character. However when this function pointer is set to point to the function reader_NextCharacter the input is already preprocessed a bit. Namely the include and import directives were processed.

This imposes some interesting feature that you may recognize now if you read the reader module and this module definition carefully. include and import works inside multi-line strings. (OK I did not talk about multi-line strings so far so do not feel ashamed if you did not realize this.)

The function pointers pfFileName and pfLineNumber should point to functions that return the file name and the line number of the last read character. This is something that a getc will not provide, but the reader functions do. This will allow the lexical analyzer to store the file name and the line number for each token.

The next group of variables seems to be frightening and unreadable at first, but here is this book to explain them. These variables define what is a string, a symbol, what has to be treated as unimportant space and so on. Usually symbols start with alpha character and are continued with alphanumeric characters in most programming languages. But what is an alpha character? Is _ one or is $ a valid alphanumeric character. Well, for the lexer module if any of these characters appear in the variable SSC then the answer is yes. The name stands for Symbol Start Characters. But lets go through all these variables one by one.

The default values for these variables are set in the function lex_InitStructure. Interestingly these default values are perfectly ok for ScriptBasic.

The field pNASymbols points to an array that contains the non-alpha symbols list. Each element of this array contains a string that is the textual representation of the symbol and a code, which is the token code of the symbol. For example the table NASYMBOLS in file `syntax.c' is:

LexNASymbol NASYMBOLS[] = { { "@\\" , CMD_EXTOPQN } , { "@`" , CMD_EXTOPQO } , { "@'" , CMD_EXTOPQP } , { "@" , CMD_EXTOPQQ } ,

...

{ "@" , CMD_EXTOPQ } , { "^" , CMD_POWER } , { "*" , CMD_MULT } , { NULL, 0 } };

When the lexical analyzer finds something that is not a string, number or alphanumeric symbol it tries to read forward and recognize any of the non-alpha tokens listed in this table. It is extremely important that the symbols are ordered in this table so that the longer symbols come first thus a symbol abc is not presented before abcd. Otherwise abcd will never be found!

The variable cbNASymbolLength is nothing to care about. This is used internally and is calculated automatically by the lexical analyzer.

The variable pASymbols is similar to the variable pNASymbols pointing to a same kind of table. This variable however should point to an array that contains the alphanumeric symbols. You can find the array ASYMBOLS in file `syntax.c' that is pointed by this variable for ScriptBasic.

The order of the words in this array is not important except that more frequent words being listed earlier result faster compilation.

The field pCSymbols points to an array that is used only for debugging purposes. I mean debugging ScriptBasic code itself and not debugging BASIC programs.

The rest of the variables are used by the functions that iterate through the list of tokens when the syntax analyzer reads the token list or to report errors during lexical analysis. Error reporting is detailed in a separate section.

The tables that list the lexical elements are not maintained "by hand". The source for ScriptBasic syntax is maintained in the file `syntax.def' and the program `syntaxer.pl' creates the C syntax file `syntax.c' from the syntax definition.

The program `syntaxer.pl' is so complex that after two years I wrote it I had hard time to understand it and I rather treat it as a holly code: blessed and untouchable. (Ok: see: that code is quite compound, but if there was any bug found in that I could understand what I did in a few hours. Anyway, the brain created that code once belonged to me.)

[Contents]

2.3.1. Functions implemented in this module

The following subsections list the functions that are implemented in this module. The source text of this documentation was extracted from the source Documentation embedded in the C source as comment. Treat these subsections more as reference documentation and less tutorial like.

This module contains the functions and structures that are used by ScriptBasic to perform lexical analysis of the source code. The module was originally developed for ScriptBasic but was developed to be general enough to be used in other projects.

[Contents]

2.3.1.1. lex_SymbolicName()

This function usually is for debug purposes. This searches the table of the predefined symbols and returns the string which is the predefined symbols for which the code was passsed.
char *lex_SymbolicName(pLexObject pLex,
                       long OpCode
  )@{

[Contents]

2.3.1.2. lex_HandleContinuationLines()

This function is called from the main function before syntax analysis is started. This function handles the usual basic continuation lines. If the last character on a line is a _ character, which is either recognised during lexical analysis as a character or as a symbol then this lexical element and the following new-line character token is removed from the list of tokens.

void lex_HandleContinuationLines(pLexObject pLex
  )@{

[Contents]

2.3.1.3. lex_RemoveSkipSymbols()

This function is called from lex_DoLexicalAnalysis() to remove the lexical elements from the list of tokens that were denoted by the preprocessors to be deleted.

Some lexical elements are used to give information to some of the preprocessors. These tokens should be deleted, because later processing can not deal with them and confuses syntax analysis.

In those cases the preprocessor should set the type of the token to be LLEX_T_SKIP or LEX_T_SKIP_SYMBOL. The type LEX_T_SKIP should be used in case the token is handled due to ProcessLexSymbol preprocessor command and LEX_T_SKIP otherwise.

When the type is set LEX_T_SKIP_SYMBOL the lexical analyzer knows to release the string holding the symbol. If the type is LEX_T_SKIP only the token record is released.

If the symbol string is not released due to erroneously setting the type to LEX_T_SKIP instead LEX_T_SKIP_SYMBOL the memory will not be released until the interpreter finishes pre execution steps. So usually if you do not know how to set the type to skip a token LEX_T_SKIP is safe.

void lex_RemoveSkipSymbols(pLexObject pLex
  )@{

[Contents]

2.3.1.4. lex_RemoveComments()

This function called from the function lex_DoLexicalAnalysis() function to remove the comments before the syntax analysis starts.

It should be called before calling the continuation line handling because usually REM lines are not continuable

void lex_RemoveComments(pLexObject pLex
  )@{

[Contents]

2.3.1.5. lex_NextLexeme()

Use this function during iteration to get the next lexeme from the list of lexemes.

void lex_NextLexeme(pLexObject pLex
  )@{

[Contents]

2.3.1.6. lex_SavePosition()

Use this function to save the current position of the iteration. This is neccessary during syntactical analysis to return to a certain position when syntactical analysis fails and the program has to go back and try a different command syntax.

void lex_SavePosition(pLexObject pLex,
                      pLexeme *ppPosition
  )@{

The second argument is a pLexeme * type variable that holds the position and should be passed as argument to the function lex_RestorePosition().

[Contents]

2.3.1.7. lex_RestorePosition()

Use this function to restore the lexeme position that was saved calling the function lex_SavePosition()

void lex_RestorePosition(pLexObject pLex,
                         pLexeme *ppPosition
  )@{

[Contents]

2.3.1.8. lex_StartIteration()

You should call this function when the list of lexemes was built up before starting the iteration of the syntax analyzer. This function sets the iteration pointer to the first lexeme.

void lex_StartIteration(pLexObject pLex
  )@{

[Contents]

2.3.1.9. lex_EOF()

Call this function to check if the iteration has reached the last lexeme.

int lex_EOF(pLexObject pLex
  )@{

[Contents]

2.3.1.10. lex_Type()

During lexeme iteration this function can be used to retrieve the typeof the current lexeme. The type of a lexeme can be: