NAME

flexc++api - Application programmer’s interface of flexc++ generated classes

DESCRIPTION

Flexc++(1) was designed after flex(1) and flex++(1). Like these latter two programs flexc++ generates code performing pattern-matching on text, possibly executing actions when certain regular expressions are recognized.

Refer to flexc++(1) for a general overview. This manual page covers the Application Programmer’s Interface of classes generated by flexc++, offering the following sections:

o

1. INTERACTIVE SCANNERS: how to create an interactive scanner

o

2. THE CLASS INTERFACE: SCANNER.H: Constructors and members of the scanner class generated by flexc++

o

3. NAMING CONVENTION: symbols defined by flexc++ in the scanner class.

o

4. CONSTRUCTORS: constructors defined in the scanner class.

o

5. PUBLIC MEMBER FUNCTION: public member declared in the scanner class.

o

6. PRIVATE MEMBER FUNCTIONS: private members declared in the scanner class.

o

7. SCANNER CLASS HEADER EXAMPLE: an example of a generated scanner class header

o

8. THE SCANNER BASE CLASS: the scanner class is derived from a base class. The base class is described in this section

o

9. PUBLIC ENUMS AND -TYPES: enums and types declared by the base class

o

10. PROTECTED ENUMS AND -TYPES: enumerations and types used by the scanner and scanner base classes

o

11. NO PUBLIC CONSTRUCTORS: the scanner base class does not offer public constructors.

o

12. PUBLIC MEMBER FUNCTIONS: several members defined by the scanner base class have public access rights.

o

13. PROTECTED CONSTRUCTORS: the base class can be constructed by a derived class. Usually this is the scanner class generated by flexc++.

o

14. PROTECTED MEMBER FUNCTIONS: this section covers the base class member functions that can only be used by scanner class or scanner base class members

o

15. PROTECTED DATA MEMBERS: this section covers the base class data members that can only be used by scanner class or scanner base class members

o

16. FLEX++ TO FLEXC++ MEMBERS: a short overview of frequently used flex(1) members that received different names in flexc++.

o

17. THE CLASS INPUT: the scanner’s job is completely decoupled from the actual input stream. The class Input, nested within the scanner base class handles the communication with the input streams. The class Input, is described in this section.

o

18. INPUT CONSTRUCTORS: the class Input can easily be replaced by another class. The constructor-requirements are described in this section.

o

19. REQUIRED PUBLIC MEMBER FUNCTIONS: this section covers the required public members of a self-made Input class

UNDERSCORES

Starting with version 2.07.00 flexc++ reserved identifiers no longer end in two underscore characters, but in one. This modification was necessary because according to the C++ standard identifiers having two or more consecutive underscore characters are reserved by the language. In practice this could require some minor modifications of existing source files using flexc++’s facilities, most likely limited to changing StartCondition__ into StartCondition_ and changing PostEnum__ into PostEnum_.

The complete list of affected names is:

Enums:

Member functions:

Protected data members:

1. INTERACTIVE SCANNERS

An interactive scanner is characterized by the fact that scanning is postponed until an end-of-line character has been received, followed by reading all information on the line, read so far. Flexc++ supports the %interactive directive), generating an interactive scanner. Here it is assumed that Scanner is the name of the scanner class generated by flexc++.

Caveat: generating interactive and non-interactive scanners should not be mixed as their class organizations fundamentally differ, and several of the Scanner class’s members are only available in the non-interactive scanner. As the Scanner.h file contains the Scanner class’s interface, which is normally left untouched by flexc++, flexc++ cannot adapt the Scanner class when requested to change the interactivity of an existing Scanner class. Because of this support for the --interactive option was discontinued at flexc++’s 1.01.00 release.

The interactive scanner generated by flexc++ has the following characteristics:

o

The Scanner class is derived privately from std::istringstream and (as usual) publicly from ScannerBase.

o

The istringstream base class is constructed by its default constructor.

o

The function lex’s default implementation is removed from Scanner.h and is implemented in the generated lex.cc source file. It performs the following tasks:

- If the token returned by the scanner is not equal to 0 it is returned as then next token;

- Otherwise the next line is retrieved from the input stream passed to the Scanner’s constructor (by default std::cin). If this fails, 0 is returned.

- A ’\n’ character is appended to the just read line, and the scanner’s std::istringstream base class object is re-initialized with that line;

- The member lex_ returns the next token. This implementation allows code calling Scanner::lex() to conclude, as usual, that the input is exhausted when lex returns 0.

Here is an example of how such a scanner could be used:

    // scanner generated using ’flexc++ lexer’ with lexer containing 


    // the %interactive directive


    int main()


    {


        Scanner scanner;        // by default: read from std::cin 


    


        while (true)


        {


            cout << "? ";       // prompt at each line


            while (true)        // process all the line’s tokens


            {


                int token = scanner.lex();


    


                if (token == ’\n’)  // end of line: new prompt


                    break;


    


                if (token == 0)     // end of input: done


                    return 0;


    


                                    // process other tokens


                cout << scanner.matched() << ’\n’;


                if (scanner.matched()[0] == ’q’)


                    return 0;


            }


        }


    }


        

2. THE CLASS INTERFACE: SCANNER.H

By default, flexc++ generates a file Scanner.h containing the initial interface of the scanner class performing the lexical scan according to the specifications given in flexc++’s input file. The name of the file that is generated can easily be changed using flexc++’s --class-header option. In this man-page we’ll stick to using the default name.

The file Scanner.h is generated only once, unless an explicit request is made to rewrite it (using flexc++’s --force-class-header option).

The provided interface is very light-weight, primarily offering a link to the scanner’s base class (see this manpage’s sections 8 through 16).

Many of the facilities offered by the scanner class are inherited from the ScannerBase base class. Additional facilities offered by the Scanner class. are covered below.

3. NAMING CONVENTION

All symbols that are required by the generated scanner class end in an underscore character (e.g., executeAction_). These names should not be redefined. As they are part of the Scanner and ScannerBase class their scope is immediately clear and confusion with identically named identifiers elsewhere is unlikely.

Some member functions do not use the underscore convention. These are the scanner class’s constructors, or names that are similar or equal to names that have historically been used (e.g., length). Also, some functions are offered offering hooks into the implementation (like preCode). The latter category of function also have names that don’t end in underscores.

4. CONSTRUCTORS

o

explicit Scanner(std::istream &in = std::cin, std::ostream &out = std::cout, bool keepCwd = true) This constructor by default reads information from the standard input stream and writes to the standard output stream. When the Scanner object goes out of scope the input and output files are closed.

By default (keepCwd == true) the directory that was active when the scanner was constructed is made current again once the scanning process ends (i.e., when lex_(), see below, returns 0).

When keepCwd == false the scanner doesn’t reset its working directory to the directory that was current when the scanner was constructed. This may be convenient in situations where a program repeatedly calls switchStream(), followed by full lexical scans of the switched-to streams.

With interactive scanners input stream switching or stacking is not available; switching output streams, however, is.

o

Scanner(std::string const &infile, std::string const &outfile, bool keepCwd = true) This constructor opens the input and output streams whose file names were specified. When the Scanner object goes out of scope the input and output files are closed. If outfile == "-" then the standard output stream is used as the scanner’s output medium; if outfile == "" then the standard error stream is used as the scanner’s output medium.

The parameter keepCwd is used as with the previous constructor.

This constructor is not available with interactive scanners.

5. PUBLIC MEMBER FUNCTIONS

o

int lex() The lex function performs the lexical scanning of the input file specified at construction time (but also see section 6.1. for information about intermediate stream-switching facilities). It returns an int representing the token associated with the matched regular expression. The returned value 0 indicates end-of-file. Considering its default implementation, it could be redefined by the user. Lex’s default implementation merely calls lex_:

inline int Scanner::lex()
{


    return lex_();
}


        

Caveat: with interactive scanners the lex function is defined in the generated lex.cc file. Once flexc++ has generated the scanner class header file this scanner class header file isn’t automatically rewritten by flexc++. If, at some later stage, an interactive scanner must be generated, then the inline lex implementation must be removed `by hand’ from the scanner class header file. Likewise, a lex member implementation (like the above) must be provided `by hand’ if a non-interactive scanner is required after first having generated files implementing an interactive scanner.

6. PRIVATE MEMBER FUNCTIONS

o

int lex_() This function is used internally by lex and should not otherwise be used.

o

int executeAction_() This function is used internally by lex and should not otherwise be used.

o

void preCode() By default this function has an empty, inline implementation in Scanner.h. It can safely be replaced by a user-defined implementation. This function is called by lex_, just before it starts to match input characters against its rules: preCode is called by lex_ when lex_ is called and also after having executed the actions of a rule which did not execute a return statement. The outline of lex_’s implementation looks like this:

int Scanner::lex_()
{


    ...


    preCode();


    while (true)


    {


        size_t ch = get_();            // fetch next char


        ...


        switch (actionType_(range))    // determine the action


        {


            ... maybe return


        }


        ... no return, continue scanning


        preCode();


    } // while
}


        

o

void postCode(PostEnum_ type) By default this function has an empty, inline implementation in Scanner.h. It can safely be replaced by a user-defined implementation. This function is called by lex_, just after a rule has been matched. Values of the enum class PostEnum_ indicate the characteristic of the matched rule. PostEnum_ has four values: PostEnum_::END, PostEnum_::POP, PostEnum_::RETURN, and PostEnum_::WIP. Refer to section 10 for their meanings.

o

void print() When the --print-tokens or %print-tokens directive is used this function is called to display, on the standard output stream, the tokens returned and text matched by the scanner generated by flexc++.

Displaying is suppressed when the lex.cc file is (re)generated without using this directive. The function actually showing the tokens (ScannerBase::print_) is called from print, which is defined in-line in Scanner.h. Calling ScannerBase::print_, therefore, can also easily be controlled by an option controlled by the program using the scanner object.

7. SCANNER CLASS HEADER EXAMPLE

#ifndef Scanner_H_INCLUDED_
#define Scanner_H_INCLUDED_
// $insert baseclass_h
#include "Scannerbase.h"
// $insert classHead
class Scanner: public ScannerBase
{


    public:


        explicit Scanner(std::istream &in = std::cin,


                                std::ostream &out = std::cout);


        Scanner(std::string const &infile, std::string const &outfile);


        


        // $insert lexFunctionDecl


        int lex();


    private:


        int lex_();


        int executeAction_(size_t ruleNr);


        void print();


        void preCode();     // re-implement this function for code that must 


                            // be exec’ed before the patternmatching starts


        void postCode(PostEnum_ type);    


                            // re-implement this function for code that must 


                            // be exec’ed after the rules’s actions.
};
// $insert scannerConstructors
inline Scanner::Scanner(std::istream &in, std::ostream &out)
:


    ScannerBase(in, out)
{}
inline Scanner::Scanner(std::string const &infile, std::string const &outfile)
:


    ScannerBase(infile, outfile)
{}
// $insert inlineLexFunction
inline int Scanner::lex()
{


    return lex_();
}
inline void Scanner::preCode() 
{


    // optionally replace by your own code
}
inline void Scanner::postCode(PostEnum_ type) 
{


    // optionally replace by your own code
}
inline void Scanner::print() 
{


    print_();
}
#endif // Scanner_H_INCLUDED_


        

8. THE SCANNER BASE CLASS

By default, flexc++ generates a file Scannerbase.h containing the interface of the base class of the scanner class also generated by flexc++. The name of the file that is generated can easily be changed using flexc++’s --baseclass-header option. In this man-page we use the default name.

The file Scannerbase.h is generated at each new flexc++ run. It contains no user-serviceable or extensible parts. Rewriting can be prevented by specifying flexc++’s --no-baseclass-header option).

9. PUBLIC ENUMS AND -TYPES

o

enum class StartCondition_ This strongly typed enumeration defines the names of the start conditions (i.e., mini scanners). It at least contains INITIAL, but when the %s or %x directives were used it also contains the identifiers of the mini scanners declared by these directives. Since StartCondition_ is a strongly typed enum its values must be preceded by its enum name. E.g.,

    begin(StartCondition_::INITIAL);


        

10. PROTECTED ENUMS AND -TYPES

o

enum class ActionType_ This strongly typed enumeration is for internal use only.

o

enum Leave_ This enumeration is for internal use only.

o

enum class PostEnum_ Values of this strongly typed enumeration are passed to the scanner’s private member postCode, indicating the scanner’s action after matching a rule. The values of this enumeration are:

PostEnum_::END: the function lex_ immediately returns 0 once postCode returns, indicating the end of the input was reached;

PostEnum_::POP: the end of an input stream was reached, and processing continues with the previously pushed input stream. In this case the function lex_ doesn’t return, it simply coontinues processing the previously pushed stream;

PostEnum_::RETURN: the function lex_ immediately returns once postCode returns, returning the next token;

PostEnum_::WIP: the function lex_ has matched a non-returning rule, and continues its rule-matching process.

11. NO PUBLIC CONSTRUCTORS

There are no public constructors. ScannerBase is a base class for the Scanner class generated by flexc++. ScannerBase only offers protected constructors.

12. PUBLIC MEMBER FUNCTIONS

o

std::string const &cwd() const return the working directory that was active when the Scanner’s constructor is called. It is also the program’s current working directory after constructing a Scanner object and when its lex member returns.

o

bool debug() const returns true if --debug or %debug was specified, otherwise false.

o

bool interactiveLine() this member is only available with interactive scanners. All remaining contents of the current interactive line buffer is discarded, and the interactive line buffer is filled with the contents of the next input line. This member can be used when a condition is encountered which invalidates the remaining contents of a line. Following a call to interactiveLine the next token that is returned by the lexical scanner will be the first token on the next line. This member returns true if the next line is available and false otherwise.

o

std::string const &filename() const returns the name of the file currently processed by the scanner object.

o

size_t length() const returns the length of the text that was matched by lex. With flex++ this function was called leng.

o

size_t lineNr() const returns the line number of the currently scanned line. This function is always available (note: flex++ only offered a similar function (called lineno) after using the %lineno option).

o

std::string const &matched() const returns the text matched by lex (note: flex++ offers a similar member called YYText).

o

void setDebug(bool onOff) Switches on/off debugging output by providing the argument true or false. Switching on debugging output only has visible effects if the debug option was specified.

o

void switchIstream(std::string const &infilename) The currently processed input stream is closed, and processing continues at the stream whose name is specified as the function’s argument. This is not a stack-operation: after processing infilename processing does not return to the original stream.

This member is not available with interactive scanners.

o

void switchOstream(std::ostream &out) The currently processed output stream is closed, and new output is written to out.

o

void switchOstream(std::string const &outfilename)

The current output stream is closed, and output is written to outfilename. If this file already exists, it is rewritten.

o

void switchStreams(std::istream &in, std::ostream &out = std::cout) The currently processed input and output streams are closed, and processing continues at in, writing output to out. This is not a stack-operation: after processing in processing does not return to the original stream.

This member is not available with interactive scanners.

o

void switchStreams(std::string const &infilename, std::string const &outfilename) The currently processed input and output streams are closed, and processing continues at the stream whose name is specified as the function’s first argument, writing output to the file whose name is specified as the function’s second argument. This latter file is rewritten. This is not a stack-operation: after processing infilename processing does not return to the original stream. If outfilename == "-" then the standard output stream is used as the scanner’s output medium; if outfilename == "" then the standard error stream is used as the scanner’s output medium.

If outfilename == "-" then the standard output stream is used as the scanner’s output medium; if outfilename == "" then the standard error stream is used as the scanner’s output medium.

This member is not available with interactive scanners.

13. PROTECTED CONSTRUCTORS

o

ScannerBase(std::string const &infilename, std::string const &outfilename, bool keepCwd = true) The scanner object opens and reads infilename and opens (rewrites) and writes outfilename. It is called from the corresponding Scanner constructor.

By default (keepCwd == true) the directory that was active when ScannerBase was constructed is made current again once the scanning process ends (i.e., when Scanner::lex_() returns 0).

When keepCwd == false the scanner doesn’t reset its working directory to the directory that was current when the ScannerBase was constructed. This may be convenient in situations where a program repeatedly calls switchStream(), followed by full lexical scans of the switched-to streams.

This constructor is not available for interactive scanners.

o

ScannerBase(std::istream &in, std::ostream &out, bool keepCwd = true) The in and out parameters are, respectively, the derived class constructor’s input stream and output streams.

The parameter keepCwd is used as with the previous constructor.

14. PROTECTED MEMBER FUNCTIONS

All member functions ending in an underscore character are for internal use only and should not be called by user-defined members of the Scanner class.

The following members, however, can safely be called by members of the generated Scanner class:

o

void accept(size_t nChars = 0) accept(n) returns all but the first `nChars’ characters of the current token back to the input stream, where they will be rescanned when the scanner looks for the next match. So, it matches `nChars’ of the characters in the input buffer, rescanning the rest. This function effectively sets length’s return value to nChars (note: with flex++ this function was called less);

o

void begin(StartCondition_ startCondition) activate the regular expression rules associated with StartCondition_ startCondition. As this enumeration is a strongly typed enum the StartCondition_ scope must be specified as well. E.g.,

        begin(StartCondition_::INITIAL);


            

o

void echo() const The currently matched text (i.e., the text returned by the member matched) is inserted into the scanner object’s output stream;

o

void leave(int retValue) actions defined in the lexical scanner specification file may or may not return. This frequently results in complicated or overlong compound statements, blurring the readability of the specification file. By encapsulating the actions in a member function readability is enhanced. However, frequently a compound statement is still required, as in:

    regex-to-match  {


                        if (int ret = memberFunction())


                            return ret;


                    }


            

The member leave removes the need for constructions like the above. The member leave can be called from within member functions encapsulating actions performed when a regular expression has been matched. It ends lex, returning retValue to its caller. The above rule can now be written like this:

    regex-to-match  memberFunction();


            

and memberFunction could be implemented as follows:

    void memberFunction()


    {


        if (someCondition())


        {                           // any action, e.g., 


                                    // switch mini-scanner


            begin(StartCondition_::INITIAL);


            leave(Parser::TOKENVALUE);    // lex returns TOKENVALUE


            // this point is never reached


        }


    


        pushStream(d_matched);      // switch to the next stream


                                    // lex continues


    }


            

The member leave should only (indirectly) be called (usually nested) from actions defined in the scanner’s specification s; calling leave outside of this context results in undefined behavior.

o

void more() the matched text is kept and will be prefixed to the text that is matched at the next lexical scan;

o

std::ostream &out() returns a reference to the scanner’s output stream;

o

bool popStream() closes the currently processed input stream. If the stream stack (cf. the member streamStack()) contains multiple elements then the most recently pushed element is removed and processing continues with the then final element of the streamStack. If this switch was successfully performed true is returned. If the stream stack contains only one element then all information has been processed and false is returned. This member is called automatically at the end of the currently processed input stream. After completely processing the initial input file the member lex returns 0.

When the <<EOF>> pattern is used in the scanner’s specification file popStream is not automatically called. In that case it should explicitly be called in the <<EOF>> pattern’s action block, returning 0 when it returns false. E.g.,

    <<EOF>> {


                if (not popStream())


                    return 0;


                cerr << "WIP on " << streamStack().size() << 


                    "file(s)\n";


            }


    

o

void push(size_t ch) character ch is pushed back onto the input stream. I.e., it will be the character that is retrieved at the next attempt to obtain a character from the input stream;

o

void push(std::string const &txt) the characters in the string txt are pushed back onto the input stream. I.e., they will be the characters that are retrieved at the next attempt to obtain characters from the input stream. The characters in txt are retrieved from the first character to the last. So if txt == "hello" then the ’h’ will be the character that’s retrieved next, followed by ’e’, etc, until ’o’;

o

void pushStream(std::istream &curStream) this function pushes curStream on the stream stack;

This member is not available with interactive scanners.

o

void pushStream(std::string const &curName) same, but the stream curName is opened first, and the resulting istream is pushed on the stream stack. If curName does not exist a std::exception is thrown. An exception is also thrown when the stream stack size exceeds (by default) 10 (the option max-depth can be used to change the default).

This member is not available with interactive scanners.

o

void redo(size_t nChars = 0) this member acts like accept but its argument counts backward from the end of the matched text. All but these nChars characters are kept and the last nChar characters are rescanned. This function effectively reduces length’s return value by nChars;

o

void setFilename(std::string const &name) this function sets the name of the stream returned by filename to name;

o

void setMatched(std::string const &text) this function stores text in the matched text buffer. Following a call to this function matched returns text.

o

StartCondition_ startCondition() const returns the currently active start condition (mini scanner);

o

std::vector<StreamStruct> const &streamStack() const returns the vector of currently stacked input streams. Starting with flexc++ version 2.08.00 the size and content of the information in this vector has been changed. Starting with version 2.08.00 its size is at least 1. Its size is incremented at every pushStream call, and decremented at every poStream call.

The StreamStruct itself is a struct having only one documented member: std::string const &pushedName, containing the absolute pathname of the initial and pushed streams.

Internally, streamStack vector is used as a stack. The pathname of the file that was specified at the scanner’s construction time is found at index position 0, and pathname of the file that’s currently being processed is found at streamStack().back(). Switching input streams changes the pathname at streamStack().back(), and if the pathname of a stream cannot be determined (which happens when constructing a scanner from or switching to a std::istream) then the program’s current working directory is not altered and pushedName contains (istream).

This member is not available with interactive scanners.

15. PROTECTED DATA MEMBERS

All protected data members are for internal use only, allowing lex_ to access them. All of them end in an underscore character.

16. FLEX++ TO FLEXC++ MEMBERS

17. THE CLASS INPUT

Flexc++ generates a file Scannerbase.h defining the scanner class’s base class, by default named ScannerBase (which is the name used in this man-page). The base class ScannerBase contains a nested class Input having this interface:

class Input
{


    public:


        Input();


        Input(std::istream *iStream, size_t lineNr = 1);


        size_t get();


        size_t lineNr() const;          


        size_t nPending() const;          


        void setPending(size_t nPending);          


        void reRead(size_t ch);


        void reRead(std::string const &str, size_t fmIdx);


        void close();
};


        

The members of this class are all required and offer an implementation level between the operations of ScannerBase and flexc++’s actual input file that’s being processed.

By default, flexc++ provides the implementation of all of Input’s required members. Therefore, in most situations this section of this man-page can safely be ignored.

However, users may define and extend their own Input class, providing flexc++’s base class with their own Input class. To do so flexc++’s rules file must contain the following two directives:

       %input-interface      = "interface"


       %input-implementation = "sourcefile"


        

Avoid using standard extension for interface and sourcefile to prevent confusion by the compiler when using program maintenance utilities: the interface specifies the interface of the class Input, like the one shown aboove, and that interface is then inserted in ScannerBase’s class as a nested class. Short, one-line implementations can be provided in-class. The sourcefile should contain standard (non-inline) implementations of members which must all be defined as members which are nested inside the surrounding class ScannerBase. E.g., the implementation of the member Input::reRead(size_t ch) must look like this:

       void ScannerBase::Input::reRead(size_t ch)


       {


           .... your implementation here


       }


        

As an advice: when implementing your own Input class start with the default class declaration and implementation as found in ScannerBase.h and lex.cc, and modify that default class to the class you need. Note that the interface of your class Input must at least offer the same members and constructors of the default provided class input, but it may define additional data members and/or member functions if required by your implementation (see also the following two section for descriptions of the class Input’s constructors and required members.

By default the class Input is defined in ScannerBase’s private section. When providing your own implementation the class Input is declared and defined in ScannerBase’s protected section so its members can also be accessed by the derived class Scanner. Moreover, in that situation the (otherwise private) ScannerBase data member Input *d_input is also declared in ScannerBase’s protected section.

When the lexical scanner generated by flexc++ switches streams using the //include directive (see also section 2. FILE SWITCHING) in the flexc++input(7) man page), then the input stream that’s currently processed is pushed on an Input stack maintained by ScannerBase, and processing continues at the file named at the //include directive. Once the latter file has been processed, the previously pushed stream is popped off the stack, and processing of the popped stream continues. This implies that Input objects must be `stack-able’. Its interface must always be designed in such a way that it satisfies this requirement.