NAME

re2c - generate fast lexical analyzers for C/C++, Go and Rust

SYNOPSIS

Note: examples are in C++ (but can be easily adapted to C).

re2c    [ OPTIONS ] [ WARNINGS ] INPUT
re2go   [ OPTIONS ] [ WARNINGS ] INPUT
re2rust [ OPTIONS ] [ WARNINGS ] INPUT

Input can be either a file or - for stdin.

INTRODUCTION

re2c works as a preprocessor. It reads the input file (which is usually a program in the target language, but can be anything) and looks for blocks of code enclosed in special-form comments. The text outside of these blocks is copied verbatim into the output file. The contents of the blocks are processed by re2c. It translates them to code in the target language and outputs the generated code in place of the block.

Here is an example of a small program that checks if a given string contains a decimal number:

// re2c $INPUT -o $OUTPUT -i --case-ranges
#include <assert.h>
bool lex(const char *s) {


    const char *YYCURSOR = s;


    /*!re2c


        re2c:yyfill:enable = 0;


        re2c:define:YYCTYPE = char;


        number = [1-9][0-9]*;


        number { return true; }


        *      { return false; }


    */
}
int main() {


    assert(lex("1234"));


    return 0;
}

In the output everything between /*!re2c and */ has been replaced with the generated code:

/* Generated by re2c */
// re2c $INPUT -o $OUTPUT -i --case-ranges
#include <assert.h>
bool lex(const char *s) {


    const char *YYCURSOR = s;


    
{


    char yych;


    yych = *YYCURSOR;


    switch (yych) {


        case '1' ... '9': goto yy2;


        default: goto yy1;


    }
yy1:


    ++YYCURSOR;


    { return false; }
yy2:


    yych = *++YYCURSOR;


    switch (yych) {


        case '0' ... '9': goto yy2;


        default: goto yy3;


    }
yy3:


    { return true; }
}
}
int main() {


    assert(lex("1234"));


    return 0;
}

SYNTAX

A re2c program consists of a sequence of blocks intermixed with code in the target language. There are three main kinds of blocks:

There are also many auxiliary blocks; see section blocks and directives for a full list of them. A block may contain the following kinds of statements:

PROGRAM INTERFACE

The generated code interfaces with the outer program with the help of primitives -- symbolic names that can be defined as variables, functions or macros in the target language (collectively referred to as the API). The definition of primitives is left for the user: this gives them both freedom in customizing the lexer and responsibility to understand how it works. Not all primitives have to be defined --- only those used by a given program. The manual provides definitions for the most popular use cases. For a full list of primitives and their meaning see the API primitives section.

There are two API flavors that define the set of primitives used by re2c:

There are two API styles that determine the form in which the primitives should be defined:

/*!re2c


  re2c:define:YYPEEK       = "*cursor";


  re2c:define:YYSKIP       = "++cursor;";


  re2c:define:YYBACKUP     = "marker = cursor;";


  re2c:define:YYRESTORE    = "cursor = marker;";


  re2c:define:YYBACKUPCTX  = "ctxmarker = cursor;";


  re2c:define:YYRESTORECTX = "cursor = ctxmarker;";


  re2c:define:YYRESTORETAG = "cursor = ${tag};";


  re2c:define:YYLESSTHAN   = "limit - cursor < @@{len}";


  re2c:define:YYSTAGP      = "@@{tag} = cursor;";


  re2c:define:YYSTAGN      = "@@{tag} = NULL;";


  re2c:define:YYSHIFT      = "cursor += @@{shift};";


  re2c:define:YYSHIFTSTAG  = "@@{tag} += @@{shift};";
*/

#define  YYPEEK()                 *cursor
#define  YYSKIP()                 ++cursor
#define  YYBACKUP()               marker = cursor
#define  YYRESTORE()              cursor = marker
#define  YYBACKUPCTX()            ctxmarker = cursor
#define  YYRESTORECTX()           cursor = ctxmarker
#define  YYRESTORETAG(tag)        cursor = tag
#define  YYLESSTHAN(len)          limit - cursor < len
#define  YYSTAGP(tag)             tag = cursor
#define  YYSTAGN(tag)             tag = NULL
#define  YYSHIFT(shift)           cursor += shift
#define  YYSHIFTSTAG(tag, shift)  tag += shift

For YYFILL definition and instructions how to customize or disable end-of-input checks see the handling the end of input and buffer refilling sections.

OPTIONS

Some of the options have corresponding configurations, others are global and cannot be changed after re2c starts reading the input file. Debug options generally require building re2c in debug configuration. Internal options are useful for experimenting with the algorithms used in re2c.

WARNINGS

Warnings can be invividually enabled, disabled and turned into an error.

BLOCKS AND DIRECTIVES

Below is the list of re2c directives (syntactic constructs that mark the beginning and end of the code that should be processed by re2c). Named blocks were added in re2c version 2.2. They are exactly the same as unnamed blocks, except that the name can be used to reference a block in other parts of the program. More information on each directive can be found in the related sections.

API PRIMITIVES

Here is a list of API primitives that may be used by the generated code in order to interface with the outer program. Which primitives are needed depends on multiple factors, including the complexity of regular expressions, input representation, buffering, the use of various features and so on. All the necessary primitives should be defined by the user in the form of macros, functions, variables, free-form pieces of code, or any other suitable form. re2c does not (and cannot) check the definitions, so if anything is missing or defined incorrectly the generated code will not compile.

CONFIGURATIONS

REGULAR EXPRESSIONS

re2c uses the following syntax for regular expressions:

Character classes and string literals may contain the following escape sequences: \a, \b, \f, \n, \r, \t, \v, \\, octal escapes \ooo and hexadecimal escapes \xhh, \uhhhh and \Uhhhhhhhh.

HANDLING THE END OF INPUT

One of the main problems for the lexer is to know when to stop. There are a few terminating conditions:

The first two conditions terminate the lexer in a "natural" way: it comes to a state with no outgoing transitions, and the matching automatically stops. The third condition, end of input, is different: it may happen in any state, and the lexer should be able to handle it. Checking for the end of input interrupts the normal lexer workflow and adds conditional branches to the generated program, therefore it is necessary to minimize the number of such checks. re2c supports a few different methods for handling the end of input. Which one to use depends on the complexity of regular expressions, the need for buffering, performance considerations and other factors. Here is a list of methods:

The following subsections contain an example of each method.

// re2c $INPUT -o $OUTPUT
#include <assert.h>
// Expect a null-terminated string.
static int lex(const char *YYCURSOR) {


    int count = 0;


    for (;;) {


    /*!re2c


        re2c:define:YYCTYPE = char;


        re2c:yyfill:enable = 0;


        *      { return -1; }


        [\x00] { return count; }


        [a-z]+ { ++count; continue; }


        [ ]+   { continue; }


    */


    }
}
int main() {


    assert(lex("") == 0);


    assert(lex("one two three") == 3);


    assert(lex("f0ur") == -1);


    return 0;
}

// re2c $INPUT -o $OUTPUT
#include <assert.h>
// Expect a null-terminated string.
static int lex(const char *str, unsigned int len) {


    const char *YYCURSOR = str, *YYLIMIT = str + len, *YYMARKER;


    int count = 0;


    for (;;) {


    /*!re2c


        re2c:define:YYCTYPE = char;


        re2c:yyfill:enable = 0;


        re2c:eof = 0;


        str = ['] ([^'\\] | [\\][^])* ['];


        *    { return -1; }


        $    { return count; }


        str  { ++count; continue; }


        [ ]+ { continue; }


    */


    }
}
#define TEST(s, r) assert(lex(s, sizeof(s) - 1) == r)
int main() {


    TEST("", 0);


    TEST("'qu\0tes' 'are' 'fine: \\'' ", 3);


    TEST("'unterminated\\'", -1);


    return 0;
}

// re2c $INPUT -o $OUTPUT
#include <assert.h>
#include <stdlib.h>
#include <string.h>
/*!max:re2c*/
static int lex(const char *str, unsigned int len) {


    // Make a copy of the string with YYMAXFILL zeroes at the end.


    char *buf = (char*) malloc(len + YYMAXFILL);


    memcpy(buf, str, len);


    memset(buf + len, 0, YYMAXFILL);


    const char *YYCURSOR = buf, *YYLIMIT = buf + len + YYMAXFILL;


    int count = 0;
loop:


    /*!re2c


        re2c:api:style = free-form;


        re2c:define:YYCTYPE = char;


        re2c:define:YYFILL  = "goto fail;";


        str = ['] ([^'\\] | [\\][^])* ['];


        [\x00] {


            // Check that it is the sentinel, not some unexpected null.


            if (YYCURSOR - 1 == buf + len) goto exit; else goto fail;


        }


        str  { ++count; goto loop; }


        [ ]+ { goto loop; }


        *    { goto fail; }


    */
fail:


    count = -1;
exit:


    free(buf);


    return count;
}
#define TEST(s, r) assert(lex(s, sizeof(s) - 1) == r)
int main() {


    TEST("", 0);


    TEST("'qu\0tes' 'are' 'fine: \\'' ", 3);


    TEST("'unterminated\\'", -1);


    TEST("'unexpected \0 null\\'", -1);


    return 0;
}

// re2c $INPUT -o $OUTPUT
#include <assert.h>
#include <stdlib.h>
#include <string.h>
static int lex(const char *str, unsigned int len) {


    // For the sake of example create a string without terminating null.


    char *buf = (char*) malloc(len);


    memcpy(buf, str, len);


    const char *cur = buf, *lim = buf + len, *mar;


    int count = 0;


    for (;;) {


    /*!re2c


        re2c:yyfill:enable = 0;


        re2c:eof = 0;


        re2c:api = custom;


        re2c:api:style = free-form;


        re2c:define:YYCTYPE = char;


        re2c:define:YYLESSTHAN = "cur >= lim";


        re2c:define:YYPEEK = "cur < lim ? *cur : 0";  // fake null


        re2c:define:YYSKIP = "++cur;";


        re2c:define:YYBACKUP = "mar = cur;";


        re2c:define:YYRESTORE = "cur = mar;";


        str = ['] ([^'\\] | [\\][^])* ['];


        *    { count = -1; break; }


        $    { break;; }


        str  { ++count; continue; }


        [ ]+ { continue; }


    */


    }


    free(buf);


    return count;
}
#define TEST(s, r) assert(lex(s, sizeof(s) - 1) == r)
int main() {


    TEST("", 0);


    TEST("'qu\0tes' 'are' 'fine: \\'' ", 3);


    TEST("'unterminated\\'", -1);


    return 0;
}

BUFFER REFILLING

The need for buffering arises when the input cannot be mapped in memory all at once: either it is too large, or it comes in a streaming fashion (like reading from a socket). The usual technique in such cases is to allocate a fixed-sized memory buffer and process input in chunks that fit into the buffer. When the current chunk is processed, it is moved out and new data is moved in. In practice it is somewhat more complex, because lexer state consists not of a single input position, but a set of interrelated positions:

Not all these are used in every case, but if used, they must be updated by YYFILL. All active positions are contained in the segment between token and cursor, therefore everything between buffer start and token can be discarded, the segment from token and up to limit should be moved to the beginning of buffer, and the free space at the end of buffer should be filled with new data. In order to avoid frequent YYFILL calls it is best to fill in as many input characters as possible (even though fewer characters might suffice to resume the lexer). The details of YYFILL implementation are slightly different depending on which EOF handling method is used: the case of EOF rule is somewhat simpler than the case of bounds-checking with padding. Also note that if -f --storable-state option is used, YYFILL has slightly different semantics (described in the section about storable state).

               <-- shift -->


             >-A------------B---------C-------------D#-----------E->


             buffer       token    marker         limit,


                                                  cursor
>-A------------B---------C-------------D------------E#->


             buffer,  marker        cursor        limit


             token


               <-- shift -->


             >-A------------B---------C-------------D#--E (EOF)


             buffer       token    marker         limit,


                                                  cursor
>-A------------B---------C-------------D---E#........


             buffer,  marker       cursor limit


             token

// re2c $INPUT -o $OUTPUT
#include <assert.h>
#include <stdio.h>
#include <string.h>
#define BUFSIZE 4095
struct Input {


    FILE *file;


    char buf[BUFSIZE + 1], *lim, *cur, *mar, *tok; // +1 for sentinel


    bool eof;
};
static int fill(Input &in) {


    if (in.eof) return 1;


    const size_t shift = in.tok - in.buf;


    const size_t used = in.lim - in.tok;


    // Error: lexeme too long. In real life could reallocate a larger buffer.


    if (shift < 1) return 2;


    // Shift buffer contents (discard everything up to the current token).


    memmove(in.buf, in.tok, used);


    in.lim -= shift;


    in.cur -= shift;


    in.mar -= shift;


    in.tok -= shift;


    // Fill free space at the end of buffer with new data from file.


    in.lim += fread(in.lim, 1, BUFSIZE - used, in.file);


    in.lim[0] = 0;


    in.eof = in.lim < in.buf + BUFSIZE;


    return 0;
}
static int lex(Input &in) {


    int count = 0;


    for (;;) {


        in.tok = in.cur;


    /*!re2c


        re2c:api:style = free-form;


        re2c:define:YYCTYPE  = char;


        re2c:define:YYCURSOR = in.cur;


        re2c:define:YYMARKER = in.mar;


        re2c:define:YYLIMIT  = in.lim;


        re2c:define:YYFILL   = "fill(in) == 0";


        re2c:eof = 0;


        str = ['] ([^'\\] | [\\][^])* ['];


        *    { return -1; }


        $    { return count; }


        str  { ++count; continue; }


        [ ]+ { continue; }


    */


    }
}
int main() {


    const char *fname = "input";


    const char content[] = "'qu\0tes' 'are' 'fine: \\'' ";


    // Prepare input file: a few times the size of the buffer, containing


    // strings with zeroes and escaped quotes.


    FILE *f = fopen(fname, "w");


    for (int i = 0; i < BUFSIZE; ++i) {


        fwrite(content, 1, sizeof(content) - 1, f);


    }


    fclose(f);


    int count = 3 * BUFSIZE; // number of quoted strings written to file


    // Initialize lexer state: all pointers are at the end of buffer.


    Input in;


    in.file = fopen(fname, "r");


    in.cur = in.mar = in.tok = in.lim = in.buf + BUFSIZE;


    in.eof = 0;


    // Sentinel (at YYLIMIT pointer) is set to zero, which triggers YYFILL.


    in.lim[0] = 0;


    // Run the lexer.


    assert(lex(in) == count);


    // Cleanup: remove input file.


    fclose(in.file);


    remove(fname);


    return 0;
}

               <-- shift -->                 <-- need -->


             >-A------------B---------C-----D-------E---F--------G->


             buffer       token    marker cursor  limit
>-A------------B---------C-----D-------E---F--------G->


             buffer,  marker cursor               limit


             token


               <-- shift -->                 <-- need -->


             >-A------------B---------C-----D-------E-F        (EOF)


             buffer       token    marker cursor  limit
>-A------------B---------C-----D-------E-F###############


             buffer,  marker cursor                   limit


             token                        <- YYMAXFILL ->

// re2c $INPUT -o $OUTPUT
#include <assert.h>
#include <stdio.h>
#include <string.h>
/*!max:re2c*/
#define BUFSIZE (4096 - YYMAXFILL)
struct Input {


    FILE *file;


    char buf[BUFSIZE + YYMAXFILL], *lim, *cur, *tok;


    bool eof;
};
static int fill(Input &in, size_t need) {


    if (in.eof) return 1;


    const size_t shift = in.tok - in.buf;


    const size_t used = in.lim - in.tok;


    // Error: lexeme too long. In real life could reallocate a larger buffer.


    if (shift < need) return 2;


    // Shift buffer contents (discard everything up to the current token).


    memmove(in.buf, in.tok, used);


    in.lim -= shift;


    in.cur -= shift;


    in.tok -= shift;


    // Fill free space at the end of buffer with new data from file.


    in.lim += fread(in.lim, 1, BUFSIZE - used, in.file);


    // If read less than expected, this is end of input => add zero padding


    // so that the lexer can access characters at the end of buffer.


    if (in.lim < in.buf + BUFSIZE) {


        in.eof = true;


        memset(in.lim, 0, YYMAXFILL);


        in.lim += YYMAXFILL;


    }


    return 0;
}
static int lex(Input &in) {


    int count = 0;


    for (;;) {


        in.tok = in.cur;


    /*!re2c


        re2c:api:style = free-form;


        re2c:define:YYCTYPE  = char;


        re2c:define:YYCURSOR = in.cur;


        re2c:define:YYLIMIT  = in.lim;


        re2c:define:YYFILL   = "if (fill(in, @@) != 0) return -1;";


        str = ['] ([^'\\] | [\\][^])* ['];


        [\x00] {


            // Check that it is the sentinel, not some unexpected null.


            return in.tok == in.lim - YYMAXFILL ? count : -1;


        }


        str  { ++count; continue; }


        [ ]+ { continue; }


        *    { return -1; }


    */


    }
}
int main() {


    const char *fname = "input";


    const char content[] = "'qu\0tes' 'are' 'fine: \\'' ";


    // Prepare input file: a few times the size of the buffer, containing


    // strings with zeroes and escaped quotes.


    FILE *f = fopen(fname, "w");


    for (int i = 0; i < BUFSIZE; ++i) {


        fwrite(content, 1, sizeof(content) - 1, f);


    }


    fclose(f);


    int count = 3 * BUFSIZE; // number of quoted strings written to file


    // Initialize lexer state: all pointers are at the end of buffer.


    // This immediately triggers YYFILL, as the check `in.cur < in.lim` fails.


    Input in;


    in.file = fopen(fname, "r");


    in.cur = in.tok = in.lim = in.buf + BUFSIZE;


    in.eof = 0;


    // Run the lexer.


    assert(lex(in) == count);


    // Cleanup: remove input file.


    fclose(in.file);


    remove(fname);


    return 0;
}

MULTIPLE BLOCKS

Sometimes it is necessary to have multiple interrelated lexers (for example, if there is a high-level state machine that transitions between lexer modes). This can be implemented using multiple connected re2c blocks. Another option is to use start conditions.

The implementation of connections between blocks depends on the target language. In languages that have goto statement (such as C/C++ and Go) one can have all blocks in one function, each of them prefixed with a label. Transition from one block to another is a simple goto. In languages that do not have goto (such as Rust) it is necessary to use a loop with a switch on a state variable, similar to the yystate loop/switch generated by re2c, or else wrap each block in a function and use function calls.

The example below uses multiple blocks to parse binary, octal, decimal and hexadecimal numbers. Each base has its own block. The initial block determines base and dispatches to other blocks. Common configurations are defined in a separate block at the beginning of the program; they are inherited by the other blocks.

// re2c $INPUT -o $OUTPUT -i
#include <stdint.h>
#include <limits.h>
#include <assert.h>
static const uint64_t ERROR = UINT64_MAX;
template<int BASE> static void add(uint64_t &u, char d) {


    u = u * BASE + d;


    if (u > UINT32_MAX) u = ERROR;
}
static uint64_t parse_u32(const char *s) {


    const char *YYCURSOR = s, *YYMARKER;


    uint64_t u = 0;


    /*!re2c


        re2c:yyfill:enable = 0;


        re2c:define:YYCTYPE = char;


        end = "\x00";


        '0b' / [01]        { goto bin; }


        "0"                { goto oct; }


        "" / [1-9]         { goto dec; }


        '0x' / [0-9a-fA-F] { goto hex; }


        *                  { return ERROR; }


    */
bin:


    /*!re2c


        end   { return u; }


        [01]  { add<2>(u, YYCURSOR[-1] - '0'); goto bin; }


        *     { return ERROR; }


    */
oct:


    /*!re2c


        end   { return u; }


        [0-7] { add<8>(u, YYCURSOR[-1] - '0'); goto oct; }


        *     { return ERROR; }


    */
dec:


    /*!re2c


        end   { return u; }


        [0-9] { add<10>(u, YYCURSOR[-1] - '0'); goto dec; }


        *     { return ERROR; }


    */
hex:


    /*!re2c


        end   { return u; }


        [0-9] { add<16>(u, YYCURSOR[-1] - '0');      goto hex; }


        [a-f] { add<16>(u, YYCURSOR[-1] - 'a' + 10); goto hex; }


        [A-F] { add<16>(u, YYCURSOR[-1] - 'A' + 10); goto hex; }


        *     { return ERROR; }


    */
}
int main() {


    assert(parse_u32("") == ERROR);


    assert(parse_u32("1234567890") == 1234567890);


    assert(parse_u32("0b1101") == 13);


    assert(parse_u32("0x7Fe") == 2046);


    assert(parse_u32("0644") == 420);


    assert(parse_u32("9999999999") == ERROR);


    return 0;
}

START CONDITIONS

Start conditions are enabled with --start-conditions option. They provide a way to encode multiple interrelated automata within the same re2c block.

Each condition corresponds to a single automaton and has a unique name specified by the user and a unique internal number defined by re2c. The numbers are used to switch between conditions: the generated code uses YYGETCONDITION and YYSETCONDITION primitives to get the current condition or set it to the given number. Use /*!conditions:re2c*/ directive or the --header option to generate numeric condition identifiers. Configuration re2c:cond:enumprefix specifies the generated identifier prefix.

In condition mode every rule must be prefixed with a list of comma-separated condition names in angle brackets, or a wildcard <*> to denote all conditions. The rule syntax is extended as follows:

The code re2c generates for conditions depends on whether re2c uses goto/label approach or loop/switch approach to encode the automata.

In languages that have goto statement (such as C/C++ and Go) conditions are naturally implemented as blocks of code prefixed with labels of the form yyc_<cond>, where cond is a condition name (label prefix can be changed with re2c:cond:prefix). Transitions between conditions are implemented using goto and condition labels. Before all conditions re2c generates an initial switch on YYGETSTATE that jumps to the start state of the current condition. The shortcut rules :=> bypass the initial switch and jump directly to the specified condition (re2c:cond:goto can be used to change the default behavior). The rules with semantic actions do not automatically jump to the next condition; this should be done by the user-defined action code.

In languages that do not have goto (such as Rust) re2c reuses the yystate variable to store condition numbers. Each condition gets a numeric identifier equal to the number of its start state, and a switch between conditions is no different than a switch between DFA states of a single condition. There is no need for a separate initial condition switch. (Since the same approach is used to implement storable states, YYGETCONDITION/YYSETCONDITION are redundant if both storable states and conditions are used).

The program below uses start conditions to parse binary, octal, decimal and hexadecimal numbers. There is a single block where each base has its own condition, and the initial condition is connected to all of them. User-defined variable cond stores the current condition number; it is initialized to the number of the initial condition generated with /*!conditions:re2c*/.

// re2c $INPUT -o $OUTPUT -ci
#include <stdint.h>
#include <limits.h>
#include <assert.h>
static const uint64_t ERROR = UINT64_MAX;
/*!conditions:re2c*/
template<int BASE> static void add(uint64_t &u, char d) {


    u = u * BASE + d;


    if (u > UINT32_MAX) u = ERROR;
}
static uint64_t parse_u32(const char *s) {


    const char *YYCURSOR = s, *YYMARKER;


    int c = yycinit;


    uint64_t u = 0;


    /*!re2c


        re2c:api:style = free-form;


        re2c:define:YYCTYPE        = char;


        re2c:define:YYGETCONDITION = "c";


        re2c:define:YYSETCONDITION = "c = @@;";


        re2c:yyfill:enable = 0;


        <*> * { return ERROR; }


        <init> '0b' / [01]        :=> bin


        <init> "0"                :=> oct


        <init> "" / [1-9]         :=> dec


        <init> '0x' / [0-9a-fA-F] :=> hex


        <bin, oct, dec, hex> "\x00" { return u; }


        <bin> [01]  { add<2>(u,  YYCURSOR[-1] - '0');      goto yyc_bin; }


        <oct> [0-7] { add<8>(u,  YYCURSOR[-1] - '0');      goto yyc_oct; }


        <dec> [0-9] { add<10>(u, YYCURSOR[-1] - '0');      goto yyc_dec; }


        <hex> [0-9] { add<16>(u, YYCURSOR[-1] - '0');      goto yyc_hex; }


        <hex> [a-f] { add<16>(u, YYCURSOR[-1] - 'a' + 10); goto yyc_hex; }


        <hex> [A-F] { add<16>(u, YYCURSOR[-1] - 'A' + 10); goto yyc_hex; }


    */
}
int main() {


    assert(parse_u32("") == ERROR);


    assert(parse_u32("1234567890") == 1234567890);


    assert(parse_u32("0b1101") == 13);


    assert(parse_u32("0x7Fe") == 2046);


    assert(parse_u32("0644") == 420);


    assert(parse_u32("9999999999") == ERROR);


    return 0;
}

STORABLE STATE

With --storable-state option re2c generates a lexer that can store its current state, return to the caller, and later resume operations exactly where it left off. The default mode of operation in re2c is a "pull" model, in which the lexer "pulls" more input whenever it needs it. This may be unacceptable in cases when the input becomes available piece by piece (for example, if the lexer is invoked by the parser, or if the lexer program communicates via a socket protocol with some other program that must wait for a reply from the lexer before it transmits the next message). Storable state feature is intended exactly for such cases: it allows one to generate lexers that work in a "push" model. When the lexer needs more input, it stores its state and returns to the caller. Later, when more input becomes available, the caller resumes the lexer exactly where it stopped. There are a few changes necessary compared to the "pull" model:

Here is an example of a "push" model lexer that simulates reading packets from a socket. The lexer loops until it encounters the end of input and returns to the calling function. The calling function provides more input by "sending" the next packet and resumes lexing. This process stops when all the packets have been sent, or when there is an error.

// re2c $INPUT -o $OUTPUT -f
#include <assert.h>
#include <stdio.h>
#include <string.h>
#define DEBUG 0
#define LOG(...) if (DEBUG) fprintf(stderr, __VA_ARGS__);
// Use a small buffer to cover the case when a lexeme doesn't fit.
// In real world use a larger buffer.
#define BUFSIZE 10
struct State {


    FILE *file;


    char buf[BUFSIZE + 1], *lim, *cur, *mar, *tok;


    int state;
};
typedef enum {END, READY, WAITING, BAD_PACKET, BIG_PACKET} Status;
static Status fill(State &st) {


    const size_t shift = st.tok - st.buf;


    const size_t used = st.lim - st.tok;


    const size_t free = BUFSIZE - used;


    // Error: no space. In real life can reallocate a larger buffer.


    if (free < 1) return BIG_PACKET;


    // Shift buffer contents (discard already processed data).


    memmove(st.buf, st.tok, used);


    st.lim -= shift;


    st.cur -= shift;


    st.mar -= shift;


    st.tok -= shift;


    // Fill free space at the end of buffer with new data.


    const size_t read = fread(st.lim, 1, free, st.file);


    st.lim += read;


    st.lim[0] = 0; // append sentinel symbol


    return READY;
}
static Status lex(State &st, unsigned int *recv) {


    char yych;


    /*!getstate:re2c*/


    for (;;) {


        st.tok = st.cur;


    /*!re2c


        re2c:api:style = free-form;


        re2c:define:YYCTYPE    = "char";


        re2c:define:YYCURSOR   = "st.cur";


        re2c:define:YYMARKER   = "st.mar";


        re2c:define:YYLIMIT    = "st.lim";


        re2c:define:YYGETSTATE = "st.state";


        re2c:define:YYSETSTATE = "st.state = @@;";


        re2c:define:YYFILL     = "return WAITING;";


        re2c:eof = 0;


        packet = [a-z]+[;];


        *      { return BAD_PACKET; }


        $      { return END; }


        packet { *recv = *recv + 1; continue; }


    */


    }
}
void test(const char **packets, Status expect) {


    // Create a "socket" (open the same file for reading and writing).


    const char *fname = "pipe";


    FILE *fw = fopen(fname, "w");


    FILE *fr = fopen(fname, "r");


    setvbuf(fw, NULL, _IONBF, 0);


    setvbuf(fr, NULL, _IONBF, 0);


    // Initialize lexer state: `state` value is -1, all pointers are at the end


    // of buffer.


    State st;


    st.file = fr;


    st.cur = st.mar = st.tok = st.lim = st.buf + BUFSIZE;


    // Sentinel (at YYLIMIT pointer) is set to zero, which triggers YYFILL.


    st.lim[0] = 0;


    st.state = -1;


    // Main loop. The buffer contains incomplete data which appears packet by


    // packet. When the lexer needs more input it saves its internal state and


    // returns to the caller which should provide more input and resume lexing.


    Status status;


    unsigned int send = 0, recv = 0;


    for (;;) {


        status = lex(st, &recv);


        if (status == END) {


            LOG("done: got %u packets\n", recv);


            break;


        } else if (status == WAITING) {


            LOG("waiting...\n");


            if (*packets) {


                LOG("sent packet %u\n", send);


                fprintf(fw, "%s", *packets++);


                ++send;


            }


            status = fill(st);


            LOG("queue: '%s'\n", st.buf);


            if (status == BIG_PACKET) {


                LOG("error: packet too big\n");


                break;


            }


            assert(status == READY);


        } else {


            assert(status == BAD_PACKET);


            LOG("error: ill-formed packet\n");


            break;


        }


    }


    // Check results.


    assert(status == expect);


    if (status == END) assert(recv == send);


    // Cleanup: remove input file.


    fclose(fw);


    fclose(fr);


    remove(fname);
}
int main() {


    const char *packets1[] = {0};


    const char *packets2[] = {"zero;", "one;", "two;", "three;", "four;", 0};


    const char *packets3[] = {"zer0;", 0};


    const char *packets4[] = {"looooooooooong;", 0};


    test(packets1, END);


    test(packets2, END);


    test(packets3, BAD_PACKET);


    test(packets4, BIG_PACKET);


    return 0;
}

REUSABLE BLOCKS

Reusable blocks are re2c blocks that can be reused any number of times and combined with other re2c blocks. They are defined with /*!rules:re2c[:<name>] ... */ (the <name> is optional). A rules block can be used in two contexts: either in a use block, or in a use directive inside of another block. The code for a rules block is generated at every point of use.

Use blocks are defined with /*!use:re2c[:<name>] ... */. The <name> is optional; if not specified, the associated rules block is the most recent one (whether named or unnamed). A use block can add named definitions, configurations and rules of its own. An important use case for use blocks is a lexer that supports multiple input encodings: the same rules block is reused multiple times with encoding-specific configurations (see the example below).

In-block use directive !use:<name>; can be used from inside of a re2c block. It merges the referenced block <name> into the current one. If some of the merged rules and configurations overlap with the previously defined ones, conflicts are resolved in the usual way: the earliest rule takes priority, and latest configuration overrides preceding ones. One exception are the special rules *, $ and (in condition mode) <!>, for which a block-local definition overrides any inherited ones. Use directive allows one to combine different re2c blocks together in one block (see the example below).

Named blocks and in-block use directive were added in re2c version 2.2. Since that version reusable blocks are allowed by default (no special option is needed). Before version 2.2 reuse mode was enabled with -r --reusable option. Before version 1.2 reusable blocks could not be mixed with normal blocks.

// re2c $INPUT -o $OUTPUT
#include <assert.h>
// This example shows how to combine reusable re2c blocks: two blocks
// ('colors' and 'fish') are merged into one. The 'salmon' rule occurs
// in both blocks; the 'fish' block takes priority because it is used
// earlier. Default rule * occurs in all three blocks; the local (not
// inherited) definition takes priority.
enum What { COLOR, FISH, DUNNO };
/*!rules:re2c:colors


    *                            { assert(false); }


    "red" | "salmon" | "magenta" { return COLOR; }
*/
/*!rules:re2c:fish


    *                            { assert(false); }


    "haddock" | "salmon" | "eel" { return FISH; }
*/
static What lex(const char *s) {


    const char *YYCURSOR = s, *YYMARKER;


    /*!re2c


        re2c:yyfill:enable = 0;


        re2c:define:YYCTYPE = char;


        !use:fish;


        !use:colors;


        * { return DUNNO; }  // overrides inherited '*' rules


    */
}
int main() {


    assert(lex("salmon") == FISH);


    assert(lex("what?") == DUNNO);


    return 0;
}

// re2c $INPUT -o $OUTPUT --input-encoding utf8
#include <assert.h>
#include <stdint.h>
// This example supports multiple input encodings: UTF-8 and UTF-32.
// Both lexers are generated from the same rules block, and the use
// blocks add only encoding-specific configurations.
/*!rules:re2c


    re2c:yyfill:enable = 0;


    "∀x ∃y" { return 0; }


    *       { return 1; }
*/
static int lex_utf8(const uint8_t *s) {


    const uint8_t *YYCURSOR = s, *YYMARKER;


    /*!use:re2c


        re2c:define:YYCTYPE = uint8_t;


        re2c:encoding:utf8 = 1;


    */
}
static int lex_utf32(const uint32_t *s) {


    const uint32_t *YYCURSOR = s, *YYMARKER;


    /*!use:re2c


        re2c:define:YYCTYPE = uint32_t;


        re2c:encoding:utf32 = 1;


    */
}
int main() {


    static const uint8_t s8[] = // UTF-8


        { 0xe2, 0x88, 0x80, 0x78, 0x20, 0xe2, 0x88, 0x83, 0x79 };


    static const uint32_t s32[] = // UTF32


        { 0x00002200, 0x00000078, 0x00000020, 0x00002203, 0x00000079 };


    assert(lex_utf8(s8) == 0);


    assert(lex_utf32(s32) == 0);


    return 0;
}

SUBMATCH EXTRACTION

re2c has two options for submatch extraction.

The first option is -T --tags. With this option one can use standalone tags of the form @stag and #mtag, where stag and mtag are arbitrary used-defined names. Tags can be used anywhere inside of a regular expression; semantically they are just position markers. Tags of the form @stag are called s-tags: they denote a single submatch value (the last input position where this tag matched). Tags of the form #mtag are called m-tags: they denote multiple submatch values (the whole history of repetitions of this tag). All tags should be defined by the user as variables with the corresponding names. With standalone tags re2c uses leftmost greedy disambiguation: submatch positions correspond to the leftmost matching path through the regular expression.

The second option is -P --posix-captures: it enables POSIX-compliant capturing groups. In this mode parentheses in regular expressions denote the beginning and the end of capturing groups; the whole regular expression is group number zero. The number of groups for the matching rule is stored in a variable yynmatch, and submatch results are stored in yypmatch array. Both yynmatch and yypmatch should be defined by the user, and yypmatch size must be at least [yynmatch * 2]. re2c provides a directive /*!maxnmatch:re2c*/ that defines YYMAXNMATCH: a constant equal to the maximal value of yynmatch among all rules. Note that re2c implements POSIX-compliant disambiguation: each subexpression matches as long as possible, and subexpressions that start earlier in regular expression have priority over those starting later. Capturing groups are translated into s-tags under the hood, therefore we use the word "tag" to describe them as well.

With both -P --posix-captures and T --tags options re2c uses efficient submatch extraction algorithm described in the Tagged Deterministic Finite Automata with Lookahead paper. The overhead on submatch extraction in the generated lexer grows with the number of tags --- if this number is moderate, the overhead is barely noticeable. In the lexer tags are implemented using a number of tag variables generated by re2c. There is no one-to-one correspondence between tag variables and tags: a single variable may be reused for different tags, and one tag may require multiple variables to hold all its ambiguous values. Eventually ambiguity is resolved, and only one final variable per tag survives. When a rule matches, all its tags are set to the values of the corresponding tag variables. The exact number of tag variables is unknown to the user; this number is determined by re2c. However, tag variables should be defined by the user as a part of the lexer state and updated by YYFILL, therefore re2c provides directives /*!stags:re2c*/ and /*!mtags:re2c*/ that can be used to declare, initialize and manipulate tag variables. These directives have two optional configurations: format = "@@"; (specifies the template where @@ is substituted with the name of each tag variable), and separator = ""; (specifies the piece of code used to join the generated pieces for different tag variables).

S-tags support the following operations:

M-tags support the following operations:

S-tags can be implemented as scalar values (pointers or offsets). M-tags need a more complex representation, as they need to store a sequence of tag values. The most naive and inefficient representation of an m-tag is a list (array, vector) of tag values; a more efficient representation is to store all m-tags in a prefix-tree represented as array of nodes (v, p), where v is tag value and p is a pointer to parent node.

Here is a simple example of using s-tags to parse semantic versions consisting of three numeric components: major, minor, patch (the latter is optional). See below for a more complex example that uses YYFILL.

// re2c $INPUT -o $OUTPUT
#include <assert.h>
#include <stddef.h>
struct SemVer { int major, minor, patch; };
static int s2n(const char *s, const char *e) { // pre-parsed string to number


    int n = 0;


    for (; s < e; ++s) n = n * 10 + (*s - '0');


    return n;
}
static bool lex(const char *str, SemVer &ver) {


    const char *YYCURSOR = str, *YYMARKER;


    // User-defined tag variables that are available in semantic action.


    const char *t1, *t2, *t3, *t4, *t5;


    // Autogenerated tag variables used by the lexer to track tag values.


    /*!stags:re2c format = 'const char *@@;\n'; */


    /*!re2c


        re2c:yyfill:enable = 0;


        re2c:define:YYCTYPE = char;


        re2c:tags = 1;


        num = [0-9]+;


        @t1 num @t2 "." @t3 num @t4 ("." @t5 num)? [\x00] {


            ver.major = s2n(t1, t2);


            ver.minor = s2n(t3, t4);


            ver.patch = t5 != NULL ? s2n(t5, YYCURSOR - 1) : 0;


            return true;


        }


        * { return false; }


    */
}
int main() {


    SemVer v;


    assert(lex("23.34", v) && v.major == 23 && v.minor == 34 && v.patch == 0);


    assert(lex("1.2.999", v) && v.major == 1 && v.minor == 2 && v.patch == 999);


    assert(!lex("1.a", v));


    return 0;
}

Here is a more complex example of using s-tags with YYFILL to parse a file with newline-separated semantic versions. Tag variables are part of the lexer state, and they are adjusted in YYFILL like other input positions. Note that it is necessary for s-tags because their values are invalidated after shifting buffer contents. It may not be necessary in a custom implementation where tag variables store offsets relative to the start of the input string rather than the buffer, which may be the case with m-tags.

// re2c $INPUT -o $OUTPUT --tags
#include <assert.h>
#include <stddef.h>
#include <stdio.h>
#include <string.h>
#include <vector>
#define BUFSIZE 4095
struct Input {


    FILE *file;


    char buf[BUFSIZE + 1], *lim, *cur, *mar, *tok;


    // Tag variables must be part of the lexer state passed to YYFILL.


    // They don't correspond to tags and should be autogenerated by re2c.


    /*!stags:re2c format = 'const char *@@;'; */


    bool eof;
};
struct SemVer { int major, minor, patch; };
static bool operator==(const SemVer &x, const SemVer &y) {


    return x.major == y.major && x.minor == y.minor && x.patch == y.patch;
}
static int s2n(const char *s, const char *e) { // pre-parsed string to number


    int n = 0;


    for (; s < e; ++s) n = n * 10 + (*s - '0');


    return n;
}
static int fill(Input &in) {


    if (in.eof) return 1;


    const size_t shift = in.tok - in.buf;


    const size_t used = in.lim - in.tok;


    // Error: lexeme too long. In real life could reallocate a larger buffer.


    if (shift < 1) return 2;


    // Shift buffer contents (discard everything up to the current token).


    memmove(in.buf, in.tok, used);


    in.lim -= shift;


    in.cur -= shift;


    in.mar -= shift;


    in.tok -= shift;


    // Tag variables need to be shifted like other input positions. The check


    // for non-NULL is only needed if some tags are nested inside of alternative


    // or repetition, so that they can have NULL value.


    /*!stags:re2c format = "if (in.@@) in.@@ -= shift;\n"; */


    // Fill free space at the end of buffer with new data from file.


    in.lim += fread(in.lim, 1, BUFSIZE - used, in.file);


    in.lim[0] = 0;


    in.eof = in.lim < in.buf + BUFSIZE;


    return 0;
}
static bool lex(Input &in, std::vector<SemVer> &vers) {


    // User-defined local variables that store final tag values.


    // They are different from tag variables autogenerated with `stags:re2c`,


    // as they are set at the end of match and used only in semantic actions.


    const char *t1, *t2, *t3, *t4;


    for (;;) {


        in.tok = in.cur;


    /*!re2c


        re2c:eof = 0;


        re2c:api:style = free-form;


        re2c:define:YYCTYPE  = char;


        re2c:define:YYCURSOR = in.cur;


        re2c:define:YYMARKER = in.mar;


        re2c:define:YYLIMIT  = in.lim;


        re2c:define:YYFILL   = "fill(in) == 0";


        re2c:tags:expression = "in.@@";


        num = [0-9]+;


        num @t1 "." @t2 num @t3 ("." @t4 num)? [\n] {


            int major = s2n(in.tok, t1);


            int minor = s2n(t2, t3);


            int patch = t4 != NULL ? s2n(t4, in.cur - 1) : 0;


            SemVer ver = {major, minor, patch};


            vers.push_back(ver);


            continue;


        }


        $ { return true; }


        * { return false; }


    */}
}
int main() {


    const char *fname = "input";


    const SemVer semver = {1, 22, 333};


    std::vector<SemVer> expect(BUFSIZE, semver), actual;


    // Prepare input file (make sure it exceeds buffer size).


    FILE *f = fopen(fname, "w");


    for (int i = 0; i < BUFSIZE; ++i) fprintf(f, "1.22.333\n");


    fclose(f);


    // Reopen input file for reading.


    f = fopen(fname, "r");


    // Initialize lexer state: all pointers are at the end of buffer.


    Input in;


    in.file = f;


    in.cur = in.mar = in.tok = in.lim = in.buf + BUFSIZE;


    /*!stags:re2c format = "in.@@ = in.lim;\n"; */


    in.eof = false;


    // Sentinel (at YYLIMIT pointer) is set to zero, which triggers YYFILL.


    *in.lim = 0;


  


    // Run the lexer and check results.


    assert(lex(in, actual) && expect == actual);


    // Cleanup: remove input file.


    fclose(f);


    remove(fname);


    return 0;
}

Here is an example of using POSIX capturing groups to parse semantic versions.

// re2c $INPUT -o $OUTPUT
#include <assert.h>
#include <stddef.h>
// Maximum number of capturing groups among all rules.
/*!maxnmatch:re2c*/
struct SemVer { int major, minor, patch; };
static int s2n(const char *s, const char *e) { // pre-parsed string to number


    int n = 0;


    for (; s < e; ++s) n = n * 10 + (*s - '0');


    return n;
}
static bool lex(const char *str, SemVer &ver) {


    const char *YYCURSOR = str, *YYMARKER;


    // Allocate memory for capturing parentheses (twice the number of groups).


    const char *yypmatch[YYMAXNMATCH * 2];


    size_t yynmatch;


    // Autogenerated tag variables used by the lexer to track tag values.


    /*!stags:re2c format = 'const char *@@;\n'; */


    /*!re2c


        re2c:yyfill:enable = 0;


        re2c:define:YYCTYPE = char;


        re2c:posix-captures = 1;


        num = [0-9]+;


        (num) "." (num) ("." num)? [\x00] {


            // `yynmatch` is the number of capturing groups


            assert(yynmatch == 4);


            // Even `yypmatch` values are for opening parentheses, odd values


            // are for closing parentheses, the first group is the whole match.


            ver.major = s2n(yypmatch[2], yypmatch[3]);


            ver.minor = s2n(yypmatch[4], yypmatch[5]);


            ver.patch = yypmatch[6] ? s2n(yypmatch[6] + 1, yypmatch[7]) : 0;


            return true;


        }


        * { return false; }


    */
}
int main() {


    SemVer v;


    assert(lex("23.34", v) && v.major == 23 && v.minor == 34 && v.patch == 0);


    assert(lex("1.2.999", v) && v.major == 1 && v.minor == 2 && v.patch == 999);


    assert(!lex("1.a", v));


    return 0;
}

Here is an example of using m-tags to parse a version with a variable number of components. Tag variables are stored in a trie.

// re2c $INPUT -o $OUTPUT
#include <assert.h>
#include <stddef.h>
#include <vector>
static const int MTAG_ROOT = -1;
// An m-tag tree is a way to store histories with an O(1) copy operation.
// Histories naturally form a tree, as they have common start and fork at some
// point. The tree is stored as an array of pairs (tag value, link to parent).
// An m-tag is represented with a single link in the tree (array index).
struct Mtag {


    const char *elem; // tag value


    int pred; // index of the predecessor node or root
};
typedef std::vector<Mtag> MtagTrie;
typedef std::vector<int> Ver; // unbounded number of version components
static int s2n(const char *s, const char *e) { // pre-parsed string to number


    int n = 0;


    for (; s < e; ++s) n = n * 10 + (*s - '0');


    return n;
}
// Append a single value to an m-tag history.
static void add_mtag(MtagTrie &trie, int &mtag, const char *value) {


    Mtag m = {value, mtag};


    mtag = (int)trie.size();


    trie.push_back(m);
}
// Recursively unwind tag histories and collect version components.
static void unfold(const MtagTrie &trie, int x, int y, Ver &ver) {


    // Reached the root of the m-tag tree, stop recursion.


    if (x == MTAG_ROOT && y == MTAG_ROOT) return;


    // Unwind history further.


    unfold(trie, trie[x].pred, trie[y].pred, ver);


    // Get tag values. Tag histories must have equal length.


    assert(x != MTAG_ROOT && y != MTAG_ROOT);


    const char *ex = trie[x].elem, *ey = trie[y].elem;


    if (ex != NULL && ey != NULL) {


        // Both tags are valid pointers, extract component.


        ver.push_back(s2n(ex, ey));


    } else {


        // Both tags are NULL (this corresponds to zero repetitions).


        assert(ex == NULL && ey == NULL);


    }
}
static bool parse(const char *str, Ver &ver) {


    const char *YYCURSOR = str, *YYMARKER;


    MtagTrie mt;


    // User-defined tag variables that are available in semantic action.


    const char *t1, *t2;


    int t3, t4;


    // Autogenerated tag variables used by the lexer to track tag values.


    /*!stags:re2c format = 'const char *@@ = NULL;'; */


    /*!mtags:re2c format = 'int @@ = MTAG_ROOT;'; */


    /*!re2c


        re2c:api:style = free-form;


        re2c:define:YYCTYPE = char;


        re2c:define:YYSTAGP = "@@ = YYCURSOR;";


        re2c:define:YYSTAGN = "@@ = NULL;";


        re2c:define:YYMTAGP = "add_mtag(mt, @@, YYCURSOR);";


        re2c:define:YYMTAGN = "add_mtag(mt, @@, NULL);";


        re2c:yyfill:enable = 0;


        re2c:tags = 1;


        num = [0-9]+;


        @t1 num @t2 ("." #t3 num #t4)* [\x00] {


            ver.clear();


            ver.push_back(s2n(t1, t2));


            unfold(mt, t3, t4, ver);


            return true;


        }


        * { return false; }


    */
}
int main() {


    Ver v;


    assert(parse("1", v) && v == Ver({1}));


    assert(parse("1.2.3.4.5.6.7", v) && v == Ver({1, 2, 3, 4, 5, 6, 7}));


    assert(!parse("1.2.", v));


    return 0;
}

ENCODING SUPPORT

It is necessary to understand the difference between code points and code units. A code point is a numeric identifier of a symbol. A code unit is the smallest unit of storage in the encoded text. A single code point may be represented with one or more code units. In a fixed-length encoding all code points are represented with the same number of code units. In a variable-length encoding code points may be represented with a different number of code units. Note that the "any" rule [^] matches any code point, but not necessarily any code unit (the only way to match any code unit regardless of the encoding is the default rule *). The generated lexer works with a stream of code units: yych stores a code unit, and YYCTYPE is the code unit type. Regular expressions, on the other hand, are specified in terms of code points. When re2c compiles regular expressions to automata it translates code points to code units. This is generally not a simple mapping: in variable-length encodings a single code point range may get translated to a complex code unit graph. The following encodings are supported:

Include file include/unicode_categories.re provides re2c definitions for the standard Unicode categories.

Option --input-encoding specifies source file encoding, which can be used to enable Unicode literals in regular expressions. For example --input-encoding utf8 tells re2c that the source file is in UTF8 (it differs from --utf8 which sets input text encoding). Option --encoding-policy specifies the way re2c handles Unicode surrogates (code points in range [0xD800-0xDFFF]).

Below is an example of a lexer for UTF8 encoded Unicode identifiers.

// re2c $INPUT -o $OUTPUT -8 --case-ranges -i
#include <assert.h>
#include <stdint.h>
/*!include:re2c "unicode_categories.re" */
static int lex(const char *s) {


    const char *YYCURSOR = s, *YYMARKER;


    /*!re2c


        re2c:define:YYCTYPE = 'unsigned char';


        re2c:yyfill:enable = 0;


        // Simplified "Unicode Identifier and Pattern Syntax"


        // (see https://unicode.org/reports/tr31)


        id_start    = L | Nl | [$_];


        id_continue = id_start | Mn | Mc | Nd | Pc | [\u200D\u05F3];


        identifier  = id_start id_continue*;


        identifier { return 0; }


        *          { return 1; }


    */
}
int main() {


    assert(lex("_Ыдентификатор") == 0);


    return 0;
}

INCLUDE FILES

re2c allows one to include other files using directive /*!include:re2c FILE */ or !include FILE ;, where FILE is a path to the file to be included. The first form should be used outside of re2c blocks, and the second form allows one to include a file in the middle of a re2c block. re2c looks for included files in the directory of the including file and in include locations, which can be specified with -I option. Include directives in re2c work in the same way as C/C++ #include: the contents of FILE are copy-pasted verbatim in place of the directive. Include files may have further includes of their own. Use --depfile option to track build dependencies of the output file on include files. re2c provides some predefined include files that can be found in the include/ subdirectory of the project. These files contain definitions that can be useful to other projects (such as Unicode categories) and form something like a standard library for re2c. Below is an example of using include directive.

typedef enum { OK, FAIL } Result;
/*!re2c


    number = [1-9][0-9]*;
*/

// floating-point numbers
frac  = [0-9]* "." [0-9]+ | [0-9]+ ".";
exp   = 'e' [+-]? [0-9]+;
float = frac exp? | [0-9]+ exp;
float { return OK; }

// re2c $INPUT -o $OUTPUT -i
#include <assert.h>
/*!include:re2c "definitions.h" */
Result lex(const char *s) {


    const char *YYCURSOR = s, *YYMARKER;


    /*!re2c


        re2c:define:YYCTYPE = char;


        re2c:yyfill:enable = 0;


        *      { return FAIL; }


        number { return OK; }


        !include "extra_rules.re.inc";


    */
}
int main() {


    assert(lex("123") == OK);


    assert(lex("123.4567") == OK);


    return 0;
}

HEADER FILES

re2c allows one to generate header file from the input .re file using option -t, --type-header or configuration re2c:flags:type-header and directives /*!header:re2c:on*/ and /*!header:re2c:off*/. The first directive marks the beginning of header file, and the second directive marks the end of it. Everything between these directives is processed by re2c, and the generated code is written to the file specified by the -t --type-header option (or stdout if this option was not used). Autogenerated header file may be needed in cases when re2c is used to generate definitions of constants, variables and structs that must be visible from other translation units.

Here is an example of generating a header file that contains definition of the lexer state with tag variables (the number variables depends on the regular grammar and is unknown to the programmer).

// re2c $INPUT -o $OUTPUT -i --header lexer/state.h
#include <assert.h>
#include <stddef.h>
#include "lexer/state.h" // the header is generated by re2c
/*!header:re2c:on*/
struct LexerState {


    const char *str, *cur;


    /*!stags:re2c format = "const char *@@;"; */
};
/*!header:re2c:off*/
long lex(LexerState& st) {


    const char *t;


    /*!re2c


        re2c:header = "lexer/state.h";


        re2c:yyfill:enable = 0;


        re2c:define:YYCTYPE = char;


        re2c:define:YYCURSOR = "st.cur";


        re2c:tags = 1;


        re2c:tags:expression = "st.@@";


        [a]* @t [b]* { return t - st.str; }


    */
}
int main() {


    const char *s = "ab";


    LexerState st = { s, s /*!stags:re2c format = ", NULL"; */ };


    assert(lex(st) == 1);


    return 0;
}

/* Generated by re2c */
typedef struct {


    const char *str, *cur, *mar;


    const char *yyt1;
} LexerState;

SKELETON PROGRAMS

With the -S, --skeleton option, re2c ignores all non-re2c code and generates a self-contained C program that can be further compiled and executed. The program consists of lexer code and input data. For each constructed DFA (block or condition) re2c generates a standalone lexer and two files: an .input file with strings derived from the DFA and a .keys file with expected match results. The program runs each lexer on the corresponding .input file and compares results with the expectations. Skeleton programs are very useful for a number of reasons:

The difficulty with generating input data is that for all but the most trivial cases the number of possible input strings is too large (even if the string length is limited). re2c solves this difficulty by generating sufficiently many strings to cover almost all DFA transitions. It uses the following algorithm. First, it constructs a skeleton of the DFA. For encodings with 1-byte code unit size (such as ASCII, UTF-8 and EBCDIC) skeleton is just an exact copy of the original DFA. For encodings with multibyte code units skeleton is a copy of DFA with certain transitions omitted: namely, re2c takes at most 256 code units for each disjoint continuous range that corresponds to a DFA transition. The chosen values are evenly distributed and include range bounds. Instead of trying to cover all possible paths in the skeleton (which is infeasible) re2c generates sufficiently many paths to cover all skeleton transitions, and thus trigger the corresponding conditional jumps in the lexer. The algorithm implementation is limited by ~1Gb of transitions and consumes constant amount of memory (re2c writes data to file as soon as it is generated).

VISUALIZATION AND DEBUG

With the -D, --emit-dot option, re2c does not generate code. Instead, it dumps the generated DFA in DOT format. One can convert this dump to an image of the DFA using Graphviz or another library. Note that this option shows the final DFA after it has gone through a number of optimizations and transformations. Earlier stages can be dumped with various debug options, such as --dump-nfa, --dump-dfa-raw etc. (see the full list of options).

SEE ALSO

You can find more information about re2c at the official website: http://re2c.org. Similar programs are flex(1), lex(1), quex(http://quex.sourceforge.net).

AUTHORS

re2c was originaly written by Peter Bumbulis in 1993. Since then it has been developed and maintained by multiple volunteers; mots notably, Brain Young, Marcus Boerger, Dan Nuffer and Ulya Trofimovich.