Parsers, although they are optional in JetPAG grammars, play the main rule in parsing input. Parsers work on tokns streams, which are fed by scanners. Tokens are matched based on their TTIDs, where text of tokens can be retrieve by using symbolic variables and embedding proper source code. Each token object holds information about its TTID, it's location in input and its text.
Token streams are linked, i.e. each token links to the next token in the stream. Linking stream tokens is done in the scanner and links cannot be modified after tokens are constructed (their are always passed as constant objects). This allows free traversal through the stream and simple seeking (going back and forth in the token stream) and overall work for the token stream manager.
It's important to know that tokens are always passed as references, and they are managed object with using smart pointers. The type of used references is the member type StreamToken, declared in both scanners and parsers.
Parsers are independent of scanner generated from the same grammar. The only required compatiblity between the parser and the scanner is to have the same set of token types. Token types are generated automatically for scanners, but can also be reused among several grammars with other scanners and parsers. If no scanner specification is available in the grammar token references in the parser grammar would rely on any externally defined token types.
For generating a file containing external definitions of token types available in the grammar use the gen_ttypes boolean grammar option which generates the file with the .jtt extension. For using token types defined in an external file use the external_ttypes option, which takes the name of the file for importing token types. In this combined example, the first grammar defines a scanner and the second grammar uses the token types from the first one (The first grammar must be interpreted by JetPAG before the second one):
grammar G1< gen_ttypes >: scanner L: A: 'a' ; B: 'b' ;
grammar G2< external_ttypes="G1_ttypes.jtt" >: parser P: rule: A B ;
Parser rules might be one of two types (in addition to the common rule types):