Runtime API

Runtime API

Generated lexers and parsers build on two small base classes from the ISPA standard library (ispastdlib). Include <ispastdlib.hpp> and you get everything. Knowing these two classes tells you most of what you can do with the generated code.

Lexer: Lexer_base<Token>

Turns text into a stream of tokens.

Giving it input

lexer.setInput(std::string("1 + 2"));   // copies the text, safe
lexer.setInput("1 + 2");                // does NOT copy, keep the buffer alive!

⚠️ The const char* version only stores a pointer. The text must outlive the lexer and anything that points into it. The std::string version keeps its own copy.

The constructors accept the same things: nothing, a std::string, a const char*, or an existing TokenFlow<Token>.

Making tokens

Method	What it does
makeTokens()	Lex the whole input at once.
makeTokens(text)	Set the input and lex it. Takes std::string or const char*.
makeTokensFromFile(path)	Read a file and lex it. Throws IspaAPIExcept if it can't be opened.
getTokens()	Read-only access to the produced TokenFlow.
clearTokens()	Throw the tokens away.
hasInput() / hasTokens()	Ask what the lexer currently holds.

If the lexer reaches a character nothing can match, it raises an internal error that includes the position and the offending character.

Two ways to walk tokens

• iterator walks tokens that have already been produced. Call makeTokens() first.
• lazy_iterator lexes on demand as you advance. This is what you want for external parser integration: eager tokenization throws if the lexer has external rules, so use lazy tokenization there.

Each token carries source metadata such as its start position, line and column.

Errors

LL parser diagnostics are available through getErrors() and getContextErrors() after parsing, including after a parse throws. A new parse clears them. Lexer failures such as an unterminated string can still throw before parser recovery runs.

Parser: Parser_base<MainNode, Token>

Turns tokens (or raw text) into your AST.

auto& tree = parser.parse("1 + 2 * 3");   // lazy: parses straight from text
auto& tree = parser.parse(lexer);         // non-lazy: parses from a lexer's tokens

• parse(const char*) runs the lazy path, with no separate tokenization step.
• parse(Lexer_base&) runs the non-lazy path from a lexer.
• Both return a reference to the tree, MainNode&, which the parser owns.
• If no input has been provided, IspaAPIExcept is thrown.
• setInput(...) and clearInput() let you reuse a parser.

As with the lexer, const char* input isn't copied, so keep the text alive while you use the tree.

Note for people extending ISPA

parseFromTokens() and lazyParse() are the bridge between the generator and the library. The generator implements them, so don't override them yourself.

For an end-to-end C++ example, see examples/main.cpp in the repository and Getting Started. The generated namespace follows the output/module naming used by the grammar; inspect the generated header when using a custom output prefix. Recovered trees can have missing captures, so inspect diagnostics before consuming them.