Generated Code

Generated Code

You ran ISPA and files appeared. What are you looking at? 🔍

Tokens and rules, and their ids

For every token, ISPA generates a namespace that mirrors the token's path in your grammar, plus an id. That id identifies the token, and the parser uses the same ids. Rules work the same way.

These are collected in Tokens and Rules. Next to them are TokensStr and RulesStr, which hold the names of tokens and rules by path.

To go from an id back to a readable name, use:

namespace Parser {
    auto TokensToString(std::size_t id) -> const char*;
    auto RulesToString(std::size_t id) -> const char*;
}

Handy for debugging and error messages.

Node types: Types and FlatTypes

Every token and rule gets a generated C++ type. They come in two flavours:

• Types is the one you should normally use. Everything is separated with ::, so the path reads naturally.
• FlatTypes holds the same types with _ as the delimiter. These are the main node structures the parser and lexer actually work with. Nested rules appear inside them as aliases to another flat type.

Where types live:

• Types live in a modules:: namespace within Tokens.
• Shared tokens, which ISPA determines are used in several productions, aren't encapsulated this way and live directly in Types.

Strange-looking token names?

Some token names are generated for you. A literal symbol like $ can't be part of a C++ identifier, so ISPA uses the ICU library to give non-literal symbols literal names. For example, $ becomes DOLLAR_SIGN. Some tokens may also be called AUTO, particularly when a production has a single literal with no other user of the same token.

What a generated type looks like

Every token and rule type has the same shape. Here's a real one, a $ token:

namespace Parser {
    namespace FlatTypes {
        struct modules_rule_regular_value_dyn_DOLLAR_SIGN {
            auto write_to_output(::ISPA_STD::ASTPrinter<std::ostream>& printer) const -> void;
            auto print(std::ostream& os) const -> void;
            auto to_string() const -> std::string;

            constexpr auto members() { return std::tie(); }
            constexpr auto members() const { return std::tie(); }
            static constexpr auto members_str() {
                return std::array<std::string_view, 0>{{}};
            }

            constexpr auto methods() {
                return std::make_tuple(&modules_rule_regular_value_dyn_DOLLAR_SIGN::write_to_output,
                                       &modules_rule_regular_value_dyn_DOLLAR_SIGN::print);
            }
            constexpr auto methods() const { /* same, const */ }
            static constexpr auto methods_str() {
                return std::array<std::string_view, 2>{{"write_to_output", "print"}};
            }
        };
    }
}

Reading it from the top:

• print, to_string, write_to_output print the node to a stream, turn it into a string, or write it through an ASTPrinter.
• members() / members_str() give compile-time access to the node's fields and their names. This token captures nothing, so both are empty. If you add named AST fields in your grammar (@{...}), the type's fields (with deduced types) show up here.
• methods() / methods_str() do the same for the node's methods.

This metadata is constexpr, so you can write generic code, like printers or visitors, that works over any node.

The parser has metadata too

The generated parser exposes the same kind of constexpr metadata:

constexpr auto methods() {
    return std::make_tuple(&Parser::parseFromTokens, &Parser::lazyParse);
}
static constexpr auto methods_str() {
    return std::array<std::string_view, 2>{{"parseFromTokens", "lazyParse"}};
}

The base classes it builds on are covered in Runtime API.

Python output

-l python -a LL -o build/Parser writes build/Parser.py. Place its directory and the target runtime directory (parser/stdlibs/target in a checkout, or installed ispatarget/python) on PYTHONPATH. Create Parser.Parser() and call parse(text). Python output uses the same grammar captures and AST shape, but its public class and runtime are Python modules rather than C++ headers and core.cpp.

External dependencies stay as separate generated modules or C++ translation units; the host does not copy their production code. See Modules and External Parsers.