The definition of fragment specifier is inconsistent

[xmh0511]

Consider this example:

macro_rules! test{
    ($id:ident, $e:expr)=>{
        $id.$e
    }
}
fn main() {
  let a = (1,);
  test!(a,0);
}

The compiler reports an error

error: unexpected token: `expr` metavariable
 --> src/main.rs:3:13
  |
3 |         $id.$e
  |             ^^
...
8 |   test!(a,0);
  |   ---------- in this macro invocation
  |
  = note: this error originates in the macro `test` (in Nightly builds, run with -Z macro-backtrace for more info)

IIUC, the reason can be interpreted by the syntax

Valid fragment specifiers are:

• expr: an Expression

$e is defined as an expr fragment specifier, which can match LiteralExpression; however, the valid syntax of Tuple indexing expressions is defined as

TupleIndexingExpression → Expression . TUPLE_INDEX

The component following . is TUPLE_INDEX in which an Expression cannot appear. So, this is the reason for the error, which is ruled by syntax.

However, see the definition of path

path: a TypePath

This is a fixed version by #2248

A TypePath is

TypePath → ::? TypePathSegment ( :: TypePathSegment )*

Instead, a path in an expression should be

PathExpression →
PathInExpression
| QualifiedPathInExpression

The path that appears in an expression doesn't expect the syntax TypePath, that is, the following code should have had a similar error as the first example:

macro_rules! test2{
    ($id:ident,$p:path)=>{
        $id = $p;
    }
}
mod S{
    pub const I:i32 = 0;
}
fn main() {
  let mut i = 0;
  test2!(i,S::I);
}

However, this code is ok.

[ehuss]

If I had to guess, I would say the conversion of a type path to an expression path is done here.

@fmease would you concur? I can't honestly claim to remember why I used the phrase "TypePath style path" back in 2018, but I want to say that this is the reason. It accepts TypePath, but when expanded it can be translated to other path kinds, like PathExpression and SimplePath.

[xmh0511]

If I had to guess, I would say the conversion of a type path to an expression path is done here.

@fmease would you concur? I can't honestly claim to remember why I used the phrase "TypePath style path" back in 2018, but I want to say that this is the reason. It accepts TypePath, but when expanded it can be translated to other path kinds, like PathExpression and SimplePath.

Maybe we want to express it in this way:

TypePath style path means that the sequence of tokens matches TypePath

which can be briefly said that it has the form of TypePath. However, we cannot say a TypePath, because it's a syntax that is limited to appearing in a specific grammar.

[fmease]

Hmm, I see. AFAICT that page doesn't precisely define match as in "fragment-specifier matches a Rust syntax fragment of the kind specified". A few paragraphs above it does briefly talk about "opaque ASTs" wrt. matching behavior but it seems this section isn't sufficient.

Right now I think I wouldn't want to go back to the "Rule-style" phrasing (at least not without properly defining it somewhere and using it all other relevant places) since there's a lot more to it, transcending path / TypePath.

Going back to

fragment-specifier matches a Rust syntax fragment of the kind specified

I think it's ambiguous as to whether it means "structural" or "nominal" match. Well, we know (from rustc's impl and as alluded to by the "opaque AST" section) it's a nominal match with several exceptions and nuances IIUC.

Now, since it's a nominal match that begs the question what the relationship is between all syntax fragment kinds. For example, we know item and expr are compatible with stmt; block and literal with expr. Would we need to explicitly specify that fact in the Reference or is that implied by the fact that the corresponding grammar rules (e.g., Statement for stmt) also generate/accept token sequences produced/accepted by the other corresp. grammar rule (e.g., Item for item)? Wouldn't that be structural matching though which we know it isn't?

An "interesting" case (similar to TypePath vs. pat, expr, …) is for example: expr is intentionally compatible with pat (!) as long as the not-so-opaque fragment contains a maybe-negated literal (structurally):

// OK!

macro_rules! test {
    ($x:expr) => {
        let /*pat ctxt!*/ $x = (loop {}) else { loop {} };
    }
}

fn scope() { test!(0); }

// OK!
macro_rules! m0 { ($x:expr) => { m1! { $x } } }
macro_rules! m1 { ($x:pat) => {} }
m0! { "" }

Furthermore, either syntax fragments branded with tt are compatible with all other kinds or they don't get "branded" in the first place; I don't know the specifics. Finally, injected identifiers and lifetimes usually aren't treated as opaque syntax fragments but as transparent ones; I'm relatively sure that that's user observable but I'm not an expert on that matter.

I'm really starting to ramble now.

Anyway, I think a consequence of that would be the following: Fragment kind lifetime (LIFETIME_TOKEN) can be used in places where LIFETIME_OR_LABEL is expected:

// OK!

macro_rules! check {
    ($l:lifetime) => { $l: loop { break $l; } };
}

fn scope() { check!('a); }

Moving towards the early expansion phase, since PR rust-lang/rust#146961 you can use $e:expr in places where ConfigurationPredicate is expected (as long as it reparses as a configuration predicate // is structurally a configuration predicate) despite it not allowing arbitrary expressions in its grammar.

[fmease]

So in summary path/TypePath being "malleable"/reinterpretable isn't a one-off, there are lots of similar cases due to most syntax fragments being treated as opaque and there being various "ad hoc" "conversion" rules in rustc (which is fragile and buggy; see for example issue rust-lang/rust#143221). IINM so far this isn't documented in an exhaustive and principled manner in the Reference.

Maybe I've missed the mark somewhere and the examples I gave aren't comparable to the path case.

[fmease]

Of course, theoretically we could precisely describe these opaque syntax fragment rules by mimicking rustc in the way it deals with metavars internally. However, that'd be rather janky I feel like.

That would entail updating the Reference grammar like this perhaps:

...

Expression ->
      ExpressionWithoutBlock
    | ExpressionWithBlock
    | `⟦expr ` Expression `⟧`
    | `⟦literal ` `-`? LiteralExpression `⟧`
    | `⟦block ` BlockExpressionNoAttributes `⟧`
    | `⟦path ` TypePath `⟧`;

...

where ⟦ and ⟧ would by definition represent tokens that are not part of the normal user-writable alphabet. They would form the basis of branded invisible delimiters. So a metavariable $x of kind block matching token sequence {} would be transcribed as ⟦block {}⟧.

Instead, a path in an expression should be

PathExpression →
    PathInExpression
   | QualifiedPathInExpression

They should not, they are still TypePaths. Consider:

// OK

macro_rules! m { ($p:path) => { $p } }

fn main() {
    let _ = m!(Option<i32>::None);
            // ~~~~~~~~~~~~~~~~~ a valid TypePath
            //                   not a valid PathInExpression (those require turbofish)
}

[xmh0511]

Option<i32>::None The token sequence is interpreted as PathExprSegment, which is a PathInExpression, which is a PathExpression, which is a ExpressionWithoutBlockNoAttrs, which is a ExpressionWithoutBlock, which is a Expression. So, it can appear in places that expect an Expression.

However, if the same form token sequence is interpreted as TypePath, even though they have the same tokens, TypePath cannot appear in places that expect an Expression. Because TypePath is not an Expression.

Commonly, in a language standard, we impose some requirements by syntax and sometimes by the semantic rules. For example, A + B, the left and right operands must be interpreted as Expression. If some tokens have already been interpreted as TypePath, it would cause a hard error if they appear as the operand.

[fmease]

Option<i32>::None The token sequence is interpreted as […] a Expression.

No, the token sequence Option<i32>::None is not an Expression¹. Option::<i32>::None would be an Expression (notice the turbofish). Rule PathExprSegment requires token :: to precede the < of GenericArgs.

However, if the same form token sequence is interpreted as TypePath, even though they have the same tokens, TypePath cannot appear in places that expect an Expression. Because TypePath is not an Expression.

While the conclusion "TypePath is not an Expression" is correct, the reasoning is not, since that's not how formal grammars work. Grammar or rather production rules are not nominally identified, they are structurally identified. If rule R and rule Q produce the same set of strings, then they are identical.

The set of strings generated by PathInExpression is a proper subset of the set of strings generated by TypePath, assuming PR #2247 has been merged². Consequently, one can (informally) state that PathInExpression is a TypePath and that TypePath isn't a PathInExpression and thus indeed not an Expression just like you said.

Now, this whole "opaque syntax fragment" business (as incompletely touched upon in macro.decl.transcription.fragment) and "nominal matching" (both discussed in #2265 (comment)) is specific to Rust's macro-by-example, not to formal grammars. #2265 (comment) provides a way to model this with formal grammars where both Option::<i32>::None and ⟦path Option<i32>::None⟧ are Expressions.

Commonly, in a language standard, we impose some requirements by syntax and sometimes by the semantic rules. For example, A + B, the left and right operands must be interpreted as Expression. If some tokens have already been interpreted as TypePath, it would cause a hard error if they appear as the operand.

While I haven't written a language standard in my life I would say I'm rather familiar with formal language theory and on the practical side, I'm decently skilled in writing parsers for context-free languages, more specifically recursive-descend and Pratt parsers, so I wouldn't call myself fully unqualified when it comes to this matter.

_{Edit: I had to rewrite several wrong statements because my very tired brain mixed up TypePath & ExpressionInPath in parts of the argument. Should be all rectified now.}

Footnotes

1. And it certainly isn't a PathExprSegment even if Rust didn't require turbofish in expressions since then there would be two path segments, not one! ↩

2. GenericArgs as defined on branch master disagrees with rustc and is thus invalid. ↩

[xmh0511]

For example, a token sequence that can be interpreted as an identifier or a keyword; however, a keyword cannot appear in a place that expects an identifier, even if they have the same token sequence.

Moreover, take a look at the C++ standard, for example, cast-expression

cast-expression:

• unary-expression
• ( type-id ) cast-expression

(int*){};

This is an invalid cast-expression; however, this is not regulated by semantics, it's specified by grammar. The token sequence {} is not a cast-expression.

[fmease]

For example, a token sequence that can be interpreted as an identifier or a keyword; however, a keyword cannot appear in a place that expects an identifier, even if they have the same token sequence.

I don't know what this is in reply to or what this is trying to prove or disprove.

This is an invalid cast-expression; however, this is not regulated by semantics, it's specified by grammar. The token sequence {} is not a cast-expression.

That's true but I don't know what you're trying to tell me, as I said, I'm deeply familiar with formal grammars, you don't need to teach me the basics :)

[fmease]

I feel like we're talking past each other. So I'd like to clarify: I agree that "[the] definition of fragment specifier is inconsistent". I agree that stating fragment specifiers of kind path accept "a TypePath" is misleading or even incorrect in light of "opaque matching".

The points I was trying to express are:

1. The macro-by-example section should precisely define match since the rule "fragment-specifier matches a Rust syntax fragment of the kind specified" is ambiguous / underspecified
2. path/TypePath→expr/Expression isn't the only problematic pairing, there are tons of cases were "branded" opaque syntax fragments can be converted into another one
   • however, the fact that TypePath Option<i32>::None is allowed to expand into an expression context despite not being an Expression is definitely unlike the other cases I've listed (expr→stmt, item→stmt, literal→expr, lifetime→LIFETIME_OR_LABEL (unless keyword lifetime), expr->pat (if maybe-negated literal), …) which are mostly widening (barring some crazy rustc bugs)
3. I don't know what the best way forward is but I'd love to see a more principled or at least exhaustive solution; by means of an "interconversion/compatibility table", by reifying opaque syntax fragments in the grammar (cue ⟦…⟧) or by doing XY

[fmease]

However, there might be a simple interim solution, a "one-line change". Feel free to cook something up yourself in a PR. I myself will revisit this topic in 1 or 2 weeks and see if there can be done something that isn't a $massive_overhaul.

[xmh0511]

However, there might be a simple interim solution, a "one-line change". Feel free to cook something up yourself in a PR. I myself will revisit this topic in 1 or 2 weeks and see if there can be done something that isn't a $massive_overhaul.

Maybe we need to invite preprocessing-syntax for the fragment-specifier in macro. Then, preprocessing-token can be converted to relevant syntax according to the context where the preprocessing-syntax is used.

For example, path denotes preprocessing-path, it matches the token sequence that could be a TypePath or PathExpression. Then, if such a preprocessing-path $p is used in fn()->$p, then it is converted to TypePath. it will be a PathExpression in _ = $p.

expr matches preprocessing-expr, that is, any token sequence that could form an Expression matches preprocessing-expr, and preprocessing-expr can only be converted to an Expression, that is, preprocessing-expr A::B::C denoted by $e cannot appear in fn()->$e because preprocessing-expr cannot be converted to TypePath.

In other words, a preprocessing-syntax doesn't have a concrete syntax meaning until it's converted to a specific syntax. Different preprocessing-syntax limits what kind of syntax it can be converted to. tt matches preprocessing-tokentree, it can be converted to any possible syntax.