Caution

You’re reading a draft of the Ferrocene Language Specification. Some parts of this document might be missing, incomplete or incorrect. Our aim is to have the specification ready by the end of 2022.

20. Macros

Legality Rules

20:1 A macro is a custom definition that extends Rust by defining callable syntactic transformations. The effects of a macro are realized through macro invocations or attribute use. Macros come in two distinct forms:

20:4 Tokens are a subset of lexical elements consumed by macros.

20.1. Declarative Macros

Syntax

MacroRulesDeclaration ::=
    macro_rules ! Name MacroRulesDefinition

MacroRulesDefinition ::=
    ( MacroRuleList ) ;
  | [ MacroRuleList ] ;
  | { MacroRuleList }

MacroRuleList ::=
    MacroRule (; MacroRule)* ;?

MacroRule ::=
    MacroMatcher => MacroTranscriber

MacroMatcher ::=
    ( MacroMatch* )
  | [ MacroMatch* ]
  | { MacroMatch* }

MacroTranscriber ::=
    DelimitedTokenTree

MacroMatch ::=
    MacroMatcher
  | MacroMatchToken
  | MacroMetavariableMatch
  | MacroRepetitionMatch

20.1:1 A MacroMatchToken is any lexical element in category LexicalElement, except punctuation $ and category Delimiter.

Legality Rules

20.1:2 A declarative macro is a macro that associates a name with a set of syntactic transformation macro rules.

20.1:3 A macro rule is a construct that consists of a macro matcher and a macro transcriber.

20.1:4 A macro matcher is a construct that describes a syntactic pattern that a macro must match.

20.1:5 A macro match is the most basic form of a satisfied macro matcher.

20.1:6 A macro transcriber is a construct that describes the replacement syntax of a macro.

20.1:7 A declarative macro is invoked using a macro invocation.

Examples

macro_rules! answer_to_life {
    () => { 42 };
}

20.1.1. Metavariables

Syntax

MacroMetavariableMatch ::=
    $ MacroMetavariable : MacroFragmentSpecifier

MacroMetavariable ::=
    Keyword
  | NonKeywordIdentifier

MacroFragmentSpecifier ::=
    block
  | expr
  | ident
  | item
  | lifetime
  | literal
  | meta
  | pat
  | pat_param
  | path
  | stmt
  | tt
  | ty
  | vis

MacroMetavariableIndication ::=
    $ MacroMetavariable

Legality Rules

20.1.1:1 Declarative macros employ metavariables to match a token of a particular kind and bind its value to a name for use during macro transcription.

20.1.1:2 A metavariable is a macro match that describes a variable.

20.1.1:3 A metavariable is visible in the macro transcriber of the macro rule of the macro matcher it is declared in.

20.1.1:4 A fragment specifier is a construct that indicates the type of a metavariable.

20.1.1:5 Fragment specifier kinds impose the following fragment specifier restructions on the tokens that follow them:

  • 20.1.1:6 expr shall only be followed by =>, ,, or ;.

  • 20.1.1:7 pat shall only be followed by =>, ,, =, |, if, or in.

  • 20.1.1:8 path shall only be followed by =>, ,, =, |, ;, :, >, >>, [, {, as, where, or a metavariable with the block fragment specifier kind.

  • 20.1.1:9 pat_param shall only be followed by =>, ,, =, |, if, or in.

  • 20.1.1:10 stmt shall only be followed by =>, ,, or ;.

  • 20.1.1:11 ty shall only be followed by =>, ,, =, |, ;, :, >, >>, [, {, as, where, or a metavariable with the block fragment specifier kind.

  • 20.1.1:12 vis shall only be followed by ,, an identifier except for priv, any token that may begin a TypeSpecification, or a metavariable with the ident, ty or block fragment specifier kind.

  • 20.1.1:13 Any other kind may be followed by any token.

20.1.1:14 A metavariable indication is a construct that indicates a metavariable.

Examples

macro_rules! square {
    ($e:expr) => { $e * $e };
}

20.1.2. Repetition

Syntax

MacroRepetitionMatch ::=
    $ ( MacroRepetitionMatchContent ) MacroRepetitionSeparator? MacroRepetitionOperator

MacroRepetitionMatchContent ::=
    MacroMatch*

MacroRepetitionTranscriber ::=
    $ ( TokenTree* ) MacroRepetitionSeparator? MacroRepetitionOperator

MacroRepetitionOperator ::=
    +
  | *
  | ?

20.1.2:1 A MacroRepetitionSeparator is any lexical element in category LexicalElement, except punctuation +, *, ?, and category Delimiter.

Legality Rules

20.1.2:2 A macro repetition in matching allows for a syntactic pattern to be matched zero or multiple times during macro matching.

20.1.2:3 A macro repetition in transcription allows for a syntactic pattern to be transcribed zero or multiple times during macro transcription.

20.1.2:4 A macro repetition is either a macro repetition in matching or a macro repetition in transcription.

20.1.2:5 A repetition operator is a construct that indicates the number of times a macro repetition in matching or a macro repetition in transcription can be repeated.

20.1.2:6 The effects of a repetition operator are as follows:

  • 20.1.2:7 * - Zero or more repetitions.

  • 20.1.2:8 + - One or more repetitions.

  • 20.1.2:9 ? - Zero or one repetition.

20.1.2:10 A macro repetition has the following additional restrictions:

Examples

macro_rules! generate_pairs {
    ( $( $first:ident )* ; $( &second:ident )* )
        =>
    { $( $first, $second )* };
}

20.2. Procedural Macros

Legality Rules

20.2:1 A procedural macro is a macro that encapsulates syntactic transformations in a function. Procedural macros consume one or more streams of tokens and produce a stream of tokens.

20.2:2 Procedural macros shall be defined in a crate subject to attribute crate_type where the type is proc-macro.

20.2:3 A macro implementation function is the function that encapsulates the syntactic transformations of a procedural macro.

20.2:4 A macro implementation function enters the name of the procedural macro into the macro namespace.

20.2.1. Function-like Macros

Legality Rules

20.2.1:1 A function-like macro is a procedural macro that consumes a stream of tokens and produces a stream of tokens.

20.2.1:2 The macro implementation function of a function-like macro shall be subject to the following restrictions:

20.2.1:9 A function-like macro is invoked using a macro invocation.

20.2.1:10 The sole parameter of the macro implementation function captures the token stream produced from the DelimitedTokenTree of the macro invocation, excluding outer Delimiters.

Examples

#[proc_macro]
pub fn make_answer_to_life(_items: TokenStream) -> TokenStream {
    "fn answer_to_life() -> u32 { 42 }".parse().unwrap()
}

20.2.2. Derive Macros

Legality Rules

20.2.2:1 A derive macro is a procedural macro that consumes a stream of tokens and produces a stream of tokens. Derive macros are used to construct new syntax for abstract data types.

20.2.2:2 The macro implementation function of a derive macro shall be subject to the following restrictions:

20.2.2:9 A derive macro is invoked using attribute derive.

20.2.2:10 The sole parameter of the macro implementation function captures the token stream produced from the related EnumDeclaration, StructDeclaration, or UnionDeclaration.

20.2.2:11 A derive macro adds all its declared derive helper attributes into the derive helper attribute scope of the abstract data type the attribute is attached to.

20.2.2:12 A derive helper attribute is an inert attribute that acts as a hint to attribute derive.

Examples

#[proc_macro_derive(Answer)]
pub fn derive_answer_to_life(_items: TokenStream) -> TokenStream {
    "fn answer_to_life() -> u32 { 42 }".parse().unwrap()
}

20.2.3. Attribute Macros

Legality Rules

20.2.3:1 An attribute macro is a procedural macro that consumes two streams of tokens to produce a single stream of tokens, and defines a new outer attribute that can be attached to items. Attribute macros are used to replace items with other items.

20.2.3:2 The macro implementation function of an attribute macro shall be subject to the following restrictions:

20.2.3:9 An attribute macro is invoked using an attribute of the form

  • 20.2.3:10 #[SimplePath], or

  • 20.2.3:11 #[SimplePath DelimitedTokenTree]

20.2.3:12 The first function parameter of the macro implementation function captures the token stream produced from the DelimitedTokenTree of the invoking attribute, excluding outer Delimiters. If no DelimitedTokenTree is provided, then the token stream is considered empty.

20.2.3:13 The second function parameter of the macro implementation function captures the token stream produced from the related item, including all outer attributes that apply to that item.

Examples

#[proc_macro_attribute]
pub fn output_and_return_item
    (attr: TokenStream, item: TokenStream) -> TokenStream
{
    println!("attr: \"{}\"", attr.to_string());
    println!("item: \"{}\"", item.to_string());
    item
}

20.3. Macro Invocation

Syntax

MacroInvocation ::=
    SimplePath ! DelimitedTokenTree

DelimitedTokenTree ::=
    ( TokenTree* )
  | [ TokenTree* ]
  | { TokenTree* }

TokenTree ::=
    DelimitedTokenTree
  | NonDelimitedToken

TerminatedMacroInvocation ::=
    SimplePath ! ( TokenTree* ) ;
  | SimplePath ! [ TokenTree* ] ;
  | SimplePath ! { TokenTree* }

20.3:1 A NonDelimitedToken is any lexical element in category LexicalElement, except delimiters (, ), [, ], {, and }.

Legality Rules

20.3:2 A macro invocation is a call of a declarative macro or function-like macro that is expanded statically and replaced with the result of the macro.

20.3:3 A terminated macro invocation is a macro invocation that may be used as a statement.

Examples

20.3:4 See Paragraph 20.1. for the declaration of answer_to_life.

answer_to_life!();

20.3:5 See Paragraph 20.1.1. for the declaration of square.

square!(5);

20.3:6 See Paragraph 20.1.2. for the declaration of generate_pairs.

generate_pairs!(1, 2, 3; 9, 8, 7);

20.3:7 See Paragraph 20.2.1. for the declaration of make_answer_to_life.

make_answer_to_life!();

20.3:8 See Paragraph 20.2.2. for the declaration of Answer.

#[derive(Answer)]
struct derive_macro_invoker;

20.3:9 See Paragraph 20.2.3. for the declaration of output_and_return_item.

#[output_and_return_item]
fn attribute_macro_invoker() {}

20.4. Macro Expansion

Legality Rules

20.4:1 Macro expansion is the process of statically executing a macro invocation and replacing it with the produced output of the macro invocation.

20.4:2 Macro expansion of declarative macros proceeds as follows:

  1. 20.4:3 The TokenTree of the macro invocation has all outer block docs and outer line docs contained within replaced by their equivalent attribute doc representation.

  2. 20.4:4 The TokenTree of the macro invocation is matched against the macro rules of the resolved macro by considering individual macro matchers. It is a static error if no macro matcher is satisfied.

  3. 20.4:5 The macro transcriber of the satisfied macro rule produces its result, with all metavariable indications resolved. It is a static error if the macro transcriber fails to produce its result.

  4. 20.4:6 The macro invocation is replaced with the result of the macro transcriber. It is a static error if the result cannot be parsed according to the expected expansion syntax of the context where the macro invocation resides. The expected expansion syntax is as follows:

    1. 20.4:7 If the macro invocation appears as part of a statement, the output is required to constitute zero or more statements.

    2. 20.4:8 If the macro invocation appears as part of an expression-without-block, the output is required to constitute an expression.

    3. 20.4:9 If the macro invocation appears as part of a pattern-without-range, the output is required to constitute zero or more patterns.

    4. 20.4:10 If the macro invocation appears as part of an associated item, an item within an external block, or another macro invocation, the output is required to constitute zero or more items.

    5. 20.4:11 If the macro invocation appears as part of a type specification without bounds, the output is required to constitute a type.

20.4:12 Expansion of function-like macros proceeds as follows:

  1. 20.4:13 The TokenTree of the macro invocation has all outer block docs and outer line docs contained within replaced by their equivalent attribute doc representation.

  2. 20.4:14 The TokenTree of the macro invocation is transformed into a corresponding proc_macro::TokenStream.

  3. 20.4:15 The macro implementation function is called with the proc_macro::TokenStream as its sole argument. It is a static error if the macro implementation function call fails.

  4. 20.4:16 The macro invocation is replaced with the returned proc_macro::TokenStream of the macro implementation function call. It is a static error if the result can not be parsed according to the expected expansion syntax of the context where the macro invocation resides. The expected expansion syntax is as follows:

    1. 20.4:17 If the macro invocation appears as part of a statement, the output is required to constitute zero or more statements.

    2. 20.4:18 If the macro invocation appears as part of an expression-without-block, the output is required to constitute an expression.

    3. 20.4:19 If the macro invocation appears as part of a pattern-without-range, the output is required to constitute zero or more patterns.

    4. 20.4:20 If the macro invocation appears as part of an associated item, an item within an external block, or another macro invocation, the output is required to constitute zero or more items.

    5. 20.4:21 If the macro invocation appears as part of a type specification without bounds, the output is required to constitute a type.

20.4:22 Expansion of derive macros proceeds as follows:

  1. 20.4:23 The item subject to the derive macro has all outer block docs and outer line docs contained within replaced by their equivalent attribute doc representation.

  2. 20.4:24 The item subject to the derive macro is transformed into a corresponding proc_macro::TokenStream without the invoking derive attribute as well as any preceding derive attributes.

  3. 20.4:25 The macro implementation function is called with the proc_macro::TokenStream as its sole argument. It is a static error if the macro implementation function call fails.

  4. 20.4:26 The returned proc_macro::TokenStream of the macro implementation function call is appended to the enclosing block expression or module where the related EnumDeclaration, StructDeclaration, or UnionDeclaration resides. It is a static error if the output proc_macro::TokenStream does not constitute zero or more items.

20.4:27 The expansion of attribute macros proceeds as follows:

  1. 20.4:28 The DelimitedTokenTree of the invoking attribute macro is transformed into a corresponding proc_macro::TokenStream without the outer Delimiters. If no DelimitedTokenTree is provided, and empty proc_macro::TokenStream is used. This proc_macro::TokenStream constitutes the first function parameter of the macro implementation function.

  2. 20.4:29 The item subject to the attribute macro has all outer block docs and outer line docs contained within replaced by their equivalent attribute doc representation.

  3. 20.4:30 The item subject to the attribute macro is transformed into a corresponding proc_macro::TokenStream without the invoking attribute. This proc_macro::TokenStream constitutes the second function parameter of the macro implementation function.

  4. 20.4:31 The macro implementation function is called with the two [proc_macro::TokenStream]s as the two arguments. It is a static error if the macro implementation function call fails.

  5. 20.4:32 The item subject to the attribute macro is replaced with the returned proc_macro::TokenStream of the macro implementation function call. It is a static error if the output proc_macro::TokenStream does not constitute zero or more items.

20.4.1. Macro Matching

Legality Rules

20.4.1:1 Macro matching is the process of performing rule matching and token matching.

20.4.1.1. Rule Matching

Legality Rules

20.4.1.1:1 Rule matching is the process of consuming a TokenTree in an attempt to fully satisfy the macro matcher of a macro rule that belongs to a resolved declarative macro.

20.4.1.1:2 Rule matching proceeds as follows:

  1. 20.4.1.1:3 The macro matchers of all macro rules that belong to a resolved macro are tried against the TokenTree of the macro invocation, in declarative order. In the event of a static error, no further attempts at selecting a subsequent macro matcher are made.

  2. 20.4.1.1:4 The macro match of a candidate macro matcher is tried against the TokenTree of the macro invocation by matching individual tokens, in left-to-right order.

  3. 20.4.1.1:5 Matching does not employ lookahead. It is a static error if matching a candidate macro matcher is ambiguous.

  4. 20.4.1.1:6 Matching does not employ backtracking. It is a static error if matching a candidate macro matcher fails while parsing into a metavariable and having consumed at least one token while parsing the metavariable.

  5. 20.4.1.1:7 It is a static error if no macro matcher is selected.

20.4.1.2. Token Matching

Legality Rules

20.4.1.2:1 Token matching is the process of consuming a TokenTree in an attempt to fully satisfy a macro match of a selected macro matcher that belongs to a resolved declarative macro.

20.4.1.2:2 Token matching proceeds as follows:

20.4.1.2:3 The outer Delimiters of a macro matcher match any outer Delimiters in the macro invocation.

20.4.1.2:4 A metavariable in a macro matcher is matched against a sequence of tokens in the macro invocation based on its fragment specifier:

20.4.1.2:19 Once a metavariable is matched, the matching sequence of tokens is bound to that metavariable.

20.4.1.2:20 Repetition in a macro matcher is matched based on how many times the pattern appears consecutively optionally separated by a separator in the TokenTree of the macro invocation, as follows:

  • 20.4.1.2:21 If the repeated pattern includes a separator, then the separator must be able to follow the repeated pattern.

  • 20.4.1.2:22 If the repeated pattern can appear multiple times, then the repeated pattern must be able to follow itself.

  • 20.4.1.2:23 If the repeated pattern can appear zero times, then the preceding pattern must be able to follow the succeeding pattern.

  • 20.4.1.2:24 The repeated pattern must be able to follow the preceding pattern.

  • 20.4.1.2:25 The succeeding pattern must be able to follow the repeated pattern.

20.4.1.2:26 A repetition index is a monotonically increasing number that is initialized to zero, and incremented by one.

20.4.1.2:27 Once a metavariable is matched, the matching sequence of tokens is treated as follows:

  1. 20.4.1.2:28 The matching sequence of tokens is stored in an ordered collection at the current repetition index.

  2. 20.4.1.2:29 The current repetition index is incremented by one.

20.4.1.2:30 Each matched metavariable in a macro repetition in matching is bound separately, where the matches are stored in an ordered collection.

20.4.1.2:31 Any other token in a macro matcher is matched literally against the TokenTree of the macro invocation.

20.4.1.2:32 It is a static error if the TokenTree of the macro invocation contains leftover tokens after macro matching.

20.4.2. Macro Transcription

Legality Rules

20.4.2:1 Macro transcription is the process of producing the expansion of a declarative macro.

20.4.2:2 Macro transcription proceeds as follows:

20.4.2:3 Every metavariable indication found in the DelimitedTokenTree of the macro transcriber that belongs to a matched macro rule is replaced by the matched sequence of tokens of the metavariable.

20.4.2:4 Unresolved metavariable indications are kept as tokens in the output verbatim.

20.4.2:5 Every macro repetition in transcription found in the DelimitedTokenTree of the macro transcriber shall be transcribed by repeatedly transcribing the tokens inside of it.

20.4.2:6 The number of transcription repetitions for a macro repetition in transcription shall depend on its repetition operator, as follows:

20.4.2:10 A metavariable indication that is matched inside of a macro repetition shall not be used outside of a macro repetition in transcription.

20.4.2:11 A metavariable indication shall be used in a macro repetition in transcription of the same nesting depth as its corresponding metavariable appears in the macro matcher.

20.4.2:12 A metavariable indication within a macro repetition in transcription shall repeat the same number of times in its matching macro repetition if the macro repetition occurs at the same nesting depth.

20.4.2:13 Multiple transcribed metavariable indications in the same macro repetition in transcription shall repeat the same number of times.

20.4.2:14 When transcribing a metavariable indication in a macro repetition in transcription, the metavariable indication is replaced with the matched sequence of tokens of the corresponding iteration of the repetition. metavariable taken from the ordered collection.

20.4.2:15 A metavariable indication in a macro repetition in transcription shall be transcribed to the matched tokens in order, as follows:

macro_rules! foo {
    ( $($expr:expr)* ) => {
        $( $expr ; )*
        // $expr is an error
    };
    ( $( $( $expr:expr )*  )*  ) => {
        $($($expr)*)*
    }
}

foo! {
  0
  1
  2
}

20.4.2:16 yields 0;1;2;

20.4.2:17 Given a macro invocation with N metavariable actuals, a macro of the form

macro_rules! m {
    ( $(param: expr)* ) => {
        $( $param )*
    }
}

20.4.2:18 is equivalent to a macro of the form

macro_rules! m {
    ( $param_1: expr $param_2: expr ... $param_N: expr) => {
        $param_1 $param_2 ... $param_N
    }
}

20.4.2:19 where the metavariable of the macro repetition in matching are repeated N times, and the metavariable indications of the macro repetition in transcription are repeated N times. Invoking such a macro relates the first metavariable actual of the macro invocation with the first metavariable of the macro repetition in matching, the second metavariable actual with the second metavariable, etc.

20.5. Hygiene

20.5:1 Hygiene is a property of macros and identifiers` that appear within them, which aims to eliminate the syntactic interference between a macro and its environment.

Legality Rules

20.5:2 Hygiene is categorized as follows:

20.5:6 Every macro has associated hygiene that depends on its kind:

20.5:9 The metavariable $crate in a declarative macro‘s expansion refers to the crate the declarative macro was declared in.