proc-macro2

proc-macro2 is a wrapper around the compiler's proc_macro crate. It provides a way to use procedural macro types in contexts where the standard proc_macro crate is unavailable or unsuitable.

Use proc-macro2 to:

1. Use macro types in non-macro contexts: Types from proc_macro can only exist inside a procedural macro. proc_macro2 types can be used in build.rs, main.rs, or any other non-macro code. This allows libraries like syn and quote to be used outside of macros.
2. Enable unit testing: Because proc_macro can only be executed within the compiler's macro expansion process, code using it cannot be unit tested. Implementing macro logic using proc_macro2 allows you to test components in isolation.

To use proc-macro2, add it to your Cargo.toml dependencies:

[dependencies]
proc-macro2 = "1.0"

By default, proc-macro2 tracks the stable compiler API. To access additional APIs available in the most recent nightly compiler, you must pass the procmacro2_semver_exempt configuration flag to rustc via RUSTFLAGS.

Warning: This flag is infectious. If you use it, any crate depending on your crate must also use it. This serves as a reminder that you are opting out of standard semver guarantees, as proc-macro2 may introduce breaking changes to these unstable APIs in minor versions.

RUSTFLAGS='--cfg procmacro2_semver_exempt' cargo build

The force() and unforce() functions allow a developer to manually control whether proc-macro2 uses the compiler's native proc-macro API or its own internal fallback implementation. This is typically used for troubleshooting or ensuring consistent behavior across different compiler environments.

proc_macro2::force();
proc_macro2::unforce();

A TokenStream is a collection of TokenTrees. When you iterate over a TokenStream using into_iter(), you are performing a shallow iteration.

This means if your TokenStream contains a delimited group (for example, { a; b }), the iterator will yield the entire group as one TokenTree rather than yielding a and b individually. To access the contents of a group, you must first inspect the TokenTree to see if it is a Group and then iterate over that group's contents.

A wrapper around the compiler's proc_macro crate. It serves two primary purposes:

1. Portability: proc_macro types can only exist inside procedural macros. proc_macro2 types can exist anywhere, including build.rs and main.rs. This allows foundational libraries like syn and quote to be used in non-macro contexts.
2. Testability: Because proc_macro types are restricted to the macro expansion context, they cannot be used in standard unit tests. Using proc_macro2 allows you to write unit tests for your macro logic in isolation.

Note on Thread-Safety: Most types in this crate are !Sync because the underlying compiler types use thread-local memory and cannot be accessed from a different thread.

The subspan method on Literal allows you to retrieve a Span that is a subset of the literal's original span, covering only the source bytes within a specific range.

Warning: The underlying proc_macro::Literal::subspan method is a nightly-only feature. When using proc-macro2 with a stable compiler, subspan will always return None.

// Returns a subset span if the range is valid, otherwise None
let sub = my_literal.subspan(0..5);

By default, proc-macro2 tracks the most recent stable compiler API. To access functionality from the most recent nightly compiler, you must pass the procmacro2_semver_exempt configuration flag to rustc.

Warning: This flag is infectious. You must apply it not only to your crate but also to any crate that depends on your crate. This serves as a reminder that you are operating outside of normal semver guarantees, as minor versions of proc-macro2 may introduce breaking changes to these unstable APIs.

RUSTFLAGS='--cfg procmacro2_semver_exempt' cargo build

When writing a procedural macro, you typically convert the input proc_macro::TokenStream into a proc_macro2::TokenStream to perform transformations, then convert the result back to proc_macro::TokenStream for the compiler.

If you are using the syn crate for parsing, use the parse_macro_input! macro to ensure parse errors are propagated correctly to the compiler.

extern crate proc_macro;

#[proc_macro_derive(MyDerive)]
pub fn my_derive(input: proc_macro::TokenStream) -> proc_macro::TokenStream {
    let input = proc_macro2::TokenStream::from(input);

    let output: proc_macro2::TokenStream = {
        /* transform input */
        # input
    };

    proc_macro::TokenStream::from(output)
}

When writing a procedural macro, you typically convert the input proc_macro::TokenStream into a proc_macro2::TokenStream to perform transformations, then convert the result back to a proc_macro::TokenStream for the compiler.

If you are using the syn crate for parsing, use the parse_macro_input! macro instead of manual conversion to ensure parse errors are correctly propagated to the compiler.

extern crate proc_macro;

#[proc_macro_derive(MyDerive)]
pub fn my_derive(input: proc_macro::TokenStream) -> proc_macro::TokenStream {
    let input = proc_macro2::TokenStream::from(input);

    let output: proc_macro2::TokenStream = {
        /* transform input */
    };

    proc_macro::TokenStream::from(output)
}

To convert a string slice into a TokenStream, use the token_stream function. This function performs lexical analysis, handling whitespace, comments (including doc comments), identifiers, literals, punctuation, and groups. If the input is not a valid token stream, it returns a LexError containing the span where the error occurred.

Note: Doc comments are transformed into a specific pattern: # or #! followed by doc = "comment" inside a group.

// Note: token_stream is marked pub(crate) in the source, 
// implying it is an internal utility for the proc-macro2 crate 
// rather than a public API for end-users. 
// However, it represents the primary parsing entrypoint.

let tokens = token_stream(cursor_input)?; 

Spans represent a location in the source code. Common operations include:

• Span::call_site(): Returns the call site span.
• Span::mixed_site(): Returns the mixed site span.
• span.resolved_at(other): Returns a span representing the resolution of the first span at the second.
• span.located_at(other): Returns a span representing the location of the first span relative to the second.