Add hygiene attributes to compile expanded source code

[zalanlevai]

Proposal

Full title: Add optional hygiene attributes to compile already expanded source code for hygienic code generation

Summary of proposed changes:

• Add unstable rustc_hygiene attribute with necessary fields to encode hygiene information.
• Add annotate_with_hygiene_attributes function to rustc_expand to add optional rustc_hygiene attributes to the AST.
• Modify macro expansion to consider rustc_hygiene attributes as the source of hygiene information, if present. (No changes to the parser necessary.)

Currently, tool developers trying to use information (types, lifetimes, etc.) from rustc to generate new code, or modifications to existing code, have to drive an instance of the compiler through parsing, macro expansion, AST -> HIR lowering, HIR analysis (and so on, until they have all the desired information from the compiler). Then, they have to convert their modified AST representation into source code, which then can be passed to a second instance of the compiler, with the new code as a string input. When dealing with macro expanded code however, this approach can fail in multiple ways, as the hygiene information required to keep the code compiling correctly is lost in the process of conversion to source code.

Take the tool mutest-rs as an example (note, I am the developer of mutest-rs), which uses rustc to produce code mutations to evaluate test suite adequacy (whether the test suite is capable of catching unintended changes to the code). Code generating tools like mutest-rs have to circumvent compilation issues by patching the macro expanded code, by undoing some expansions, adding flags for internals to compile, and still leaving some common cases completely unresolved. Not to mention the potential hygiene pitfalls of such an approach.

To solve this issue, I propose that the compiler defines a new unstable hygiene attribute, rustc_hygiene, which can be optionally added to every AST node. This attribute contains all the necessary fields required to retain the hygiene information of each scope, identifier, etc. (help is needed in identifying exactly what information needs to be serialized in the attributes to ensure correctness). To add these annotations, I propose the addition of a new function in rustc_expand, annotate_with_hygiene_attributes(&mut rustc_ast::ast::Crate, <ADDITIONAL_DEPENDENCIES>), which can be called by tool developers to add these optional attributes to their AST. Then, they can print the source code with the already existing rustc_ast_pretty::pprust facilities, but this will also now print the hygiene attributes that were added previously. The compiler can then make use of these annotations, when present, without any changes to the parser. The rustc_hygiene attribute will be treated as inert in the parser, and will remain for processing during the macro expansion phase. Then, during macro expansion, these attributes can be read, if present, and treated as the source of hygiene information. Afterwards, the rest of the compilation passes can be performed as usual, without any additional changes.

In effect, this would allow for tools like mutest-rs to perform code generation driven by rustc on properly expanded code (including module expansion and local macro expansion), while still retaining the necessary hygiene information to compile the majority of crates out of the box, and do so correctly. For mutest-rs, this means much expanded crate support, and hopefully increased adoption of mutation testing in the Rust community. Other tools doing similar work will also get the same benefits.

Mentors or Reviewers

Needs mentors and reviewers.

@davidtwco suggested that I turn this into an MCP, the original Zulip discussion can be found here.

In addition, based on the experts map, it seems the following people might be able to help mentor certain aspects of the work:

• macro expansion, and name resolution and hygiene: @petrochenkov,
• parser: @estebank,
• (AST -> HIR lowering: @oli-obk).

Process

The main points of the Major Change Process are as follows:

• File an issue describing the proposal.
• A compiler team member or contributor who is knowledgeable in the area can second by writing @rustbot second.
  • Finding a "second" suffices for internal changes. If however, you are proposing a new public-facing feature, such as a -C flag, then full team check-off is required.
  • Compiler team members can initiate a check-off via @rfcbot fcp merge on either the MCP or the PR.
• Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

Comments

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

[rustbot]

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

cc @rust-lang/compiler @rust-lang/compiler-contributors

[apiraino]

@rfcbot concern added-complexity-to-frontend

From Zulip

[nnethercote]

If it's just done via attributes, that seems ok to me.

@rustbot resolve added-complexity-to-frontend

[apiraino]

trying again

@rfcbot resolve added-complexity-to-frontend