Skip to content

#[newtype_oparg] derive macro#7489

Open
ShaharNaveh wants to merge 6 commits intoRustPython:mainfrom
ShaharNaveh:macro-newtype-oparg
Open

#[newtype_oparg] derive macro#7489
ShaharNaveh wants to merge 6 commits intoRustPython:mainfrom
ShaharNaveh:macro-newtype-oparg

Conversation

@ShaharNaveh
Copy link
Contributor

@ShaharNaveh ShaharNaveh commented Mar 23, 2026

Reason for creating a new crate for this is because rustpython-derive-impl has rustpython-compiler as a dependency, so it created a circular dependency:/

Summary by CodeRabbit

  • New Features

    • Added a new procedural-macro crate to support macro-driven enum/newtype generation.
  • Refactor

    • Modernized compiler opcode/newtype handling by adopting the new macro system.
  • Bug Fixes

    • Fixed bytecode instruction parsing for containment operations.
  • Chores

    • Added a workspace dependency entry to enable the new macro crate.

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Mar 23, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: 3d456821-89e6-447c-8fb4-d08b70837232

📥 Commits

Reviewing files that changed from the base of the PR and between f5c2f9b and 9fbc6d3.

📒 Files selected for processing (1)
  • crates/macros/src/newtype_oparg.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/macros/src/newtype_oparg.rs

📝 Walkthrough

Walkthrough

Adds a new procedural-macro crate rustpython-macros providing #[newtype_oparg], refactors many bytecode oparg enums/newtypes to use it, updates Cargo workspace deps, and fixes Invert flag parsing in the VM frame handler.

Changes

Cohort / File(s) Summary
Workspace Dependencies
Cargo.toml, crates/compiler-core/Cargo.toml
Add workspace dependency rustpython-macros = { path = "crates/macros", version = "0.5.0" }.
New Procedural Macro Crate
crates/macros/Cargo.toml, crates/macros/src/lib.rs, crates/macros/src/newtype_oparg.rs
Introduce rustpython-macros proc-macro crate and #[newtype_oparg] implementation that generates newtype wrappers, enum conversions (From/TryFrom), Display impls, and validation for structs/enums.
Bytecode Oparg Refactoring
crates/compiler-core/src/bytecode/oparg.rs
Replace local oparg_enum!/newtype_oparg! expansions with #[newtype_oparg] on many enums/newtypes (e.g., ResumeType, RaiseKind, BinaryOperator, SpecialMethod, CommonConstant, ConstIdx, VarNum, VarNums, LoadAttr, LoadSuperAttr, Label), removing manual From/Display impls and prior macro-generated code.
VM Frame Invert Parsing Fix
crates/vm/src/frame.rs
Fix parsing of Invert flag by converting full u32 arg (no truncation to u8) in ContainsOp* instruction handling.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Suggested reviewers

  • youknowone
  • coolreader18

Poem

🐰 I hopped into crates with a wink and a cheer,
I stitched opargs with tokens both crisp and clear,
Macros hummed softly, conversions took flight,
Bytecode now tidy, beneath moonlit byte,
A carrot of code — hop, compile, delight! 🥕

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: introducing a new #[newtype_oparg] derive macro to handle opcode-argument type definitions.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
crates/vm/src/frame.rs (1)

5467-5469: Extract invert once and reuse shared inversion logic.

These branches all re-parse invert and repeat the same boolean inversion pattern. Please compute invert once per opcode path and apply common logic once.

♻️ Proposed refactor
+                let invert_yes = matches!(
+                    bytecode::Invert::try_from(u32::from(arg)),
+                    Ok(bytecode::Invert::Yes)
+                );
                 if let Some(dict) = b.downcast_ref_if_exact::<PyDict>(vm) {
                     let a = self.nth_value(1); // needle
                     let found = dict.get_item_opt(a, vm)?.is_some();
                     self.pop_value();
                     self.pop_value();
-                    let invert =
-                        bytecode::Invert::try_from(u32::from(arg)).unwrap_or(bytecode::Invert::No);
-                    let value = match invert {
-                        bytecode::Invert::No => found,
-                        bytecode::Invert::Yes => !found,
-                    };
+                    let value = if invert_yes { !found } else { found };
                     self.push_value(vm.ctx.new_bool(value).into());
                     Ok(None)
                 } else {
                     let b = self.pop_value();
                     let a = self.pop_value();
-                    let invert =
-                        bytecode::Invert::try_from(u32::from(arg)).unwrap_or(bytecode::Invert::No);
-                    let value = match invert {
-                        bytecode::Invert::No => self._in(vm, &a, &b)?,
-                        bytecode::Invert::Yes => self._not_in(vm, &a, &b)?,
-                    };
+                    let found = self._in(vm, &a, &b)?;
+                    let value = if invert_yes { !found } else { found };
                     self.push_value(vm.ctx.new_bool(value).into());
                     Ok(None)
                 }

As per coding guidelines, “When branches differ only in a value but share common logic, extract the differing value first, then call the common logic once to avoid duplicate code.”

Also applies to: 5478-5480, 5497-5499, 5508-5510

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@crates/vm/src/frame.rs` around lines 5467 - 5469, Compute the
bytecode::Invert value once into a local named invert (e.g., let invert =
bytecode::Invert::try_from(u32::from(arg)).unwrap_or(bytecode::Invert::No));
then replace the duplicate try_from/unwrap_or calls in the branches that handle
inversion with that single variable and apply the shared boolean inversion logic
one time (use a single match or if let on invert to decide whether to flip the
computed boolean value). Update all locations that currently re-parse invert
(the occurrences around the current match/branches handling inversion) to use
this extracted invert variable and a common inversion step so the branches only
differ by the value used, not the inversion logic.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@crates/macros/Cargo.toml`:
- Around line 14-17: The syn dependency in Cargo.toml lacks the "full" feature
required by the procedural macro because the code uses syn types Item, ItemEnum,
ItemStruct and Spanned; update the syn dependency entry to enable the full
feature (i.e., change the syn = { workspace = true } line to include features =
["full"] ) so the procedural macro can compile with those types available.

In `@crates/macros/src/newtype_oparg.rs`:
- Around line 128-141: The code currently calls
attr.parse_nested_meta(...).unwrap(), which panics on bad attribute syntax;
change this to propagate a syn::Result error instead: make the closure passed to
attr.parse_nested_meta return syn::Result<()> (using ? for nested parsing so
unknown/invalid attributes produce a syn::Error), and replace the surrounding
.map/.unwrap pattern with a fallible loop or an iterator collected into Result
(e.g., iterate variants, parse attributes with ? and push VariantInfo into
variants_info), so any parse error from parse_nested_meta bubbles up as a
compile-time error rather than panicking; update handling of display and
catch_all inside the closure to use ? for meta.value().parse and return
Err(meta.error(...)) as syn::Error.

---

Nitpick comments:
In `@crates/vm/src/frame.rs`:
- Around line 5467-5469: Compute the bytecode::Invert value once into a local
named invert (e.g., let invert =
bytecode::Invert::try_from(u32::from(arg)).unwrap_or(bytecode::Invert::No));
then replace the duplicate try_from/unwrap_or calls in the branches that handle
inversion with that single variable and apply the shared boolean inversion logic
one time (use a single match or if let on invert to decide whether to flip the
computed boolean value). Update all locations that currently re-parse invert
(the occurrences around the current match/branches handling inversion) to use
this extracted invert variable and a common inversion step so the branches only
differ by the value used, not the inversion logic.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: a3f37a06-ba28-4da8-a664-be77f97960bc

📥 Commits

Reviewing files that changed from the base of the PR and between 8c01615 and f5c2f9b.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (7)
  • Cargo.toml
  • crates/compiler-core/Cargo.toml
  • crates/compiler-core/src/bytecode/oparg.rs
  • crates/macros/Cargo.toml
  • crates/macros/src/lib.rs
  • crates/macros/src/newtype_oparg.rs
  • crates/vm/src/frame.rs

Comment on lines +14 to +17
[dependencies]
proc-macro2 = { workspace = true }
quote = { workspace = true }
syn = { workspace = true }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Check how syn is configured in the workspace and what features the macro implementation requires
echo "=== Workspace syn configuration ==="
rg -n 'syn\s*=' Cargo.toml

echo -e "\n=== Macro implementation usage ==="
# Check what syn types/functions are used in the macro crate
fd -e rs . crates/macros/src --exec cat {} \; 2>/dev/null | head -100

Repository: RustPython/RustPython

Length of output: 2835


🏁 Script executed:

#!/bin/bash
# Check workspace Cargo.toml for syn definition with features
echo "=== Workspace syn definition ==="
rg -A 3 'syn = ' Cargo.toml | head -20

# Also check if there are any feature specifications
echo -e "\n=== Checking for syn features in workspace ==="
rg 'syn.*features' Cargo.toml

Repository: RustPython/RustPython

Length of output: 227


🏁 Script executed:

#!/bin/bash
# Check if workspace Cargo.toml has a [dependencies] section with syn features
echo "=== Workspace Cargo.toml content around syn ==="
rg -B 5 -A 5 'syn = "2"' Cargo.toml

# Also check the exact structure of how dependencies are defined in workspace
echo -e "\n=== Workspace Cargo.toml dependencies section ==="
sed -n '/\[workspace.dependencies\]/,/^\[/p' Cargo.toml | head -50

Repository: RustPython/RustPython

Length of output: 2976


🏁 Script executed:

#!/bin/bash
# Check if any other crates in the repo specify features for syn
echo "=== How syn is used in other Cargo.toml files ==="
fd -name Cargo.toml -type f . | xargs rg -l 'syn' | head -10

# Check specific crates to see if they specify features
echo -e "\n=== Syn usage in other crates ==="
fd -name Cargo.toml -type f . | xargs grep -A 2 'syn.*=' | grep -v "^--$" | head -30

Repository: RustPython/RustPython

Length of output: 853


🏁 Script executed:

#!/bin/bash
# Find all Cargo.toml files and check how syn is specified
echo "=== All syn dependencies in Cargo.toml files ==="
find . -name Cargo.toml -type f -exec grep -H 'syn' {} \; | grep -v '.pyc' | head -20

# Also check the macros crate Cargo.toml directly
echo -e "\n=== crates/macros/Cargo.toml full content ==="
cat crates/macros/Cargo.toml

Repository: RustPython/RustPython

Length of output: 841


Specify required syn features for the procedural macro.

The macro code uses Item, ItemEnum, ItemStruct, and Spanned from the syn crate, all of which require the "full" feature to be enabled. Update the dependency to:

syn = { workspace = true, features = ["full"] }

This aligns with how crates/derive-impl specifies syn features for its procedural macro implementation.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@crates/macros/Cargo.toml` around lines 14 - 17, The syn dependency in
Cargo.toml lacks the "full" feature required by the procedural macro because the
code uses syn types Item, ItemEnum, ItemStruct and Spanned; update the syn
dependency entry to enable the full feature (i.e., change the syn = { workspace
= true } line to include features = ["full"] ) so the procedural macro can
compile with those types available.

Comment on lines +128 to +141
attr.parse_nested_meta(|meta| {
if meta.path.is_ident("display") {
let value = meta.value()?.parse::<syn::LitStr>()?;
display = Some(value.value());
Ok(())
} else if meta.path.is_ident("catch_all") {
catch_all = true;
Ok(())
} else {
Err(meta.error("unknown oparg attribute"))
}
})
.unwrap();
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Propagate parse error instead of panicking.

Using .unwrap() on the parse_nested_meta result will cause a panic if the attribute syntax is invalid (e.g., #[oparg(display)] without = "value"). This should return a proper compile-time error.

🐛 Proposed fix
                     attr.parse_nested_meta(|meta| {
                         if meta.path.is_ident("display") {
                             let value = meta.value()?.parse::<syn::LitStr>()?;
                             display = Some(value.value());
                             Ok(())
                         } else if meta.path.is_ident("catch_all") {
                             catch_all = true;
                             Ok(())
                         } else {
                             Err(meta.error("unknown oparg attribute"))
                         }
-                    })
-                    .unwrap();
+                    })?;

Note: This requires changing the closure to return syn::Result and adjusting the outer map to collect errors properly, or restructuring the iteration to use a fallible pattern:

let mut variants_info = Vec::with_capacity(variants.len());
for variant in variants.iter() {
    // ... parsing logic with ? operator ...
    variants_info.push(VariantInfo { ... });
}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@crates/macros/src/newtype_oparg.rs` around lines 128 - 141, The code
currently calls attr.parse_nested_meta(...).unwrap(), which panics on bad
attribute syntax; change this to propagate a syn::Result error instead: make the
closure passed to attr.parse_nested_meta return syn::Result<()> (using ? for
nested parsing so unknown/invalid attributes produce a syn::Error), and replace
the surrounding .map/.unwrap pattern with a fallible loop or an iterator
collected into Result (e.g., iterate variants, parse attributes with ? and push
VariantInfo into variants_info), so any parse error from parse_nested_meta
bubbles up as a compile-time error rather than panicking; update handling of
display and catch_all inside the closure to use ? for meta.value().parse and
return Err(meta.error(...)) as syn::Error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant