Skip to content

GROOVY-12332: release the parse graph before class generation - #3230

Merged
paulk-asert merged 3 commits into
apache:masterfrom
daniellansun:GROOVY-12332
Nov 6, 2026
Merged

paulk-asert merged 3 commits into
apache:masterfrom
daniellansun:GROOVY-12332

Conversation

@daniellansun

Copy link
Copy Markdown
Contributor

https://issues.apache.org/jira/browse/GROOVY-12332

Since 6.0.0-beta-3, compiling one very large class needed more heap than Gradle's default forked compiler (512m). The same sources spread across ordinary-sized classes still compiled. Bisect lands on GROOVY-12318 (1b0ebf6102), which moved the parser prediction cache onto a private ATN and stopped clearing it in place.

Two strong references then survived parsing:

  • Each parameter kept its parse context so a record header could be lowered. That context's parent chain is the whole concrete syntax tree, so one parameter pinned every token of the file.
  • The recognizer kept its interpreter. After the cache was replaced, a finished parser still pointed at the retired ATN, so the prediction graph stayed live through class generation.

Both are dropped once the AST is built. The shared cache remains softly reachable for later files, which is what keeps an idle compiler copy collectable.

Checked against the groovy-verify sources from the report (VerifyChecker at 15,580 lines, plus the rest of src/main) on JDK 21:

compiler 512m
6.0.0-beta-3 compiles
master before this change OutOfMemoryError: Java heap space during class generation
this change compiles (peak during parse, then the graph is released; about 135m retained afterwards)

./gradlew :lintGate and :test --tests bugs.Groovy12332 --tests bugs.Groovy12142 --tests org.codehaus.groovy.classgen.RecordTest passed. bugs.Groovy12332 fails if the release is removed: parameters still hold parser contexts, and the prediction cache stays reachable from the finished class.

Testing

  • bugs.Groovy12332

A single large class kept its concrete syntax tree and the parser
prediction cache reachable through class generation, so a forked
compiler at the default 512m heap ran out of memory. Drop the
parameter parse-context pins and detach the recognizers once the
AST is built. The shared cache stays softly reachable for later files.
@codecov-commenter

codecov-commenter commented Nov 5, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.47619% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.3387%. Comparing base (22bb46c) to head (82d5e5c).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
...va/org/apache/groovy/parser/antlr4/AstBuilder.java 83.3333% 0 Missing and 2 partials ⚠️
Additional details and impacted files

Impacted file tree graph

@@                Coverage Diff                 @@
##               master      #3230        +/-   ##
==================================================
+ Coverage     74.3343%   74.3387%   +0.0044%     
- Complexity      44299      44312        +13     
==================================================
  Files            1699       1699                
  Lines          150855     150877        +22     
  Branches        28754      28756         +2     
==================================================
+ Hits           112137     112160        +23     
+ Misses          28441      28440         -1     
  Partials        10277      10277                
Files with missing lines Coverage Δ
...g/apache/groovy/parser/antlr4/GroovyLangLexer.java 100.0000% <100.0000%> (ø)
.../apache/groovy/parser/antlr4/GroovyLangParser.java 60.0000% <100.0000%> (+20.0000%) ⬆️
...va/org/apache/groovy/parser/antlr4/AstBuilder.java 87.3049% <83.3333%> (-0.0470%) ⬇️

... and 12 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@paulk-asert

Copy link
Copy Markdown
Contributor

Thanks, this makes a real difference. I compiled a generated 17,504-line @CompileStatic class in a fresh JVM (FileSystemCompiler, -XX:+UseSerialGC), comparing this PR with its parent (ee802a1):

-Xmx before this PR
256m OOM OK
320m OOM OK
384m OOM OK
448m OK (15.0s) OK (6.3s)
512m OK (9.1s) OK (5.5s)

One concern: removing _PARAMETER_CONTEXT by walking the finished AST can miss parameters, and one missed parameter keeps the whole parse tree reachable. I probed 17 shapes; two still keep a context after a full compile:

  • closure parameters in an instance initializer block: class C { def r; { r = { int x -> x }(1) } }
  • closure parameters in a default parameter value: def m(f = { int x -> x }) { f(1) }

Adding one such initializer, { q = { int x -> x }(1) }, to the generated class brings it back to failing at 256m and 320m. It passes at 384m, after 34s, mostly in GC.

Since the context is only read during the visit (transformRecordHeaderToProperties), could it be kept by the builder rather than on the Parameter? That removes the walk entirely:

private final Map<Parameter, GroovyParserRuleContext> parameterContexts = new IdentityHashMap<>();
private final Map<Parameter, ModifierManager> parameterModifierManagers = new IdentityHashMap<>();

// processFormalParameter
parameterModifierManagers.put(parameter, modifierManager);
parameterContexts.put(parameter, ctx);

// transformRecordHeaderToProperties
FormalParameterContext parameterCtx = (FormalParameterContext) parameterContexts.get(parameter);
ModifierManager parameterModifierManager = parameterModifierManagers.get(parameter);

// buildAST, finally
releaseRecognizers();
parameterContexts.clear();
parameterModifierManagers.clear();

releaseParameterContexts, releaseClassParameterContexts, clearParameterContexts, PARAMETER_CONTEXT_RELEASER and the two metadata keys then go away: +11/−60 in AstBuilder. Moving the modifier manager too drops the remaining link from every parameter to the AstBuilder.

With that change:

  • all 17 shapes keep no context;
  • the class with the initializer closure compiles at 256m;
  • Groovy12332, the parser tests, gls.*, groovy.transform.stc.*, RecordTest and RecordSpecificationTest pass (114 classes, 4,487 tests).

The parameter assertion in Groovy12332 would then pass trivially. It might be worth adding the initializer-closure case, or asserting that no _PARAMETER_CONTEXT metadata remains anywhere.

Releasing the parser and lexer looks safe:

  • Antlr4ParserPlugin doesn't use them after buildAST();
  • getOriginalText only runs during the visit;
  • tokens keep their own character stream, so token text still works.

I'd also suggest this for 6.0.1, since GROOVY-12332 is a 6.0.0 regression that breaks builds at Gradle's default heap.

@paulk-asert

Copy link
Copy Markdown
Contributor

Hold FormalParameterContext and ModifierManager in IdentityHashMaps on
AstBuilder, keyed by Parameter, and clear them after the visit. Leaving
either on the parameter keeps the CST alive, and the post-parse walk
missed closure parameters in instance initializers and default values.

Co-authored-by: Daniel Sun <sunlan@apache.org>
@daniellansun

Copy link
Copy Markdown
Contributor Author

Thanks Paul - applied your IdentityHashMap patch as tip 25cbbc0 (Author Paul King, Committer me, Co-authored-by me). Holding FormalParameterContext / ModifierManager on AstBuilder and clearing after the visit drops the CST for the initializer-block and default-value shapes you called out; Groovy12332 covers those cases. CI is green on this tip, including Windows LTS. Happy to take this for 6.0.1 as well.

@testlens-app

testlens-app Bot commented Nov 6, 2026

Copy link
Copy Markdown

✅ All tests passed ✅

🏷️ Commit: 82d5e5c
▶️ Tests: 78845 executed
⚪️ Checks: 26/26 completed


Learn more about TestLens at testlens.app/docs.

@paulk-asert
paulk-asert merged commit 04ced7c into apache:master Nov 6, 2026
27 checks passed
paulk-asert added a commit that referenced this pull request Nov 6, 2026
* GROOVY-12332: release the parse graph before class generation

A single large class kept its concrete syntax tree and the parser
prediction cache reachable through class generation, so a forked
compiler at the default 512m heap ran out of memory. Drop the
parameter parse-context pins and detach the recognizers once the
AST is built. The shared cache stays softly reachable for later files.

* GROOVY-12332: keep parameter parse state off the AST

Hold FormalParameterContext and ModifierManager in IdentityHashMaps on
AstBuilder, keyed by Parameter, and clear them after the visit. Leaving
either on the parameter keeps the CST alive, and the post-parse walk
missed closure parameters in instance initializers and default values.

Co-authored-by: Daniel Sun <sunlan@apache.org>

---------

Co-authored-by: Paul King <paulk@asert.com.au>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants