Skip to content

fix ConcurrentModificationException in AuthXManager token renewal - #4601

Merged
ggivo merged 3 commits into
redis:masterfrom
insaf021:authx-connection-iteration
Jul 10, 2026
Merged

ggivo merged 3 commits into
redis:masterfrom
insaf021:authx-connection-iteration

Conversation

@insaf021

@insaf021 insaf021 commented Jul 8, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #4606

Problem

AuthXManager.authenticateConnections runs on the token-renewal thread and walks a Collections.synchronizedList of WeakReference<Connection> to push the fresh credentials onto each pooled connection. Entries whose Connection has since been garbage collected turn up as cleared references during ordinary pool churn.

I ran into it exercising renewal against a pool that had recycled a connection: the renewal walk aborts with ConcurrentModificationException, so every connection after the cleared entry never receives the renewed credentials and starts failing auth once the old token expires.

Root cause

The walk uses an enhanced-for loop and calls connections.remove(connectionRef) in place whenever it meets a cleared reference:

for (WeakReference<Connection> connectionRef : connections) {
    Connection connection = connectionRef.get();
    if (connection != null) {
        connection.setCredentials(credentialsFromToken);
    } else {
        connections.remove(connectionRef);  // bumps modCount mid-iteration
    }
}

The structural modification bumps the backing ArrayList's modCount, and the next iterator.next() throws. Collections.synchronizedList only guarantees safe iteration when you synchronise on the list yourself and mutate through the iterator.

Fix

Move the loop under synchronized (connections) and prune cleared references via Iterator.remove(), per the synchronizedList javadoc. This also closes the narrower race where addConnection structurally modifies the list from another thread mid-walk. setCredentials just swaps an AtomicReference, so holding the monitor across the loop stays cheap.

Testing

Added authenticateConnectionsPrunesClearedReferences: it seeds the connection list with a cleared WeakReference ahead of two live connections and drives authenticateConnections directly. On master it errors with ConcurrentModificationException; with the fix it passes, the cleared entry is pruned, and both live connections carry the renewed credentials. A standalone snippet showing the same failure shape is in #4606.


Note

Medium Risk
Changes authentication credential propagation on the token-renewal path; incorrect locking could affect renewal under pool churn, but the change follows synchronizedList usage and is narrowly scoped.

Overview
Fixes token renewal aborting with ConcurrentModificationException when AuthXManager.authenticateConnections walks pooled connections and prunes garbage-collected WeakReference entries.

The renewal loop now runs under synchronized (connections) and drops cleared references with Iterator.remove() instead of mutating the list during an enhanced-for iteration. Several AuthXManager fields are marked final.

Adds authenticateConnectionsPrunesClearedReferences to assert cleared weak refs are pruned and live connections still receive renewed credentials.

Reviewed by Cursor Bugbot for commit 2533458. Bugbot is set up for automated code reviews on this repo. Configure here.

@ggivo

ggivo commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

HI @insaf021
Will take a look as soon as we find some free cycles.

Meaning will appreciate opening an issue and providing a reproducer/code example to make sure we are dicussing same issue. It will help triage it faster.

PR description is hard to read; some formatting will help reviewers navigate faster

@insaf021

insaf021 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Opened #4606 with a standalone reproducer: the same enhanced-for plus in-place remove pattern throws ConcurrentModificationException on any synchronizedList, no server needed, and the unit test in this PR errors the same way on current master. Also restructured the description into problem / root cause / fix / testing sections, should be easier to navigate now.

@ggivo ggivo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ggivo
ggivo merged commit 2a70b9c into redis:master Jul 10, 2026
17 checks passed
@ggivo

ggivo commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

@insaf021
Thanks for spotting & fixing it. Now merged.

ggivo added a commit that referenced this pull request Jul 10, 2026
…Manager token renewal (#4614)

fix ConcurrentModificationException in AuthXManager token renewal (#4601)

* fix ConcurrentModificationException in AuthXManager token renewal

* clean up

---------


(cherry picked from commit 2a70b9c)

Co-authored-by: insaf021 <Insaf@digiscrypt.com>
@ggivo ggivo added the bug label Aug 7, 2026
@ggivo ggivo modified the milestones: 7.5.4, 8.0.0 Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ConcurrentModificationException in AuthXManager.authenticateConnections when pruning GC'd connection references

2 participants