Skip to content

CQ: Count each recovered shared message store reference once - #17351

Merged
lhoguin merged 1 commit into
rabbitmq:mainfrom
cloudamqp:msg-store-ref-count
Sep 8, 2026
Merged

lhoguin merged 1 commit into
rabbitmq:mainfrom
cloudamqp:msg-store-ref-count

Conversation

@gomoripeti

Copy link
Copy Markdown
Contributor

Proposed Changes

There was a bug in the CQ shared message store, the recovery process after an unclean shutdown counted every live message one too many, when it goes through every per-queue index to look for shared messages that are still referenced by any queue.

Bug introduced in commit 18acc01 included in 4.0.0

Details: rabbit_msg_store:count_msg_refs/3 registers references with ets:update_counter/4 and a default object carrying ref_count=1. The increment is applied to a freshly inserted default as well, so every message came out of dirty recovery with one reference too many. That reference was never released: after a crash, every message that was in the store at the time stayed on disk for good, with its index entry, once all queues had consumed it. Fan-out messages were affected the same way (N queues gave N+1 references).

Start the default at 0. Add a regression test, and make msg_store_v1_compat remove each message once, as its second remove of a zero-reference entry now crashes the store the way it would in production.

Types of Changes

What types of changes does your code introduce to this project?
Put an x in the boxes that apply

  • Bug fix (non-breaking change which fixes issue #NNNN)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause an observable behavior change in existing systems)
  • Documentation improvements (corrections, new content, etc)
  • Cosmetic change (whitespace, formatting, etc)
  • Build system and/or CI

Checklist

Put an x in the boxes that apply.
You can also fill these out after creating the PR.
This is simply a reminder of what we are going to look for before merging your code.

  • Mandatory: I (or my employer/client) have have signed the CA (see https://github.com/rabbitmq/cla)
  • I have read the CONTRIBUTING.md document
  • I have added tests that prove my fix is effective or that my feature works
  • All tests pass locally with my changes
  • If relevant, I have added necessary documentation to https://github.com/rabbitmq/rabbitmq-website
  • If relevant, I have added this change to the first version(s) in release-notes that I expect to introduce it

Further Comments

count_msg_refs/3 registers references with ets:update_counter/4 and a
default object carrying ref_count=1. The increment is applied to a
freshly inserted default as well, so every message came out of dirty
recovery with one reference too many. That reference was never
released: after a crash, every message that was in the store at the
time stayed on disk for good, with its index entry, once all queues
had consumed it. Fan-out messages were affected the same way (N queues
gave N+1 references).

Start the default at 0. Add a regression test, and make
msg_store_v1_compat remove each message once, as its second remove of
a zero-reference entry now crashes the store the way it would in
production.
@mergify

mergify Bot commented Sep 8, 2026

Copy link
Copy Markdown

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟒 Approval recommended

The focused fix matches ets:update_counter/4 semantics and is covered by appropriate recovery tests.

Pull request overview

Fixes dirty recovery so each shared message-store reference is counted exactly once.

Changes:

  • Initializes recovered reference counters at zero before incrementing
  • Adds regression coverage for single and fan-out references
  • Updates legacy compatibility expectations
File summaries
File Description
deps/rabbit/src/rabbit_msg_store.erl Corrects recovered reference counting
deps/rabbit/test/backing_queue_SUITE.erl Adds regression coverage and updates compatibility tests
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0
  • Review effort level: Balanced

πŸ’‘ Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@lhoguin
lhoguin merged commit dc482f2 into rabbitmq:main Sep 8, 2026
378 of 379 checks passed
@lhoguin

lhoguin commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Thank you!!

@lhoguin

lhoguin commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Did manual backport in #17354

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants