Skip to content

Stabilize search-on-timeout FAIL tests against worker-thread race - #4721

Merged
atakavci merged 6 commits into
redis:masterfrom
atakavci:ali/queryOnTimeout
Sep 30, 2026
Merged

atakavci merged 6 commits into
redis:masterfrom
atakavci:ali/queryOnTimeout

Conversation

@atakavci

@atakavci atakavci commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The searchOnTimeoutFailReturnsError / aggregateOnTimeoutFailReturnsError tests fail nightly against Redis unstable since 2026-08-27. As of RediSearch/RediSearch#10961, the FAIL on-timeout policy is enforced by a blocked-client timeout callback racing the worker thread running the query β€” confirmed as intended behavior in RediSearch#11274: when the worker wins, an over-budget query returns full results instead of an error. With 10k documents the query overran the 1ms timeout by only a few milliseconds, leaving the outcome to scheduling jitter. Raising ON_TIMEOUT_DOC_COUNT to 100k gives a ~200x runtime-to-timeout margin so the callback deterministically wins.

Key Decisions & Assumptions

  • Keep testing the realistic workers-enabled path with TIMEOUT 1, rather than forcing search-workers 0 (inline execution sidesteps the race).
  • Flakiness was measured below ~10x margin; the constant's javadoc records this threshold.

Testing

Test-only change, verified via redis-cli against the failing image (unstable-33278213642-debian): at 100k docs the FAIL error returned 50/50 (search) and 20/20 (aggregate), versus 1/10 at 10k. The shared fixture also serves the RETURN-policy warning tests, which remain valid.


Note

Low Risk
Test-only changes to search timeout fixtures; no production client or runtime behavior is modified.

Overview
Stabilizes RediSearch search-on-timeout FAIL integration tests by indexing 100k documents for searchOnTimeoutFailReturnsError and aggregateOnTimeoutFailReturnsError, while RETURN warning tests still use 10k.

The shared helpers populateOnTimeoutIndex and timingOutSearchParams now take a document count so FAIL and RETURN scenarios can use different index sizes. New javadoc documents that with search-workers > 0, the FAIL policy depends on a race between the blocked-client timeout callback and the query workerβ€”queries that only slightly exceed the 1ms limit may still succeed, which made the old 10k setup flaky on Redis unstable.

Reviewed by Cursor Bugbot for commit 7982373. Bugbot is set up for automated code reviews on this repo. Configure here.

@atakavci
atakavci requested a review from ggivo September 1, 2026 12:21
@atakavci atakavci self-assigned this Sep 1, 2026
@atakavci atakavci changed the title Stabilize search on-timeout FAIL tests against worker-thread race Stabilize search-on-timeout FAIL tests against worker-thread race Sep 1, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit 28211db. Configure here.

@ggivo ggivo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM,
Will suggest leaving test running for all Redis versions, if not, causing a delay in CI. Using @SinceRedisVersion implies it is availble/compatible with the given version

Otherwise, it's probably better to introduce a dedicated "slow" test job

@ggivo ggivo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@atakavci
atakavci merged commit 84b8828 into redis:master Sep 30, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants