Skip to content

fix : [ClusterPipeline] ExecutorService/thread is created and destroyed too frequently in ClusterPipeline - #4479

Merged
ggivo merged 4 commits into
masterfrom
topic/ggivo/4141-cluster-pipeline-with-external-executor
Apr 2, 2026
Merged

ggivo merged 4 commits into
masterfrom
topic/ggivo/4141-cluster-pipeline-with-external-executor

Conversation

@ggivo

@ggivo ggivo commented Apr 1, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Multi-node cluster pipelines create a new ExecutorService on every sync() call, causing significant overhead in high-throughput scenarios.

Solution

Added support for externally managed ExecutorService in cluster pipelines:

  • New API: cluster.pipelined(ExecutorService) - allows users to provide a shared executor
  • Backward compatible: cluster.pipelined() retains existing auto-managed behavior

Implementation

  • Modified MultiNodePipelineBase to accept optional ExecutorService
  • Added pipelined(ExecutorService) methods to RedisClusterClient and JedisCluster

Usage Example:

ExecutorService executor = Executors.newFixedThreadPool(3);
try (RedisClusterClient cluster = ...) {
    // Reuse executor across multiple pipelines
    try (ClusterPipeline pipeline = cluster.pipelined(executor)) {
        pipeline.set("key", "value");
        pipeline.sync();
    }
} finally {
    executor.shutdown(); // User controls lifecycle
}

Closes #4141


Note

Medium Risk
Changes the concurrency/executor lifecycle for multi-node cluster pipelining; incorrect executor management or shared-executor contention could impact performance or cause hangs, though behavior is covered by new integration tests.

Overview
Cluster pipelining can now reuse a caller-provided ExecutorService instead of creating/shutting down a thread pool on every multi-node sync().

MultiNodePipelineBase accepts an optional shared executor and only shuts down internally-created executors, while JedisCluster and RedisClusterClient add pipelined(ExecutorService) (with pipelined() delegating to it) to pass the executor through to ClusterPipeline. New integration tests verify the shared executor is not shut down and that many concurrent pipelines sharing a small executor complete without deadlock.

Written by Cursor Bugbot for commit 38fa2a5. This will update automatically on new commits. Configure here.

@github-actions

github-actions Bot commented Apr 1, 2026 •

Copy link
Copy Markdown

Test Results

  176 files  ± 0    176 suites  ±0   11m 1s ⏱️ +45s
8 527 tests + 9  8 316 ✅ +1 272  211 💤  - 1 263  0 ❌ ±0 
3 233 runs  +15  3 170 ✅ +  445   63 💤  -   430  0 ❌ ±0 

Results for commit 38fa2a5. ± Comparison against base commit a3cfede.

This pull request removes 1 and adds 10 tests. Note that renamed tests count towards both.
redis.clients.jedis.commands.unified.search.FTHybridCommandsTestBase$SupportedScorersTest ‑ testScorer(Scorer, double, double)
redis.clients.jedis.ClusterPipeliningTest ‑ sharedExecutorConcurrentPipelinesNoDeadlock
redis.clients.jedis.ClusterPipeliningTest ‑ sharedExecutorPipelineDoesNotShutdownSharedExecutor
redis.clients.jedis.ClusterPipeliningTest ‑ sharedExecutorPipelineKeysAtSameNode
redis.clients.jedis.commands.unified.search.FTHybridCommandsTestBase$SupportedScorersTest ‑ testScorer(Scorer, double, double)[1]
redis.clients.jedis.commands.unified.search.FTHybridCommandsTestBase$SupportedScorersTest ‑ testScorer(Scorer, double, double)[2]
redis.clients.jedis.commands.unified.search.FTHybridCommandsTestBase$SupportedScorersTest ‑ testScorer(Scorer, double, double)[3]
redis.clients.jedis.commands.unified.search.FTHybridCommandsTestBase$SupportedScorersTest ‑ testScorer(Scorer, double, double)[4]
redis.clients.jedis.commands.unified.search.FTHybridCommandsTestBase$SupportedScorersTest ‑ testScorer(Scorer, double, double)[5]
redis.clients.jedis.commands.unified.search.FTHybridCommandsTestBase$SupportedScorersTest ‑ testScorer(Scorer, double, double)[6]
redis.clients.jedis.commands.unified.search.FTHybridCommandsTestBase$SupportedScorersTest ‑ testScorer(Scorer, double, double)[7]

♻️ This comment has been updated with latest results.

@jit-ci

jit-ci Bot commented Apr 1, 2026 •

Copy link
Copy Markdown

🛡️ Jit Security Scan Results

CRITICAL HIGH MEDIUM

✅ No security findings were detected in this PR


Security scan by Jit

@ggivo
ggivo requested a review from Copilot April 1, 2026 12:11
@ggivo
ggivo marked this pull request as ready for review April 1, 2026 12:12
@ggivo
ggivo requested a review from atakavci April 1, 2026 12:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds an option for Redis Cluster pipelining to reuse a caller-provided ExecutorService, avoiding the overhead of creating/shutting down a thread pool on every multi-node sync().

Changes:

  • Added pipelined(ExecutorService) overloads to RedisClusterClient and JedisCluster.
  • Updated MultiNodePipelineBase/ClusterPipeline to accept and use an optional shared executor for multi-node sync().
  • Added tests validating shared-executor lifecycle behavior and concurrent pipeline usage.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
src/main/java/redis/clients/jedis/RedisClusterClient.java Adds pipelined(ExecutorService) API and delegates existing pipelined() to it.
src/main/java/redis/clients/jedis/JedisCluster.java Adds pipelined(ExecutorService) API (deprecated client) and delegates existing pipelined() to it.
src/main/java/redis/clients/jedis/MultiNodePipelineBase.java Implements shared-vs-dedicated executor selection and avoids shutting down shared executors.
src/main/java/redis/clients/jedis/ClusterPipeline.java Wires the optional executor through to the base pipeline implementation.
src/test/java/redis/clients/jedis/ClusterPipeliningTest.java Adds coverage for shared-executor behavior and concurrency/deadlock regression.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

@atakavci atakavci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MultiNodePipelineBase is somewhat in need of care in general.
but approving if you like to proceed with this PR as is.

@ggivo
ggivo merged commit 7744cd4 into master Apr 2, 2026
31 of 32 checks passed
@ggivo ggivo added this to the 8.0.0 milestone Apr 2, 2026
ggivo added a commit that referenced this pull request Apr 2, 2026
…troyed too frequently in ClusterPipeline #4479

(cherry picked from commit 7744cd4)
ggivo added a commit that referenced this pull request Apr 2, 2026
#4480)

backport: [ClusterPipeline] ExecutorService/thread is created and destroyed too frequently in ClusterPipeline  #4479

(cherry picked from commit 7744cd4)
@ggivo
ggivo deleted the topic/ggivo/4141-cluster-pipeline-with-external-executor branch April 3, 2026 12:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[ClusterPipeline] ExecutorService/thread is created and destroyed too frequently in ClusterPipeline

3 participants