fix(anthropic): SDK auto-retry + Retry-After 120s cap deadlock rate-limited Anthropic accounts

Bug Description

Two interacting defaults turn an Anthropic 429 (soft rate limit) into a hard outage:

agent/anthropic_adapter.py build_anthropic_client (and build_anthropic_bedrock_client) construct Anthropic(...) / AnthropicBedrock(...) without setting max_retries, so the SDK's default max_retries=2 kicks in. The SDK retries 429/5xx with a 1-2s exponential backoff and ignores Retry-After.
run_agent.py:14462 caps Retry-After at 120s: _retry_after = min(float(_ra_raw), 120) # Cap at 2 minutes. On Anthropic Tier 1 accounts the input-token bucket resets in ~171s.

Net effect on a rate-limited account:

SDK fires 2 silent retries inside hermes's outer retry loop → burns through 2-3 request slots against a bucket that won't refill for minutes
Outer hermes retry sleeps 120s (capped) and fires again before the actual reset window → another nested SDK retry burst
Soft per-minute throttle escalates into hard request-count exhaustion; the agent appears hung

Steps to Reproduce

Anthropic API key on Tier 1 (or any account where reset window > 120s)
Saturate input-token budget (long conversation / many parallel turns)
Receive 429 with Retry-After: 171 (or any value > 120)
Observe: SDK retries fire silently inside hermes's retry, hermes sleeps 120s instead of 171s, request slots exhaust before the actual reset window

Expected Behavior

Single retry strategy: hermes's outer loop owns rate-limit backoff (it already reads Retry-After)
Retry-After honored up to a sane ceiling (e.g. 600s — covers all realistic provider windows while still rejecting absurd values)

Actual Behavior

Two retry layers fighting each other
120s cap turns a 171s reset window into a death loop

Affected Component

Agent Core (conversation loop)

Root Cause Analysis

Verified on origin/main HEAD 77276070f5 (2026-05-15):

agent/anthropic_adapter.py build_anthropic_client — no max_retries kwarg passed to Anthropic(**kwargs) → SDK default 2
agent/anthropic_adapter.py build_anthropic_bedrock_client — no max_retries kwarg passed to AnthropicBedrock(...) → SDK default 2
run_agent.py:14462 — _retry_after = min(float(_ra_raw), 120)

Proposed Fix

Set max_retries=0 on both SDK constructors (Anthropic + AnthropicBedrock) — delegate retries to hermes's outer loop entirely. The outer loop already honors Retry-After; the SDK layer just doubles up.
Raise the Retry-After cap from 120s to 600s (covers Anthropic Tier 1 and similar providers; still rejects pathological hours/days values).

I have working commits in my fork that survived rebase onto v2026.5.7 (tests included). Happy to open a PR — wanted to surface this first since max_retries=0 is a behavior change for any user relying on SDK-level retry semantics.

Environment

Hermes: v2026.5.7 base + verified against origin/main 2026-05-15
Python: 3.11+
OS: macOS 15.x (bug is platform-independent — pure provider retry logic)

We have observed a related issue with custom providers using Anthropic transport. Beyond the Anthropic-specific SDK retry described here, run_agent.py has a broader 429 handling problem:

On custom Anthropic-compatible providers, the 429 retry logic eagerly falls back to the next provider instead of retrying with jittered backoff on the original. With a single primary provider, this creates a loop: 429 -> try fallback (may not exist) -> fail -> retry primary -> 429 again.

We patched run_agent.py to use jittered exponential backoff (base 5s, max 60s) on the original provider before falling back. Combined with the SDK fix (max_retries=0 to prevent double-retry), this resolves the deadlock for all Anthropic-compatible providers.

For Retry-After cap: suggest 600s instead of 120s. Tier 1 reset windows are ~171s, so 120s cap causes death loops.

fix(anthropic): stop SDK auto-retry double-firing and raise Retry-Aft…

fix(agent): disable OpenAI SDK auto-retry that double-fires inside th…

Fixed on main via PR #53918 (rebase-merge, your commit 1ab35ba25 preserved with your authorship).

Both halves of your diagnosis landed:

max_retries=0 on all three Anthropic SDK constructors (Anthropic, AnthropicBedrock, bearer-hook client) — the outer loop owns retry and honors Retry-After.
Retry-After cap raised 120s → 600s so Tier-1 reset windows (~171s) are honored instead of re-tripping the limit.

During the salvage we audited the same bug class on the OpenAI/aggregator path and found it there too (SDK default max_retries=2, never overridden), so we also set max_retries=0 at the single create_openai_client chokepoint. auxiliary_client is intentionally left alone — it builds its own clients and isn't wrapped by the conversation loop.

Thanks for the precise root-cause writeup — line-level references on main made this an easy verify-and-ship.

#53918