[Bug]: a memory provider's initialize() on the turn critical path can wedge the gateway forever — drain cannot interrupt it, guard won't restart it, and the process becomes unkillable while holding gateway.lock #145889

Bug Description

Hermes 0.22.5 (Windows, native, Python 3.14) can get into a state where a single conversation turn never reaches the model, the gateway then never drains, the desktop app can no longer start any gateway, and the wedged process cannot be killed — only a reboot recovers.

The immediate trigger is a third-party memory provider (memtensor / MemOS 2.0.12) whose initialize() blocked on a non-timeout lock. But the reason one provider's bug takes down the whole gateway is host-side, and that's what this report is about:

1. MemoryManager.initialize_all() runs synchronously on the turn's init path, with no deadline, on a non-daemon thread.
2. _each_provider() catches exceptions, but nothing catches "never returns" — there is no watchdog around plugin lifecycle hooks.
3. gateway.drain "interrupting remaining work" only cancels async tasks; a thread blocked in a syscall is not interruptible, so drain never completes.
4. gateway-guard decides liveness from PID existence, so it sees the wedged process and refuses to restart it.
5. While wedged, the process holds gateway.lock, so every later serve --port 0 exits before the port announcement; the desktop retries forever with only Gateway starting (deadline) and no reason.

Steps to Reproduce

A deterministic repro should be possible with a stub provider — I could not run the CLI in this environment, so I'm describing the sequence I observed plus the minimal stub:

1. Install any memory provider plugin whose initialize() blocks (e.g. time.sleep(10**9), or a lock that is never released) and select it as memory.provider.
2. Start the gateway / desktop app and send one message in a chat.
3. The turn never advances: active_work … api_calls: 0, elapsed_s grows without bound (observed 1812 s, then 106 min across two days).
4. After the turn wedges: gateway_state.json goes running → draining and stays there (active_agents ≥ 1). gateway.log receives nothing new.
5. Quit and reopen the desktop app: hermes --profile <p> serve --host 127.0.0.1 --port 0 exits with status 1 before announcing a port (exited before port announcement (1)).
6. Try to recover: taskkill /F /T /PID <pid> → "No running instance of the task" for every PID in the tree, while Get-Process -Id <pid> still lists it (10 MB working set), the port is still LISTENING, and gateway.lock cannot be renamed ("The process cannot access the file because it is being used by another process").

Expected Behavior

• A plugin that never returns from initialize() must not stop the agent from answering. The provider should be marked degraded after a bounded wait and the turn should proceed without memory.
• gateway stop/restart must be able to end a gateway whose drain is stuck (today it waits forever).
• The guard must restart a gateway whose loop heartbeat has stopped advancing, regardless of whether the PID still exists.
• Startup failures should name their cause (gateway.lock held by pid N, heartbeat age 44 h) instead of only Gateway starting (deadline).
• A gateway in this state should not be unkillable. If a thread is stuck in a kernel wait, at minimum drain should say which thread it is (faulthandler.dump_traceback(all_threads=True)) before giving up.

Actual Behavior

Turn wedged forever. Traceback captured in agent.log (trimmed):

File "...\agent\agent_init.py", line 1372, in _init_memory
File "...\agent\memory_manager.py", line 929, in initialize_all
File "...\agent\memory_manager.py", line 244, in _each_provider
File "...\plugins\memtensor\__init__.py", line 491, in initialize
File "...\plugins\memtensor\__init__.py", line 1959, in _open_session
File "...\memos-plugin\adapters\hermes\memos_provider\shared_bridge_runtime.py", line 438, in request
File "...\memos-plugin\adapters\hermes\memos_provider\shared_bridge_runtime.py", line 183, in request
File "...\memos-plugin\adapters\hermes\memos_provider\bridge_client.py", line 313, in request
        with self._lock:

The plugin's own 60 s timeout only covers waiter.wait(timeout) after the lock is released, so lock acquisition is unbounded — that part is the plugin's bug (MemOS 2.0.12, bridge_client.request). But it should be containable.

State left behind:

{"gateway_state":"draining","active_agents":1,
 "active_work":[{"elapsed_s":628.3,"api_calls":0}],
 "drain_since":"2026-10-28T03:38:52Z"}

Earlier the same class of wedge exited the gateway with "exit_reason":"loop_liveness_watchdog", after which gateway-guard.log logged every 5 minutes:

heartbeat expired / no such process (pid=6672, age=...), but the process is running pid=6672 — not restarting

i.e. heartbeat expired, but the process is running — not restarting. That is the false negative in step 4.

Environment: Windows 11 ARM64, native (no WSL), Hermes 0.22.5, Python 3.14.9, desktop app + serve gateway, profiles default and one named profile.

Debug Report

hermes debug share / hermes debug could not be run here (the CLI launcher is not executable from the agent's sandboxed shell), so the log excerpts above are pasted verbatim instead. Happy to attach the full desktop.log / agent.log / gateway-guard.log if you tell me where to send them privately.

Suggested fixes (host side)

1. Bound plugin lifecycle hooks. In MemoryManager.initialize_all() (or _each_provider), run each provider's initialize() on a daemon thread and join(timeout) (30–45 s). On timeout: log provider_init_timeout provider=<name>, mark the provider degraded for the session, continue with memory disabled. Never join without a timeout, never run it on a non-daemon thread.
2. Prefer off-path init. Initialize memory providers after the turn starts (lazily, on first memory use) or in a background warm-up, so a slow provider costs nothing to the response.
3. Make drain able to fail. After gateway.drain exceeds its budget: faulthandler.dump_traceback(all_threads=True) into gateway.log naming the stuck thread, then hard-exit with a distinct code so guard restarts. (fix(gateway): add hard-exit watchdog to prevent drain hang on stop/restart #35676 added a hard-exit watchdog for drain; this state shows a path that still bypasses it, because the block is in a thread the drain doesn't own.)
4. Guard liveness = heartbeat advancing, not PID existence. A heartbeat that stops advancing for N minutes should mean "dead", even if the PID is still in the process table (adjacent to fix(gateway): re-stamp gateway_state.json from the loop heartbeat #103215).
5. Diagnosable startup refusal. When a new gateway can't start because gateway.lock is held by an owner whose heartbeat is stale, write the owner pid + heartbeat age into the desktop log; today it's only Gateway starting (deadline).
6. Make GatewayClientError("Gateway connection failed; no local fallback") (hermes_cli/gateway_client.py:313) carry the discovery state (draining, foreign_stale_lock, pid) — right now it sends users toward "another instance is running" instead of "the gateway is wedged".

Related issues

• Graceful gateway restart blocked indefinitely by a stuck cron agent turn (drain never completes) #93711 — graceful gateway restart blocked indefinitely by a stuck cron agent turn (drain never completes)
• fix(gateway): add hard-exit watchdog to prevent drain hang on stop/restart #35676 — hard-exit watchdog to prevent drain hang on stop/restart (closed; this state still slips through)
• bug(gateway): Background Honcho timeouts synchronously deadlock Discord platform adapter and freeze chat threads #36858 — background Honcho timeouts synchronously deadlock the Discord adapter (same shape: a memory provider's blocking call stalls the host)
• fix(gateway): re-stamp gateway_state.json from the loop heartbeat #103215 — re-stamp gateway_state.json from the loop heartbeat

Workaround applied locally

Two local patches, which I'm happy to turn into a PR if the direction is acceptable:

• Plugin bridge (memos_provider/bridge_client.py): blocking pipe writes go through a bounded queue (maxsize=64, put(timeout=5.0)) drained by daemon writer threads, so a stalled pipe can no longer hold self._lock or a non-daemon thread.
• Host (agent/memory_manager.py): initialize_all() gained a bounded path (deadline + circuit breaker, skip-and-continue on timeout) so a stalled provider can't wedge the turn.

I have also kept the failing logs and a rewritten patch that applies to stock 0.22.5 files, so a maintainer can verify against it.

Reporter note: this report was assembled by an assistant agent (DSH) from the user's own logs on their Windows machine; the local workaround patches were applied with the user's approval.