Hermes Agent Community Reports Webhook Scheduling Deadlock That Can Stall Gateway

On October 27, the Hermes Agent community reported a gateway deadlock: when multiple Webhooks trigger cron_job at the same time, jobs may remain in running indefinitely, and later triggers may be skipped because the previous run has not finished. The reporter provided thread stacks and a minimal program that does not depend on a model service. Another project participant added the issue to the gateway fix tracker. As of review, the issue remained open, with no linked patch branch or PR. Issue report

According to the report’s analysis, the Webhook uses asyncio.to_thread to hand scheduling execution to the default thread pool. That thread then calls run_coroutine_threadsafe(...).result(), synchronously waiting for the main event loop to complete the job. But while handling state writes and job execution, the main loop also needs an available worker from the same pool. Once all threads are waiting, the required work cannot start, creating a circular wait. Call path and reproduction

The official Python documentation explicitly warns that a task in a thread pool waiting on another Future can cause a deadlock. Starting with Python 3.13, the default thread limit is min(32, (os.process_cpu_count() or 1) + 4). So the six-core environment in the report has only ten default worker threads; “ten concurrent jobs” should not be treated as a fixed threshold across all deployments. Thread pool documentation

This also shows why asynchronous interfaces still require scrutiny of wait relationships: the Future returned by run_coroutine_threadsafe can be awaited by another operating system thread, while the official example sets a timeout on .result(). Based on these mechanisms, deployment monitoring should track not only whether the event loop responds, but also whether job states continue to progress, so it can detect when execution resources have stalled. Cross-thread scheduling documentation

The reporter said they did not encounter further stalls after staggering Webhooks and reducing overlap; this remains an observation from a single environment. The engineering team should next consider whether to use a dedicated executor, await main-loop operations directly, and add timeouts and resource liveness checks. A formal fix still needs verification; tracking the issue does not mean a fix has been released. Mitigation recommendations