Dify Community Reports Scheduler Connections Not Being Released, Potentially Accumulating Idle Redis Connections in Long-Running Workers

A resource lifecycle issue has been reported in Dify’s scheduled AI workflows. An issue filed on October 22 says long-running Celery workers using a gevent pool and consuming the schedule_poller queue may retain Redis result-subscription connections after dispatching work completes. The reporter said their production environment accumulated more than 10,000 connections; this is an observation from an individual deployment and does not claim that a service outage or data loss occurred. Original issue

The report initially came from a deployment based on Dify 1.16.1, with Celery 5.6.3, gevent 26.4.0, and redis-py 7.4.0. The author traced the issue to the result callback path of group(...).apply_async(): after unsubscribe, the drainer responsible for receiving results may remain alive, continuing to hold the backend and connection. According to the author’s analysis, setting task_ignore_result=True did not prevent this path from creating a subscription. Technical analysis

This distinction matters for operations. Celery’s official documentation defines task_ignore_result as whether to store task return values, so this setting alone does not establish that all result consumers have completed resource cleanup. When checking scheduler health, engineering teams should monitor whether connections continue to increase after work finishes, alongside success rates and execution times. Celery configuration documentation

The report recommends tracking Redis client IDs, idle times, and last commands across multiple real dispatch batches with a continuously running test worker. A short-lived test worker may close its sockets on exit and mask the issue. Connection counts should also not be interpreted as increasing by a fixed one connection per task. Reproduction steps

The author also proposed isolating the result backend and explicitly cleaning up resources as a direction for a fix, but the verifiable evidence currently remains primarily the original author’s report. The next things to watch are maintainer reproduction, patch review, and long-duration load validation; at this stage, the impact should not be generalized to all Dify deployments or cloud services.