GitHub Repo
TensorFlow 2.22 Release Candidate Adds Tiered Batch Waiting and a Separate Warmup Thread Pool
BatchFunction adds configurable wait times by request criticality and a control for handling warmup work in a separate thread pool. No performance benchmarks have been published; deployments still need to validate scheduler settings and tail latency.
On September 24, TensorFlow published the 2.22.0-rc0 release notes on GitHub, adding tiered batch waiting and a separate warmup thread pool setting for BatchFunction. This is a release candidate announcement: the PyPI package was already available on the 22nd, and platform files were uploaded on the 23rd, so the announcement date differs from when the package first became available. [Release notes](https://github.com/tensorflow/tensorflow/releases/tag/v2.22.0-rc0), [PyPI record](https://pypi.org/project/tensorflow/2.22.0rc0/)
BatchFunction combines inputs from multiple concurrent calls along the first dimension, then runs a specified function, making it suitable for model inference. It already has controls such as maximum batch size, timeout, and queue capacity. The fuller the batch, the more likely it is to improve compute utilization, but waiting for other requests also adds latency. Low traffic and bursty traffic therefore call for different tradeoffs. [Operator documentation](https://www.tensorflow.org/mlir/tf_ops#tfbatchfunction)
The new `per_criticality_batch_timeout_micros` lets different criticality levels use different wait times; the official documentation lists it as an integer array. From a service-design perspective, teams could send interactive requests sooner while letting less urgent work wait longer to fill batches. The effect still depends on how requests are classified and whether the execution path uses a corresponding scheduler. This change directly affects services that use batching functions, and existing applications should confirm that the setting is propagated all the way to the execution node. [Interface definition](https://www.tensorflow.org/mlir/tf_ops#tfbatchfunction)
Another attribute, `num_warmup_batch_threads`, can configure a separate thread pool for warmup requests. The scheduler source explains that this design addresses warmup work occupying normal batching threads when the global scheduler is used. For services that handle traffic while loading a model, it provides a way to manage warmup and online work separately. [Scheduler source](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/kernels/batching_util/shared_batch_scheduler.h)
These controls address scheduling resources and batch formation timing. Under the scheduler's semantics, once the timeout expires, a thread must still be available to dispatch a partially filled batch. The configured timeout therefore does not directly guarantee end-to-end latency, and a separate thread pool does not establish hardware resource isolation. [Scheduler semantics](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/kernels/batching_util/shared_batch_scheduler.h)
The official release notes provide no throughput or tail-latency benchmarks for these two settings. Engineering teams can first replay mixed traffic, verify the operator attributes, criticality classifications, and warmup path, then compare high-percentile latency by request class, average batch size, and model readiness time. In particular, they should test resource contention during model loading to see whether faster warmup comes with worse online latency, and wait for the stable release to confirm the interface and behavior. [Release candidate notes](https://github.com/tensorflow/tensorflow/releases/tag/v2.22.0-rc0)