Ultralytics 8.4.162 adds image batch prefetching to overlap loading with CUDA inference

Ultralytics released version 8.4.162 on September 24, enabling CUDA image inference to preload the next batch of images while processing the current batch. The corresponding package is also available on PyPI. The change targets the data-loading pipeline and is relevant to teams performing offline analysis of large image collections. [Release notes](https://github.com/ultralytics/ultralytics/releases/tag/v8.4.162), [PyPI package](https://pypi.org/project/ultralytics/8.4.162/)

Previously, the iterator had to finish reading and decoding the next batch of images before the GPU had data to process. The new workflow uses a single background worker thread for prefetching, overlapping loading with processing of the current batch. In the main branch, this path is enabled when the device is CUDA, the source uses LoadImagesAndVideos, all files are images, and there is more than one batch. Videos and single batches do not use this path. Based on the pipeline design, the benefit depends on whether computation can hide file-reading time; it does not directly imply that a model’s forward pass on a single image will become faster. [Source code](https://raw.githubusercontent.com/ultralytics/ultralytics/main/ultralytics/engine/predictor.py)

The developer’s tests in the PR used YOLO26n with a 640-pixel input size, running five paired trials in separate processes for each configuration. On an L4, processing 1,000 COCO images with batch=1 reduced the median elapsed time from 23.012 seconds to 19.521 seconds. Processing 5,000 images with batch=16 reduced it from 31.224 seconds to 27.608 seconds. Timing included model creation, result collection, output hashing, and CUDA synchronization, so these figures represent the complete test workflow. They come from a specific commit before the merge and still need to be verified against the released package. [Test conditions](https://github.com/ultralytics/ultralytics/pull/26319)

There is a documentation discrepancy concerning scope: the release summary still describes the feature as limited to PyTorch detection, but the PR’s final commit indicates that it was extended to other tasks and backends. The current main branch also lacks these two restrictions. Engineers should check the enabling conditions in their installed version and avoid extrapolating the early benchmarks to every affected path. [Merge history](https://github.com/ultralytics/ultralytics/pull/26319)

Prefetching retains additional decoded images in memory, so peak system RAM usage should be included in the evaluation. For large volumes of results, stream=True also allows results to be consumed one at a time, but developers still need to check whether their application accumulates results itself. [Inference interface](https://docs.ultralytics.com/reference/engine/predictor/)

Based on the pipeline design, background loading may still be constrained by bandwidth and tail latency when images reside on a network drive. Benefits may also shrink if writing results to disk accounts for most of the runtime. Practical validation should keep image order, batch size, and output settings fixed while measuring total elapsed time, result consistency, and resource cleanup after the stream is closed early. Comparisons should also distinguish initial model creation from subsequent repeated runs to avoid conflating initialization differences with prefetching gains.