多模態模型與開發者平台
Gemini Turns Long-Video Understanding Into a Tool Loop, Dynamically Selecting Clips and Sampling Rates Based on the Query
Google has added agentic video processing to three Gemini Flash models, allowing them to load frames, audio, or transcripts on demand instead of processing an entire video at a fixed frame rate in advance. The company claims token savings of up to 88% and cost reductions of 66%, though time to first token for short videos and evaluation reproducibility remain to be tested.
Google has launched agentic video understanding for Gemini 3.7 Flash, 3.6 Flash, and 3.5 Flash-Lite, transforming video input from a one-time, fixed-sampling process into a model-controlled tool loop. The traditional static mode captures frames at a default rate of one frame per second and places them in context alongside the audio. The new mode instead lets the model search the timeline based on the query, choose whether to read the transcript, audio, or specific frames, and resample fast-moving action at a higher frame rate and resolution.
Developers can set `processing` to `agentic` within video content in the Gemini Interactions API. The response’s `steps` will include `processing_call` and `processing_result`, recording clip-loading requests and their results, respectively. This is more than an internal optimization: in stateless, multi-turn workflows, applications must send these steps back together with previous responses, or the video context will be lost. That content also counts toward input tokens in subsequent requests. Navigation tokens generated during reasoning are classified as thought tokens, while loaded media is counted as tool-use tokens, so cost-tracking systems need to monitor the two separately.
Google says tests on long videos reduced token usage by up to 88%, cut analysis costs by 66%, and improved quality by as much as approximately 7%. Potential use cases include searching across hours-long videos, anomaly detection, action counting, and locating subsecond events. The feature uses standard API token pricing and supports both uploaded videos and public YouTube URLs.
The engineering tradeoff is that the agentic loop must first plan and invoke tools, which may increase time to first token for videos shorter than five minutes. Auditing tasks that require complete frame-by-frame coverage may also be poorly suited to selective loading. Google has disclosed only best-case improvements and selected LongVideoBench demonstrations, without providing a comprehensive breakdown by model, video length, and query type. Before deployment, teams should use their actual video distribution to compare static and agentic modes in terms of recall, latency, token composition, and the risk of missing critical segments.