See, Reason, Act: Multimodal Agents That Operate the World

For most of the last three years, "multimodal model" meant one thing: a vision-language model (VLM) that could look at a picture and answer questions about it. An image encoder fed an LLM; the LLM described. The loop ended at language. That definition is now obsolete, and the change is not "the model got better at pictures." The change is that multimodal systems have crossed from understanding the world to operating it — closing a loop of See → Understand → Reason → Act across vision, voice, and tools.

This is the quiet reorganization behind a cluster of 2024–2026 releases: Anthropic's Computer Use, OpenAI's Operator and the Realtime API, Google's Project Mariner, and open-source agent models like ByteDance's UI-TARS. They look like separate products. They are one shift seen from different angles.

Three threads, now fused

Computer-use agents. The cleanest proof that multimodal AI can act came from agents that drive a computer the way a person does — by reading the screen and issuing mouse and keyboard actions. Anthropic shipped the first major commercial version in October 2024 with Claude 3.5 Sonnet, scoring 14.9% on the OSWorld benchmark of real desktop tasks (humans: ~72.4%). The trajectory since is the story: Sonnet 4 → 42.2%, Sonnet 4.5 → 61.4%, and by February 2026 Sonnet 4.6 reached 72.5% — essentially at the human line. OpenAI's Operator (January 2025), powered by its Computer-Using Agent (CUA) model on GPT-4o with reinforcement learning, posted 38.1% on OSWorld, 58.1% on WebArena, and 87% on WebVoyager, running a perceive–reason–act loop directly over raw pixels and asking for confirmation only on sensitive steps. Google's Project Mariner (December 2024) hit 83.5% on WebVoyager but ran at roughly five seconds per action. ByteDance's open-source UI-TARS-1.5 reframed the hard part as grounding: 61.6% on ScreenSpotPro (dense, high-resolution professional UIs) versus 27.7% for Claude and 23.4% for CUA, and it "thinks before it acts" via an explicit reasoning step.

Continuous voice. OpenAI's Realtime API (public beta October 2024, general availability August 28, 2025 with the gpt-realtime model) is the counterexample to the old "say a sentence → wait for the model → get a sentence back" pattern. It uses a single speech-to-speech model over a persistent connection instead of chaining speech-to-text, an LLM, and text-to-speech. OpenAI reports roughly 300 ms to first audio over WebRTC and 250–500 ms end-to-end, and the connection handles interruptions natively — you can cut the agent off mid-sentence and it reroutes. The traditional cascade typically lands at 1.5–3 seconds and, worse, flattens tone and emotion into text before the model ever sees it.

Tool use. Function calling has matured into protocol-level plumbing. gpt-realtime added remote MCP server support and SIP for phone calls; Operator calls tools mid-task. The agent is no longer a chatbot with a screen — it is a node on the same tool graph as the rest of your software.

Insight 1: The convergence is the event, not the components

The interesting fact is not that any one capability improved. It is that they fused. gpt-realtime can see an image you send, hear your continuous speech, call an external tool over MCP, and respond — all inside one speech-to-speech model. Operator can see a webpage and click. Voice, vision, and action used to be three product categories with three roadmaps. In 2025 they became one capability: a closed perception–reasoning–action loop. The meaningful unit of "multimodal" stopped being modality coverage and became agency.

Insight 2: Latency is the moat — and it is an architecture problem

Continuous voice only feels continuous because OpenAI collapsed three models into one. That is an engineering decision about the loop, not a bigger model. The numbers explain why it matters: under ~500 ms a conversation flows and the pause reads as a thoughtful person; between 500 ms and 1 s it is tolerable; past 1 s the conversation breaks down and people start talking over the agent. For an agent that also acts, latency is not a polish metric — it is the difference between a system that can hold a real-time loop with the world (react to a screen change, interrupt, recover) and one that is a slow batch processor wearing a chatbot skin. The frontier work in voice agents is happening in the transport and architecture, not the weights.

Insight 3: Grounding is the bottleneck that "understanding images" hides

Saying "I see a button" and executing "click at (x, y) reliably" are different problems, and the second one is where agents still struggle. OSWorld's human ceiling is ~72.4%; the best agents only reached it after sixteen months of near-fivefold gains, and only on that benchmark. On dense, unfamiliar, high-resolution software (ScreenSpotPro), even the leader sits at 61.6% while frontier chat models fall to the mid-20s. Meanwhile browser tasks are effectively solved — WebVoyager has climbed past 90% and leaders sit near 97%. So "operating the world" is real but partitioned: easy and web tasks are largely done; precise manipulation of stateful, novel, high-DPI software is the live frontier. The gap is perception-grounding, not reasoning.

Insight 4: The stack is collapsing from framework to model

Early agents were frameworks: a base model wrapped in prompts, tools, and orchestration code. They were flexible but brittle and expensive to maintain. Native agent models like UI-TARS and CUA instead train perception, reasoning, memory, and action as one model through reinforcement learning on interaction trajectories. This is the same pattern seen in reasoning models — capability moves from orchestration engineering into the weights. The scarce resource becomes high-quality interaction trajectories and a reliable environment to train them in. Whoever owns the data loop owns the agent.

Insight 5: Trust, not capability, gates deployment

Every vendor shipped these agents behind guardrails, because acting changes the risk model. Operator confirms sensitive actions; Mariner was locked to the active tab and refused checkouts; CUA runs in sandboxes. The open wound is prompt injection: a malicious webpage can hijack an agent mid-task, and OpenAI has stated this risk is "unlikely to ever be fully solved" for agents on the open web. So the rate of real-world autonomy is capped by safety and trust, not by model IQ.

The throughline

The word "multimodal" is doing too much work. The real story of 2024–2026 is agency: systems that can hold a real-time, closed See → Understand → Reason → Act loop across vision, voice, and tools. We are not primarily waiting for smarter models. We are waiting for faster loops, sharper grounding, and guardrails trustworthy enough to widen the domains. The agents that operate the world already exist in narrow ones; the next eighteen months are about expanding the map while keeping the loop safe.