Observation, memory, action, runtime: the four costs of efficient GUI agents
A deep dive into a new EMNLP 2026 survey that maps GUI-agent efficiency across observation, memory, action, and runtime, then shows which costs each optimization moves elsewhere.
A GUI agent can complete a task and still be too slow, too expensive, or too intrusive to deploy. Bizhe Bai and ten co-authors' Efficient GUI Agents: A Systems Survey of Observation, Memory, Action, and Runtime Optimization makes the reason measurable: deployment cost accumulates across the whole observe–remember–plan–act–verify trajectory, so a local reduction counts only after its own retrieval, parsing, verification, and orchestration costs are included. The paper is an arXiv v1 survey submitted on September 2, 2026, with an author note that it was accepted at EMNLP 2026's Grounding Language Models workshop. 1
The survey is worth reading in full for anyone building computer-use systems. Its durable contribution is a four-axis accounting frame, rather than a claim that one architecture wins everywhere: observation efficiency, context and memory efficiency, action efficiency, and planner-side and system efficiency. The frame turns an attractive speedup into a question an engineer can answer: which part of the trajectory became cheaper, which new cost appeared, and whether task quality stayed matched? 2
The loop the survey makes measurable
A GUI agent receives a goal, observes a rendered interface or a structured representation, retains enough history to interpret the current state, chooses an action, executes it, and checks the result. Screenshots, DOM or HTML, accessibility trees, focused elements, and window metadata can all enter the observation. Clicking, typing, scrolling, dragging, hotkeys, and application switches can all become actions. 3
The loop matters because every step can multiply the next step's cost. A long DOM increases the planner's input. A screenshot-heavy history increases cache pressure. A mistaken click creates recovery work. A verifier can prevent a cascade while adding another model call. The paper's appendix writes the episode as a trajectory with observation, memory update, policy, environment transition, and verification terms; Sections 3–6 then reduce different terms in that trajectory. 5
That formulation creates a real comparison among the methods. Observation reduction changes what the agent sees. Memory compression changes what the agent carries. Action abstraction changes how many interactions the agent needs. Planner and runtime methods change how the system allocates computation and privacy across the loop. A method can improve one term while shifting cost into another.
Four axes, four trade-offs
| Axis | Main bottleneck | Representative mechanism | Direct signal | Cost that still needs measurement |
|---|---|---|---|---|
| Observation | Redundant DOM, AxTree, HTML, pixels, or visual tokens | Retrieval, pruning, region focus, parsing, or hybrid visual-plus-structure inputs | Context length, parser calls, crops, visual tokens, grounding latency | Retriever/filter latency, extra zooms, parser model cost, grounding errors 6 |
| Context and memory | Replaying long trajectories and retaining large caches | Summaries, selective look-back, continuous memory, or GUI-specific KV compression | History length, retrieved memory, cache size, FLOPs, memory footprint | Lost state, recomputation, prefill cost, retrieval latency, memory-quality trade-off 7 |
| Action | Too many primitive steps, unsafe actions, and unstructured exploration | Skills, compiled programs, action filtering, verification, rollback, or guided search | Step count, model calls, cost, latency, recovery length | Skill discovery, verifier calls, failed actions, routing errors, setup cost 8 |
| Planner and system | Reasoning, orchestration, privacy, and backend overhead around the action predictor | Fast/slow planning, specialist routing, local/cloud cooperation, GUI-plus-code execution | End-to-end latency, TTFT, TPOT, tool calls, privacy exposure, human intervention | Coordination, privacy detectors, specialist calls, fallback behavior, unmatched task conditions 9 |
Observation: spend perception where actionability lives
Textual GUI agents often receive far more interface structure than the next action requires. The survey cites Agent-E's reports of raw DOM inputs reaching 800,000 tokens, typical tasks taking 150–220 seconds, and roughly 25 LLM calls. LineRetriever reports observation reductions of 61%, 72%, and 73% across its settings, while FocusAgent reports more than 50% average AxTree reduction, often above 80%, with a 2,000-token context cap. Prune4Web pushes filtering into executable programs: examples reduce DOM trees of 10,000–100,000 tokens and more than 500 candidate elements to fewer than 20 actionable candidates. 10
These methods differ in what they preserve. A semantic retriever may find text related to the goal, while a planning-aware retriever selects lines that support the next navigation decision. A filter program can reduce both context length and the grounding search space, while its own generation and runtime cost require separate accounting. The paper's table marks several of those costs as unreported, so the reported reduction is a signal from the authors' baseline rather than a cross-paper ranking. 11
Screenshots create a different allocation problem. Small targets, repeated widgets, and irrelevant background occupy the same image as the action target. Region-aware methods split screens, propose crops, or zoom progressively. R-VLM reports about 5.6 seconds per sample and up to 2x inference-latency cost in the cited setting; RegionFocus reports 66.8% average trajectory overhead and a 19.74% step-count increase. The survey uses these figures to show why visual focus is an allocation mechanism, not a free reduction. 12
Parsing and hybridization address a second problem: screenshots preserve rendered context, while DOM and accessibility trees preserve structure. OmniParser and Tree-of-Lens convert images or pointed regions into semi-structured representations. GUI-Actor adds a grounding verifier and coordinate-free action regions. Hybrid systems combine visual context with accessibility or structural signals. The useful design question is therefore whether the representation is merely smaller or becomes more immediately actionable. 13
Memory: preserve the state the next action needs
Long-horizon execution makes raw history replay expensive and noisy. Agent-S, ColorBrowserAgent, and GUI-Rise replace full trajectory replay with task-progress memories or summaries. PAL-UI combines dual-level summarization with active look-back. HiconAgent and SimpAgent use dynamic sampling, anchor-guided compression, or consistency-guided pruning; the survey reports 25.21T FLOPs for compressed history against 35.75T in an uncompressed 3B setting and 62.31T in a 7B setting for HiconAgent's cited comparison. 14
Serving-level methods move the compression boundary into the representation used during inference. GUI-KV exploits spatial and temporal redundancy in GUI trajectories and reports 38.9% fewer MFLOPs per decoded token at five screenshots, with cache budgets of 5–20%. ST-Lite reports a 2.45x decoding speedup and 1.40x end-to-end speedup at 10–20% cache budgets, while prefill remains close to 1.0x in the cited setting. The split between decode and prefill matters: a decode improvement cannot be treated as an end-to-end result without the rest of the serving path. 15
Continuous memory takes a more radical route. Auto-scaling Continuous Memory compresses a trajectory into eight embeddings even when the raw trajectory exceeds 15,000 tokens. The survey records the method's reported data-collection cost of about $4,000 and tuning of 1.2% of parameters. The numbers describe the cost of building the memory mechanism, while the fixed-length representation describes its serving behavior. A deployment comparison needs both. 15
Action: reduce steps and make failure recoverable
Action abstraction turns repeated primitive operations into reusable skills, routines, or programs. SkillWeaver discovers website procedures and distills them into callable APIs. PolySkill separates an abstract goal from a site-specific implementation, with learned functions typically covering 2–5 GUI steps in the cited description. Mobile-Agent-E uses reusable Shortcuts with explicit preconditions. These methods save planning work when the same control pattern recurs, but they also introduce discovery, transfer, and precondition costs. 16
ActionEngine compiles repeated interaction patterns into state-machine memory and executable programs. Its reported comparison moves cost from $0.71 to $0.06, latency from 237.5 to 118.3 seconds, input tokens from 62.3k to 8.1k, and model calls from 10.2 to 1.8. CoAct-1 adds code execution as a second action modality and reports 10.15 average steps, compared with 15.22 for GTA-1 and 14.90 for UI-TARS in the cited comparison. These figures support a narrower conclusion: recurring subtasks can become cheaper when the system can compile or route them. The figures do not establish that compilation or code routing wins on every task. 16
Action pruning, verification, and recovery solve a different failure mode. V-Droid reports 0.7 seconds per decision and 4.3 seconds per step in its cited comparison, while typical mobile agents in that comparison exceed 20 seconds per step. BacktrackAgent adds explicit backtracking with verifier, judger, and reflector modules. LongHorizonUI combines reflection with rollback across trajectories averaging 24.6 steps and reaching 37 steps at maximum in the cited setting. A shorter path matters, but a path that detects and repairs an early mistake can matter more than a path that is short only when every action succeeds. 17
Exploration methods make search itself more structured. LASER models web interaction as state-space exploration with backtracking. Auto-Intent uses compact intents to guide self-exploration. OpenWebVoyager uses an exploration–feedback–optimization loop, and WebOperator combines pre-execution filtering, best-first search, and safe backtracking. The common mechanism is a reduction in blind rollout, achieved through reusable structure, predicted intent, or transition logic. 18
Planner and system: price the parts around the model call
Planner-side efficiency includes reasoning depth and model-call scheduling. System efficiency includes memory reuse, privacy-aware routing, specialist cooperation, and the choice between GUI actions and programmatic execution. MMBench-GUI is cited as an efficiency-aware evaluation framework with a 50-step budget, redundant-step cost values of 7–8, privacy noise of 40%, and specialist cost of 16%. The numbers belong to that benchmark's definitions and conditions; they demonstrate why a success score alone omits relevant costs. 19
The survey treats privacy as a runtime constraint rather than an afterthought. CORE routes work between cloud and local language models to reduce UI exposure. GUIGuard introduces privacy-preserving stages around GUI interaction. A local model can reduce exposure while adding specialist or detection work. The correct comparison therefore records both the content sent to each backend and the latency and cost of the routing path. 20
CoAct-1 and related systems show the value of hybrid execution: a subtask that needs visual interaction can stay in the GUI channel, while file operations or data processing can move to code. Hybrid execution changes the action space and the coordination problem at the same time. A benchmark result becomes meaningful only when it reports the routing policy, handoffs, tool setup, and failure behavior alongside the average step count. 20
How the field got here
The survey's chronology is a sequence of changes in the unit of optimization. It is a map of the mechanisms represented in the paper, not a claim that every research group moved through the same stages.
- 2023: environments and search. WebArena and LASER established realistic web tasks and state-space exploration as central objects for agent research. The question was how an agent could navigate a changing interface and recover from a wrong branch. 18
- 2024: visual grounding and computer-use benchmarks. SeeClick, ScreenAI, OmniParser, Agent-S, OSWorld, AndroidWorld, and related work expanded the observation problem from text navigation to screenshots, multimodal grounding, mobile control, and open-ended desktop tasks. 6
- 2025: selective context and recoverable control. LineRetriever, FocusAgent, ShowUI, HiconAgent, GUI-KV, SimpAgent, skill discovery, verification, rollback, and reasoning-aware systems moved attention toward the cost of the trajectory rather than the next action alone. 9
- 2026: system-level accounting. ST-Lite, continuous memory, SecAgent, PolySkill, ActionEngine, CoAct-1, and related work pushed compression, reusable actions, and hybrid execution into serving and orchestration. The new survey gathers these mechanisms under one efficiency-centered taxonomy and makes their shifted costs part of the comparison. 21
The progression changes the question a builder should ask. Early work asked whether a GUI agent could perform the task. Later work asked how much interface state, history, action budget, and runtime the agent consumed while doing so. The survey's answer is a measurement frame that keeps those questions attached to the same trajectory.
The evaluation correction
Task completion is an effectiveness measure. Efficiency measures the resources spent to reach that outcome. The paper separates latency metrics such as TTFT, TPOT, inference latency, per-step latency, and end-to-end latency from observation metrics such as perception-call count, DOM/HTML/AxTree length, and image-crop count. It also tracks history length, retrieved memory, KV-cache size, memory footprint, actions, tool calls, and human intervention. 22
The paper's appendix labels its tables an evidence ledger. Reported percentages, speedups, token counts, costs, and step counts are copied from the cited authors' baseline comparisons. The survey does not turn those heterogeneous measurements into a direct meta-analysis. This limitation protects the reader from a false leaderboard: a cache speedup, a token reduction, and a step-count reduction answer different questions under different models, tasks, hardware, precision, and serving conditions. 23
The same correction applies to verification. A verifier can improve recoverability while increasing calls and wall-clock latency. The survey marks verifier-call and latency fields as unreported for several methods. A system that reports only the final success rate leaves the price of its safety mechanism outside the result. 17
A builder's checklist
- Define the success condition. Record the task outcome, the allowed action budget, the acceptable latency, and the privacy boundary before comparing systems. A success rate without those conditions cannot describe deployability.
- Measure the observation path. Log DOM, HTML, AxTree, screenshot, crop, OCR, parser, and VLM calls separately. Record the representation size that reaches the planner and the time spent producing it. 6
- Measure retained state. Record history length, retrieved memory, cache budget, prefill and decode costs, and the information discarded by summarization or pruning. A smaller context deserves a state-retention test.
- Count actions and recovery. Track primitive actions, compiled skills, failed actions, rollback length, backtracking, and recomputation. A lower step count with more expensive failures may increase total cost.
- Price verification. Log verifier calls, verifier latency, judgment accuracy, false recoveries, and the fraction of tasks that use verification. A post-condition check belongs in the cost ledger.
- Test routing and handoffs. Record when a task moves from GUI interaction to code, from local to cloud inference, or from a generalist to a specialist. Include setup, coordination, privacy filtering, and fallback behavior.
- Match the experiment. Keep model, task, hardware, precision, benchmark, context policy, action budget, and stopping rule aligned before comparing a number across methods. The survey's evidence-ledger warning makes this requirement explicit. 23
- Report the full trajectory. Publish effectiveness together with latency, calls, tokens, cache, memory, actions, recovery, cost, privacy exposure, and human intervention. The resulting record lets another team see where the saving came from and what it displaced.
Where the evidence stops
The survey supplies a strong map and a disciplined vocabulary. It does not supply a universal best architecture. The paper's own tables contain heterogeneous author baselines, and several methods leave parser, retriever, verifier, orchestration, or wall-clock costs unreported. Cross-benchmark comparability and co-design across observation, memory, and execution remain open problems. 24
That boundary is exactly why the paper is useful. A reader designing a GUI agent can now ask whether a claimed gain lowered total trajectory cost, preserved the state needed for the next action, and kept verification and privacy inside the deployment budget. The full paper is the right next read for the taxonomy tables and the cited method papers: Efficient GUI Agents: A Systems Survey of Observation, Memory, Action, and Runtime Optimization.
References
- 1
- 2Bai et al., Sections 1–2
arxiv.org
- 3Bai et al., Sections 2.1–2.2
arxiv.org
- 4Bai et al., Figure 1
arxiv.org
- 5Bai et al., Appendix A
arxiv.org
- 6Bai et al., Section 3
arxiv.org
- 7Bai et al., Section 4
arxiv.org
- 8Bai et al., Section 5
arxiv.org
- 9Bai et al., Section 6
arxiv.org
- 10Bai et al., Section 3.1
arxiv.org
- 11Bai et al., Appendix D, Table 2
arxiv.org
- 12Bai et al., Section 3.2
arxiv.org
- 13Bai et al., Section 3.3
arxiv.org
- 14Bai et al., Section 4.1
arxiv.org
- 15Bai et al., Section 4.2
arxiv.org
- 16Bai et al., Section 5.1
arxiv.org
- 17Bai et al., Section 5.2
arxiv.org
- 18Bai et al., Section 5.3
arxiv.org
- 19Bai et al., Section 6.1
arxiv.org
- 20Bai et al., Section 6.2
arxiv.org
- 21
- 22
- 23Bai et al., Appendix D
arxiv.org
- 24Bai et al., Abstract and Section 7
arxiv.org
This story was produced automatically by a channel. One sentence is all it takes for Neodrop to keep producing for you.