01 Context & Constraints

Cloud-based agents controlling mobile devices fail in the real world for three fundamental reasons: network jitter on cellular connections destroys tactile feedback loops, transmitting real-time screen captures off-device violates basic personal privacy, and commercial API latency (typically 800ms–2400ms per step) makes typing and scrolling feel broken.

In this project, we built a fully autonomous, zero-cloud agent loop executing entirely on-device. An open-weight quantized model running via llama.cpp in Termux translates user intent directly into low-level Android kernel input events via an ADB Unix domain socket.

02 Agent Architecture

The system is organized into a four-stage cyclic pipeline designed to eliminate IPC overhead between perception and execution:

USER GOAL CLI / Voice Intent LOCAL LLM llama.cpp Q4_K_M Vulkan / OpenCL Kernels Structured JSON-RPC AGENT DAEMON Python Dispatcher UI Node Indexer Action Validator ANDROID HW /dev/input/event* screencap -p perception loop
Figure 01 — Pipeline architecture of on-device model inference to kernel multi-touch injection.

03 ADB Touch Transport

Standard adb shell input tap x y launches a new zygote JVM process for every invocation, taking 320ms–540ms per touch event. To achieve interactive speeds, the daemon communicates directly with the kernel via native binary events:

TERMUX — /dev/input/event3 injection
$ python3 -c "import struct, time, socket; ..."
[SYN_REPORT] Event code 0x00 Type 0x00 Value 0 [ABS_MT_POSITION_X] X=540 (normalized 1080p width) [ABS_MT_POSITION_Y] Y=1120 (normalized 2400p height) [ABS_MT_PRESSURE] P=78 Value written to /dev/input/event3 Latency: 4.8ms from socket packet to kernel touch acknowledgment.

04 Local Inference Benchmarks

Testing on a Snapdragon 8+ Gen 1 platform using llama-bench with OpenCL shader backend:

LLAMA.CPP BENCHMARK — TERMUX
$ ./llama-cli -m models/qwen2.5-3b-instruct-q4_k_m.gguf -ngl 33 -b 512 -p "Action: tap Settings"
llama_model_loader: loaded 33 layers to GPU compute buffers system_info: n_threads = 4 / 8 | Android Termux PRoot aarch64 | OpenCL Adreno 730 prompt eval time = 284.12 ms / 16 tokens ( 17.42 tokens per second) eval time = 882.35 ms / 6 tokens ( 6.80 tokens per second) total time = 1166.47 ms / 22 tokens TOOL_CALL: {"action":"tap", "target":{"class":"TextView", "text":"Settings", "bounds":[48, 620, 240, 710]}}

05 Execution Logs & Perception

Real-time execution log from the agent control loop executing a multi-step navigation flow:

[02:41:08.102] GOAL User requested: "Toggle Bluetooth off and launch WireGuard"
[02:41:08.215] PERCEPT Dumping UI hierarchy via uiautomator: 42 visible leaf nodes
[02:41:08.490] INFERENCE Prompting local Qwen2.5-3B (temp=0.1, max_tokens=64)
[02:41:09.112] TOOL Matched action: adb_swipe(540, 0, 540, 800, 150) -> Quick Settings shade
[02:41:09.280] TOUCH Direct kernel write /dev/input/event3 -> 12 events dispatched
[02:41:09.640] VERIFY Quick Settings shade confirmed active. Next step: Toggle Bluetooth.

06 Device Benchmark Matrix

Measured performance across physical test devices under thermal equilibrium:

Hardware Platform Kernel / OS Prompt Proc Generation Peak Memory Status
Snapdragon 8+ Gen 1 (Nothing Phone 2) Linux 5.10 / Android 14 17.4 tok/s 6.8 tok/s 1,840 MB PASS
Tensor G3 (Pixel 8) Linux 5.15 / Android 14 14.1 tok/s 5.2 tok/s 1,910 MB PASS
Dimensity 9200+ Linux 5.15 / Android 14 16.8 tok/s 6.1 tok/s 1,870 MB PASS
Snapdragon 865 (Legacy Pixel 4 XL) Linux 4.19 / Android 13 8.2 tok/s 3.1 tok/s 1,820 MB OOM RISK

07 Failure Modes & Race Conditions

Through 300 automated execution trials, we identified two severe failure modes:

  • Soft Keyboard Occlusion: When an input field gains focus, the soft keyboard animates upward over 250ms. If the agent emits an immediate subsequent tap on a target coordinate calculated prior to the animation, the tap hits the keyboard rather than the intended submit button.
  • Accessibility Node Cache Staleness: Android's uiautomator frequently returns cached accessibility node bounds for dynamically populated RecyclerView items, causing the agent to attempt taps on off-screen views.
Root Cause Analysis

The Android window manager does not broadcast an atomic completion event for soft keyboard transitions to unprivileged shell sessions. The agent must verify window metrics after any input event before issuing coordinates.

08 Lessons & Architecture Decisions

To eliminate these race conditions without adding artificial sleep delays:

  1. We implemented a heuristic bounding-box validator that clips all proposed target coordinates against the active display minus the current IME (Input Method Editor) frame height.
  2. We replaced full UI hierarchy XML dumps with an incremental binary diffing algorithm running in C, reducing perception extraction time from 420ms down to 38ms.

09 Source Code & Reproduction

The entire daemon and quantized models are available in the repository below. Ensure you enable Developer Options and grant USB debugging (Security settings) for simulated input permission.

Inspect Repository axe01010/android-ai-agent ↗

K
Krish / axe01010
Systems engineer and security researcher. Eight years building and shipping production software directly from mobile Linux environments.
RELATED RESEARCH & BUILDS