Control protocol

Every ashlar app can be driven by an agent in three ways, all speaking one method vocabulary:

Protocol version: 1 (reported by app.version).

Framing

One JSON object per line (\n, an optional \r is stripped), each way, one reply per request, in order.

Request:

{"id": 7, "key": "<64 hex digits>", "method": "engine.execute", "params": {"command": "edit.undo"}}

• id (optional, any JSON): echoed in the reply; omitted from the reply when absent.
• key: the app’s control key (TCP only; see Keys). Not needed after auth on the same connection.
• method (required string).
• params (optional; null or absent means {}).

Reply:

{"id": 7, "ok": true, "result": …}
{"id": 7, "ok": false, "error": "human-readable message", "detail": …}

detail is optional machine-readable context: for a failed command, the command error {"kind": "unknown"|"disabled"|"bad_params"|"failed"|"panicked", "id": …, …}; for a failed batch, {"steps": [...]}.

Hardening (TCP)

The server is built so that a hostile local process or web page cannot drive or crash the app:

• Loopback only. Binds 127.0.0.1; the client only connects there.
• Key. Once the port is bound the app writes 32 random bytes (64 hex digits) to <dir>/control-key.<instance>, created with mode 0600, where <dir> is a folder the app chooses (its settings folder) and <instance> is the Flatpak instance id or local-<port>. The file is removed when the server is dropped; a stale one is replaced. Every request must carry the key, as the top-level key member or once per connection with auth. A missing or wrong key gets {"ok":false,"error":"unauthorized"} and the connection is closed. Keys are compared in constant time. params.key is an ordinary parameter for every method but auth (ui.key {"key":"Enter"}).
• Strict lines. A line that is not a JSON object with a string method (invalid UTF-8, bad JSON, an array, a number, an HTTP request line) gets one error reply and the connection is closed, so nothing after it runs. This is what stops a browser’s cross-origin fetch POST: its request line POST / HTTP/1.1 is rejected before the JSON body is read.
• Bounded lines. Longer than max_line (4 MiB default): error reply, the rest of the line is drained (at most 64 MiB, 2 s) so the client can read the error, then close. A peer can never make the server buffer more than max_line.
• Connection cap. At most max_connections (16) at once; extra ones get {"ok":false,"error":"too many control connections"} and are closed.
• First-request deadline. A connection must send an authorised request within 30 s. It is one overall deadline: blank lines or a slow drip of bytes do not extend it. After that, no idle limit.
• Reply timeout. The app thread answers within 60 s or the client gets timeout: ….
• Never panics. No unwrap/indexing on input; junk never reaches the app thread; a request the app drops unanswered replies with an error by itself.

Standard methods

Answered by ashlar_control::handle from any Dispatch (headless or windowed):

UI methods

Routed to the app’s UiHandler. A handler may keep a request and answer it on a later frame. Headless sessions refuse them with a hint. The windowed runtime answers the app-level ones itself and hands the debugger methods (next section) to ashlar_ui::devtools:

app.clock holds the time the app’s frames see, to look at an animation at an exact moment. pause freezes it at the current frame’s time; step_ms (0 < ms ≤ 60000; pauses first) moves the frozen time on and runs one frame there; resume lets time run again from the frozen value, so nothing jumps (the app’s time then lags the wall clock by the time spent paused). time is the egui time in seconds the next frame sees. While paused:

• input still runs frames, at the frozen time. Two clicks are then 0 s apart, so egui counts them as a double click; step between clicks that must stay single.
• everything timed by egui time stands still: egui’s animations, ashlar_ui::motion, tooltip delays, toast timeouts, the caret’s blink phase and other timed wakes. A timed wake still runs a frame about once per its delay (the frame finds it not yet due), and an egui animation that is mid-flight keeps asking for frames that draw the same picture.
• settings and autosave run on the host’s clock, so saves are not held up. Background jobs and control requests run as usual.

ashlar-debug clock pause|step <ms>|resume (also a .ui script line) and the MCP tool ui_clock send it. A headless session refuses it.

Apps may add their own methods (ui.timeline.hit…); list them in UiHandler::methods so they show in app.version.

Debugger methods

Drive the UI the way a browser automation tool drives a page: snapshot the element tree with short refs, act on refs, wait, read, screenshot. Implemented by ashlar_ui::devtools (an egui plugin), so an app that does not use ashlar-app gets the same methods (the ashlar-debug CLI and the MCP ui_* tools speak them; see debugging-uis.md) by passing requests to devtools::handle(ctx, request) (see that module’s docs). app.version lists them when available.

Elements and refs

The element tree is egui’s AccessKit output (every egui widget: role, label, value, state, bounds) merged with the ashlar automation registry (dotted ids, component roles). An element:

{"ref": "e12", "id": "notes.title", "role": "textbox", "name": "Title", "value": "Groceries",
 "enabled": true, "focused": false, "selected": false, "checked": null, "visible": true,
 "rect": [x, y, w, h], "parent": "e3"}

id, value, selected, checked and parent are omitted when they do not apply. rect is the on-screen part in logical points. Roles: button checkbox switch radio radiogroup textbox searchbox slider spinbutton combobox link label image colorwell panel window dialog scrollbar separator progressbar menu menubar menuitem tab tablist list listitem tree treeitem table row cell columnheader toolbar status alert tooltip group generic.

A ref stays the same for an element across snapshots while it exists (keyed by its automation id, else by its egui widget id and role); every ui.snapshot increments the generation. The first debugger request turns egui’s AccessKit output on for the rest of the session.

Targets

Every method that acts on an element takes a target, as target: {...} or flat in the params:

The error text starts with the kind (stale_ref: …, not_found: …, timeout: …) and detail.kind carries it: stale_ref ({kind, ref, generation}: the element of that ref is gone; take a new snapshot), not_found ({kind, generation}), timeout ({kind, elapsed_ms}).

Methods

All answer once the frames they need have run; input methods return {} after the injected input has been processed by a frame.

The text outline of ui.snapshot, one element per line, two spaces of indent per level:

- button "New Note" [ref=e5] [id=notes.add]
- list [ref=e8] [id=notes.list]
  - listitem "Groceries" [ref=e9] [id=notes.list.row.0] [selected]
- textbox "Title" [ref=e14] [id=notes.title] [value="Groceries"] [focused]

Flags: [disabled] [focused] [selected] [checked]│[unchecked] [offscreen]; names and values are cut at 80 characters. interactive lists only visible controls; compact leaves out unnamed non-interactive and off-screen elements. Both leave out unnamed generic elements (drag areas, click catchers) that have no automation id.

Waits are checked on frames the app runs anyway plus one wake at the deadline: a pending wait never makes the app repaint in a loop. idle means a frame asked for no repaint within 200 ms, no injected input is pending and the app reports no work of its own (devtools::set_busy; the runtime reports its background jobs). Screenshot annotations[].rect is in image pixels relative to the returned image; annotate and highlight are painted into the captured frame. Screenshots are rendered off screen when the app provides a renderer (devtools::set_offscreen; ashlar-app uses eframe’s wgpu device), so they work for a window that is covered, in the background, minimized behind others or on a locked screen; ashlar-app also runs the UI passes other ui.* requests need while such a window draws none (Runtime::hidden_pass). Otherwise the backend captures a presented frame: the request is repeated every 0.25 s, the window is brought to the front once (unless raise: false), and after 1 s without a frame it fails with an error saying why. Either way it never hangs and never spins: a failed screenshot costs a bounded handful of frames. Pictures travel inline, never through a predictable temp file.

The inspector

F12 (or devtools.toggle) opens an inspector window in the app: Pick (hover outlines an element with its role, name, ref, id and rect; a click selects it and copies @eN), the element tree, frame and wake counters with a live repaint counter, and panels the app provides (devtools::section: the runtime adds its recent logs and the undo history). The inspector is not part of snapshots. Closed, it costs nothing: no frames, no allocations.

MCP tools

ashlar-mcp exposes the same vocabulary as tools. Arguments marked ? are optional; a target is a ref string ("@e3", "e3"), "id=…", "text=…", or an object as in the debugger methods.

screenshot, the ui_* tools (ui_clock and ui_scene included) and app_stats/app_logs are listed only when the backend’s app.version reports their method, so an agent never sees a tool the app cannot serve. The documentation tools are listed when the server has a library or a user folder. With ToolMode::PerCommand every command is also its own tool (dots become _) with its real input schema. crates/mcp/tests/protocol_doc.rs checks this table against the code.

CLI reply contract

<tool> exec <id> [json], describe, commands --json, run, version --json print one line: {"ok":true,"result":…} or {"ok":false,"error":"…"} (plus detail). Exit codes: 0 ok, 1 the command failed, 2 usage error.