` is an optional docked tree (`root`) plus a list of floating groups, back to front. The tree has three node kinds: - `Node::Tabs { panels, active }`: a tab group, its panels in order and the index of the one shown. - `Node::Split { axis, size, first, second }`: two children side by side (`Horizontal`) or stacked (`Vertical`). `size` is a `SplitSize`: a ratio of the space, a fixed first size, or a fixed second size. - `Node::Stack { entries }`: an accordion. Each `StackEntry` has a panel, an `open` flag and an optional preferred body height. A `Floating

` is a tab group with a rect `[x, y, width, height]`. `P` is the app's panel id. It must be stable across sessions and languages (an enum or a string key, never a translated title) and implement `Clone + Eq + Hash`; the renderer also needs `Debug + Send + Sync + 'static`. A panel appears at most once. Per-document tabs (one tab per open file, say) belong inside a panel, not in the layout. ```rust use ashlar_ui::docking::{Action, DockError, Floating, Layout, Node, Placement}; #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] enum Panel { Outline, Inspector, History } fn arrange() -> Result, DockError> { let mut layout = Layout { root: Some(Node::Tabs { panels: vec![Panel::Outline, Panel::Inspector], active: 0 }), floating: vec![Floating { panels: vec![Panel::History], active: 0, rect: [40.0, 60.0, 260.0, 300.0] }], }; // Dock History into the main group, in front of Inspector. layout.apply(Action::Move { panel: Panel::History, anchor: Panel::Outline, placement: Placement::Tab { before: Some(Panel::Inspector) }, })?; Ok(layout) } ``` ## Transactions Every change is an `Action`, applied with `Layout::apply` or `Layout::apply_with_permissions`. A transaction checks the current layout, makes the change on a copy, checks the copy, and only then replaces the layout. When it fails, the layout is exactly as before and a `DockError` says why: a missing panel or target, a panel moved onto itself, a duplicate id, an active index out of range, too deep or too many (`MAX_DEPTH`, `MAX_PANELS`), or geometry that is not finite or out of bounds. What a successful move guarantees: - the moved panel is never lost on the way; - each group keeps its selected panel by identity, not by index; - groups left empty disappear, and a split left with one child is replaced by that child; - an accordion entry keeps its open state and height when it moves to another accordion; - sizes of splits the move did not touch stay as they were. `Open` adds a panel that is not in the layout yet (or activates it if it is). `OpenAt` places a new panel at an exact spot, and moves it there if it already exists. Whether a panel id is known to the app at all is for the app's panel registry to check before it asks. ## Permissions and fixed panels `apply_with_permissions` takes a function from panel to `Permissions` (`movable`, `floatable`, `closable`, `accepts_tabs`). A document canvas is typically `Permissions::PROTECTED`: it cannot be moved, closed or floated, and nothing can join it as a tab, but other panels can still be split off beside it. A panel that must never leave the docked tree is the app's rule to enforce. An empty root (`root: None`, or a `Tabs` node with no panels) is valid, for example when every panel floats. ## Rendering Apps that draw their own chrome can describe their groups as `DropTarget`s and call `resolve_drop` with the pointer. Targets are listed back to front, floating groups in their current stacking order after the docked ones. The frontmost group under the pointer decides alone: if it refuses the drop, groups behind it are not considered. `drop_zone` and `zone_rect` give the zone rules on their own. `DockArea` is the optional built-in renderer. It fits tabs into the strip and moves the rest into an overflow menu, previews drops, cancels drags on Escape or when the window loses focus, and draws split and accordion dividers and the floating groups' resize corners. It never changes the layout: `DockOutput::actions` lists what the user asked for, in order, for the app to send through its own command path. With the `serde` feature `Action` serializes, so pointer gestures, keyboard commands and remote control can all go through the same path. Panel bodies get an egui id derived from the dock area and the panel only, so their widget state follows a panel when it moves; durable data should still live in the app, not in egui memory. `DockStyle` holds the colours, font, metrics and the localized labels of the stock controls (`DockStyle::from_ui` gives defaults). `show_with_limits` takes per-panel `PanelLimits`. The renderer fits the view into the viewport: when there is not enough room, minimums shrink in proportion, and the preferred sizes stored in the layout are never overwritten by that. Divider gestures produce explicit `ResizeSplit` or `ResizeStack` actions. A custom header that starts dragging a panel can hand the gesture to the renderer with `begin_drag`, using the same area id. Floating groups are egui areas inside one viewport. Native windows, moving panels between viewports, and monitor coordinates are for the host to build. ## Persistence The `serde` feature derives serialization for the model; apps can also write their own adapter. The library picks no file location. Before trusting a loaded layout, validate panel ids against the app's registry, cap the payload size, version the schema, and bound the depth of the deserializer: `validate` rejects deep trees, but it cannot protect a parser that has already recursed without limit. ## Hiding and restoring panels To hide a panel, capture `layout.location(&panel)`, apply `Close`, and keep the `Location` alongside the app's own visibility state. `restore(panel, &location)` puts it back using its neighbours' ids, fixed split sizes, the floating rect, and recreates a closed accordion entry. If the neighbour is gone, the layout is left unchanged and `MissingTarget` is returned; the app then picks a fallback such as `Open`. Panels hidden from the same group come back correctly when restored in reverse order of closing. ## What the tests cover The module's unit tests cover moves, floats and merges, rejected transactions, selection by identity, reordering, accordion state, permissions, location and restore for splits, floating groups and accordions, validation limits, drop resolution, 800 random moves, serde round trips, and the renderer's gestures: drags with Escape and focus loss, size limits, the resize corner, accordion dividers, custom content, blockers, `cancel_drag` ownership, `chrome_has_focus`, and accessible names of the stock menu actions. `crates/ui/tests/docking_workflows.rs` drives a document canvas and three populated panels with real pointer and keyboard input: it edits a value, merges a floating panel into a tab group, detaches it again and redocks it, at two widths, two scales and both themes, round-tripping every action through JSON. Set `ASHLAR_UI_DOCK_VISUAL_DIR` to an absolute path to keep a PNG per stage: ```sh ASHLAR_UI_DOCK_VISUAL_DIR=/abs/path cargo test -p ashlar-ui --all-features --test docking_workflows ``` These checks run offscreen. They do not replace an app's own tests of its panels and commands. ## Custom chrome and content `show_customized` takes one `DockContent` adapter instead of closures: `body` draws a panel, `panel_menu` adds entries at the top of a tab's context menu, and `group_header_width` / `group_header` reserve and draw a strip at the right end of a group's tab bar (clipped to it). Queue commands from these callbacks rather than changing the layout inside them. Collapsed accordion entries still have their context menu. Collapsing whole tab groups is a host policy. `label` returns `WidgetText`, so a tab can show a shaped galley (icons, mixed styles); the same text is used to measure the tab, to paint it and in menus. The plain string from the `label` closure stays the accessible name. `action_label` shapes the text of the stock Float, Move and Close entries; it gets no panel id. Visible text never replaces an accessible name. `show` and `show_with_limits` remain for apps without an adapter. ## Activation and focus To bring a panel forward, apply `Activate` and pass `raise_panel(panel)` on the next frame: a floating group is raised and the panel's tab takes keyboard focus. Stacking order is not stored in the layout. A floating group is identified by its first panel. The 12-point grip at the left of a floating group's tab strip moves the whole group with `MoveFloating`; dragging a tab detaches just that panel. `DockOutput::group_drag` and `group_drop` report which group is being dragged and where it was released, for hosts that turn a drop outside the window into a native window, say. Escape or focus loss puts the group back where it started. `blocked_rects` marks screen areas (menus, dialogs, overlays) where previews, drops and the float fallback are blocked, including overlays that appear in the middle of a drag. Hosts pass the current overlay geometry every frame, or call `cancel_drag` while something modal is up. `DockArea::chrome_has_focus` says whether keyboard focus is on the dock's own controls: tabs, collapse and overflow buttons, dividers, resize corners and grips. Panel content and custom header or menu widgets do not count, and an invalid layout gives `false`. A host can use it to let global shortcuts through while Enter, Space, Tab, Escape and the arrow keys stay with the focused control. # Visual regressions `crates/ui/tests/visual_regression.rs` renders real components and independent reference painters with the same offscreen wgpu renderer and compares the results. The reference painters never call the components: they draw literal geometry with plain egui painter calls. Together they define the visual contract of each control; they are not screenshots of an app. ## Coverage | Suite | Pairs | What is compared | |---|---|---| | Controls | 372 | icon button, single-shape button, tab, horizontal and vertical split × light/dark × scales 1, 1.5, 2 × normal and compact sizes × the states normal, hover, press, disabled, focus, plus selected and selected-press for buttons and tabs | | Controls and content kit | 282 | `controls::Button` (primary, secondary, ghost, destructive), `Switch`, a `RadioGroup` option, `Progress::bar` (25 %, 100 %, the reduced-motion indeterminate pose), the `content::Collapsible` header and a dismissible `content::Banner` (info, warning, danger) × light/dark × scales 1, 1.5, 2 × the states normal, hover, press, disabled, focus, plus selected (on, chosen, open) and selected-press for the switch, radio and collapsible; drawn with the built-in token sets and no labels | | Toolbar overflow | 18 | three widths × two themes × three scales against literal rects | | Widget palette | 12 | cells painted from `WidgetPalette::apply_to` against literal painting | | SVG icons | 12 | catalog output rasterized by the SVG loader against literal, pre-substituted SVGs | All 696 positive pairs must give identical RGBA pixels. The 684 pairs rendered with wgpu must also give identical meshes: vertices (position, UV, colour), indices, texture ids and clip rects. The SVG pairs compare the loader's rasters and must contain visible pixels. Each state is reached with real input, and the test asserts it was reached: the control is hovered, held down (and dragging, for split handles), or focused. Disabled controls must stay unhovered and unfocused under the same input. Selection wins over press. Split handles are driven in both orientations. The compact fixtures repeat every contract at a smaller size. 48 negative comparisons change one thing in the reference (fill colour, spacing, stroke width; the fill of a kit button, switch, radio, bar and banner, the collapsible's chevron) and must differ in both pixels and meshes, which shows the comparison can fail. There is no tolerance, mask, crop, perceptual metric or automatic update. The toolbar fitting itself (malformed input, reachable overflow, groups kept in order) is covered by unit tests in the `layout` module. ## Running ```sh cargo test -p ashlar-ui --test visual_regression ``` With `ASHLAR_UI_VISUAL_DIR=/absolute/path`, each suite writes into its own subdirectory: `-before.png` (reference), `-after.png` (component), `-difference.png` (changed pixels in magenta, 255,0,128) and `comparison.tsv` with the columns `case`, `width`, `height`, `changed_pixels`, `mesh_equal` and `expected_change`. Rows are appended as each pair finishes, so the file is useful even when a later pair fails. Images are at most 256×144 pixels. A renderer failure fails the test. ## Limits The comparison is same-machine only: both sides go through the same driver in the same run, so it says nothing about pixel identity across GPUs or drivers. It does not replace app screenshots, complete themes, text layout checks, command dispatch, window resizing, tests against real accessibility services, or cross-platform runtime tests. Apps that keep their own baseline images keep their own record of where those came from. # ashlar-trace: telemetry Crash, performance and usage telemetry for Rust desktop apps, with a backend you host yourself. - **`ashlar-trace`** — a small client you embed. Safe Rust only, no async runtime, no UI toolkit. It never blocks your frame loop and never prints. - **`ashlar-trace-server`** — one binary and one data folder: ingest API, SQLite storage, and a dashboard for crashes, performance and usage. - **`ashlar-trace-proto`** — the wire types both share. > Status: release candidate, load-tested; not yet published. See [roadmap.md](roadmap.md). Apps built on ashlar use it through `ashlar-telemetry`, which wires it to the command chokepoint, logging, the frame loop and native dialogs. ## What it collects | Signal | What you get | |---|---| | Crashes | Panics with backtrace, handled errors, unclean exits found on next launch, abnormal child-process exits, each with recent log lines and device details. | | Hangs | A report when the main loop stops responding. | | Performance | Frame-time histogram, slow and frozen frames, startup time, process CPU and memory, plus your own counters and gauges. | | Usage | Sessions, daily opens, focused / active / idle time, and the events you declare. | The host app decides what is tracked. Events are declared as enums and checked at build time; anything not declared and listened to is never recorded. End users can switch each category off, and `DO_NOT_TRACK=1` is honoured. [docs/privacy.md](privacy.md) lists every field. ## Sketch ```rust ashlar_trace::events! { pub enum Category { File = "file", Edit = "edit" } pub enum AppEvent { FileOpen = "file.open" in File, Undo = "edit.undo" in Edit, } } let tracer = ashlar_trace::Tracer::builder("app-write-key", "my-app", env!("CARGO_PKG_VERSION")) .data_dir(my_data_dir) .endpoint("https://telemetry.example.com") .listen(Category::File) .init(); // Nothing is sent until the user has seen your first-run notice. tracer.notice_shown(); // Once per turn of the main loop: let _turn = tracer.turn(); // a turn that takes too long is reported as a hang tracer.track(AppEvent::FileOpen); tracer.frame(frame_duration); // once per frame ``` ## Documentation - [Integrating the SDK into a host app](integration.md) - [Self-hosting the server](self-hosting.md) - [Architecture](architecture.md) - [Wire format](wire-format.md) - [Privacy and collected fields](privacy.md) ## Development The crates are part of the ashlar workspace and its gate (`cargo xtask ci`). Heavier runs: `cargo xtask trace-stress` (the SDK's benchmarks and stress tests, release build), `cargo xtask trace-load` (a short load profile against a spawned release server) and `cargo xtask trace-demo` (checks the windowed demo, which is its own workspace). # Integrating the SDK into a host app This guide walks through adding `ashlar-trace` to an application. Every snippet is taken from, or was checked against, one of the two demo hosts in this repository: - [`examples/trace-demo-cli`](../../examples/trace-demo-cli): a scriptable command-line process. It is the one the end-to-end tests drive, so everything it does is tested against a real server. - [`examples/trace-demo-gui`](../../examples/trace-demo-gui): a small `eframe`/`egui` window that wires the SDK the way a real GUI host would. It compiles and passes clippy (`cargo xtask trace-demo`); it has not been run, because it needs a display. What the SDK may collect is listed in [privacy.md](privacy.md). How the pieces fit together is in [architecture.md](architecture.md). ## 1. Add the dependency ```toml [dependencies] ashlar-trace = "0.2" ``` Default features are `http` (a blocking HTTPS client on the SDK's own thread) and `sysinfo` (process CPU and memory, device details). Without `http`, nothing leaves the machine unless you supply a `Transport` through `Builder::transport`. Without `sysinfo`, the CPU, memory and device fields are simply absent. The SDK needs Rust 1.90, has no async runtime, forbids `unsafe`, and never writes to stdout. ## 2. Declare the catalog Events are declared once, as two enums. A name that is not lower-case `snake_case` (dots allowed, at most 64 bytes), or that is used twice, fails the build. ```rust ashlar_trace::events! { pub enum Category { File = "file", Edit = "edit", View = "view", } pub enum DemoEvent { FileOpen = "file.open" in File, FileSave = "file.save" in File, EditUndo = "edit.undo" in Edit, ViewZoom = "view.zoom" in View, } } ``` Only what you declare and listen to is ever recorded. `Tracer::track` takes the enum, so a misspelled event does not compile. ## 3. Build the tracer Do it once, early in `main`, and keep the returned `Tracer` alive for the whole process. It is cheap to clone and safe to share across threads. Dropping the last clone ends the session and sends what is waiting. ```rust let mut builder = Tracer::builder(&key, "demo", env!("CARGO_PKG_VERSION")) .data_dir(&data_dir) .listen_all::() .logger(LoggerMode::Install) .process(ProcessKind::Gui); if let Some(endpoint) = &endpoint { builder = builder.endpoint(endpoint); } let tracer = builder.init(); ``` What each setting means: - **Key, name, version.** `Tracer::builder(write_key, app_name, app_version)`. The write key comes from `ashlar-trace-server app create` (see [self-hosting.md](self-hosting.md)). It is not a secret: it ships inside the app and only allows ingest. - **`data_dir`.** The host's own data directory. The SDK keeps its files in a `tracer` folder inside it (see section 3.2 of the architecture). Each host picks its own location, so this is required. If it is empty or the folder cannot be created, `init` returns a disabled tracer rather than failing. - **`endpoint`.** The server's base URL, such as `https://telemetry.example.com`, with no path. Without one the SDK records into its spool and sends nothing. - **`build`.** An optional build identifier such as a commit hash, shown next to the version. - **`process`.** `Gui` (the default), `Cli`, `Service` or `Child`. The dashboard counts "daily opens" from `Gui` sessions only, so give tools and helper processes their own kind. - **`listen`.** `listen(category)`, `listen_events(&[...])` and `listen_all::()` choose what is recorded in this run. An event that is not listened to is dropped before it reaches the queue. Each of them also tells the server what the app declares (so it can show events that were never fired and set apart names that were never declared), so they can come in any order and nothing else is needed. `catalog::()` exists for the rare build that declares events without listening to any. - **`send_before_notice`.** By default nothing is sent until you call `notice_shown()` (section 9). Set this to `true` only if the user sees your notice somewhere else, such as in an installer. - **`gpu`.** Graphics adapter details are only known to the host. If you have them, pass `Gpu { name, vendor, backend, driver }` to `Builder::gpu`. They go into the device details on every envelope. - **`signals` / `without`.** All built-in signals are on by default; `without(Signal::Hang)` switches one off. - **`configure`.** Adjusts limits and intervals (`flush_interval`, `heartbeat_interval`, `perf_window`, `hang_threshold`, `spool_max_bytes` and so on). The demo-cli uses it to shrink every interval to milliseconds for tests: ```rust builder = builder.configure(|config| { config.tick_interval = Duration::from_millis(20); config.heartbeat_interval = Duration::from_millis(100); config.hang_threshold = Duration::from_millis(400); }); ``` ### GPU details In the GUI demo the tracer is built inside eframe's app-creator callback, because that is where the toolkit's creation context exists. With the `wgpu` renderer the adapter is in `cc.wgpu_render_state` (`adapter.get_info()` gives the name, vendor and backend). With the `glow` renderer used by the demo, the renderer string can only be read through an `unsafe` GL call, which the demo's workspace forbids, so `demo-gui` passes no GPU. The SDK does not need one. ## 4. Feed it from the main loop The performance and hang signals need a few calls from the loop that draws the UI. All of them are an atomic operation or a short lock; none blocks on disk or network. | Call | Where | Why | |---|---|---| | `turn()`, or `begin_turn()` / `end_turn()` | Around each piece of work the main loop does | Hang watchdog: a turn that takes longer than `hang_threshold` is a hang | | `frame(duration)` | Around one UI pass | Frame-time histogram, slow and frozen frames | | `first_frame()` | After the first frame is on screen | Startup time since `init` (recorded once) | | `set_focused(bool)` | When the window gains or loses focus, or every turn | Focused versus unfocused time | | `note_input()` | When the user does something | Active versus idle time (idle after 120 s of no input by default) | ### Turns and the hang watchdog A hang is the main loop being stuck in one piece of work. So tell the SDK when a piece of work starts and ends: ```rust let _turn = tracer.turn(); // ends when the guard is dropped handle_the_event(); ``` A turn that stays open longer than `hang_threshold` (2 s by default) is reported once, with how long it has been blocked. The time between turns, while the loop waits for the next event, is never a hang, however long it lasts. That makes this work for hosts that repaint only on input, and for hosts whose event loop belongs to a GUI framework: you do not need to see the wait, only your own callbacks. No timed repaint or wake-up is needed. `begin_turn()` and `end_turn()` are the same thing without the guard. Each is one atomic update. A second `begin_turn` while a turn is open keeps the original start, and `end_turn` without an open turn does nothing, so nesting and early returns are harmless. ### An immediate-mode GUI This is the wiring from `demo-gui`. eframe owns the event loop and runs it only on input. It calls `logic` once per loop turn (including turns where nothing is drawn) and `ui` for each frame, and each is marked as a turn: ```rust impl eframe::App for DemoApp { fn logic(&mut self, ctx: &eframe::egui::Context, _frame: &mut eframe::Frame) { let _turn = self.tracer.turn(); let (focused, active) = ctx.input(|input| (input.focused, !input.events.is_empty())); self.tracer.set_focused(focused); if active { self.tracer.note_input(); } } fn ui(&mut self, ui: &mut eframe::egui::Ui, _frame: &mut eframe::Frame) { let _turn = self.tracer.turn(); let started = Instant::now(); self.contents(ui); self.tracer.frame(started.elapsed()); if !self.first_frame_done { self.first_frame_done = true; self.tracer.first_frame(); } } } ``` `frame` measures the cost of producing a frame (here: building the UI), not the time between frames. That is what matters for hosts that repaint only on input. ### A loop that runs continuously A game-style loop that never waits can call `heartbeat()` once per iteration instead: a last beat older than the threshold is a hang. Do not use it for a loop that waits for input, because every wait longer than the threshold would look like a hang. Once a host has begun a turn, heartbeats no longer count, so the two are not mixed. Hosts that call neither get no hang reports at all, never false ones. `.without(Signal::Hang)` switches the watchdog off. ### Pausing hang detection When the main loop is legitimately blocked, for example by a native modal dialog, pause detection around it. Pauses nest, and resuming restarts the measurement, so the paused time is not a hang: ```rust self.tracer.pause_hang_detection(); std::thread::sleep(Duration::from_secs(3)); self.tracer.resume_hang_detection(); ``` The GUI demo has two buttons that block the UI thread for 3 s: one leaves detection on (and produces a `hang` issue with `blocked_ms` of about the threshold or more), one pauses it (and produces nothing). ## 5. Track usage `track` records a declared event. `track_with` adds up to 16 properties, each a bool, integer, float or `&'static str` of at most 64 bytes. There is deliberately no way to pass a runtime string, so file names and typed text cannot get in by accident. ```rust tracer.track(DemoEvent::FileOpen); tracer.track_with( DemoEvent::FileSave, &[("count", Prop::Int(3)), ("format", Prop::Str("native")), ("ratio", Prop::Float(0.5)), ("recent", Prop::Bool(true))], ); ``` ### A central command dispatcher Hosts that run every user action through one function keyed by a command id can record the id there, in one place: ```rust tracer.track_id(Category::Edit, "edit.palette_command"); ``` `track_id` records an id that is not in the catalog, marked `dynamic`, under a category you listen to. Rules: - Pass ids only, and only fixed program strings (`&'static str`). Never pass parameters, arguments, paths or anything the user typed. - The id must follow the same naming rule as event names. An invalid one is dropped and counted. - Dynamic ids are exempt from the server's catalog check, so a command registered at run time does not have to be in the enum. Declare an event in the catalog when you want it to appear in the "declared but never fired" list; use `track_id` for the long tail. ## 6. Logs, handled errors and child processes ### Log lines as breadcrumbs The SDK keeps the last 100 log lines at `Info` and above, scrubbed, and attaches them to a crash. They leave the device only inside a crash report. - **The host has no logger.** `.logger(LoggerMode::Install)` installs the SDK's logger globally. This is what `demo-cli` does by default. - **The host already installs its own logger.** Wrap it and install the wrapper in its place. Nothing is lost: every record still reaches the wrapped logger. This is `demo-cli --wrap-logger`, tested end to end: ```rust let tracer = builder.init(); // LoggerMode::None, the default let wrapped = tracer.logger(Some(Box::new(MyLogger))); if log::set_boxed_logger(wrapped).is_ok() { log::set_max_level(log::LevelFilter::Info); } ``` Only one global logger can exist, so the host must not also install its own. Set the maximum level yourself: `log` ignores every record until you do. With a disabled tracer, `Tracer::logger(Some(inner))` simply returns `inner`. ### Handled errors For an error you caught and survived. `context` is a fixed label such as a command id, never the message's variable parts: ```rust let failure = std::io::Error::other("could not write the preview"); tracer.report_handled(&failure, Some("preview.write")); ``` It becomes an issue of kind `handled`, counts towards the session's `errors`, and carries the breadcrumbs. The message is scrubbed (home directory, account name) and truncated, but it is still text your code wrote: do not put user content in error messages. ### Child processes If the host spawns helpers that can abort, report how they ended. The SDK records it only when the status is not success: ```rust let status = command.status()?; tracer.report_child_exit("demo-child", status); ``` `program` is a fixed label. The report carries the exit code or the signal (on Unix). A child that links the SDK itself, with `ProcessKind::Child` and the same data directory, also gets its own session. If it dies without ending it, the next process that starts reports an `unclean_exit` for it. ## 7. Counters and gauges on a realtime thread Register once, on a normal thread (registration locks and allocates). Record from the realtime thread: each call is one atomic operation, with no lock, allocation or system call. ```rust let saves = tracer.counter("saves"); // a Counter, Arc-backed and cheap to clone saves.add(7); let layers = tracer.gauge("layers"); // a Gauge layers.set(4.0); ``` Counters are reported per performance window as the increase since the previous window (zero increases are left out). A gauge is reported when it was set during the window. There is a cap on how many one process can register; handles over the cap work but are never reported. On a disabled tracer, both return detached handles that do nothing. ## 8. Short-lived tools A command-line tool that runs for a moment never reaches the periodic send, and one that ends with `std::process::exit` never drops its `Tracer`, so dropping cannot end its session. Use `ProcessKind::Cli` and end the session yourself on every exit path: ```rust let clean = tracer.shutdown_within(Duration::from_millis(25)); std::process::exit(code); ``` `shutdown_within` does what dropping the last handle does: it closes the session as `exited`, on disk, before it waits; then it gives the worker at most the time you pass for one last send. Later calls, and the final drop, do nothing more, and nothing is recorded after it. It returns true when nothing is left that could be sent. `shutdown` is the same with the configured `shutdown_timeout`. What could not be sent is kept (within the spool caps) and goes out with the next process of the same host. `flush_blocking` still works for a tool that wants to send before it ends. `ProcessKind::Cli` also leaves out what only makes sense for an app with a main loop: the hang watchdog, performance windows (frames) and CPU and memory sampling (`CLI_OFF`), whatever `signals` says. And it writes events and signal reports to the spool on the calling thread (a file append) instead of queueing them for the worker: a tool may exit while the worker waits on a slow request, and what only the queue held would be lost with it. A send costs at least one round trip to the server, often several (a new connection each run), which is far more than a tool that agents call many times in a row should add. Keep the wait at exit short and hand what is left to a **courier**: a background process, started from the same program, built with `Builder::courier(true)`. A courier only delivers what other processes of the host left in the spool, reporting the ones that ended uncleanly as any process does; it opens no session of its own (no start, end, heartbeat or marker) and starts no panic hook, watchdog or logger. `ashlar-telemetry` packages this (`Telemetry::end`, `Telemetry::hand_off`, `serve_courier`). A process that records neither sessions nor crashes leaves no session marker, so however it ends it is never reported as an unclean exit. A stdio protocol server can use the SDK freely: nothing it does touches stdout. ## 9. Consent, the notice, and "forget me" Recording is on by default (opt-out), with three independent categories: `Crash`, `Performance` and `Usage`. **Nothing is sent until the user has seen your first-run notice.** The SDK records from the first moment and keeps the data on disk, but it makes no request until you call `notice_shown()` once. That call is remembered, so later runs send from the start. The host owns the user interface: ```rust if self.tracer.notice_pending() { // Show your first-run notice, with the three switches if you like. When the user closes it: self.tracer.notice_shown(); } // Settings page: one switch per category. let mut allowed = self.tracer.consent(ConsentCategory::Usage); if ui.checkbox(&mut allowed, "Usage").changed() { self.tracer.set_consent(ConsentCategory::Usage, allowed); } // "Forget me". self.tracer.forget(); ``` - If you never call `notice_shown()`, no data ever arrives. The SDK says so once through the `log` facade (a warning with target `ashlar_trace`), and `flush_blocking` returns false while data is waiting for the notice. A tool without a user interface of its own that shares the data directory with your app needs nothing: it sends once the app has shown the notice. - A category the user switches off before closing the notice is deleted from disk and never sent. - If your notice is shown elsewhere, for example by an installer, build with `.send_before_notice(true)`. - `set_consent(category, false)` takes effect at once and deletes that category's unsent items from the spool. The choice is saved and survives restarts. Other running processes of the same host pick it up within a tick. - `forget()` deletes everything stored locally, sends a request asking the server to delete the old install's data (retried on the next launch if the server is unreachable), and issues a new install id. - `DO_NOT_TRACK` set to anything but empty, `0` or `false`, or `TRACER_DISABLED` set the same way, turns everything off for that process: nothing is written to disk, no folder is created, no request is made, and the consent file is not changed. Document this for your users. - A host whose telemetry is not optional builds with `.force_consent(true)`: every category is allowed and the notice counts as shown from `init`, whatever an earlier run saved in `consent.json` (the file is left as it is), and `set_consent` and `notice_shown` do nothing. Nothing in the session is held back, the session start included. - Such a host can also build with `.honor_do_not_track(false)`: `DO_NOT_TRACK` is then ignored. `TRACER_DISABLED` is always honoured, so developers and CI can still switch the SDK off. If you do this, say so plainly in your own privacy text: users who set `DO_NOT_TRACK` expect it to work. ## 10. Inspect mode and the local report `.inspect(true)` mirrors every item that is accepted for sending into `outbox.jsonl` in the SDK's folder (capped in size), so a user can read exactly what leaves the machine. A "show what is sent" menu item can simply open that file. `write_report(path)` writes a plain-text report the user can attach to a bug report: app and version, device, session ids and totals, the last performance window and the recent log lines. It works without a server and without sending anything: ```rust match self.tracer.write_report(&path) { Ok(()) => format!("wrote {}", path.display()), Err(error) => format!("could not write the report: {error}"), } ``` ## 11. Tests and builds without telemetry `Tracer::disabled()` is a tracer whose every method does nothing and touches no files or threads. Pass it into code under test, or use it in builds that ship without telemetry: ```rust let tracer = Tracer::disabled(); assert!(!tracer.is_enabled()); tracer.track(DemoEvent::FileOpen); // nothing happens assert!(tracer.flush_blocking(Duration::from_millis(1))); // nothing to send assert_eq!(tracer.install_id(), ""); ``` `demo-cli` runs its whole `normal` scenario against a disabled tracer in a unit test. To test against a real server, run `ashlar-trace-server` in-process on `127.0.0.1:0` with a temporary data directory, as `examples/trace-demo-cli/tests/e2e.rs` does. Set `.configure(|c| ...)` intervals to milliseconds so tests need no long sleeps. ## 12. The web build On `wasm32` the SDK compiles but does nothing: `Builder::init` always returns a disabled tracer, so the same code path works unchanged and costs nothing. There is no data directory, no thread and no request on the web target. (`cargo xtask wasm` keeps `ashlar-trace-proto` and `ashlar-trace` building for it.) ## 13. Cost and limits Measured with `cargo xtask trace-stress` (release build, a 15-core laptop, a real `Tracer` on a temporary folder and a transport that accepts at once). Run it on your own hardware; the tables it prints are the source of these numbers. **Per call on a host thread, one thread** (median, 99th percentile): | Call | Median | p99 | Allocations | |---|---|---|---| | `track`, listened to | 35 ns | 42 ns | 0 | | `track_with`, 4 properties | 44 ns | 60 ns | 1 | | `track_id` | 38 ns | 42 ns | 0 | | `track`, not listened to | 10 ns | 15 ns | 0 | | `frame` | 5 ns | 10 ns | 0 | | `turn()` or `begin_turn` + `end_turn` | 25 ns | 38 ns | 0 | | `heartbeat`, `note_input` | 20 ns | 48 ns | 0 | | `set_focused`, `Counter::inc`, `Gauge::set` | 1–2 ns | 2 ns | 0 | | a log line kept as a breadcrumb | 88 ns | 187 ns | 4 | | a log line below the kept level | 3 ns | 7 ns | 0 | | `report_handled` | 8 µs | 26 µs | many; it is an error path | | anything on a disabled tracer | 1–2 ns | 2 ns | 0 | The allocation counts are from reading the code, not from a counting allocator (that needs `unsafe`, which this workspace forbids). **What a host thread can wait for.** Nothing that touches the disk or the network. `track` is an atomic check, a set lookup and a `try_send` into a queue that was allocated at startup; when the queue is full the event is dropped and counted with one atomic add. `frame`, the turn calls, `note_input`, counters and gauges are atomic operations only. Over ten million calls of each, with the worker flushing every 20 ms, the longest single call was a few milliseconds, and so was the longest gap in an empty loop that only reads the clock: on a desktop operating system the scheduler can take a thread away at any point, and no library can promise otherwise. What the SDK promises is that it adds no wait of its own. Calls that do touch the disk, briefly, on the calling thread: `init`, `set_consent`, `notice_shown`, `forget`, and the panic hook; in a `ProcessKind::Cli` process also `track`, `track_with`, `track_id` and `report_handled`. `flush_blocking`, `shutdown_within` and dropping the last `Tracer` wait for the worker, up to the timeout you give and `shutdown_timeout`. **With many threads at once** the calls stay lock-free but share memory, so they slow each other down: with 8 threads calling the same thing as fast as they can, `track` is about 55 ns, `frame` about 200 ns, a kept log line about 2 µs. Counters and gauges on a realtime thread are one atomic operation on memory nothing else writes to, if you give that thread its own counter. **Startup and shutdown.** `init` returns in under 1 ms, whether the folder is empty or holds 1,000 session folders left by earlier runs or a full 8 MiB spool; dealing with those is the worker's first job (about 0.4 s of its time for the 1,000 folders). Dropping the last `Tracer` takes about 1 ms when the server answers at once or refuses at once, as long as one request when the server is slow, and never longer than `shutdown_timeout` (3 s by default). **In the background.** An idle app costs about 0.02% of one core for the worker and the watchdog together. At 120 frames a second with 50 events a second the whole process, pacing loop included, used 0.3% of one core. The worker writes about 500,000 items a second to the spool. With the server unreachable for five minutes at that load, resident memory stayed flat and the spool stayed at its cap. **Limits**, all enforced, all counted in the `drops` item when they throw something away: | Limit | Default | When it is reached | |---|---|---| | Queue between host threads and the worker | 1,024 items (`queue_capacity`) | The new item is dropped: `queue_full` | | Spool on disk, all sessions together | 8 MiB (`spool_max_bytes`) | The oldest segments go: `spool_evicted` | | Age of spooled data | 30 days (`spool_max_age`) | Older segments go: `spool_evicted` | | One request | 1 MiB before compression | A larger backlog is sent as several requests | | One item | 256 KiB | Dropped: `item_too_large`; an unclean-exit report that big goes without its log lines | | Properties on one event | 16 | The rest are left out: `prop_dropped` | | Event, category and property names | 64 bytes, lower-case `snake_case`, dots allowed | Build error for declared names; `invalid_id` or `prop_dropped` at run time | | String property values | 64 bytes | Left out: `prop_dropped` | | Panic and error messages | 2 KiB | Cut, on a character boundary | | Backtraces | 32 KiB | Cut | | Log lines kept as breadcrumbs | 100 lines of 512 bytes (`breadcrumb_capacity`) | Oldest lines go; long lines are cut | | Counters and gauges | 256 names in all | Further names get a handle that records nothing | | Forget requests still owed | 16 | The oldest is given up | | Inspect mirror `outbox.jsonl` | 1 MiB | The older half is discarded | **What a power loss can cost.** Spooled items are not flushed to the disk one by one, so the last seconds before a power loss can be lost; a crash of the app alone loses nothing that was already written. A consent change or a "forget me" is made durable by the worker within a tick (one second by default). ## 14. A short checklist 1. Add the dependency; declare the catalog with `events!`. 2. Build the tracer once at startup, with data dir, endpoint, key, version, process kind. 3. Mark each turn of the main loop with `turn()`, call `frame` around each UI pass, `first_frame` once, and `set_focused` / `note_input` from your input state. 4. Call `track` / `track_id` where user actions are dispatched. 5. Install the SDK's logger or wrap yours. 6. Report handled errors and abnormal child exits. 7. Show the first-run notice and call `notice_shown()` when the user closes it (nothing is sent before that); add the three switches and a "forget me" action. 8. In short-lived tools, end with `shutdown_within` before every exit. 9. Update the host's own privacy text from [privacy.md](privacy.md). # Privacy and collected fields This page lists everything the SDK can send. If a field is not here, it is not collected. Change this page in the same commit as any change to what is collected. ## Principles - **The host app chooses.** It declares its events at build time and picks which to listen to at startup. Undeclared events cannot be recorded. - **Nothing is sent before the notice.** Telemetry is on by default with a first-run notice. Until the host app reports that the user has seen that notice, data is recorded on the device but nothing leaves it. A category the user switches off in the notice is deleted and never sent. (A host app that shows its notice elsewhere, such as in an installer, can lift this.) - **The user can switch it off.** Each category — crash reports, performance, usage — has its own switch. `DO_NOT_TRACK=1` or `TRACER_DISABLED=1` in the environment turns everything off and nothing is written or sent. A host can choose to ignore `DO_NOT_TRACK` (`honor_do_not_track(false)`) and to make telemetry not optional (`force_consent(true)`: no switches, no notice); it must then say so in its own privacy text. `TRACER_DISABLED` cannot be ignored. - **No user content.** Event names and property text are fixed strings compiled into the app. There is no API that accepts a file name, project name, document text or typed input. - **No stable identity.** The install id is random, is not derived from hardware, and can be reset. The server never stores the IP address. - **Visible.** With inspect mode on, everything that would be sent is also written to `outbox.jsonl` in the SDK's folder. - **Forgettable.** "Forget me" deletes local data, including recent log lines and the last crash held in memory, asks the server to delete everything for the install, and issues a new id. ## Fields | Field | Example | Category | |---|---|---| | App name, version, build id, process kind | `my-app`, `1.2.3`, `abc1234`, `gui` | all | | Install id | random 32 hex characters | all | | Session id | random 32 hex characters | all | | Operating system name and version, CPU architecture | `macos`, `26.0`, `aarch64` | all | | CPU model and core count, RAM size | `Example CPU`, `10`, `32768` MB | all | | Graphics adapter name, vendor, backend, driver version | supplied by the host app | all | | Locale | `en-US` | all | | Session start, end, how it ended | `exited`, `crashed`, `abnormal` | usage | | Session duration; focused, active and idle seconds | `62`, `50`, `40`, `22` | usage | | Declared event name and category | `file.open`, `file` | usage | | Event properties: booleans, numbers, fixed short strings | `format = "native"` | usage | | The list of declared events and which are listened to | | usage | | Frame-time histogram, slow and frozen frame counts | | performance | | Startup time | `850` ms | performance | | Process CPU percentage and memory use | `42.5`, `512` MB | performance | | Host-defined counters and gauges | `audio.xruns = 2` | performance | | Panic or error message, source location, thread name, backtrace | scrubbed, see below | crash | | Hang duration | `2400` ms | crash | | Child process label, exit code, signal | `exporter`, `134` | crash | | Recent log lines (level, target, message) — only attached to a crash | scrubbed, see below | crash | | Counts of data the client discarded | `queue_full = 4` | all | ## Scrubbing Applied on the device, before anything is queued or written, to messages, backtraces and log lines: - The user's home directory becomes `~`. - The user's account name is removed where it appears in a path. - Long values are truncated. Log lines and error messages are written by the host app, so the host is responsible for not logging user content. ## Retention The server keeps raw data for 90 days by default and daily totals indefinitely. Both are configurable. # Self-hosting the server `ashlar-trace-server` is one binary and one data folder. It serves the ingest API, stores everything in a SQLite file, computes daily rollups, and renders the dashboard. There is no other service to run. ## 1. Run it ### The binary ``` cargo build --release -p ashlar-trace-server ./target/release/ashlar-trace-server serve ``` With no settings it listens on `127.0.0.1:8090` and keeps its database in `./data`. It logs to stderr at `info` level and stops on ctrl-c, letting in-flight requests finish. ### The container `deploy/trace/Dockerfile` builds a small image (build from the repository root) and `deploy/trace/compose.yml` runs it with its data in a named volume: ``` docker compose -f deploy/trace/compose.yml up -d ``` The image tags (`rust:1-slim-bookworm`, `debian:bookworm-slim`) are left as they are so that security updates arrive on rebuild. If you need reproducible builds, pin the base images by digest (`FROM debian:bookworm-slim@sha256:...`) and update the digests on a schedule. The image runs as an unprivileged user (uid 10001), listens on `0.0.0.0:8090` inside the container, and keeps its data in `/data`. The compose file publishes the port on `127.0.0.1:8090` only, on purpose: put a TLS proxy in front of it (section 5) instead of exposing it directly. Check that it is up: ``` curl http://127.0.0.1:8090/v1/health ``` ## 2. Settings All settings are environment variables and all are optional. | Variable | Default | Meaning | |---|---|---| | `TRACER_BIND` | `127.0.0.1:8090` | Listen address | | `TRACER_DATA_DIR` | `./data` | Folder for the database | | `TRACER_RETENTION_DAYS` | `90` | Age at which raw rows are pruned | | `TRACER_RATE_PER_MIN` | `120` | Requests per minute per client address on the public routes (ingest, forget and health, unknown keys included); `0` switches the limit off. Dashboard sign-in has its own limit (10 a minute per address) that stays on | | `TRACER_KEY_RATE_PER_MIN` | `6000` | Ingest requests per minute for one write key over all addresses. A capacity guard, far above the per-address limit, so that one client cannot use up a key's allowance for everyone; `0` switches it off | | `TRACER_MAX_DB_MB` | `10240` | Ingest answers `507 Insufficient Storage` once the database is larger than this many MiB; clients keep their data and retry. The dashboard and sign-in keep working so you can fix it (lower the retention, then `prune`); `0` switches the ceiling off | | `TRACER_TRUST_PROXY` | unset | When `1`, take the client address from the last entry of the last `X-Forwarded-For` header line | | `TRACER_ADMIN_PASSWORD` | unset | Password for `admin create`. Read by that command only; the server never reads it | | `TRACER_SECURE_COOKIES` | on | Dashboard cookies are `Secure`, and the session cookie is named `__Host-tracer_session`. Set `0` to sign in over plain HTTP, for example on `localhost` without a proxy | A value that cannot be parsed stops the command with a message naming the variable. An empty value counts as unset. The repository's `.env.example` lists the same names with placeholders; copy it, fill it in, and keep the real file out of version control. With compose, `TRACER_RETENTION_DAYS`, `TRACER_RATE_PER_MIN`, `TRACER_KEY_RATE_PER_MIN`, `TRACER_MAX_DB_MB`, `TRACER_TRUST_PROXY` and `TRACER_SECURE_COOKIES` are passed through from the shell or from a `.env` file next to the compose file. The bind address and data folder are fixed by the image. ## 3. First setup: an app, a key and an admin Every command below works on the same data folder as `serve`. Run them with the same `TRACER_DATA_DIR`. They can run while the server is running. **Create an app.** The slug is lower-case letters, digits, `-` or `_`. The command prints the app's first write key: ``` ashlar-trace-server app create my-app "My App" created app my-app (My App) write key: tk_... ``` Put that key into the host app's `Tracer::builder`. Write keys are public by nature (they ship inside the app) and only allow sending data. More keys: ``` ashlar-trace-server key create my-app ashlar-trace-server key revoke tk_... # stops it working at once ashlar-trace-server app list ``` Revoking a key makes the server answer 401 to it. The SDK keeps its data and stops sending until the app restarts, so rotate keys by shipping a new one first, then revoking the old one when most installs have updated. **Create a dashboard login.** The password is read from `TRACER_ADMIN_PASSWORD` or, if that is unset, from one line of standard input. It must be at least 8 characters: ``` TRACER_ADMIN_PASSWORD='a long passphrase' ashlar-trace-server admin create alice ``` Then open the server's address in a browser and sign in. The dashboard has an overview, crashes, performance, usage, release reports (downloadable as HTML, Markdown or JSON) and a settings page for keys. **With compose**, the container's entrypoint is the binary, so: ``` docker compose -f deploy/trace/compose.yml exec tracer ashlar-trace-server app create my-app "My App" docker compose -f deploy/trace/compose.yml exec -e TRACER_ADMIN_PASSWORD='a long passphrase' tracer ashlar-trace-server admin create alice ``` ## 4. Point a host app at it In the host app: ```rust Tracer::builder("tk_...", "my-app", env!("CARGO_PKG_VERSION")).endpoint("https://telemetry.example.com") ``` The endpoint is the base URL with no path; the SDK appends `/v1/ingest` and `/v1/forget`. See [integration.md](integration.md) for everything else. ## 5. TLS and a reverse proxy The server speaks plain HTTP. Terminate TLS in front of it with any proxy. The SDK uses `https` and verifies certificates, so use a real certificate. Caddy, which obtains certificates itself: ``` telemetry.example.com { reverse_proxy 127.0.0.1:8090 } ``` nginx: ``` server { listen 443 ssl; server_name telemetry.example.com; # ssl_certificate and ssl_certificate_key as usual location / { proxy_pass http://127.0.0.1:8090; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } } ``` Then set: - `TRACER_TRUST_PROXY=1`, so the rate limiter sees each client's address instead of the proxy's. Without it every request appears to come from the proxy and shares one bucket, and busy installs will see 429. The server uses the **last** `X-Forwarded-For` entry, the one the proxy itself appended, so earlier entries a client forges are ignored. Only turn this on when a proxy you control is the only way to reach the server. - Leave `TRACER_SECURE_COOKIES` unset (cookies are `Secure` by default, which needs HTTPS). Only set it to `0` for plain-HTTP use on a local machine. The server never stores client addresses; it uses them for the rate limit and nothing else. Do not expose the port directly. Bind to `127.0.0.1` (the default) or to a private interface, and let the proxy be the public side. The server logs a warning at startup when it is bound beyond loopback without `TRACER_TRUST_PROXY=1`: behind a proxy that would make every client look like the proxy, so the per-address and sign-in limits would be shared by all of them. The server itself cuts off clients that take longer than 15 seconds to send request headers (this also closes idle keep-alive connections), and serves at most 1,024 connections at once, waiting for a slot beyond that. Request bodies must arrive within 30 seconds. You can still add stricter limits in the proxy (`client_header_timeout`, `limit_conn`, `client_max_body_size 1m` in nginx), but the server no longer depends on it for slow clients. ## 6. Backups Everything is in `TRACER_DATA_DIR`: the database `tracer.db`, and while the server runs, two companion files `tracer.db-wal` and `tracer.db-shm`. SQLite runs in WAL mode, so recent writes may live only in the `-wal` file. **Copying `tracer.db` alone while the server is running can give you a stale or inconsistent copy.** Use one of these: 1. **Ask SQLite for a snapshot.** Safe while the server runs: ``` sqlite3 /path/to/data/tracer.db ".backup '/backups/tracer-$(date +%F).db'" ``` or `VACUUM INTO '/backups/tracer-2026-01-01.db'` for a compacted copy. The result is a single, self-contained file. 2. **Stop the server, then copy the folder.** After a clean stop the `-wal` file is empty or gone, and copying the whole folder is fine. If you copy while stopped, copy all three files together. 3. **Snapshot the volume** (LVM, ZFS, a cloud disk snapshot). These capture the three files at one instant and are safe. For compose, option 3 or a stopped copy is simplest: ``` docker compose -f deploy/trace/compose.yml stop docker run --rm -v tracer_tracer-data:/data -v "$PWD":/backup debian:bookworm-slim tar czf /backup/tracer-data.tgz -C /data . docker compose -f deploy/trace/compose.yml start ``` (The volume is named `_tracer-data`; `docker volume ls` shows the real name.) Test a restore now and then: stop the server, put the files back into the data folder, start it. What is lost if you lose the database: history. Clients keep unsent data on their own disks and resend it, and the server ignores anything it has already stored, so a restore plus a few days of resends is not harmful. ## 7. Retention and housekeeping - **Raw rows** (events, performance windows, crashes, sessions) are deleted when they are older than `TRACER_RETENTION_DAYS` (default 90). The ids used to ignore duplicates are kept for 35 days. After pruning, SQLite reclaims the space. - **Daily totals** (opens, active installs, event counts, frame histograms, drops) are kept indefinitely, so long-term charts survive pruning. Lowering the retention period does not remove them. - **Timers.** `serve` recomputes the rollups for the last three days every five minutes and runs retention hourly. A session that is still open a day after its last item is closed as `abnormal`. - **By hand**, for example from cron on a machine that runs the binary directly: ``` ashlar-trace-server rollup --days 7 # recompute recent rollups now ashlar-trace-server prune # run retention once ``` ## 8. Upgrading 1. Take a backup (section 6). 2. Build or pull the new version and restart. Database migrations are applied at startup and tracked in the file (`PRAGMA user_version`). They only ever add; a shipped migration is never edited. 3. Check `/v1/health`, which reports the server version. Run one server process per data folder. Do not point an older binary at a database that a newer one has migrated. The wire format only grows, so older apps keep working against a newer server, and an app newer than the server may send fields the old server ignores. If a client sends a `schema` higher than the server understands it gets 422 and drops that batch, so upgrade the server before shipping an app that depends on a new schema. ## 9. Troubleshooting **The SDK stops sending, the server answers 401.** The key is missing, mistyped or revoked. The SDK keeps its data and stops sending until the host app restarts. Check the key on the dashboard's settings page, or create a new one (`key create`). The key header is `x-tracer-key`. **429 Too Many Requests.** Requests are limited per client address (`TRACER_RATE_PER_MIN`, default 120 a minute), and each write key also has a much larger overall allowance (`TRACER_KEY_RATE_PER_MIN`, default 6000). Behind a proxy, set `TRACER_TRUST_PROXY=1`, otherwise all clients share the proxy's address. The SDK honours `Retry-After` and keeps its data until the server accepts it, so an occasional 429 loses nothing. Sustained 429 usually means the proxy setting above, or a test loop. Raise the limit, or set `0` to switch it off. **507 Insufficient Storage.** The database passed `TRACER_MAX_DB_MB`. The SDK keeps its data and retries. Lower `TRACER_RETENTION_DAYS` and run `ashlar-trace-server prune`, or raise the ceiling. **400, 413 or 422.** The batch was malformed, over 1 MiB after decompression, or from a newer schema. The SDK deletes that batch and counts a `server_rejected` drop, shown on the dashboard. A proxy that rewrites or truncates bodies (or a body-size limit smaller than 1 MiB) causes this too. **Times look wrong, or events land on the wrong day.** The server compares the client's `sent_at` with its own clock. If they differ by more than five minutes, item times are shifted by the difference, and no time is accepted more than an hour in the future or older than the retention period. Make sure the *server's* clock is right (use NTP); the client's can be wrong without harm. **Nothing arrives.** 1. `curl https://your-host/v1/health` from outside. If that fails, it is the network, DNS, TLS or the proxy. 2. In the host app: is `endpoint` set (the SDK records locally and sends nothing without it), and is it the base URL with no path? 3. Is the user's `DO_NOT_TRACK` or `TRACER_DISABLED` set? Then nothing is written or sent, by design. Are all three consent categories switched off? 4. The SDK does not send at once: it sends when 50 items are waiting, every minute while items are waiting, and at the end of the session. A short run that was killed is sent by the next launch. 5. Look at the `tracer` folder inside the host's data directory. `spool/` holds what has not been delivered; `outbox.jsonl` (inspect mode) shows what would be sent. 6. The dashboard has a version filter and a date range; check them. Events that are not in the app's declared catalog are kept, but listed on the usage page under "Undeclared events" rather than among the declared ones. 7. Watch the server's stderr while the host app starts. **Sessions show up as "abnormal".** A process ended without closing its session (killed, crashed natively, power loss). The next launch of the app reports it. A session that never ends and never reports again is closed as `abnormal` after a day. **The container cannot write its data.** The image runs as uid 10001. A bind-mounted data folder must be writable by that user; the named volume in `compose.yml` already is. ## 10. Capacity and tuning Numbers below were measured with `examples/trace-loadgen` on one machine (15 cores, an NVMe disk, release build, client and server on the same host over loopback, other jobs running on the machine). Treat them as an order of magnitude for your own hardware and re-measure with the commands at the end. A "typical" envelope in the generator carries about 40 items (a session start with a catalog hash, heartbeats, 20 to 50 events, a performance window with a real histogram, now and then a crash with a 30-frame backtrace, a session end) and is about 1.2 KB gzipped. **Ingest.** | Load (64 connections) | Result | |---|---| | Flat out | about 4,300 envelopes a second (170,000 items a second), p50 13 ms, p99 about 100 ms | | Fixed 2,000 a second | p50 8 ms, p99 about 60 ms, no errors | | Fixed 500 a second | p50 7 ms, p99 about 14 ms | | 1,000 a second for 5 minutes | no errors; p50 steady at 9 ms; p99 drifts from 70 to 230 ms as the database grows to 2.7 GB; server memory 28 to 38 MiB | | 2,000 a second for 5 minutes | held for about 3 minutes (a 3 GB database), then latency climbs: see "When the database outgrows memory" | | 9,000 a second for 15 seconds (over twice the capacity) | the server answers `503` with `Retry-After: 1` to what it cannot take, keeps accepting the rest at its normal rate, answers `/v1/health` and the dashboard in under 5 ms, and is back to no errors within a second of the load dropping | Why it is fast: ingest requests are queued to one writer thread that stores up to 64 envelopes per transaction and commits once, a separate thread checkpoints the write-ahead log, statements are prepared once, and duplicate detection is one statement per item. The queue holds 256 envelopes; when it is full the server answers `503` and the SDK retries later, so memory stays bounded however hard clients push. **Durability.** The database runs in WAL mode with `synchronous = NORMAL`. A request is acknowledged only after its data is committed to the log, so stopping the process, even with `kill -9`, loses nothing that was acknowledged (a test does exactly this under load). What `NORMAL` does not do is `fsync` each commit, so a power failure or operating-system crash can lose the last moments of acknowledged data. The database is never left corrupt. If you cannot accept that, run the database on a volume with a battery-backed cache or a replicated disk. **The dashboard on a large database.** On a database with 1,000,000 sessions, 10,000,000 events, 2,000,000 performance windows, 100,000 crashes in 5,000 issues, 50 versions, 200,000 installs and 90 days (2.5 GB): | | Before | After | |---|---|---| | Slowest page, warm | 2.1 s (reports), 1.4 s (overview) | 18 ms | | Slowest page, first request after start | 2.2 s | 290 ms | | Rollup of 3 days (writer held) | 2.3 s | 3.5 s (a few ms of it holding the writer) | | Full 90-day recompute | 60 s | 64 s | Ingest at 300 envelopes a second while the 3-day rollup runs as another process: p99 58 ms against 47 ms without it. While retention deletes half the data it never stops ingest, but it does hurt the tail (p99 about 300 ms against about 40 ms). **When the database outgrows memory.** Each stored item touches several B-tree indexes in random order, so once the database is much larger than the memory the operating system can use for caching, every commit reads and writes random pages and throughput falls. On the test machine that started at roughly 3 GB at 2,000 envelopes a second. The remedies are the usual ones: more memory, a shorter `TRACER_RETENTION_DAYS`, and a fast local disk. Dropping the four per-install indexes delayed the problem a little but did not remove it, so they are kept. `TRACER_MAX_DB_MB` stops ingest with `507` before the disk fills. **Settings that matter for capacity.** `TRACER_RETENTION_DAYS` (database size is about 220 bytes per stored item including its indexes and dedup record: 12 million items took 2.7 GB); `TRACER_RATE_PER_MIN` (per client address) and `TRACER_KEY_RATE_PER_MIN` (per key, a capacity guard); `TRACER_MAX_DB_MB`. Retention runs hourly in small chunks (100 rows) that yield the writer, so the first run on a very large database can take a long while; that is harmless, and later runs only remove a day's worth. **Measuring your own server.** ``` cargo xtask trace-load # a short standard profile, prints a table cargo build --release -p ashlar-trace-server -p loadgen target/release/loadgen run --spawn --concurrency 64 --duration 30 # flat out target/release/loadgen run --spawn --rate 500 --duration 60 # fixed rate, open loop target/release/loadgen run --url http://host:8090 --key tk_... --db /data/tracer.db --rate 200 --duration 60 target/release/loadgen run --spawn --phases 1500:10,9000:15,1500:10 --concurrency 600 --probe # overload target/release/loadgen seed --db /tmp/big/tracer.db --scale 1.0 # 10M events, 1M sessions, ... (2 min, 2.5 GB) target/release/loadgen read --dir /tmp/big --mode rollup # rollup timings target/release/loadgen read --dir /tmp/big --mode pages # every page, cold and warm ``` The generator is open loop with `--rate`: requests leave on a clock and latency is measured from when they were due, so a slow server shows up as latency. It ends with a check that the number of items the server stored equals the number it acknowledged. Do not point it at a server you care about: it writes a lot. # Architecture This is the spec. Code follows it; when they disagree, fix one of them in the same change. ``` host app ──► ashlar-trace ──► bounded queue ──► worker thread ──► disk spool ──► HTTPS batch (gzip) │ ashlar-trace-server: ingest ► SQLite ► rollups ► dashboard + reports ``` ## 1. What the SDK must fit The host apps set these constraints: - Rust edition 2024, minimum Rust 1.90, `unsafe` forbidden, no `unwrap`/`expect`/`panic` outside tests. - Immediate-mode GUI with a per-frame callback. Some hosts repaint only on input, so frame *cost* matters more than frame *rate*. - The `log` facade only. A host may have no logger, or its own. Only one global logger can exist, so the SDK must be able to wrap one. - No async runtime in the GUI process. Background work is a plain thread; HTTP is blocking. - Each host also ships a short-lived command-line binary, a stdio protocol server (stdout must stay clean), a web build, and sometimes child processes that can abort. - One host has a realtime audio thread, where the SDK must never lock, allocate or make a system call. - Each host chooses its own data directory, so the SDK takes it as a parameter. ## 2. Crates | Crate | Role | May depend on | |---|---|---| | `ashlar-trace-proto` | Wire types, constants, name validation. No I/O. | `serde` | | `ashlar-trace` | The client. | `ashlar-trace-proto`, `serde_json`, `log`, `thiserror`, `flate2`, `getrandom`, optional `ureq`, optional `sysinfo` | | `ashlar-trace-server` | The backend binary. | `ashlar-trace-proto`, `axum`, `tokio`, `rusqlite`, `maud`, `argon2`, and small helpers | ## 3. SDK ### 3.1 Shape `client.rs` holds the public `Tracer` handle and `Builder`. Behind it are two halves that meet only at the `Sink` trait (`sink.rs`): - **`pipeline`** implements `Sink`. It owns consent, identity, the session, the queue, the spool, the worker thread and sending. - **`signals`** produces data: crash, hang, perf, sys, logs, realtime counters, scrubbing, the local report. It calls `Sink` and nothing else. The public API in `client.rs` and the `Sink` trait are frozen. Changing either needs a note in `PROGRESS.md`. `Builder::init` never fails. On the web target, with no data directory, or when the SDK's folder cannot be created, it returns a disabled tracer whose methods do nothing. ### 3.2 Files on disk Everything lives in `/tracer/`: | Path | Owner | Content | |---|---|---| | `install.json` | pipeline | `{"install_id": "…", "created": ms, "pending_forget": ["…"]}`. `pending_forget` lists earlier install ids whose forget request has not been delivered yet (at most 16); it is absent when empty. | | `install.lock` | pipeline | Empty. Locked for the moment it takes to change `install.json`. | | `consent.json` | pipeline | `{"crash": bool, "performance": bool, "usage": bool, "notice_shown": bool}` | | `selection.json` | pipeline | Catalog hash, listened categories and events, enabled signals for the last run. Also the catalog hashes the server has acknowledged, per app version. | | `spool//lock` | pipeline | Empty. The process that runs the session holds an exclusive advisory lock on it for its whole life. | | `spool//header.json` | pipeline | The envelope header for that session (app, install id, device), plus `created`, which orders sessions. | | `spool//session.json` | pipeline | Marker for the running session: envelope header, start time, time of the last rewrite, last heartbeat totals, error count, `panicked` flag. Removed on clean shutdown. | | `spool//.jsonl` | pipeline | Items in segments, one per line: a consent-category byte (`c`, `p`, `u`, or `-` for none) followed by the item as JSON. | | `spool//breadcrumbs.jsonl` | signals | Recent log lines of that session, rewritten periodically. | | `outbox.jsonl` | pipeline | Inspect mode only: a capped mirror of every item that was accepted for sending. | Files are written to a temporary name and renamed. A file that fails to parse is treated as absent, never as an error. **Nothing in the SDK's folder is trusted.** Every file is read as possibly damaged or planted: a state file that does not parse is treated as absent and replaced; a file is read only if it is a regular file of plausible size (state files up to 1 MiB, segments up to 8 MiB), so a symbolic link is never followed out of the folder and an oversized file is never loaded into memory; a session folder is only ever a real folder, never a link; a segment far larger than the SDK writes is deleted unread when its folder is adopted and counted as `spool_damaged`. If the SDK's folder, or the running session's folder, is deleted under a running session, the worker puts back the folder, lock, header, marker and the install and consent files on its next pass; what was recorded in between is lost uncounted. **Durability.** Nothing waits for the disk on a host thread. Spool segments are appended with one write per item and are never flushed with `fsync`: a crash of the process loses nothing that was written, but a power loss or a crash of the operating system can lose the last seconds of items, or leave a cut-off last line, which readers skip. `consent.json` and `install.json` are written by temporary file and rename on the calling thread and then flushed to the disk, with their folder, by the worker within a tick; until then a power loss can bring back the previous consent choice or install id, or an empty file, which reads as absent (consent then starts from the defaults). The marker and `selection.json` are not flushed. **Several processes, one directory.** A host's GUI, command-line tool and child processes may all run on the same data directory at once, so every session folder has exactly one owner: - A running process owns `spool//`. It creates the folder at `init`, takes the lock, and appends only there. The operating system releases the lock when the process ends, however it ends. - A folder whose lock is held belongs to a live process and is left completely alone by everyone else: not sent, not evicted, not purged, not deleted. - A folder whose lock can be taken is an *orphan*. The process that takes the lock adopts it and keeps the lock until the folder is gone, so no two processes work on the same orphan. Orphans are looked for at startup, before every send, and every `flush_interval`. - An orphan recorded under an install id other than the current one was forgotten by the user and is deleted unsent. A folder with neither header nor marker is deleted once it is a minute old (younger, it may belong to a process that is still starting). A scan never creates a lock file in a folder younger than that, since the folder may be one its owner is deleting at that moment. - `install.json`, `consent.json`, `selection.json` and `outbox.jsonl` are shared. Processes that start together on a fresh directory agree on the install id: the file is created with create-if-absent, and the losers read what the winner wrote. The worker re-reads `consent.json` and `install.json` on every tick, so a consent change or a `forget` in one process reaches the others within a tick. Changes to `install.json` are read-modify-write under `install.lock`; changes to `consent.json` start from the file's current content, without a lock. - Where the filesystem does not support locks (`try_lock` fails with an error other than "would block"), a folder counts as an orphan only when nothing in it has been written for 24 hours, and no unclean exit is ever reported from it: its marker is discarded and its items are sent. A running session rewrites its marker every `heartbeat_interval`, which keeps its folder fresh. ### 3.3 Consent and kill switches - Default is on for all three categories. `Signal::consent()` maps each signal to its category. - `DO_NOT_TRACK` (any value other than empty, `0` or `false`) or `TRACER_DISABLED` set the same way disables everything for the process: `allows` is false for every signal, nothing is written to the spool, and no request is made. The consent file is not changed. - `Sink::allows(signal)` is true only when the host enabled the signal, the kill switches are off, and the category is allowed. - `set_consent(category, false)` takes effect at once and also deletes that category's unsent items from the spool. - A choice made in this process is the authority in memory until it has been saved. If `consent.json` cannot be written, the worker tries again on every tick and does not read the file in the meantime, so a file that could not be updated never switches an opt-out back on. - `notice_pending()` is true until `notice_shown()`. Until then the SDK records and spools as usual (opt-out model) but **sends nothing**: no ingest request and no forget request leaves the machine before the user has seen the host's notice. `notice_shown()` is saved in `consent.json`, so later runs send from the start, and other running processes of the host see it within a tick. A category the user switches off before the notice is acknowledged is deleted from the spool and so is never sent. `flush_blocking` returns false while sending is held and data is waiting. The first time that happens the worker logs one `log::warn!` with target `ashlar_trace`, so a host that forgot to call `notice_shown()` can find out why nothing arrives; the SDK's own logger forwards lines with that target but never keeps them as breadcrumbs. - `Builder::send_before_notice(true)` lifts the hold, for hosts that show their notice somewhere else, such as an installer. It does not mark the notice as shown. - Under a kill switch `set_consent` and `notice_shown` change nothing on disk. - `forget()` deletes this process's unsent items, every orphan, its breadcrumbs and the outbox, sends a `ForgetRequest` for the old install id, and creates a new install id. It also calls `Signals::forget()`, which empties what the signal modules hold in memory from before: the breadcrumb ring, the last crash and the last performance window. Other running processes of the host see the new id on their next tick and do the same to their own session folder and memory; items they had queued under the old id are discarded, which is decided under the spool lock so that none can land under the new header. - The rotated install record is the authority in memory. If `install.json` cannot be written, the new id and the pending forget stay in memory, the worker tries the save again on every tick, and a file that still names an id this process has rotated away from is treated as behind and overwritten, never taken back. - The forget request is best effort and never holds up ingest. A network error, a 5xx or a 429 is retried later with its own backoff, in this run and after later launches. A 401 or 403 stops all sending, as for ingest, and keeps the request. Any other 4xx is final: the request is given up and counted as `forget_rejected`. ### 3.4 Identity and ids Install and session ids are 16 random bytes from `getrandom`, as 32 lower-case hex characters. If the random source fails, the SDK falls back to a mix of time, process id and a counter rather than failing. Item ids have the same form but are not drawn from the system each time, because that is a system call per item: the first half is random and fixed for the process, the second half is a per-process counter passed through a mixing function that gives every count a different value. They are unique, which is all the server's duplicate check needs. The two items that report an adopted session's unclean end are the exception: their ids are derived from the session id (§3.5). ### 3.5 Session - One session per process run. `session_start` is submitted during `init`. - The worker emits `session_heartbeat` every `heartbeat_interval` with running totals, and rewrites the marker, `spool//session.json`. A heartbeat is never written after the session's end. - Time accounting uses a monotonic clock, sampled once a second by the worker: a second counts as *focused* if the host last said `set_focused(true)` (hosts that never call it count as focused), as *active* if focused and `note_input()` was called within `idle_after`, otherwise *idle*. `note_input` and `set_focused` are single atomic stores. - Clean shutdown (last `Tracer` clone dropped) writes `session_end` on the calling thread, removes the marker, tries one final send, and releases the session folder. If everything was sent the folder is deleted; otherwise it stays, without a marker, as an orphan that the next process to look simply sends. Status is `crashed` if a panic was seen and the thread dropping the tracer is panicking, otherwise `exited`. - `errors` counts handled errors and survived panics. The panic hook counts every panic when it happens, because it cannot know then whether the process will survive it. When a session ends `crashed`, by clean shutdown while unwinding or as reported from a marker with `panicked` set, the panic that ended it is taken off again: `crashed` already says so, and it is not counted twice. - The time one tick may add to the session's totals is capped at five tick intervals, and at no less than a minute. A longer gap means the machine was asleep (some monotonic clocks keep counting through sleep), and that time is not credited as focused, active or idle. - An orphan (§3.2) that still has its marker belonged to a process that ended uncleanly. The process that adopts it writes into that folder, under **that** session's header: a `session_end` with status `crashed` if the marker's `panicked` flag was set, else `abnormal`, with the last saved totals; and, when not panicked, a `crash` of kind `unclean_exit` carrying `signals::saved_breadcrumbs()`. Both carry the time of the marker's last rewrite, or the present if the marker claims a later time. It then removes the marker, then the saved breadcrumbs, and sends the folder's items like any other. The breadcrumbs are deleted last on purpose: if the adopter dies after reading them and before the report is on disk, the next adopter still finds them. - An unclean end is reported once: only the lock holder reports, and the marker is removed after the report is on disk. The ids of the two items are derived from the session id, so if the adopter itself dies between those two steps and the report is written again, the server stores it once. - Short-lived processes (`ProcessKind::Cli`, `Child`) behave the same; they should call `flush_blocking` before exit. - `Sink::session_summary()` gives the local report the install id, session id, start time, running totals and error count. ### 3.6 Event catalog The host declares events once: ```rust ashlar_trace::events! { pub enum Category { File = "file", Edit = "edit" } pub enum AppEvent { FileOpen = "file.open" in File, Undo = "edit.undo" in Edit, } } ``` - A declarative macro; no procedural macro crate. - It generates both enums (`Copy`, `Clone`, `Debug`, `PartialEq`, `Eq`, `Hash`), and implements `EventCategory` and `Event` with `ALL` lists. - At build time, through `const` assertions using `ashlar_trace_proto::validate_name` and `const_str_eq`, it rejects an invalid name and a duplicate name within either enum. `trybuild` compile-fail tests cover both. - `listen(category)`, `listen_events(&[..])` and `listen_all::()` choose what is recorded, and each declares the catalog itself, so their order does not matter and no separate call is needed. `listen(category)` reaches the events through `EventCategory::EVENTS`, which the macro fills in. `Builder::catalog::()` declares the events without listening to any. A hand-written `EventCategory` that leaves `EVENTS` empty needs that call; `init` logs one warning when categories are listened to and no event is declared. - `track` drops an event that is not listened to before it reaches the queue. The check must be cheap: a lookup in a small set built at startup. - `track_id(category, id)` records an id that is not in the catalog, marked `dynamic`, when its category is listened to. The id is validated with `validate_name` at run time; an invalid one is dropped and counted. - Properties: at most `MAX_PROPS`, keys validated like names, string values `&'static str` of at most `MAX_PROP_STR_LEN` bytes. Anything over a limit is dropped and counted. - `session_start` always carries `catalog_hash` (a stable hash of the sorted catalog), `listening` and `signals`. It carries the full `catalog` until the server has acknowledged that hash for this app version; an ingest reply with `need_catalog: true` clears the acknowledgement so the next session resends it. ### 3.7 Queue, spool and sending - `submit` does `try_send` on a bounded channel of `queue_capacity`. When full it drops the item and counts `queue_full` with one atomic add. It never blocks and takes no lock. - `track` hands the worker exactly what it was given: the two `&'static str` names, a timestamp, and the properties copied into one boxed slice (no allocation at all without properties). Checking the limits, building the wire event, assigning the id, serialising and writing are done by the worker. `frame` is three atomic operations on a histogram of atomics. No call on a host thread takes a lock the worker holds across disk or network I/O. - `Builder::init` does only what must precede the first call: creates the folders, reads the three small state files, takes the session folder's lock, writes header and marker and the `session_start`, and starts the worker. Looking through what earlier runs left behind, however much, is the worker's first job. - When the worker drains the queue for a flush or at shutdown it takes at most one queue's worth, so host threads that keep the queue full cannot keep it from sending, ticking or answering the flush. At shutdown the drop counts are written to the spool whether or not anything can be sent, so they are not lost with the process. - One worker thread drains the queue, appends items to its own session's spool segment, and ticks every `tick_interval` (1 s by default): session accounting, re-reading the shared consent and install files, `Signals::tick`. Every other interval is effectively rounded up to a multiple of the tick. - `submit_now` serialises the item and appends it to the spool on the calling thread, without going through the queue. It is used from the panic hook, so it must not panic, must tolerate a poisoned lock, and gives up (counting `spool_busy`) rather than wait on a lock the panicking thread may itself hold. - The spool is capped by `spool_max_bytes` and `spool_max_age`, across all session folders: the folders of other live processes count towards the size, as of the last look at them. To make room a process evicts the oldest segments it may touch (adopted orphans first, oldest session first, then its own oldest) and never anything of another live process; if that is not enough, the new item is dropped. Evicted and expired items are counted as `spool_evicted`. - The worker sends when `batch_items` are waiting or `flush_interval` has passed, and once at startup for anything earlier runs left. It sends the orphans it has adopted, oldest session first, then its own session. One request per session folder, at most `MAX_ENVELOPE_BYTES` after serialisation; larger backlogs are split. What a request carried is deleted when the server has answered for it, and an orphan's folder is deleted when it is empty. - Bodies are gzip-compressed JSON, at the fastest level: on a 1 MiB batch of events it takes an eighth of the time of the default level for 5% more bytes. - Reply handling follows the table in `docs/wire-format.md`. Retries use exponential backoff with jitter, from `retry_min` (5 s) doubling up to `retry_max` (15 min), each wait a random point in the upper half of its step. A `Retry-After` given in seconds is honoured, never for less than `retry_min` (so `Retry-After: 0` does not mean "on every tick") and up to a day; an explicit flush skips the backoff but never the server's `Retry-After`. - Only an ingest reply that parses says anything about the catalog. A 2xx with an empty or foreign body counts as delivered but neither acknowledges the catalog nor asks for it. - A segment is read as bytes and each line is judged on its own. A tail without a newline (a write cut short by a crash) and a line that is not valid UTF-8 are skipped and counted as `spool_damaged`; the lines around them are sent. A segment that cannot be read at all is left out of the batch and stays on disk; it does not hold up the sessions after it. - No interval can overflow the clock: `tick_interval` is clamped to at most an hour and the retry waits to at most a week when the worker starts, and a `flush_blocking` timeout or `shutdown_timeout` too long to add to the clock means "no deadline". - Drop counts are sent as a `drops` item with the next batch, while at least one consent category is allowed. The reasons are: `queue_full`, `spool_evicted`, `spool_busy`, `item_too_large`, `io_error`, `server_rejected` (a batch the server refused with 400, 413 or 422), `spool_damaged` (a stored line that was cut short or is not valid UTF-8), `forget_rejected`, `invalid_id`, `prop_dropped`, `tick_panicked`, and from the signal modules `breadcrumbs_write`. - With no endpoint configured the SDK still spools (within the caps), still adopts orphans, and inspect mode still works, but nothing is sent. This holds even when the host supplies its own `Transport`. - `flush_blocking(timeout)` asks the worker to drain and send now and waits for it. It is true when this process has nothing left that it could send. `shutdown` does the same with `shutdown_timeout`, then stops the worker. - The built-in transport is `ureq` 3 with rustls, behind the `http` feature, with connect and overall timeouts and redirects off. Hosts and tests can supply their own `Transport`. ### 3.8 Signals | Module | Behaviour | |---|---| | `crash` | A panic hook that chains to the previous hook. It captures message, location, thread name and `Backtrace::force_capture`, scrubs them, attaches breadcrumbs and the last perf window, calls `submit_now` with a `crash` of kind `panic`, then `count_error` and `mark_panicked`. It must never panic itself and must guard against re-entry. `report_handled` submits kind `handled` and calls `count_error`. `report_child_exit` submits kind `child_exit` with exit code or signal, only for an unsuccessful status. **`fatal`** is set where it can be known: `true` on a `child_exit` and on the `unclean_exit` the pipeline writes for a dead session; `false` on `handled` and `hang`; and `false` on a `panic` at capture, because the hook cannot know whether the host catches it. The server marks a panic fatal when its session ends `crashed`. | | `hang` | A watchdog thread, driven in one of two ways. **Turns** (preferred): `begin_turn()` when the main loop starts handling something and `end_turn()` when it is done, or `turn()` for a guard that does both. A turn open for longer than `hang_threshold` is a hang; the time between turns, when the loop waits for work, never is. This is the only way that works for a loop that runs only on input or that a GUI framework owns. **Heartbeats**: `heartbeat()` once per iteration of a loop that runs continuously; a last beat older than the threshold is a hang. Once a host has begun a turn, heartbeats no longer count for that process. Each call is one atomic update. A second `begin_turn` while a turn is open keeps the original start; `end_turn` without an open turn does nothing. Either way one `crash` of kind `hang` is submitted per hang, with `blocked_ms` as observed so far, and nothing more until the next turn or beat. Watching starts with the first turn or beat, so hosts that call neither get no reports. `pause`/`resume` nest; resuming restarts the open turn or the last beat, and stores that before lifting the pause. If the watchdog finds far more time has passed than its own poll interval, it was suspended with the rest of the machine: it restarts the measurement instead of reporting. | | `perf` | `frame(duration)` records into a `FrameStats` histogram behind a lock that is held only for the update. `first_frame()` records startup time since `init` once. Every `perf_window` the tick submits one `perf` item if it has anything to report, and keeps a copy as "last perf" for crash reports. | | `sys` | With the `sysinfo` feature: this process's CPU and memory every `sys_interval`, folded into the perf window (mean CPU, last and peak memory), and static device details once for the envelope header. Without the feature, those fields are absent. | | `logs` | A `log::Log` that keeps the last `breadcrumb_capacity` lines at or above `breadcrumb_level` in a ring, scrubbed, and forwards every record to the wrapped logger if there is one. `LoggerMode::Install` installs it globally; otherwise the host wraps its own logger with `Tracer::logger`. The ring is saved to `breadcrumbs.jsonl` in the session folder on the tick when it changed. Log lines leave the device only inside a crash. Lines with the SDK's own target, `ashlar_trace`, are forwarded but not kept. | | `rt` | `Counter` and `Gauge` are `Arc`. Registration allocates and locks; recording is one atomic operation. The tick reads them into the perf window (counters as the increase since the last window). | | `scrub` | Replaces the home directory with `~`, removes the account name from paths, truncates long values (messages 2 KiB, backtraces 32 KiB, log lines 512 bytes). | | `report` | `write_report(path)` writes a plain-text report: app, version, device, session ids and totals, last perf window, recent breadcrumbs. | Known limit for 0.1: a backtrace from a release build stripped of symbols contains unresolved frames. Symbolication arrives with native crash support. ### 3.9 Threads | Thread | Does | |---|---| | Host threads | Call `Tracer` methods; each is an atomic store, a short lock, or a `try_send`. | | `tracer-worker` | Drains the queue, writes the spool, ticks, adopts orphaned session folders, sends. The only thread that touches the network. | | `tracer-watchdog` | Hang detection only. Started when the `Hang` signal is enabled. | ## 4. Server ### 4.1 Shape One binary, `ashlar-trace-server`, with subcommands: | Command | Does | |---|---| | `serve` | Runs the HTTP server, the rollup timer and the retention timer. | | `app create ` | Creates an app and prints its first write key. | | `app list` | Lists apps. | | `key create ` / `key revoke ` | Manages write keys. | | `admin create ` | Creates a dashboard login; reads the password from `TRACER_ADMIN_PASSWORD` or stdin. | | `prune` | Runs retention once. | | `rollup [--days N]` | Recomputes rollups once. | Settings come from the environment (all optional): | Variable | Default | Meaning | |---|---|---| | `TRACER_BIND` | `127.0.0.1:8090` | Listen address | | `TRACER_DATA_DIR` | `./data` | Folder for the database | | `TRACER_RETENTION_DAYS` | `90` | Age at which raw rows are pruned | | `TRACER_RATE_PER_MIN` | `120` | Requests per minute per client address on the public routes (ingest, forget, health), unknown keys included; `0` switches the limit off. Dashboard sign-in has its own limit, which stays on | | `TRACER_KEY_RATE_PER_MIN` | `6000` | Ingest requests per minute for one write key over all addresses: a capacity guard far above the per-address limit; `0` switches it off | | `TRACER_MAX_DB_MB` | `10240` | Ingest answers `507` once the database is larger than this many MiB; the dashboard and sign-in keep working; `0` switches the ceiling off | | `TRACER_TRUST_PROXY` | unset | When `1`, take the client address from the last `X-Forwarded-For` entry of the last such header line | | `TRACER_ADMIN_PASSWORD` | unset | Password for `admin create`; read by that command only | | `TRACER_SECURE_COOKIES` | on | Cookies are `Secure` and the session cookie uses the `__Host-` prefix. Set `0` to allow plain-HTTP use on a local machine | ### 4.2 Storage - SQLite in WAL mode, schema in `migrations/`, applied at startup and tracked with `PRAGMA user_version`. `0001_init` is the schema; `0002_dashboard` adds indexes for the dashboard queries; `0003_hardening` makes catalogs additive (`catalog_events.hash`), adds `events.declared`, `daily_events.declared`, `sessions.catalog_hash` and `issues.created_day`; `0004_scale` adds the indexes the large-database measurements called for (`sessions_by_version`, the partial `perf_with_metrics`), `daily_active` (daily, weekly and monthly active installs) and `dirty_days`. - All access goes through a `Store` trait so a second backend can be added. - **Writes: group commit.** Ingest requests go through a bounded queue (256 envelopes) to one `tracer-writer` thread. It takes whatever has queued up (at most 64 envelopes), stores them in one transaction, commits once and only then answers each caller, so a request is acknowledged only after its data is committed. When the queue is full the answer is `503` with `Retry-After: 1`: memory cannot grow, and the client keeps its data. If a group fails, its envelopes are stored one by one, so a bad envelope cannot take the others down; a failure that concerns the whole database (locked past the busy timeout, disk full, read-only) is reported at once instead (`503` for a lock, `500` otherwise). - **Durability.** WAL with `synchronous = NORMAL`: a commit is in the log file but not fsynced. A crash or `kill -9` of the process loses nothing that was acknowledged; a power failure or operating-system crash can lose the last moments of acknowledged data (the database stays consistent, never half-written). A separate `tracer-checkpoint` thread folds the log into the database file every second, so the writer never pays for that fsync inside a commit; the log is allowed to grow to about 200 MiB before the writer would checkpoint it itself. - **Reads** use a small pool of `query_only` connections and never wait for the writer (WAL). - **Signals.** `SIGINT` and `SIGTERM` both stop the server gracefully: it stops accepting, lets in-flight requests finish (up to 10 s) and exits with status 0. - Conventions are in the header of `0001_init.sql`. ### 4.3 Ingest `POST /v1/ingest`, in order: 1. Take one request from the client address's token bucket (in memory, bounded, IPv6 by /64) before anything else, so unknown keys and the health route are limited too; over the limit is 429 with `Retry-After`. The address is used here, for the sign-in limit and nowhere else. A request the address bucket refuses does not touch the key bucket. 2. Look up `x-tracer-key`; unknown or revoked is 401. Then take one request from the key's own bucket (`TRACER_KEY_RATE_PER_MIN`, a capacity guard); over it is 429. 3. When the database is over `TRACER_MAX_DB_MB`, answer 507. The client keeps its data and retries. 4. Read at most `MAX_ENVELOPE_BYTES` after gunzip; larger is 413. Parse; failure is 400. A `schema` above the server's is 422. 5. In one transaction: - Upsert the install (first/last seen, versions, device). - Upsert the session **only when the envelope carries a session item** (`session_start`, `session_heartbeat` or `session_end`). Crashes, performance windows and events keep their `session_id` as plain text on their own rows, so a user who switched session reporting off never creates a session that the stale closer would turn `abnormal`. - For each item, skip it if `(app_id, id)` is in `items_seen` (a duplicate), else store the item by type and record the id: - `session_start`: remember the named `catalog_hash` on the session. A carried catalog is stored **next to** the others, keyed by `(app, version, hash)`, never in place of one (at most 8 per version and 200 per app); a hash the server does not hold and did not receive sets `need_catalog`. - `session_heartbeat`: raise the session totals (never lower them) and `last_seen`. - `session_end`: raise the totals and set the status, unless the stored status is already terminal. When the status is `crashed`, the session's most recent `panic` crash is marked `fatal`. - `event`: validate the name. **An event is never refused for being undeclared**: the catalog is client input, and a refused event is deleted by the client. A non-dynamic name is stored with `declared = 1` when it is in any catalog held for that app version (which includes the one the session's `session_start` named), else `declared = 0`. With no catalog held it is `declared = 1` and `need_catalog` is set. - `perf`: store. - `crash`: compute the fingerprint, upsert the issue, store. At most 20 crashes are taken per envelope (the rest count as rejected). An app creates at most 200 new issues per day; further new fingerprints share one "Too many distinct crashes" issue for that day, so nothing is dropped silently. A `panic` is marked `fatal` when its session has ended `crashed` (in either arrival order; the SDK cannot know at capture time whether the host survives). A fatal panic from the client also marks an `ok` session `crashed`. - `drops`: add to `daily_drops`. Each count is clamped to 1,000,000, reasons must be valid names, at most 64 distinct reasons are kept per app per day (the rest count as `other`) and a stored total is capped. - unknown types: skip, count as rejected. 6. Item times are corrected for clock drift by `received_at - sent_at` when the difference is over five minutes, then clamped to the retention window in the past and to one hour in the future. Item ids longer than 64 bytes are rejected. **Fingerprint:** kind, plus the first frames of the backtrace that belong to the app (skipping `std`, `core`, `alloc`, panic and backtrace machinery, and with hex addresses and hash suffixes removed); when there is no usable backtrace, the location; when there is neither, the message with digits removed. The issue title is the message's first line, or the kind. `POST /v1/forget` deletes the install's rows from `installs`, `sessions`, `events`, `perf`, `crashes` and `daily_installs`, and removes issues left with no crashes. It answers `200 {"ok":true}` whether or not the install existed, so it cannot be used to probe for install ids. **Connections.** The server runs hyper directly. A client has 15 seconds to send a request's headers, which also bounds an idle keep-alive connection, and at most 1,024 connections are served at once; further clients wait in the accept queue. The body of a request must arrive within 30 seconds. ### 4.4 Rollups and retention - Ingest marks every (app, day) it stores data for in `dirty_days`. A timer (every five minutes) recomputes the `daily_*` tables for exactly those days, oldest first, and clears each mark unless new data arrived meanwhile; `rollup --days N` recomputes a fixed window by hand. Late data for an old day (a client replaying a spool) is therefore rolled up too. - One day of one app is built on a private connection that only reads the raw tables and collects the results in an in-memory scratch database; the write is then one short transaction that replaces that day's rows (for the install list only the difference is applied). Ingest is never held up by the aggregation itself. - `daily_active` holds the daily, weekly and monthly active installs per day. The weekly and monthly figures cost a distinct count over up to 30 days, so they are computed for the newest two days only (older days store `-1`, and readers fall back to counting). - Per app, only the 50 busiest versions, 20 busiest operating systems and 500 busiest event names keep their own rollup rows; the long tail is folded into `other`, so made-up values cannot grow the rollups. - `daily_events` is keyed by `declared` as well: declared and undeclared uses of one name are separate rows. - Retention deletes raw rows older than `TRACER_RETENTION_DAYS`, `daily_drops` and catalogs not seen within that time, and `items_seen` older than 35 days, then lets SQLite reclaim space. Rollups other than drops are kept. Big tables are deleted in chunks of 2,000 rows, each its own short transaction with a pause of three times the chunk's duration, so ingest keeps getting the writer; space is returned 2,000 pages at a time. - Sessions still `ok` whose `last_seen` is over a day old are closed as `abnormal`. ### 4.5 Dashboard Server-rendered HTML with `maud`, charts as inline SVG generated on the server, one small embedded stylesheet, no script build. Login is a username and password (argon2), with a random session token in an `HttpOnly`, `SameSite=Lax` cookie; only its hash is stored. Every state-changing form carries a CSRF token. The login cookie lasts 14 days. The login form itself is protected by a short-lived `SameSite=Strict` cookie paired with a hidden field, and by a limit of 10 attempts a minute per address. Dashboard responses carry `Content-Security-Policy: default-src 'self'` (no inline script or style), `X-Frame-Options: DENY`, `X-Content-Type-Options: nosniff`, `Referrer-Policy: no-referrer` and `Cache-Control: no-store`. | Page | Shows | |---|---| | Overview | Per app: crash-free sessions and installs, daily opens, daily / weekly / monthly active installs, newest issues. | | Crashes | Issue list with counts, affected installs and versions; filter by kind, version, status. Issue page with backtrace, breadcrumbs, device, last perf window, and resolve / ignore. | | Performance | Frame-time percentiles (from merged histograms), slow and frozen frame rates, startup time, CPU and memory, by version and by operating system. | | Usage | Active installs, sessions per day, session length, focused / active / idle time, events ranked by use, and declared events that were never fired. | | Reports | A stability and usage report for one release: summary, top issues, performance, feature use. Download as HTML, Markdown or JSON. | | Settings | Apps and write keys. | Definitions: - **Crash-free sessions** = 1 − (crashed + abnormal) ÷ sessions that have ended. - **Crash-free installs** = 1 − installs with a crashed or abnormal session ÷ active installs. Counted per day and summed over the selected range (install-days), because rollups cannot give distinct installs across a range. - **Daily opens** = sessions started that day with process kind `gui`. Kept in the rollup as `daily_sessions.opens`, since raw sessions are pruned. ## 5. Later phases See `ROADMAP.md`. The designs above leave room for them: the `Store` trait for Postgres, the `Transport` trait for the web target, and an attachment route for minidumps. # Wire format Schema version **1**. The types live in `crates/trace-proto`; the fixtures in its `tests/fixtures/` are the reference examples. ## Transport | | | |---|---| | Ingest | `POST /v1/ingest`, body is one envelope as JSON, usually `Content-Encoding: gzip` | | Forget | `POST /v1/forget`, body `{"schema":1,"install_id":"…"}` | | Health | `GET /v1/health` | | Auth | `x-tracer-key: ` on ingest and forget | | Limits | 1 MiB per body after decompression | Replies to ingest are JSON: `{"accepted":n,"duplicates":n,"rejected":n,"need_catalog":bool}`. `rejected` counts items the server refused as malformed (a bad name, an unknown type, an over-long id, more than 20 crashes in one envelope); an event is **never** refused for not being in the catalog. `POST /v1/forget` replies `200 {"ok":true}` whether or not the install existed. The server limits requests per client address (`429`), and the address is used for that and nothing else. | Status | Meaning | Client does | |---|---|---| | 200 | Stored (see counts) | Deletes the batch | | 400, 413, 422 | Malformed or too large | Deletes the batch and counts a drop | | 401, 403 | Unknown or revoked key | Keeps data, stops sending until restart | | 429 | Rate limited | Waits for `Retry-After`, then retries | | 507 | Server storage is full | Keeps data, retries with backoff (like any 5xx) | | other 5xx, network error | Server trouble | Retries with backoff | ## Envelope One envelope carries items from **one session of one install**. Data from an earlier session, such as a crash found on the next launch, goes in its own envelope with that session's header. | Field | Type | Notes | |---|---|---| | `schema` | integer | `1` | | `app` | object | `name`, `version`, optional `build`, `process` (`gui`, `cli`, `service`, `child`) | | `install_id` | string | Random, resettable. Never derived from hardware. | | `session_id` | string | One per process run | | `device` | object, optional | `os_name`, `os_version`, `arch`, `cpu_brand`, `cpu_cores`, `ram_mb`, `gpu`, `locale` | | `sent_at` | integer | Client clock in Unix milliseconds, for drift correction | | `items` | array | See below | Every item has `id` (unique, client-generated, at most 64 bytes; the server ignores an id it has already stored), `ts` (Unix milliseconds) and `type`. The server moves `ts` onto its own clock when the client clock is more than five minutes off, and clamps it to the server's retention window in the past and one hour ahead. The server creates or updates the session record only for envelopes that carry a `session_start`, `session_heartbeat` or `session_end`. Events, `perf`, `crash` and `drops` in an envelope without one are stored with their `session_id` as plain text, so a client that has switched session reporting off creates no session. ## Item types | `type` | Fields | |---|---| | `session_start` | `catalog_hash`, `catalog` (`events[]` of `name`, `category`), `listening[]`, `signals[]` | | `session_heartbeat` | `seq`, `duration_s`, `focused_s`, `active_s`, `idle_s` — running totals | | `session_end` | `status` (`exited`, `crashed`, `abnormal`), the same totals, `errors` | | `event` | `name`, `category`, `dynamic`, `props` (bool, integer, float or short string values) | | `perf` | `window_s`, `frames`, `cpu_pct`, `rss_mb`, `rss_peak_mb`, `startup_ms`, `counters`, `gauges`, `incidents[]` | | `crash` | `kind`, `fatal`, `message`, `location`, `thread`, `backtrace`, `blocked_ms`, `exit_code`, `signal`, `context`, `breadcrumbs[]`, `perf` | | `drops` | `reasons` — what the client discarded, by reason | **`crash.fatal`.** A client cannot know at capture time whether a panic ends the host, so it sends `fatal: false` for `panic`, `handled` and `hang`, and `fatal: true` for `unclean_exit` and `child_exit` (which are found after the fact). The server decides for panics: when a session ends with `status: "crashed"` (a `session_end`, in the same or a later envelope, before or after the crash item arrives), that session's most recent `panic` is stored as fatal. A `fatal: true` panic from a client also marks its `ok` session `crashed`. **Catalog.** `session_start.catalog` is client input and is treated as unverified: catalogs are stored next to each other by `catalog_hash`, never replacing one. Each stored event carries a `declared` flag, set when its name is in a catalog held for the app version. `declared = false` events are kept and shown separately; they are not an error. `crash.kind` is one of `panic`, `handled`, `hang`, `unclean_exit`, `child_exit`. `crash.fatal` says the process could not continue. The client sets it where it can know: `true` for `unclean_exit` and `child_exit`, `false` for `handled` and `hang`. A `panic` is sent with `fatal: false`, because at capture the client cannot know whether the host catches it; the server marks it fatal when the session it belongs to ends `crashed`. `session_end.errors` counts handled errors and panics the process survived. The panic that ends a `crashed` session is not included. `perf.frames` is a histogram: `count`, `sum_us`, `max_us`, `slow` (over 16.667 ms, the 60 Hz frame budget; slow includes frozen), `frozen` (over 700 ms), `buckets`, one count per upper bound in `FRAME_BUCKET_BOUNDS_US` plus an overflow bucket, and the optional `over_budget` (over 8.333 ms, `DISPLAY_BUDGET_US`, the 120 Hz frame budget; it includes slow). Histograms merge by adding buckets. `over_budget` was added within schema 1. Clients that predate it leave it out, and older servers ignore it. Because 8.333 ms is one of the bucket bounds, a reader counts it from the buckets when it is missing (`FrameStats::over_budget_frames`), and the server always counts it from the buckets it stores, so old and new clients report the same figure. `perf.incidents` lists the incident rules that fired in the window, at most one entry per rule: `rule` (`long_frames`: 3 frames over 8.333 ms within 1 s; `stall`: one frame of 250 ms or more; `memory_growth`: 5 rising per-minute memory floors, the last at least 64 MiB and 20% above the first), `at_ms` (when it first fired, in milliseconds since process start), `count` (how often it fired in the window) and `value` (the worst figure: the slowest frame in microseconds for `long_frames` and `stall`, the growth in MiB for `memory_growth`). It was added within schema 1: clients that predate it leave it out, older servers ignore it, and readers treat an unknown `rule` as `other`. ## Names Event and category names are lower-case `snake_case`, optionally dotted (`file.open`), start with a letter, and are at most 64 bytes. The same rule is checked at build time in the SDK and again by the server. ## Compatibility - Fields are only added, and new fields are optional. - Readers ignore unknown fields, and treat an unknown item `type` as something to skip. - Unknown enum values map to `other` where the enum has one. - A breaking change increases `schema`. # Telemetry (trace) roadmap **Status, 2026-09-09:** release candidate, load-tested. The SDK (`ashlar-trace`), server (`ashlar-trace-server`), dashboard, demo hosts and guides are built; the SDK is tested end to end against the real server; a bug review of the SDK and a security review of the server are done and their findings fixed. Moved into ashlar with its history; apps use it through `ashlar-telemetry`. Not yet published. ## Version 0.1 | Id | Milestone | State | |---|---|---| | M0 | Foundation: workspace, gate, wire format with fixtures, SDK and server skeletons, schema, docs | Done | | M1 | SDK pipeline: consent, identity, session, queue, spool, transport, event catalog, usage | Done | | M2 | SDK signals: panic hook, handled errors, unclean exit, child exit, hang watchdog, frame and system metrics, breadcrumbs, local report | Done | | M3 | Server core: ingest, write keys, rate limits, dedup, SQLite store, rollups, retention, admin commands | Done | | M4 | Dashboard and reports: overview, crashes, performance, usage, per-release report export | Done | | M5 | Demo hosts and end-to-end tests | Done | | M6 | Guides and packaging: integration guide, self-hosting guide, container image | Done | | M7 | Hardening and 0.1.0 release | Release candidate | | M8 | Tests, stress and load: edge-case and fuzz suites, SDK hot-path and server ingest optimisation, large-database read path | Done; two load targets missed (below) | ## Still open - Find why ingest latency collapses at 2,000 envelopes a second once the database passes about 3 GB. Suspected: index inserts outgrowing the cache; not proven. - Cut ingest tail latency while retention deletes a large backlog (p99 about 300 ms against about 50 ms). - Repeat the load runs on Linux and on a quiet machine; all numbers so far are from one busy macOS machine. - Build and run the container image once (`deploy/trace/Dockerfile`, `deploy/trace/compose.yml`). Written but never built. - Run the windowed demo (`examples/trace-demo-gui`). It compiles and passes lints but has never been opened. - Check the dashboard on a narrow (360 px) screen. - Check the login cookie (`Secure`, `__Host-` prefix) in a browser over real HTTPS. - Decide whether to prove "no allocation" for the realtime counters, which needs an `unsafe` exemption for one test file. ## Later | Milestone | Notes | |---|---| | Native crashes | Out-of-process monitor, minidumps, symbol upload and server-side symbolication, in an isolated crate that is the only one allowed `unsafe`. | | Postgres | A second `Store` implementation for larger deployments. | | Web target | A transport for wasm32, where the SDK is currently a no-op. | | Alerts | Email or webhook on a new issue or a crash-rate spike. | | OTLP export | Forward stored data to OpenTelemetry collectors. | # Releasing Every crate in `crates/` is published to crates.io with the same version (the workspace `version` in the root `Cargo.toml`). The umbrella crate is published as **`ashlar-rs`**, because the `ashlar` name on crates.io belongs to an unrelated project; its library name is still `ashlar`, so apps write `use ashlar::prelude::*`. Examples and `xtask` are `publish = false`. ## One-time setup 1. On crates.io, create an API token with the `publish-new` and `publish-update` scopes. Pick no expiry (or a long one), so it is set once. 2. Store it as the repository secret `CARGO_REGISTRY_TOKEN`: ```sh gh secret set CARGO_REGISTRY_TOKEN -R Himasnhu-AT/ashlar ``` (paste the token when asked). The token never needs to be on a developer machine. 3. Optional, later: switch to crates.io [trusted publishing](https://crates.io/docs/trusted-publishing) and delete the token. ## What triggers a release Only pushing a tag of the form `vMAJOR.MINOR.PATCH` (for example `v0.2.0`). Nothing else publishes: pushes to `main`, pull requests and Dependabot never do, and running the `release` workflow by hand (Actions → release → Run workflow) is always a dry run. Before anything is uploaded, the workflow refuses to go on unless: | Guard | Why | |---|---| | the tag equals `v` + the workspace version | no mismatched or mistyped tags | | the tagged commit is on `main` | nothing from an unmerged branch | | the full CI gate passes on that exact commit (all three OSes, MSRV) | no broken or half-done state | | `CHANGELOG.md` has a `## [x.y.z] - YYYY-MM-DD` section | every release is documented | | that version of `ashlar-rs` is not on crates.io yet | no accidental re-release | Then it runs `cargo publish --workspace --locked` (every crate in dependency order), skipping the crates that already have this version on crates.io, and creates the GitHub release with that version's changelog section as the text. crates.io lets an account create only a few new crates at once and then about one every 10 minutes, so a release that adds new crates (the first one adds all of them) waits and tries again until every crate is there; that can take a few hours. Any other publish error stops the job. ## Cutting a release 1. Make sure `main` is green on CI and everything for the release is merged. 2. Run `cargo xtask release x.y.z` (add `--date YYYY-MM-DD` to date it other than today, UTC). It bumps `version` in `[workspace.package]` and every internal crate's `version = "…"` in `[workspace.dependencies]` together, renames "Unreleased" in `CHANGELOG.md` to `## [x.y.z] - YYYY-MM-DD` under a fresh empty "Unreleased", updates the compare links at the bottom, and runs `cargo check` to update `Cargo.lock`. It refuses when the versions disagree, the section already exists or "Unreleased" is empty, and it never commits, tags or publishes. 3. Read the diff; bump any version shown in docs snippets (for example `docs/trace/integration.md`). 4. Check that everything packages: `cargo publish --workspace --locked --dry-run`. 5. Commit to `main` and push. Wait for CI to pass. 6. Tag and push the tag: ```sh git tag -a v0.2.0 -m "ashlar 0.2.0" git push origin v0.2.0 ``` 7. Watch the `release` workflow. If a guard fails, nothing was published: fix the cause, delete the tag (`git push origin :refs/tags/v0.2.0`, `git tag -d v0.2.0`) and tag again. If publishing stops halfway (a network error, a timeout), crates.io keeps the crates that made it. Fix the cause and run the failed `publish` job again from the workflow run: it publishes only what is missing. If the fix changes the workflow itself, a re-run still uses the old one; push the fix to `main` and move the tag to that commit (`git tag -fa vX.Y.Z -m "ashlar X.Y.Z"`, `git push -f origin vX.Y.Z`). That is safe only while the crates' own files are unchanged, since the published crates came from the old commit. ## Versioning ashlar follows [Semantic Versioning](https://semver.org). Before 1.0, a minor version (0.x) may break the API; patch versions never do. The items marked `CONTRACT` in `ashlar-command` change only additively. # Security policy ## Supported versions ashlar is in early development. Security fixes go into the latest released minor version and into `main`. | Version | Supported | |---|---| | 0.1.x (latest) | yes | | older | no | ## Reporting a vulnerability **Please don't report security vulnerabilities in public issues, discussions or pull requests.** Report them privately through GitHub: [open a private security advisory](https://github.com/Himasnhu-AT/ashlar/security/advisories/new). Please include: - the affected crate(s) and version or commit - a description of the issue and its impact - steps to reproduce, or a proof of concept - any suggested fix You can expect an acknowledgement within a few days. We'll keep you updated while we work on a fix, and credit you in the advisory unless you'd rather stay anonymous. Please give us reasonable time to release a fix before you disclose the issue publicly. ## Scope Areas where security matters most in ashlar: - the agent control port (`ashlar-control`) and MCP over HTTP (`ashlar-mcp-http`): authentication, Host and Origin checks, input limits - downloading and verifying integrations (`ashlar-integrations`) and update checks (`ashlar-update`) - telemetry (`ashlar-trace`, `ashlar-trace-server`): data scrubbing, ingest limits, the dashboard - parsing of untrusted documents and files Dependencies are checked against the RustSec advisory database on every CI run. # Changelog All notable changes to ashlar are documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html). Before 1.0, a minor version (0.x) may contain breaking changes. ## [Unreleased] ### Added - Documentation site (`site/`, mdBook, deployed at https://ashlar.himanshuat.com): a user guide (first app, documents, commands, views, jobs, native features, agents, testing, web, a cookbook, an FAQ, building with an AI assistant), the developer docs and this changelog, plus `llms.txt`, `llms-full.txt` and each page's markdown for AI agents. ## [0.2.0] - 2026-11-11 ### Added - `cargo xtask release [--date YYYY-MM-DD]`: bumps the workspace and internal crate versions together, dates the changelog's "Unreleased" section, updates the compare links and refreshes Cargo.lock (docs/releasing.md step 2). It never commits, tags or publishes. - `ashlar_ui::sidebar` (IMP-5.14): a source list of folding sections with counts and rows with an icon, detail and badge, built on `list::Row` and virtualized. One Tab stop: arrows, Home/End, Left/Right to fold, Enter to activate, F2 to rename in place, type-ahead; click, double click and context clicks reported in `SidebarResponse`. Automation ids ``, `.section.`, `.section..item.`, `.rename`; labels `UiLabels::sidebar` (`ui.sidebar.rename`, `ui.sidebar.items`, English and German). `list::Row` gains `icon`, `detail`, `badge` and `focusable`. The gallery has a Sidebar page. - CodeView and DiffView follow-ups (IMP-5.20): unwrapped long lines scroll sideways (horizontal wheel or Shift+wheel, and a draggable bar along the bottom; a split diff's sides move together); both views register the new automation role `Role::Document` (`"document"` in `ui.inspect`). The gallery has a Code page. An open dialog no longer copies its shapes for the exit animation on frames where they did not change, and reuses the copy's buffer when they did (`perf_dialog_open_frame_allocs`: per-frame overhead over a plain area 173 → 21 allocations, 24.9 → 5.5 KB). - Controls follow-ups (IMP-5.19): `controls::Skeleton` placeholder rows (pulse while visible, static and frame-free under reduced motion, a busy progress node named by the new `UiLabels::controls.loading`, English and German); `content::Accordion` keeps at most one section open; `Segmented::disabled` and `RadioGroup::disabled` dim options that cannot be picked and that the keys skip; the `⇧` keycap is drawn as an outlined arrow (the default fonts lack the glyph). Visual regression adds reference-painter pairs for the progress circle and the spinner's static poses. - Toasts (IMP-5.17): F6 moves keyboard focus into the newest toast (its first action, else its close button) and back to where it was; `Toasts::take_actions_where` drains only the actions a caller accepts; `App::run_toast_actions(true)` makes the runtime run a clicked toast action whose command is registered (other actions stay for the view). The gallery's Overlays page demos action and progress toasts. - `ashlar-rs`: the `tray` and `hotkey` features pass through to `ashlar-app`, so an app on the umbrella crate can use `App::tray` and `App::global_shortcut`. - Windows menu shortcuts (IMP-6.17): `NativeMenuBar::attach_window` subclasses the window and runs `TranslateAcceleratorW` with the bar's table on key-downs, so a native menu's key equivalents fire as item activations (once; the key press doesn't also reach the window). Compile-checked for `x86_64-pc-windows-msvc`; not run on Windows yet. - Context menus and drag-out (IMP-6.15): `ashlar_native::menu::popup::show(window, &spec, at, waker)` opens a `MenuSpec` as a native context menu (macOS: posted to the run loop so it never opens inside the UI pass; Windows: opens at once, call it between frames) and reports `Some(id)` or `None` on a receiver. `ashlar_native::drag_out::start(window, DragItem { files, png, text }, on_end)` (feature `drag-out`, in `full`) drags files, a PNG or text into other apps on macOS; `Unsupported` elsewhere. - `ashlar_native::mac_shell` (default feature `mac-shell`, IMP-6.14): `guard_appearance()` checks the window delegate's `effectiveAppearance` changes before winit casts them, so a nil appearance while the screen is captured no longer ends the app; `listen_reopen(waker)` reports Dock reopen clicks; `quiet_requested(prefix)` reads `_QUIET`. An ashlar app installs both after its first frame, shows and focuses its window on reopen, and with `_QUIET=1` opens without taking the focus. - `ashlar_native::hotkey` (feature `hotkey`, IMP-6.16): system-wide shortcuts on global-hotkey (macOS, Windows, Linux on X11). `GlobalShortcuts::new(waker)`, `register(id, &chord)` with the keymap's chords (a chord without Ctrl, Alt or ⌘ is refused), `unregister`, `drain()`; dropping it releases every chord. `system::bring_to_front(window)` restores and focuses a window through the OS (Windows borrows the foreground thread's input when the foreground lock refuses), and a second launch on Windows now calls `AllowSetForegroundWindow` for the first (its pid travels in the endpoint file). `App::global_shortcut(command, chord)` (feature `hotkey` of `ashlar-app`) runs the command and brings the window forward; forwarded launches and tray clicks also bring it forward through the OS. - `ashlar_native::tray` (feature `tray`, IMP-6.11): a tray / status-bar icon with a menu built from a `MenuSpec` (macOS status bar, Windows notification area, Linux StatusNotifierItem over D-Bus with no GTK). `Tray::new(TraySpec { id, tooltip, icon, template, menu }, waker)`, `events()` (`Activate`, `Menu(id)`), `set_tooltip` / `set_icon` / `set_menu`; dropping it removes the icon. The menu bar, trays and later popups now share one process-wide menu-event router, so a tray's items never reach the menu bar. `App::tray(spec)` (feature `tray` of `ashlar-app`) runs the chosen item's command and shows the window on a click. - `cargo xtask launch --memory` launches 5 times (`--runs N`) and prints the min and max idle and peak footprint and how many runs fell in the high group; one launch landed in either of the two macOS groups at random (IMP-2.19). - `ashlar_native::memory` reports the OS's lifetime peak footprint (`ProcessMemory::peak`: macOS lifetime max phys footprint, Windows peak private commit) and `thread_count()` (macOS, Windows, Linux); `app.stats` → `process.peak_bytes` uses the OS peak, so spikes between samples count, and `process.threads` is filled on macOS and Windows too (IMP-6.18). - `ashlar_native::display::monitors()` lists the monitors before any window exists (macOS, Windows); a restored window placement is checked against them at launch instead of only on the first frame (IMP-6.18). - Web (IMP-6.8): `App::canvas_id`; when a web app cannot start (no such canvas, no WebGL) a visible error (`role="alert"`) appears next to the canvas or at the top of the page; `examples/web-hello` (with an `index.html` for trunk) is checked for wasm32 by `cargo xtask wasm`. - i18n follow-ups (IMP-6.19): plural forms for the toast "+N more" line and the keymap editor's "N commands share a shortcut" (new key `ui.keymap.shared`), through `labels::Plurals`, `labels::install_i18n`, `labels::count_text` and `labels::PLURAL_KEYS` (the label structs are unchanged); `ashlar_native::system::preferred_languages()` (macOS `NSLocale.preferredLanguages`, Windows `GetUserPreferredUILanguages`), which `ashlar_i18n::system_locales()` now appends after the environment's languages in apps (`set_native_languages`, `with_native`). - Record and replay (IMP-4.11): `ASHLAR_RECORD=path` (or `Runtime::record_to`) records each frame's time, screen and input events plus the control requests served to a JSONL ring of at most 64 MiB (written off the UI thread), and `Driver::replay(path)` plays it into a fresh driver; see docs/debugging-uis.md. - `Cx` polish (IMP-3.8): `cx.tokens()`; `cx.motion(id, target, curve)`, which animates unless the document just changed by the user's own hand; `Origin` (`User`, `Drag`, `Undo`, `Agent`, `Job`) recorded per change (`Session::origin`, `Session::with_origin`, `Cx::origin`), with control requests marked `Agent` and job applies `Job`; `ashlar_ui::layout::{row, column, inset}` taking a `tokens::Space`, `SpaceScale::get`; `Radii::{sm, md, lg}()` as `f32`. - Telemetry incident rules (IMP-2.13): `ashlar-trace` flags 3 frames over the 120 Hz budget within 1 s (`long_frames`), a frame of 250 ms or more (`stall`) and 5 rising per-minute memory floors that grow by at least 64 MiB and 20% (`memory_growth`), once per rule and window, in the new optional `perf.incidents` list (`Incident`, `IncidentRule` in `ashlar-trace-proto`; older readers ignore it). The trace server counts them as `incidents.` metrics and lists them on a crash's performance panel. - `ashlar_ui::markdown` v2 (IMP-5.4): links take keyboard focus (Tab, then Enter or Space opens them) with a focus ring, have the AccessKit `Link` role and automation role `Link` (new), and answer the pointer on every row they wrap onto; tables size columns to their content and wrap them down to their longest word, with a "N more rows" footer (`.table..more`) past `MAX_TABLE_ROWS` (500) instead of dropping rows silently; the `Markdown` builder adds `.on_task(|index, checked| ..)` (clickable task checkboxes, `.task.`), `.find(query)` (highlighted matches, counted in `MarkdownResponse::matches`; `find_ranges` gives the byte ranges), `.streaming(true)` (an unclosed `**`, `*`, `~~`, `` ` `` or link at the end is drawn as already styled, so markers never flash; also `parse_streaming`) and `.labels(&MarkdownLabels)` for "Copy", "Copied", "code" and the table footer, by default from the installed `UiLabels` (`ui.markdown.copy`, `.copied`, `.code`, `.more_rows`; English and German). `MarkdownStyle` gains `find_bg`. - `ashlar_ui::markdown::show_with_revision` and `Markdown::revision`: a caller that keeps a revision of the text skips hashing it each frame. Inline parsing is linear in the paragraph's length (a 50 KB paragraph: `bench: md_long_paragraph_parse`, about 0.4 ms), and a drawing snapshot (`tests/markdown_snapshot.rs`) checks what markdown paints (IMP-5.18). - Rich clipboard: `clipboard::set_rich` / `rich` (text, HTML, PNG, file lists; read HTML is sanitized by `clipboard::sanitize_html`, every form size-capped), `clipboard::set_secret(text, clear_after)` (kept out of clipboard history where supported, cleared only if still ours) and a `Backend` trait with `MemoryClipboard` for tests. - Dock ▸ Quit, logout, restart and shutdown go through the "Save changes?" check and flush settings, autosave and telemetry (`ashlar_native::session_end`, feature `session-end`, on by default): macOS answers `applicationShouldTerminate:` later until the user chooses; Windows handles `WM_QUERYENDSESSION` / `WM_ENDSESSION` and, since it can't wait for a dialog, keeps unsaved changes for recovery. The flow is the pure `session_end::step`. - Gesture handoff: value fields, sliders and knobs keep going when a drag is let go while moving and ease to rest (clamped, snapped; the commit carries the resting value, so one drag is still one undo step); `.fling(false)` turns it off. `VirtualList::fling(true)` drags the list to scroll it, with fling and rubber band past either end. A pause of 40 ms before release, or reduced motion, means no fling. - `ashlar_ui::rolling::RollingText`: counters and readouts that roll instead of jumping. Only the changed characters move (prefix/suffix, then LCS for short text), up when `.value(..)` rises and down when it falls; the width follows; `.tabular(true)` gives digits one width. Reduced motion crossfades. The accessible value is the whole new text at once; nothing animates once settled. - Windows reopen where they were: `App::remember_window(bool)` (default on) saves position, size, maximized and full screen state in the settings under `"window"` when they change, and the next launch opens the window there, sizes bounded and a window left on no monitor moved back (`ashlar_app::window_state::clamp`). `App::viewport(|builder| …)` changes the window builder (minimum size, decorations…) and `App::icon(png_bytes)` sets the window icon. - `ashlar-i18n` (new crate, rank 0, builds for wasm32): translations without a full Unicode library. Flat JSON catalogs per locale, parsed on first use and layered (later catalogs override key by key); a bad catalog is reported and acts as empty, never a panic. `I18n::{text, format, plural, integer, decimal, set_locale, negotiate}`: `{name}` templates, CLDR plural categories for about 50 languages (`key.one`, `key.few`, …), per-locale group and decimal separators (including Indian grouping), fallback locale → language → source (`zh-TW` → `zh-Hant`), POSIX and BCP 47 tags (`de_DE.UTF-8` → `de-DE`), `system_locales()` from `LANGUAGE`/`LC_ALL`/`LANG`, and `I18n::check()` for missing keys and mismatched placeholders. - `ashlar_ui::labels`: one `UiLabels` installed on the context holds every built-in component string (toasts, form, chat, diff view, keymap editor, tab strip, print panel). Components use it unless the app passes its own labels, which still win. English and German catalogs ship for the `ui.*` keys; `UiLabels::from_i18n` builds the labels for any locale. - Perf warnings in debug builds (`ashlar_app::perf_warn`): one stderr line per site when a frame waits (≥ 8 ms wall at ≥ 4× its CPU time; names the command or wake reason) or when 60 identical frames in a row are drawn for non-input wakes. `ASHLAR_FRAME_BUDGET_MS=N` (any build) prints at most one line a second for frames over `N` ms with the phase split; `ASHLAR_PERF_WARN=0` silences. - `ASHLAR_FRAME_LOG=FILE` and `Runtime::log_frames(path)`: one JSON line per frame (wall and UI-thread CPU time, phases, wake reasons, commands run, pass number), written by a background thread and capped at 64 MiB with a final `"end"` line (`ashlar_app::frame_log`). - `app.stats` (and the MCP `app_stats` tool) reports `process`: `footprint_bytes`, `peak_bytes`, `cpu_s` and `threads` (`ProcessStats`; `null` where the platform cannot say). - `ashlar-native` notifications: stable ids (`Notification::id`; the same id replaces a shown notification), `notify::show` (returns the id), `notify::remove`, `notify::on_click` (a `Receiver` with the notification's `info` map), `silent`, and `notify::set_app_identity`. Windows now shows toasts (AppUserModelID registered under HKCU for unpackaged apps, escaped XML, tags over 64 characters hashed); macOS installs a notification-center delegate so banners show while the app is frontmost and clicks are reported; Linux sends the `desktop-entry` hint and reports clicks from `ActionInvoked`. - `ashlar_ui::chat`: a chat kit over plain data. `Transcript` is a virtualized, variable-height message list (heights kept per message key, revision and width; a frame costs O(visible)) that follows the newest message while it streams, stays put once the user scrolls up and then offers "Jump to latest"; it is a polite AccessKit log. `Message` draws a role and a markdown body with a streaming state, `ToolCard` a tool call with folded details and keyboard-reachable Approve/Deny (`.approve`, `.deny`), and `Composer` a text field where Enter sends, Shift+Enter adds a line, input-method composition never sends, and `@` / `/` open a `PopupList` of suggestions. Strings come from `ChatLabels`. The gallery has a Chat page. - `ashlar-diff` (new crate, rank 0, no dependencies, builds for wasm32): line diffs with Myers' algorithm under a cost cap (past the cap, a split at lines unique to both sides, then a coarse delete/insert, flagged `coarse`), word-level emphasis for paired changed lines, hunks with context and the unchanged ranges between them (`FileDiff::gaps`, expandable when the new text is known), and `parse_patch` / `write_patch` for unified and `git diff` patches (renames, copies, modes, binary markers, quoted paths, `\ No newline at end of file`); malformed patches are an error, never a panic. About 2 ms for a 10k-line file. - `ashlar_ui::diff_view::DiffView`: unified and split diffs, virtualized (about 0.4 ms per scrolled frame at 50k lines), syntax-coloured per file language, word-level emphasis, "expand N lines" on collapsed ranges, foldable files, a sticky file header, line selection, `N`/`P` (or `F7`) between hunks. Automation `.file..hunk.` and friends for visible rows; AccessKit document, headings and labelled lines. - `app.stats` reports `launch`: `to_window_ms`, `to_first_frame_ms` and `launch_to_present_ms`, counted from the first line of `run()` (`FrameStats::launch`, `LaunchTimes`). Native windows only. - `_READY_FILE=path`: the app writes `{"pid", "ready_ms"}` there atomically once its first frame is presented. - `cargo xtask launch` measures `examples/hello` launch time over 9 launches (p50/p95), and `cargo xtask launch --memory` its idle RSS and physical footprint (macOS). Manual, opens windows. - `App::frame_latency(frames)`: how many frames the GPU may queue ahead of the screen (1 to 3). - Motion and jank gate for tests: `ashlar_ui::motion_gate::MotionRecorder` records values animated through `ashlar_ui::motion` and per-frame automation rects (`Driver::record` does both for an app), and `check(&MotionRules)` flags overshoot of opacity and colour, reversals without bounce, pops, flicker and frames requested after everything settled, with a per-id allow-list. The gallery's dialog, toast, theme crossfade and tab switches pass it. - `ashlar_ui::overlay`: `Select` (a button with a list of options), `Combobox` (a text field whose options are filtered with the command palette's fuzzy matcher; input-method composition never chooses), and `Popover` (anchored, flips above at the window edge, Escape or an outside click closes, focus moves in and returns). Option lists are virtualized, fade with the `enter_exit` motion token and register `.option.` with ListBox/option roles. - `ashlar_ui::help(response, text)`: a tooltip after the new `motion.tooltip_delay` token, also given to screen readers as the control's description. - `ashlar_ui::commands::menu::context(response, &spec)`: a context menu built from a `MenuSpec`. - Automation roles `ComboBox`, `ListBox`, `ListOption` and `Tooltip`. - `ashlar_ui::form`: the settings form pattern. Sections of rows (label, description, control) with a modified marker and reset button, a filter that hides non-matching rows and empty sections, highlights matches and shows "No settings match", and `Field` with a required marker and an announced error. Row controls get automation ids `