# Lesavka Architecture And Media Readiness

Status snapshot: 2026-08-12

This document explains what Lesavka is, how traffic moves through it, why the
webcam uplink has been unreliable, why the downstream eye feeds are currently
blank, and what evidence is still required before the system can be considered
finished.

## 1. What Lesavka Is Supposed To Do

Lesavka makes a physically attached lab computer usable from an operator
workstation as though its important peripherals were local. It is not merely a
screen-sharing application. It joins network transport to real USB, HDMI,
audio, camera, and input hardware.

There are three roles:

1. **Operator workstation (client)** - runs the Lesavka launcher and relay,
   displays the two remote eye feeds, plays remote audio, captures the selected
   webcam and microphone, and captures keyboard and mouse input.
2. **Theia / `titan-jh` (server)** - terminates the authenticated gRPC session,
   reads the two HDMI capture cards and remote audio device, and presents the
   operator's camera, microphone, keyboard, and mouse to the controlled host as
   physical devices.
3. **Remote controlled host (RCT)** - sees Theia as USB HID, UVC webcam, UAC
   microphone/speaker, and display-related hardware. Applications such as a
   browser or Google Meet should be able to use those devices without knowing
   that the operator is elsewhere.

The intended result is:

- two live, low-latency remote display feeds at the operator workstation;
- remote speaker audio at the operator workstation;
- keyboard and mouse control of the RCT;
- the operator's selected webcam and microphone appearing on the RCT;
- device/profile controls in the launcher remaining authoritative; and
- bounded latency, explicit health information, and recoverable failures.

## 2. Direction Names

The direction names are from the operator workstation's point of view.

### Downstream: Theia To The Operator

Downstream carries the RCT's outputs back to the operator:

```text
RCT display outputs
    -> GC311 capture cards on Theia
    -> /dev/lesavka_l_eye and /dev/lesavka_r_eye
    -> CaptureVideo gRPC streams
    -> client H.264 decoder
    -> launcher previews / breakout windows

RCT USB speaker output
    -> Theia UAC capture
    -> CaptureAudio gRPC stream
    -> client audio sink
```

### Upstream: The Operator To Theia And The RCT

Upstream carries the operator's inputs toward the RCT:

```text
operator webcam + microphone
    -> one client capture timeline
    -> StreamWebcamMedia bundled gRPC stream
    -> Theia freshness/sync planner
    -> UVC webcam + UAC microphone presented to the RCT

operator keyboard + mouse
    -> streaming gRPC input RPCs
    -> Theia HID gadget
    -> RCT input
```

The upstream and downstream video systems are separate. A healthy webcam
uplink does not prove the two downstream eye feeds work, and healthy eye feeds
do not prove that webcam video reaches the RCT.

## 3. Connection, Security, And Session Shape

The client connects to Theia over TLS with a private CA and a client
certificate. The server exposes a handshake service plus the relay service.
The handshake reports versions and physical capabilities; the relay service
owns the live streams and control RPCs.

Important RPCs are:

- `Handshake/GetCapabilities` for version, profile, and feature discovery;
- `Relay/CaptureVideo` for each downstream eye;
- `Relay/CaptureAudio` for downstream speaker audio;
- `Relay/StreamKeyboard` and `Relay/StreamMouse` for input;
- `Relay/StreamWebcamMedia` for bundled upstream camera and microphone;
- `Relay/StreamMicrophone` for the explicit microphone-only path;
- capture-power, calibration, recovery, and diagnostics RPCs.

The client starts long-running tasks for the requested media and input paths.
Queues are deliberately bounded. Live interaction is more important than
draining every packet that was once captured.

## 4. How Downstream Video Works

Theia expects two stable udev-created device names:

- `/dev/lesavka_l_eye`
- `/dev/lesavka_r_eye`

The server installer discovers the two GC311 capture devices, identifies them
by their physical udev path tags, and writes `/etc/udev/rules.d/85-gc311.rules`
so left and right do not change when `/dev/videoN` numbering changes.

When a client opens `CaptureVideo`, the server:

1. validates the requested eye and source;
2. acquires a capture-power lease;
3. waits a bounded time for the expected V4L2 device;
4. opens the capture card with GStreamer;
5. preserves the card's H.264 elementary stream instead of decoding and
   re-encoding it on Theia;
6. publishes packets through a shared per-source hub; and
7. drops packets under pressure rather than building interactive latency.

The client opens one RPC for each eye. It sends packets to a bounded display
queue. If that queue overflows, it drops predicted frames and waits for an IDR
frame before resuming, which prevents a corrupt decoder recovery sequence.
The installed workstation currently exposes `vah264dec`, backed by Intel's iHD
VA-API driver, and also has `avdec_h264` for explicit software diagnostics.

## 5. How Bundled Upstream Webcam Media Works

Webcam sessions use the v2 bundled path by default. The camera and microphone
are captured by one client-owned media pipeline and share the client's capture
clock. A bundle contains an optional video frame and the audio packets captured
beside it.

The important rules are:

- camera-enabled sessions imply microphone capture when UAC is available;
- camera audio is not flushed as an unrelated standalone stream;
- source and handoff queues keep fresh media rather than old media;
- client capture timestamps are the A/V sync truth;
- Theia maps that timeline to one new local playout epoch;
- only explicit UVC/UAC device-path offsets may move one output relative to the
  other; and
- stale or internally inconsistent mixed bundles are dropped coherently.

Theia's planner schedules audio and video from the shared timeline, then hands
approved packets to UAC and UVC sinks. The sinks should be simple consumers:
they must not accumulate private multi-second backlogs or create a second sync
model.

Microphone-only operation remains a separate supported mode. The old split
camera/microphone webcam uplink exists only as a compatibility escape hatch.

## 6. Why The Upstream Webcam Feed Has Been A Long-Running Problem

The original upstream implementation sent camera and microphone through
independent streams. Each side could establish or retain its own timing anchor,
queue, recovery state, and output delay. That made several failure modes
possible:

- a new session could inherit an old stream's playout epoch;
- one side could start while the other waited;
- stale audio or video could be drained to "catch up";
- capture timestamps could be replaced by late enqueue or network time;
- independent freshness decisions could preserve smooth playback while losing
  lip sync; and
- large static calibration values could hide architectural backlog.

Those failures produced seconds-scale delay and A/V skew, including observed
browser-visible failures near eight to ten seconds. The bundled v2 redesign
removed the independent-stream timing contract, added latest/freshness-bounded
queues, reset timing state on session replacement, made audio-master/video-
follower behavior explicit, and added device-level sync probes.

That redesign fixed the class of problem, but a newer sink handoff regression
reintroduced a different kind of video backlog.

### The Latest Confirmed Upstream Failure

Before the current patch, live diagnostics showed:

- client camera and microphone capture was connected and streaming;
- client delivery age was roughly `40-79 ms`;
- Theia's microphone sink lateness p95 was about `1.4 ms`; but
- Theia's camera sink lateness p95 was about `7661.5 ms`.

This proves the webcam, bundled network transport, and shared capture clock were
not the bottleneck. Video was becoming late inside Theia at the sink handoff.

The immediate cause was the newly auto-enabled optional HDMI camera mirror.
The camera relay synchronously performed the equivalent of:

```text
push frame to UVC
push the same frame to HDMI/KMS
```

If the KMS/HDMI mirror blocked, the task could not return to feed the next UVC
frame. A 32-frame FIFO handoff and a direct-MJPEG normalization pull that could
wait up to 50 ms amplified the stall into a deep video backlog. Audio continued
through its own sink handoff, so the result was missing or many-seconds-late
video rather than a transport disconnect.

### The `0.27.6` Correction

The correction is pushed to Gitea `master` at revision `d5b1eca`:

- UVC remains the synchronous primary camera output;
- HDMI mirroring runs in an isolated worker with only one pending latest frame;
- a blocked mirror can replace/drop its own stale frame but cannot stall UVC;
- the v2 scheduled-video handoff is a latest-value watch channel rather than a
  32-frame FIFO; and
- direct MJPEG normalization is nonblocking by default (`0 ms` pull timeout).

The client installer also now accepts a kernel module that is already loaded in
`/sys/module` even when `modinfo` cannot find metadata for the running kernel.
That installer fix is why the complete patch release is `0.27.6` rather than
`0.27.5`.

At the start of the 0.27.7 integrity work, the operator workstation was running
`lesavka-client 0.27.6` and Theia was still running `lesavka-server 0.27.4` at
revision `7011b5c`. Release artifacts and live `lesavka-relayctl version`
output, rather than this historical incident snapshot, are authoritative for
the current deployment state.

### The Post-Spool UVC Integrity Failure

The seconds-late sink handoff was not the only upstream video defect. A
simultaneous boundary capture on 2026-07-19 proved that
`/run/lesavka-uvc-frame.mjpg` was clean while the RCT-side UVC capture contained
black frames, grey lower bands, multi-frame splices, and a 184-second freeze.
The corruption was therefore introduced after the spool, in the helper,
gadget, USB, or host-consumption leg. The preserved evidence is under
`artifacts/webm-tear-analysis/20260719-014411/boundary-spool-vs-uvc/`.

The 0.27.6 code had five concrete contract defects that could create or hide
that failure:

- bulk sizing was selected from the existence of configfs `streaming_bulk`,
  even when its value was `0`;
- the gadget script swallowed a failed `streaming_bulk=1` write;
- explicit frame-size overrides could exceed the descriptor/wire cap;
- invalid, oversized, stale, or unreadable frames immediately replaced the
  last good frame with black idle video; and
- core, spool, and helper sizing calculations could silently drift apart.

### The `0.27.7` Integrity Contract

Release 0.27.7 makes the helper-published live contract authoritative. The UVC
helper reads configfs values and the browser's actual UVC probe/commit, then
atomically publishes `/run/lesavka-uvc-contract.json`. Both helper and server
spool enforce the smallest of the requested cap, descriptor cap, committed cap,
transport budget, and hard safety maximum. Missing configfs is isochronous and
unverified, never implicitly bulk. `lesavka-core.sh` now verifies bulk writeback
and restores isochronous packet sizing if the requested mode does not stick.

The output behavior is also explicit:

- a transient invalid, oversized, stale, or unreadable spool update freezes the
  last verified frame;
- black idle appears only after `LESAVKA_UVC_IDLE_AFTER_MS`, default `2000`;
- every thirtieth accepted frame receives a complete JPEG structure and APP4
  CRC check by default; and
- helper DQBUF/QBUF failures, `V4L2_BUF_FLAG_ERROR`, substitutions, strict
  validation failures, and kernel UVC/UDC events are counted.

Operational evidence is available without an RCT:

- `/run/lesavka-uvc-contract.json` is the live negotiated payload contract;
- `/run/lesavka-uvc-video-stats.json` contains helper and kernel counters;
- `relayctl upstream-sync` prints the contract and integrity status;
- the launcher's UVC chip reports `Holding`, `Integrity fault`, `Contract
  fault`, or `Unverified` from real server evidence; and
- `LESAVKA_UVC_HANDOFF_AUDIT=1` enables a bounded frame-by-frame JSONL ring at
  `/run/lesavka-uvc-handoff.jsonl` for lab diagnosis.

Marked synthetic frames now carry sequence/band blocks every 64 rows and an
APP4 sequence plus payload CRC. `scripts/manual/analyze_marked_capture.py`
classifies native MJPEG frames as intact, truncated, spliced, stale repeat, or
idle black and reports the first bad row or splice seam when recoverable. The
synthetic RCT probe's `--deep-capture` mode preserves native MJPEG, uvcvideo
debug logs, usbmon traffic, USB power state, and before/after device snapshots.

These changes close the known software defects and observability gaps. They do
not by themselves prove the RCT-side USB path is clean. The 0.27.7 hardware A/B,
mode-matrix soak, browser probe, and Google Meet validation remain release
gates. A host-controller drop or host decoder bug can still occur after the
gadget has successfully queued a verified frame. The honest long-term answer is
to retain a small Linux verification host in the lab when possible; otherwise
the calibrated server telemetry is the available integrity contract.

### The `0.27.8` Proof Candidate

End-to-end preparation exposed a separate client hardware-selection defect:
FFmpeg advertising `hevc_nvenc` did not prove that CUDA could encode a frame.
On the operator workstation the NVIDIA device was unavailable while the Intel
VA-API HEVC/H.264 route was healthy. Release 0.27.8 runtime-probes NVENC before
selection, falls back to the working GStreamer hardware encoder, and uses the
modern `key-int-max` property for `vah264enc` and `vah265enc`. The hardware
smoke harness now exercises the same usable route and retains encoded streams
plus hardware-decoded proof frames.

### The `0.27.9` Capture Power Fix

The controlled 2026-08-13 Theia reboot made the downstream outage concrete:
`relay.service` could not import `gpiod` from an obsolete home-directory
virtualenv, so GPIO27 never powered either GC311 capture card. The system
`libgpiod` Python binding remained healthy. Release 0.27.9 moves the helper
into the Lesavka tree, installs it with the server, requires `libgpiod`, and
runs it with `/usr/bin/python` before capture discovery. This removes the
unversioned virtualenv from the physical eye-feed power path.

## 7. The Current Downstream Video Failure

The current blank downstream feeds fail before transport or decoding.

On 2026-08-12, direct mutually authenticated `CaptureVideo` probes returned:

```text
eye-l device /dev/lesavka_l_eye was not ready within 5000 ms:
No such file or directory

eye-r device /dev/lesavka_r_eye was not ready within 5000 ms:
No such file or directory
```

Both RPCs reached the live server and produced a precise host-side error. The
server acquired the normal capture path but neither expected V4L2 device became
available. No H.264 packets were sent to the workstation. The workstation's
hardware decoder is present and working, so changing the local renderer cannot
fix this incident.

The capture-power control provides a second independent confirmation. Theia
reports the configured GPIO unit as `relay.service`, with state
`failed/failed`. A `ForceOn` control request momentarily changed the unit to
`active/running`, but it returned to `failed/failed` within twelve seconds. A
second attempt followed immediately by a `CaptureVideo` request still produced
no left-eye device during the five-second readiness window, and the power unit
again ended in `failed/failed`. This is not simply the normal auto-power idle
state: the unit cannot remain healthy long enough for either card to enumerate.

Theia's SSH listener is also unhealthy. TCP reaches the real Theia SSH port
`2208` over the configured route, but every connection is reset before the SSH
banner/authentication exchange. At the same time:

- Lesavka gRPC on `50051` remains available;
- node exporter on `9100` remains available;
- the host reports ample free memory, negligible load, and no socket/resource
  exhaustion; and
- the NVMe filesystem and normal network interface remain visible.

This combination points to stuck host service/GPIO/device state, not a dead
host or a client connectivity problem. A controlled Theia reboot is justified
because it can restore the SSH daemon, capture-power unit, and capture-card
enumeration in one bounded operation. It is not yet proof that the capture
hardware itself is healthy. If the devices remain absent after reboot, the next
checks are the `relay.service` failure reason, capture-card power, USB
enumeration, V4L2 nodes, udev path tags/rules, and physical card/cable state.

The 2026-08-13 controlled reboot completed that branch of the diagnosis. SSH
returned, but both cards remained absent and `relay.service` failed with
`ModuleNotFoundError: No module named 'gpiod'`. Its unit referenced a private
Python 3.14 virtualenv that had also lost `pip`; meanwhile `/usr/bin/python`
successfully imported the installed `libgpiod` binding. This proves the common
downstream failure was capture-card power automation, not two simultaneous
card failures, H.264 transport, or client decoding. Release 0.27.9 makes the
working system binding and a versioned helper the installed service contract.

## 8. Recovery And Deployment Sequence

The safe completion sequence for this incident is:

1. Perform a controlled reboot of Theia. Do not hard-reset the UVC/UAC gadget
   separately unless later evidence specifically requires it.
2. Wait for SSH, gRPC, and node-exporter health.
3. Over SSH, inspect `lsusb`, `v4l2-ctl --list-devices`, `/dev/video*`, and the
   two `/dev/lesavka_*_eye` symlinks.
4. Use `ssh theia`, whose alias targets `38.28.125.112:2208`. Port `2277` is a
   jump route for other titan hosts and is not a Theia health probe.
5. Pull Gitea `master` into Theia's existing checkout without overwriting
   unrelated local work.
6. Run `scripts/install/server.sh` as the trusted deployment path. Preserve the
   already-attached USB gadget unless a controlled rebuild is explicitly
   required.
7. Confirm Theia reports server version `0.27.9`, the pushed release revision,
   direct MJPEG normalizer timeout `0`, and a coherent UVC contract.
8. Open both downstream RPCs and prove that each emits changing, decodable H.264
   frames.
9. Run a real bundled webcam session and confirm that camera sink lateness no
   longer grows into seconds.
10. Run marked native-UVC A/B capture and the four-mode soak with zero
    truncated, spliced, or unexpected-black frames.
11. Verify the RCT receives live UVC video and UAC audio, then run the mirrored
   browser probe and a manual Google Meet check.

## 9. What Is Still Missing Before Lesavka Is Done

Lesavka is feature-rich and heavily tested, but it is not done until the live
hardware contract is repeatable. The remaining work falls into five groups.

### A. Install And Version Parity

- Push and deploy `0.27.9` through the client/server install scripts.
- Confirm client/server version and revision in every hardware probe artifact.
- Eliminate the current state where a fixed client talks to an unfixed server.

### B. End-To-End Media Acceptance

- Both downstream eye feeds must open, keep changing, decode without a black
  screen, and recover from an interrupted capture source.
- The webcam must appear on the RCT as nonblack, supported UVC video.
- Microphone audio must remain intelligible and synchronized with camera video.
- The mirrored browser probe must meet the project thresholds: preferred median
  near `35 ms`, acceptable p95 absolute skew within `80 ms`, and no sustained
  split near `1000 ms`.
- Manual Google Meet testing must agree with the device-level probes.

### C. Remaining Runtime/Diagnostics Work

- Add first-bundle, first-video-feed, first-audio-push, stale-drop, and queue-age
  counters for bundled sessions.
- Add concise per-session first camera/microphone packet and first UVC/UAC sink
  handoff evidence.
- Surface `Starting`, `Healing`, `Flowing`, `Lagging`, `Dropping`, and `Stale`
  from real path evidence rather than inferred UI state.
- Flush and stop UAC cleanly on close, replacement, and recovery.
- Prove disconnect closes all media tasks owned by that session.

### D. Compatibility And Profile Closure

- Prove normal bundled-capable webcam sessions never use the legacy split RPCs.
- Complete focused server bundled-stream acceptance tests.
- Run and record the Theia/Tethys mode matrix for `1280x720@20/30` and
  `1920x1080@20/30` before changing advertised profiles.
- Preserve the small operator calibration nudges after the static hardware
  baselines are locked.

### E. Operational Reliability

- Restore and verify Theia SSH, since the required installer and incident
  diagnostics depend on it in this lab.
- Add a bounded, documented way to recover a wedged SSH daemon without turning
  an application outage into an improvised host power operation.
- Make capture-card absence a first-class health state visible before the user
  opens a blank preview.
- Keep device-level probe artifacts as release evidence; internal queue and
  planner telemetry alone cannot declare the product fixed.
- Calibrate the helper/kernel integrity counters against simultaneous RCT
  fault-injection captures so every visible corruption class has a server-side
  signal and documented residual blind spot.

## 10. Definition Of Done

Lesavka can be called done when a freshly installed client and server can pass
this sequence without source edits, secret manual timing adjustments, or hidden
host repair:

1. handshake and versions match;
2. both eye feeds show changing remote video;
3. downstream audio is audible;
4. keyboard and mouse control the RCT;
5. selected webcam/microphone settings remain authoritative;
6. RCT applications receive live UVC/UAC media;
7. mirrored-probe sync/freshness/smoothness gates pass;
8. Google Meet passes manual audio, video, and lip-sync validation;
9. disconnect/reconnect and device changes recover without stale media; and
10. diagnostics identify the failed physical stage when any item breaks.

Until that sequence passes on the installed `0.27.9` client/server pair, the
current release should be described as a validated code correction awaiting
hardware deployment and end-to-end acceptance, not as a completed product fix.
