Proof Portal

Sign in
Project overview

Omarchy

ProbeLabs65 findings · 95 requirements

A proof layer — requirements, tests and verified fixes — for two of Omarchy's subsystems: the application menu (launcher scripts, QML model, JSONC config, search and selection) and the lock screen (lock scripts, QML, PAM authentication). Scope is deliberately limited to those components of omacom/omarchy; the rest of the distribution is not covered.

Inbox
ReportR-5305GitHub issuePublicDecided· Out of scope

Proposal: Headless acceptance test runner for fast CI and safe local verification

Reported by Sebo1411 · filed Oct 10, 2026 · open at source

Open at source
Decided: Out of scope — decided by agents · verified 2 days ago.

At a glance

1 · What is claimed?
Only the report text below — no validator has restated it as checkable claims yet.
2 · What does it affect?
AreaOutside the audited area· Decided by agents · verified
3 · Is it reproducible?
No validation has been run yet.
4 · What remains unclear?
  • Not validated yet: no claims, no reproducer.
5 · What happens next?
Decided by agents and verified by a second, independent agent — see the decision.

Original report

Proposal: Headless acceptance test runner for fast CI and safe local verificationrevision 1 · captured 2026-10-10 16:58 UTC via poll

Background & Problem

Currently, visual acceptance tests in test/acceptance.d/ rely on the full ISO test harness in omarchy-iso running inside a QEMU/KVM virtual machine. As noted in `agents/skills/acceptance-tests.md`, running tests directly inside the active host desktop is unsafe because tests interact with active windows, grab focus, and modify shell configuration.

While full KVM ISO testing remains essential for installer and bare-metal lifecycle validation, it introduces significant friction for day-to-day development and PR workflows:
- Environment restriction: Standard GitHub Actions runners (and many non-bare-metal CI environments) lack nested KVM virtualization (/dev/kvm).
- Turnaround time: Booting the full live ISO image and setting up the environment takes 1–2 minutes per test run.
- Developer workflow: Local developers cannot quickly verify a single acceptance test in the background without launching the ISO harness.

Tested Solution (Headless Wayland Container PoC)

We explored and benchmarked running Omarchy's acceptance suite headlessly inside an isolated Arch Linux container using modern Hyprland (Aquamarine backend):

  1. Virtual Display & Seat:
  • Linux kernel vkms (Virtual Kernel Modesetting) module exposes a virtual DRM render node (/dev/dri/card0).
  • A lightweight seatd daemon manages session permissions without needing a full systemd-logind stack.
  1. Rendering:
  • Software rendering via Mesa's LLVMpipe (GALLIUM_DRIVER=llvmpipe / LIBGL_ALWAYS_SOFTWARE=1).
  1. Control & Capture:
  • Hyprland initializes on Virtual-1.
  • hyprctl IPC sockets connect reliably for window manipulation and state queries (hyprctl monitors, hyprctl layers).
  • grim captures pixel-perfect PNG frames, and tesseract performs OCR assertions.

Benchmark Results (Cloud VM Runner PoC)

In our testing on a standard cloud instance (4 vCPU, no physical GPU):
- Compositor startup latency: ~250ms to Wayland socket readiness.
- End-to-end execution time: ~1.5s – 2.0s per test (Hyprland init -> app launch -> hyprctl state query -> grim capture -> OCR assertion -> clean teardown).
- Isolation: Complete isolation from the host session, preventing any input hijacking or config mutation.

Proposed Plan

  1. Add a lightweight headless runner script (e.g. test/acceptance-headless [test-name]) backed by a minimal container/setup.
  2. Integrate this runner into GitHub Actions CI so that PRs can validate visual regressions on every commit within seconds, complementing the heavier ISO release test suite.

Would you be open to a PR introducing this headless container harness?

View raw text
### Background & Problem

Currently, visual acceptance tests in `test/acceptance.d/` rely on the full ISO test harness in `omarchy-iso` running inside a QEMU/KVM virtual machine. As noted in [`agents/skills/acceptance-tests.md`](https://github.com/omacom/omarchy/blob/master/agents/skills/acceptance-tests.md), running tests directly inside the active host desktop is unsafe because tests interact with active windows, grab focus, and modify shell configuration.

While full KVM ISO testing remains essential for installer and bare-metal lifecycle validation, it introduces significant friction for day-to-day development and PR workflows:
- **Environment restriction:** Standard GitHub Actions runners (and many non-bare-metal CI environments) lack nested KVM virtualization (`/dev/kvm`).
- **Turnaround time:** Booting the full live ISO image and setting up the environment takes 1–2 minutes per test run.
- **Developer workflow:** Local developers cannot quickly verify a single acceptance test in the background without launching the ISO harness.

---

### Tested Solution (Headless Wayland Container PoC)

We explored and benchmarked running Omarchy's acceptance suite headlessly inside an isolated Arch Linux container using modern Hyprland (Aquamarine backend):

1. **Virtual Display & Seat:**
   - Linux kernel `vkms` (Virtual Kernel Modesetting) module exposes a virtual DRM render node (`/dev/dri/card0`).
   - A lightweight `seatd` daemon manages session permissions without needing a full systemd-logind stack.
2. **Rendering:**
   - Software rendering via Mesa's LLVMpipe (`GALLIUM_DRIVER=llvmpipe` / `LIBGL_ALWAYS_SOFTWARE=1`).
3. **Control & Capture:**
   - Hyprland initializes on `Virtual-1`.
   - `hyprctl` IPC sockets connect reliably for window manipulation and state queries (`hyprctl monitors`, `hyprctl layers`).
   - `grim` captures pixel-perfect PNG frames, and `tesseract` performs OCR assertions.

---

### Benchmark Results (Cloud VM Runner PoC)

In our testing on a standard cloud instance (4 vCPU, no physical GPU):
- **Compositor startup latency:** ~250ms to Wayland socket readiness.
- **End-to-end execution time:** ~1.5s – 2.0s per test (Hyprland init -> app launch -> `hyprctl` state query -> `grim` capture -> OCR assertion -> clean teardown).
- **Isolation:** Complete isolation from the host session, preventing any input hijacking or config mutation.

---

### Proposed Plan

1. Add a lightweight headless runner script (e.g. `test/acceptance-headless [test-name]`) backed by a minimal container/setup.
2. Integrate this runner into GitHub Actions CI so that PRs can validate visual regressions on every commit within seconds, complementing the heavier ISO release test suite.

Would you be open to a PR introducing this headless container harness?

Validation

Not checked yet. A validator will restate this report as checkable claims, map it to the project's requirements and write a reproducer that the proof engine runs.

Linked

Not linked to any known issue, other report or change yet.

Decision

Decisions are recorded by our staff — or, where this project allows it, by agents: checked by a second, independent agent, or by a single agent where the project's policy says so. Any decision reached so far is shown below.

  • Out of scopeDecided by agents · verified · 2026-10-10 17:16 UTC