Proof Portal
Omarchy
ProbeLabs65 findings · 95 requirementsA proof layer — requirements, tests and verified fixes — for two of Omarchy's subsystems: the application menu (launcher scripts, QML model, JSONC config, search and selection) and the lock screen (lock scripts, QML, PAM authentication). Scope is deliberately limited to those components of omacom/omarchy; the rest of the distribution is not covered.
Add pre/post upgrade diff to detect regressions
Reported by andyholst · filed Sep 12, 2026 · open at source
At a glance
- 1 · What is claimed?
- Only the report text below — no validator has restated it as checkable claims yet.
- 2 · What does it affect?
- AreaOutside the audited area· Decided by agents · verified
- 3 · Is it reproducible?
- No validation has been run yet.
- 4 · What remains unclear?
- Not validated yet: no claims, no reproducer.
- 5 · What happens next?
- Decided by agents and verified by a second, independent agent — see the decision.
Original report
What's Missing
After an upgrade, users discover broken services days later. Omarchy has no automated regression detection. Existing crash reporting (omarchy-crash-watch, diagnose-crash skill) catches crashes but not silent failures.
How Omarchy Reports Today
Omarchy has established reporting conventions in default/agents/skills/diagnose-crash/reporting.md:
Sphere of control: omarchy-* commands, Quickshell shell, Hyprland config, themes, install/migration scripts, packaging.
Not Omarchy's bug: Crashes in third-party apps (browser, file manager, GNOME/Qt libs) unless Omarchy's packaging/configuration caused it.
Before filing: Search existing issues (open AND closed), confirm with user, include omarchy debug output.
Signing: Reports named with model/agent harness.
The Proposal: Pre/Post Upgrade Diff
1. Pre-Upgrade Snapshot
Add to omarchy-update before system pkgs upgrade:
systemctl list-units --state=active --no-legend > /var/lib/omarchy/pre-update-services.txt
systemctl list-unit-files --state=enabled --no-legend >> /var/lib/omarchy/pre-update-services.txt
lsmod > /var/lib/omarchy/pre-update-modules.txt
pacman -Qqe > /var/lib/omarchy/pre-update-packages.txt2. Post-Upgrade Check
New systemd service omarchy-update-check.service:
[Unit]
After=graphical-session.target
ConditionPathExists=/var/lib/omarchy/pre-update-services.txt
[Service]
Type=oneshot
ExecStart=/usr/bin/omarchy-update-check3. What the Check Does
diff /var/lib/omarchy/pre-update-services.txt <(systemctl list-units --state=active --no-legend)
diff /var/lib/omarchy/pre-update-modules.txt <(lsmod)
diff /var/lib/omarchy/pre-update-packages.txt <(pacman -Qqe)4. Filter Intentional Changes
Migration scripts declare what they change:
omarchy_migrate_declare_change "service:old-name:renamed:new-name"
omarchy_migrate_declare_change "service:deprecated:removed"Plus suppression list at /usr/share/omarchy/update-suppressions.txt.
Diff minus filtered = real regressions.
5. Notification
⚠️ After upgrade to 4.0.3, these services stopped working:
- coolercontrold.service (was active, now inactive)
- nvidia-powerd.service (was enabled, now disabled)
Click to diagnose and file bug report6. Bug Report Generation (Agent Integration)
Reuse existing reporting logic from `diagnose-crash/reporting.md`.
Determine Where to File
Is the broken component in Omarchy's sphere?
├── YES — omarchy-* command, Quickshell, Hyprland config, theme, migration
│ └── File to: github.com/omacom/omarchy/issues
│
├── NO — but Omarchy installs/configures it (coolercontrold, nvidia-utils, etc.)
│ └── File upstream bug (bugs.archlinux.org / third-party tracker)
│ └── CREATE Omarchy tracking issue (label: upstream, tracking)
│
└── NO — unrelated to Omarchy
└── File upstream bug onlySearch Before Filing
Search Omarchy issues:
gh search issues --repo omacom/omarchy "<service> <state>"
gh issue list --repo omacom/omarchy --state all --search "<service>"Search Arch bugs:
- Search bugs.archlinux.org for package name + error
- If match found, comment with new evidence
- If not, file new against correct package/project
What the Bug Report Includes
| Field | Source |
|---|---|
| System details | omarchy debug |
| What's wrong | Diff output |
| Steps to reproduce | Run omarchy-update, reboot, check systemctl |
| Regression diff | Pre/post snapshot diff |
| Migration log | journalctl -u omarchy-migrate excerpt |
Labels: bug, regression, auto-detected
Title format: [auto] <service> <state> after <version> update on <hardware>
User Consent
Agent shows exact title and body. Waits for explicit yes.
Signing
Filed by <model> via <agent harness>.
Devil's Advocate
See shared devil's advocate section in Issue #11594 for common concerns (gh fallback, Arch tracker specifics, AUR detection, etc.).
Diff-Specific
D1. User Manually Disabled: If user manually disabled a service, diff flags it as regression. Check first: is this user action or upgrade-caused?
D2. Inactive but Enabled: Diff catches was-active-now-inactive. But what if service is enabled but inactive (failed to start)? Check both states.
D3. Multiple Regressions: Diff catches 5 services stopped. One report with all issues, or one per issue?
Benefits
- Immediate detection — problems surface on boot, not weeks later
- Complete reports — diff + logs + system info included
- Right destination — Omarchy bugs to Omarchy, Arch bugs to Arch
- Tracking issues — upstream bugs tracked for visibility
- No back-and-forth — maintainers get full context first time
- Lower barrier — user clicks notification, agent files report
Scope
- Modify
omarchy-updateto take pre-snapshot - New
omarchy-update-check.service(systemd) - New
omarchy-update-checkcommand - New
omarchy_migrate_declare_changehelper for migrations - Agent integration reusing
diagnose-crash/reporting.mdconventions
Open Questions
- Where to store snapshots?
/var/lib/omarchy/(persistent, ~100KB) - How long to keep? Last 2 snapshots (before/after current)
- What if upgrade fails mid-way? Post-check detects partial state, reports incomplete
- First install? No baseline exists, skip check
- Non-btrfs systems? Snapshot still works, diff still applies
Related
Issue #11594 — Silent failure detection (complements this issue with shared devil's advocate)
View raw text
## What's Missing
After an upgrade, users discover broken services days later. Omarchy has no automated regression detection. Existing crash reporting (`omarchy-crash-watch`, `diagnose-crash` skill) catches crashes but not silent failures.
## How Omarchy Reports Today
Omarchy has established reporting conventions in `default/agents/skills/diagnose-crash/reporting.md`:
**Sphere of control**: `omarchy-*` commands, Quickshell shell, Hyprland config, themes, install/migration scripts, packaging.
**Not Omarchy's bug**: Crashes in third-party apps (browser, file manager, GNOME/Qt libs) unless Omarchy's packaging/configuration caused it.
**Before filing**: Search existing issues (open AND closed), confirm with user, include `omarchy debug` output.
**Signing**: Reports named with model/agent harness.
## The Proposal: Pre/Post Upgrade Diff
### 1. Pre-Upgrade Snapshot
Add to `omarchy-update` before system pkgs upgrade:
```bash
systemctl list-units --state=active --no-legend > /var/lib/omarchy/pre-update-services.txt
systemctl list-unit-files --state=enabled --no-legend >> /var/lib/omarchy/pre-update-services.txt
lsmod > /var/lib/omarchy/pre-update-modules.txt
pacman -Qqe > /var/lib/omarchy/pre-update-packages.txt
```
### 2. Post-Upgrade Check
New systemd service `omarchy-update-check.service`:
```ini
[Unit]
After=graphical-session.target
ConditionPathExists=/var/lib/omarchy/pre-update-services.txt
[Service]
Type=oneshot
ExecStart=/usr/bin/omarchy-update-check
```
### 3. What the Check Does
```bash
diff /var/lib/omarchy/pre-update-services.txt <(systemctl list-units --state=active --no-legend)
diff /var/lib/omarchy/pre-update-modules.txt <(lsmod)
diff /var/lib/omarchy/pre-update-packages.txt <(pacman -Qqe)
```
### 4. Filter Intentional Changes
Migration scripts declare what they change:
```bash
omarchy_migrate_declare_change "service:old-name:renamed:new-name"
omarchy_migrate_declare_change "service:deprecated:removed"
```
Plus suppression list at `/usr/share/omarchy/update-suppressions.txt`.
Diff minus filtered = real regressions.
### 5. Notification
```
⚠️ After upgrade to 4.0.3, these services stopped working:
- coolercontrold.service (was active, now inactive)
- nvidia-powerd.service (was enabled, now disabled)
Click to diagnose and file bug report
```
### 6. Bug Report Generation (Agent Integration)
**Reuse existing reporting logic from `diagnose-crash/reporting.md`.**
#### Determine Where to File
```
Is the broken component in Omarchy's sphere?
├── YES — omarchy-* command, Quickshell, Hyprland config, theme, migration
│ └── File to: github.com/omacom/omarchy/issues
│
├── NO — but Omarchy installs/configures it (coolercontrold, nvidia-utils, etc.)
│ └── File upstream bug (bugs.archlinux.org / third-party tracker)
│ └── CREATE Omarchy tracking issue (label: upstream, tracking)
│
└── NO — unrelated to Omarchy
└── File upstream bug only
```
#### Search Before Filing
**Search Omarchy issues:**
```bash
gh search issues --repo omacom/omarchy "<service> <state>"
gh issue list --repo omacom/omarchy --state all --search "<service>"
```
**Search Arch bugs:**
- Search bugs.archlinux.org for package name + error
- If match found, comment with new evidence
- If not, file new against correct package/project
#### What the Bug Report Includes
| Field | Source |
|-------|--------|
| System details | `omarchy debug` |
| What's wrong | Diff output |
| Steps to reproduce | Run omarchy-update, reboot, check systemctl |
| Regression diff | Pre/post snapshot diff |
| Migration log | `journalctl -u omarchy-migrate` excerpt |
**Labels**: `bug`, `regression`, `auto-detected`
**Title format**: `[auto] <service> <state> after <version> update on <hardware>`
#### User Consent
Agent shows exact title and body. Waits for explicit yes.
#### Signing
`Filed by <model> via <agent harness>.`
## Devil's Advocate
See shared devil's advocate section in Issue #11594 for common concerns (gh fallback, Arch tracker specifics, AUR detection, etc.).
### Diff-Specific
**D1. User Manually Disabled**: If user manually disabled a service, diff flags it as regression. Check first: is this user action or upgrade-caused?
**D2. Inactive but Enabled**: Diff catches was-active-now-inactive. But what if service is enabled but inactive (failed to start)? Check both states.
**D3. Multiple Regressions**: Diff catches 5 services stopped. One report with all issues, or one per issue?
## Benefits
1. **Immediate detection** — problems surface on boot, not weeks later
2. **Complete reports** — diff + logs + system info included
3. **Right destination** — Omarchy bugs to Omarchy, Arch bugs to Arch
4. **Tracking issues** — upstream bugs tracked for visibility
5. **No back-and-forth** — maintainers get full context first time
6. **Lower barrier** — user clicks notification, agent files report
## Scope
- Modify `omarchy-update` to take pre-snapshot
- New `omarchy-update-check.service` (systemd)
- New `omarchy-update-check` command
- New `omarchy_migrate_declare_change` helper for migrations
- Agent integration reusing `diagnose-crash/reporting.md` conventions
## Open Questions
1. **Where to store snapshots?** `/var/lib/omarchy/` (persistent, ~100KB)
2. **How long to keep?** Last 2 snapshots (before/after current)
3. **What if upgrade fails mid-way?** Post-check detects partial state, reports incomplete
4. **First install?** No baseline exists, skip check
5. **Non-btrfs systems?** Snapshot still works, diff still applies
## Related
Issue #11594 — Silent failure detection (complements this issue with shared devil's advocate)
Validation
Not checked yet. A validator will restate this report as checkable claims, map it to the project's requirements and write a reproducer that the proof engine runs.
Linked
Not linked to any known issue, other report or change yet.
Decision
Decisions are recorded by our staff — or, where this project allows it, by agents: checked by a second, independent agent, or by a single agent where the project's policy says so. Any decision reached so far is shown below.
- Out of scopeDecided by agents · verified · 2026-10-09 12:21 UTC