Proof Portal

Project overview

Omarchy

ProbeLabs65 findings · 95 requirements

A proof layer — requirements, tests and verified fixes — for two of Omarchy's subsystems: the application menu (launcher scripts, QML model, JSONC config, search and selection) and the lock screen (lock scripts, QML, PAM authentication). Scope is deliberately limited to those components of omacom/omarchy; the rest of the distribution is not covered.

Inbox
ReportR-626GitHub issuePublicDecided· Out of scope

Add pre/post upgrade diff to detect regressions

Reported by andyholst · filed Sep 12, 2026 · open at source

Decided: Out of scope — decided by agents · verified 3 days ago.

At a glance

1 · What is claimed?
Only the report text below — no validator has restated it as checkable claims yet.
2 · What does it affect?
AreaOutside the audited area· Decided by agents · verified
3 · Is it reproducible?
No validation has been run yet.
4 · What remains unclear?
  • Not validated yet: no claims, no reproducer.
5 · What happens next?
Decided by agents and verified by a second, independent agent — see the decision.

Original report

Add pre/post upgrade diff to detect regressionsrevision 1 · captured 2026-09-28 21:23 UTC via poll

What's Missing

After an upgrade, users discover broken services days later. Omarchy has no automated regression detection. Existing crash reporting (omarchy-crash-watch, diagnose-crash skill) catches crashes but not silent failures.

How Omarchy Reports Today

Omarchy has established reporting conventions in default/agents/skills/diagnose-crash/reporting.md:

Sphere of control: omarchy-* commands, Quickshell shell, Hyprland config, themes, install/migration scripts, packaging.

Not Omarchy's bug: Crashes in third-party apps (browser, file manager, GNOME/Qt libs) unless Omarchy's packaging/configuration caused it.

Before filing: Search existing issues (open AND closed), confirm with user, include omarchy debug output.

Signing: Reports named with model/agent harness.

The Proposal: Pre/Post Upgrade Diff

1. Pre-Upgrade Snapshot

Add to omarchy-update before system pkgs upgrade:

systemctl list-units --state=active --no-legend > /var/lib/omarchy/pre-update-services.txt
systemctl list-unit-files --state=enabled --no-legend >> /var/lib/omarchy/pre-update-services.txt
lsmod > /var/lib/omarchy/pre-update-modules.txt
pacman -Qqe > /var/lib/omarchy/pre-update-packages.txt

2. Post-Upgrade Check

New systemd service omarchy-update-check.service:

[Unit]
After=graphical-session.target
ConditionPathExists=/var/lib/omarchy/pre-update-services.txt

[Service]
Type=oneshot
ExecStart=/usr/bin/omarchy-update-check

3. What the Check Does

diff /var/lib/omarchy/pre-update-services.txt <(systemctl list-units --state=active --no-legend)
diff /var/lib/omarchy/pre-update-modules.txt <(lsmod)
diff /var/lib/omarchy/pre-update-packages.txt <(pacman -Qqe)

4. Filter Intentional Changes

Migration scripts declare what they change:

omarchy_migrate_declare_change "service:old-name:renamed:new-name"
omarchy_migrate_declare_change "service:deprecated:removed"

Plus suppression list at /usr/share/omarchy/update-suppressions.txt.

Diff minus filtered = real regressions.

5. Notification

⚠️ After upgrade to 4.0.3, these services stopped working:
   - coolercontrold.service (was active, now inactive)
   - nvidia-powerd.service (was enabled, now disabled)

Click to diagnose and file bug report

6. Bug Report Generation (Agent Integration)

Reuse existing reporting logic from `diagnose-crash/reporting.md`.

Determine Where to File

Is the broken component in Omarchy's sphere?
  ├── YES — omarchy-* command, Quickshell, Hyprland config, theme, migration
  │   └── File to: github.com/omacom/omarchy/issues
  │
  ├── NO — but Omarchy installs/configures it (coolercontrold, nvidia-utils, etc.)
  │   └── File upstream bug (bugs.archlinux.org / third-party tracker)
  │   └── CREATE Omarchy tracking issue (label: upstream, tracking)
  │
  └── NO — unrelated to Omarchy
      └── File upstream bug only

Search Before Filing

Search Omarchy issues:

gh search issues --repo omacom/omarchy "<service> <state>"
gh issue list --repo omacom/omarchy --state all --search "<service>"

Search Arch bugs:
- Search bugs.archlinux.org for package name + error
- If match found, comment with new evidence
- If not, file new against correct package/project

What the Bug Report Includes

FieldSource
System detailsomarchy debug
What's wrongDiff output
Steps to reproduceRun omarchy-update, reboot, check systemctl
Regression diffPre/post snapshot diff
Migration logjournalctl -u omarchy-migrate excerpt

Labels: bug, regression, auto-detected

Title format: [auto] <service> <state> after <version> update on <hardware>

User Consent

Agent shows exact title and body. Waits for explicit yes.

Signing

Filed by <model> via <agent harness>.

Devil's Advocate

See shared devil's advocate section in Issue #11594 for common concerns (gh fallback, Arch tracker specifics, AUR detection, etc.).

Diff-Specific

D1. User Manually Disabled: If user manually disabled a service, diff flags it as regression. Check first: is this user action or upgrade-caused?

D2. Inactive but Enabled: Diff catches was-active-now-inactive. But what if service is enabled but inactive (failed to start)? Check both states.

D3. Multiple Regressions: Diff catches 5 services stopped. One report with all issues, or one per issue?

Benefits

  1. Immediate detection — problems surface on boot, not weeks later
  2. Complete reports — diff + logs + system info included
  3. Right destination — Omarchy bugs to Omarchy, Arch bugs to Arch
  4. Tracking issues — upstream bugs tracked for visibility
  5. No back-and-forth — maintainers get full context first time
  6. Lower barrier — user clicks notification, agent files report

Scope

  • Modify omarchy-update to take pre-snapshot
  • New omarchy-update-check.service (systemd)
  • New omarchy-update-check command
  • New omarchy_migrate_declare_change helper for migrations
  • Agent integration reusing diagnose-crash/reporting.md conventions

Open Questions

  1. Where to store snapshots? /var/lib/omarchy/ (persistent, ~100KB)
  2. How long to keep? Last 2 snapshots (before/after current)
  3. What if upgrade fails mid-way? Post-check detects partial state, reports incomplete
  4. First install? No baseline exists, skip check
  5. Non-btrfs systems? Snapshot still works, diff still applies

Related

Issue #11594 — Silent failure detection (complements this issue with shared devil's advocate)

View raw text
## What's Missing

After an upgrade, users discover broken services days later. Omarchy has no automated regression detection. Existing crash reporting (`omarchy-crash-watch`, `diagnose-crash` skill) catches crashes but not silent failures.

## How Omarchy Reports Today

Omarchy has established reporting conventions in `default/agents/skills/diagnose-crash/reporting.md`:

**Sphere of control**: `omarchy-*` commands, Quickshell shell, Hyprland config, themes, install/migration scripts, packaging.

**Not Omarchy's bug**: Crashes in third-party apps (browser, file manager, GNOME/Qt libs) unless Omarchy's packaging/configuration caused it.

**Before filing**: Search existing issues (open AND closed), confirm with user, include `omarchy debug` output.

**Signing**: Reports named with model/agent harness.

## The Proposal: Pre/Post Upgrade Diff

### 1. Pre-Upgrade Snapshot

Add to `omarchy-update` before system pkgs upgrade:

```bash
systemctl list-units --state=active --no-legend > /var/lib/omarchy/pre-update-services.txt
systemctl list-unit-files --state=enabled --no-legend >> /var/lib/omarchy/pre-update-services.txt
lsmod > /var/lib/omarchy/pre-update-modules.txt
pacman -Qqe > /var/lib/omarchy/pre-update-packages.txt
```

### 2. Post-Upgrade Check

New systemd service `omarchy-update-check.service`:

```ini
[Unit]
After=graphical-session.target
ConditionPathExists=/var/lib/omarchy/pre-update-services.txt

[Service]
Type=oneshot
ExecStart=/usr/bin/omarchy-update-check
```

### 3. What the Check Does

```bash
diff /var/lib/omarchy/pre-update-services.txt <(systemctl list-units --state=active --no-legend)
diff /var/lib/omarchy/pre-update-modules.txt <(lsmod)
diff /var/lib/omarchy/pre-update-packages.txt <(pacman -Qqe)
```

### 4. Filter Intentional Changes

Migration scripts declare what they change:
```bash
omarchy_migrate_declare_change "service:old-name:renamed:new-name"
omarchy_migrate_declare_change "service:deprecated:removed"
```

Plus suppression list at `/usr/share/omarchy/update-suppressions.txt`.

Diff minus filtered = real regressions.

### 5. Notification

```
⚠️ After upgrade to 4.0.3, these services stopped working:
   - coolercontrold.service (was active, now inactive)
   - nvidia-powerd.service (was enabled, now disabled)

Click to diagnose and file bug report
```

### 6. Bug Report Generation (Agent Integration)

**Reuse existing reporting logic from `diagnose-crash/reporting.md`.**

#### Determine Where to File

```
Is the broken component in Omarchy's sphere?
  ├── YES — omarchy-* command, Quickshell, Hyprland config, theme, migration
  │   └── File to: github.com/omacom/omarchy/issues
  │
  ├── NO — but Omarchy installs/configures it (coolercontrold, nvidia-utils, etc.)
  │   └── File upstream bug (bugs.archlinux.org / third-party tracker)
  │   └── CREATE Omarchy tracking issue (label: upstream, tracking)
  │
  └── NO — unrelated to Omarchy
      └── File upstream bug only
```

#### Search Before Filing

**Search Omarchy issues:**
```bash
gh search issues --repo omacom/omarchy "<service> <state>"
gh issue list --repo omacom/omarchy --state all --search "<service>"
```

**Search Arch bugs:**
- Search bugs.archlinux.org for package name + error
- If match found, comment with new evidence
- If not, file new against correct package/project

#### What the Bug Report Includes

| Field | Source |
|-------|--------|
| System details | `omarchy debug` |
| What's wrong | Diff output |
| Steps to reproduce | Run omarchy-update, reboot, check systemctl |
| Regression diff | Pre/post snapshot diff |
| Migration log | `journalctl -u omarchy-migrate` excerpt |

**Labels**: `bug`, `regression`, `auto-detected`

**Title format**: `[auto] <service> <state> after <version> update on <hardware>`

#### User Consent

Agent shows exact title and body. Waits for explicit yes.

#### Signing

`Filed by <model> via <agent harness>.`

## Devil's Advocate

See shared devil's advocate section in Issue #11594 for common concerns (gh fallback, Arch tracker specifics, AUR detection, etc.).

### Diff-Specific

**D1. User Manually Disabled**: If user manually disabled a service, diff flags it as regression. Check first: is this user action or upgrade-caused?

**D2. Inactive but Enabled**: Diff catches was-active-now-inactive. But what if service is enabled but inactive (failed to start)? Check both states.

**D3. Multiple Regressions**: Diff catches 5 services stopped. One report with all issues, or one per issue?

## Benefits

1. **Immediate detection** — problems surface on boot, not weeks later
2. **Complete reports** — diff + logs + system info included
3. **Right destination** — Omarchy bugs to Omarchy, Arch bugs to Arch
4. **Tracking issues** — upstream bugs tracked for visibility
5. **No back-and-forth** — maintainers get full context first time
6. **Lower barrier** — user clicks notification, agent files report

## Scope

- Modify `omarchy-update` to take pre-snapshot
- New `omarchy-update-check.service` (systemd)
- New `omarchy-update-check` command
- New `omarchy_migrate_declare_change` helper for migrations
- Agent integration reusing `diagnose-crash/reporting.md` conventions

## Open Questions

1. **Where to store snapshots?** `/var/lib/omarchy/` (persistent, ~100KB)
2. **How long to keep?** Last 2 snapshots (before/after current)
3. **What if upgrade fails mid-way?** Post-check detects partial state, reports incomplete
4. **First install?** No baseline exists, skip check
5. **Non-btrfs systems?** Snapshot still works, diff still applies

## Related

Issue #11594 — Silent failure detection (complements this issue with shared devil's advocate)

Validation

Not checked yet. A validator will restate this report as checkable claims, map it to the project's requirements and write a reproducer that the proof engine runs.

Linked

Not linked to any known issue, other report or change yet.

Decision

Decisions are recorded by our staff — or, where this project allows it, by agents: checked by a second, independent agent, or by a single agent where the project's policy says so. Any decision reached so far is shown below.

  • Out of scopeDecided by agents · verified · 2026-10-09 12:21 UTC