A smoke test's first run caught a real bug

Why a schedule, when CI already exists

Unit tests exercise your assumptions; a schedule exercises the world's. Vendor APIs change underneath you, and a weekly read-only scan with real credentials catches reachability problems and credential drift before a deploy depends on them.

The first dispatch reported the GitHub adapter unreachable. The credential it was using was the workflow's own GITHUB_TOKEN, minted by GitHub seconds earlier.

The bug: validity checked through identity

The adapter's health check called GET /user. For a personal access token that works fine. But the token GitHub Actions injects into a workflow is an app installation token, scoped to the repository, with no user identity at all. For that entire credential class, /user answers 403.

# user PAT                          # Actions installation token
GET /user                           GET /user
200 OK {"login": "you"}             403 Resource not accessible
                                        by integration

GET /rate_limit                     GET /rate_limit
200 OK                              200 OK

So the check was really testing whether the token belonged to a human. Every CI pipeline, ours included, authenticates with exactly the credential class it rejected.

Our unit suite couldn't have caught this: the mocks answered /user the way we assumed GitHub would. Only a real token from a real workflow could contradict the assumption.

The fix, and the line it refused to cross

The repaired check probes validity where identity is not assumed. GET /rate_limit answers 200 for any live token, user or installation. When /user succeeds, the adapter reports the human identity as before; when it answers 403, it falls back to the rate-limit probe and reports what the credential actually is: an app installation token, repo-scoped, without a user identity.

One property mattered more than the convenience: if both probes fail, the adapter still refuses. A fallback that turns two failures into a pass is not a fallback, it is an open door. The fix shipped as launchops-adapters 0.12.0 the same day. The re-dispatched run verified it against the same real token that found the bug: ok, github, 366 ms, four resource groups read.

What generalizes

• Validity and identity are different questions. Probe liveness on an endpoint that answers for every credential class you accept; report identity separately when it exists.
• A small scheduled smoke with real credentials, weekly and read-only, catches what mocks structurally cannot.
• When you add a fallback, decide first what total failure should do. Here it still refuses.

Sources: the fix is published as @enterprise-skills/launchops-adapters · how our release gates work · The production-ready guide