Break-Glass Access That Does Not Quietly Become Normal Access

Break-glass access exists for the moment when the normal control plane is unavailable or untrustworthy. That same property makes it dangerous.

An emergency identity may bypass federation, conditional access, approval workflow, or privileged access tooling. It may carry enough authority to repair the identity provider, revoke a compromised administrator, restore policy, or recover a secrets service. If I make that path easy enough for daily work, it becomes a shadow administration system. If I make it too obscure or dependent on the systems it must repair, it fails during the emergency.

The goal is not simply to create an account called emergency-admin. The goal is to engineer an independently usable recovery path with narrow activation criteria, strong custody, time-bounded authority, immediate visibility, and a tested return to zero use.

NIST SP 800-53 Rev. 5 addresses emergency and temporary accounts under AC-2(2), least privilege under AC-6, separation of duties under AC-5, and privileged-function auditing under AC-6(9). Those controls point in the right direction. The operational design still has to answer who can activate access, which dependencies may be unavailable, what the account can repair, how use is detected, and how credentials and sessions are invalidated afterward.

Define the emergency before defining the account

I start with activation scenarios, not products.

Valid examples include:

• the primary identity provider is unavailable and blocks authorized restoration
• the identity provider is suspected compromised and federated administrator sessions cannot be trusted
• a policy error denies every ordinary administrator
• the privileged access broker is unavailable while a critical service requires repair
• federation signing or encryption material must be replaced through an independent path
• the secrets service is sealed or inaccessible and its documented recovery procedure requires offline-held material

Convenience is not an emergency. Neither are a slow approval queue, an inconvenient maintenance window, a forgotten role assignment, or a vendor that wants unrestricted access while troubleshooting.

I put the activation boundary in policy:

break_glass:
  purpose: restore_identity_or_security_control_plane
  permitted_conditions:
    - primary_idp_unavailable
    - primary_idp_integrity_suspect
    - all_standard_privileged_paths_denied
    - pam_control_plane_unavailable
  prohibited_uses:
    - routine_maintenance
    - change_window_avoidance
    - approval_bypass
    - general_vendor_support
    - permanent_automation
  incident_required: true
  maximum_initial_duration: 60m
  extension_requires_new_approval: true

The list is intentionally short. If an activation reason appears every week, I treat that as evidence that the normal access design is broken.

Independence is a dependency property

A break-glass path is not independent because it has a different username. It is independent only if its critical dependencies differ from the failure it is meant to recover.

Consider an identity-provider emergency account that:

• authenticates through the same external federation provider
• requires the same push MFA service
• stores its password only inside the same SSO-protected vault
• sends approval to the same unavailable collaboration platform
• can be used only from a workstation whose login depends on the failed directory
• records all evidence only in the identity provider being investigated

That account is a second label on the first path.

I map dependencies explicitly:

NIST SP 800-207’s policy decision and enforcement model is useful here because it makes dependencies visible. If emergency authorization still requires the failed policy decision point, it is not a recovery path for that failure. If a separate decision path feeds the same compromised enforcement point, its independence may still be insufficient.

Independence does not mean isolation from every control. The emergency path still needs authentication, authorization, network restriction, logging, and review. It means those controls are chosen so the specified failure does not remove the recovery capability.

Use separate identities, not special flags on normal identities

I prefer dedicated emergency identities over adding an emergency role to ordinary administrator accounts.

A separate identity provides a clean audit principal, can use different authentication, can remain disabled or noninteractive between tests, and does not inherit ordinary federation or group complexity. It also makes use conspicuous. Daily administration under an emergency identity is visibly wrong.

The identity record should have:

• no mailbox, productivity license, or ordinary application access
• no membership in general workforce groups
• no use as a service or automation identity
• an explicit owner accountable for the control, not the credential alone
• a narrowly documented privilege set
• a creation and last-test date
• a next-review and retirement trigger
• independent contact and activation procedures

Two emergency identities can improve resilience when they use separately held credentials and the platform supports them. They should not share one password or one authenticator. Multiple copies of the same secret increase exposure without providing meaningful identity separation.

For Keycloak, a realm-local recovery administrator can remain independent of an upstream identity provider. The exact role mapping should be tested against the administrative operations needed for recovery rather than defaulting permanently to every realm-management role. Keycloak’s authorization capabilities can define resources, scopes, policies, and permissions for protected services, but the administrative console and administrative API need their own careful role analysis.

For Authentik, keep at least one recovery path that does not depend on the source, federation flow, proxy provider, or outpost being repaired. A local account alone is not sufficient if its login flow invokes a broken policy stage. Test the complete login and administrative operation while the simulated failed dependency is unavailable.

For FreeIPA, directory recovery has a different shape. Host access may depend on Kerberos, LDAP, SSSD caches, HBAC, DNS, certificates, and enrolled host state. FreeIPA documents HBAC rule evaluation with ipa hbactest, which is useful for proving normal host restrictions. Directory recovery also needs console or host-level access that survives a directory outage. Do not treat a cached directory login as the only emergency path for the directory servers themselves.

Scope emergency authority to the repair

It is tempting to grant the broadest available administrator role because nobody wants authorization to block recovery. That transfers uncertainty from availability to compromise.

I identify a minimum recovery action set:

recovery_capability:
  target: identity-control-plane
  required_actions:
    - inspect_system_health
    - inspect_recent_admin_changes
    - disable_suspect_admin_identity
    - revoke_active_sessions
    - restore_reviewed_policy_revision
    - rotate_federation_signing_material
    - create_time_bounded_recovery_operator
  prohibited_actions:
    - access_application_data
    - modify_audit_retention
    - disable_external_audit_forwarding
    - create_permanent_global_admin
    - alter_backup_retention
    - delete_recovery_evidence

Some platforms do not offer permissions granular enough to express that set. In that case, I document the residual authority and add containment around it: dedicated management network, restricted source device, session recording where feasible, dual custody, immediate alerting, and a shorter activation window.

There is also a bootstrap problem. The identity with authority to repair role assignments can often grant itself more authority. Technical least privilege has a hard limit at that tier. Separation of duties, evidence, and review become essential because authorization alone cannot contain an actor who legitimately controls authorization.

Credential custody must survive the incident

Emergency credentials are not ordinary password-manager entries.

If the organization uses a password vault, I want a dedicated emergency collection with access controls independent of the target system, explicit checkout evidence, and tested offline or alternate access where the scenario requires it. For small environments, tamper-evident physical storage under controlled custody may be more reliable than a complex online chain. The design should match the threat and staffing model, not copy a ceremonial process nobody can execute at 03:00.

Custody options include:

• two sealed components held by separate custodians
• separate authenticator and password custody
• threshold recovery shares where the platform supports them
• a hardware-backed private key stored in a controlled location
• one-time recovery codes protected as secrets
• an offline encrypted package whose decryption material is separately held

Split knowledge is useful only when the components are actually required. Giving two people identical complete copies is dual possession, not dual control.

The custody record should never contain the secret itself. It should record package identifier, creation date, custodians, seal or integrity references, storage locations at an appropriate level of abstraction, last inventory, last test, and next rotation. Public documentation must not disclose real locations, account names, key fingerprints, phone trees, or recovery answers.

Authentication factors require lifecycle treatment. Rotate a password or key after any activation, suspected exposure, custody change, failed seal check, or test that revealed the full material. Replace lost authenticators immediately and investigate the custody failure. Schedule rotation even without use where the platform, cryptographic policy, or storage process requires it. Avoid needless secret handling solely to satisfy an arbitrary calendar, but do not let an old untested package become institutional folklore.

Separate activation from authorization duration

An always-enabled emergency account is standing privilege even if nobody plans to use it.

Where the platform allows it, I keep the identity disabled or without its powerful role and activate it through a separately controlled mechanism. Activation sets an absolute expiry. The expiry should not depend on the operator remembering to remove access after a stressful incident.

A representative state machine is:

SEALED -> REQUESTED -> APPROVED -> ACTIVE -> EXPIRED
   ^                                     |        |
   |                                     v        v
   +------------------------------- ROTATED <- REVIEW

Important properties are:

• ACTIVE has a hard deadline.
• Extensions are new decisions, not silent renewal.
• Expiry removes the role or disables the identity.
• Session and token revocation follows entitlement removal.
• Credential rotation follows activation.
• Review cannot be skipped before returning to SEALED.

Account expiry, role expiry, token expiry, session expiry, and secret lease expiry are different controls. Setting an account’s expiration time does not necessarily invalidate already issued access tokens or active sessions. Removing group membership may not evict an existing console session. Rotating a password may not revoke API tokens. The procedure must call each relevant revocation mechanism and test the result.

OpenBao is useful when emergency work needs short-lived credentials or controlled secret retrieval. OpenBao leases associate dynamic secrets and service tokens with a time-to-live and support renewal or revocation. Its audit devices record API requests and responses, including errors, subject to documented exceptions. Those capabilities help, but an OpenBao-backed recovery procedure must account for OpenBao itself being unavailable or sealed. A vault cannot be its own only recovery dependency.

A practical design might use OpenBao for normal just-in-time administration and separately protected offline recovery material for OpenBao recovery. Once OpenBao is available, issue a short lease for the remaining work rather than continuing to use the root recovery capability.

Require a declared activation record

Emergency access should begin with a record, even when prior approval is impossible.

The minimum record is:

{
  "activation_id": "BG-example-correlation-id",
  "declared_condition": "primary_idp_integrity_suspect",
  "target": "identity-control-plane",
  "requested_scope": [
    "session-revocation",
    "policy-restoration"
  ],
  "requester_role": "incident-commander",
  "approver_roles": [
    "security-duty-officer",
    "platform-duty-officer"
  ],
  "activated_at": "example-timestamp",
  "expires_at": "example-timestamp",
  "evidence_destinations": [
    "independent-security-log",
    "incident-record"
  ]
}

If life safety, major outage, or active compromise makes prior approval impossible, the policy can permit a single authorized role to activate and require immediate retrospective notification. That exception must be explicit. Pretending every emergency will wait for two available approvers produces bypasses rather than control.

The activation ID should appear in checkout records, session banners, administrative change notes, audit events where supported, and the incident timeline. Never put credentials in the ticket or chat thread.

Make use noisy and evidence durable

An emergency account should generate a high-priority alert on authentication attempt, successful login, failed login, activation, role assignment, sensitive operation, session creation, and expiry failure.

I send those events to a destination outside the recovery target. If the IdP is compromised, its local audit log is not independent evidence. Forwarding to a SIEM helps only if the route, parser, index, retention, and alert have been tested.

Useful fields include:

• emergency identity identifier
• activation correlation ID
• source device and management zone
• authentication method
• target system
• session identifier
• requested and effective action
• authorization result
• policy or role revision
• event time from the source and collector
• termination and revocation result

Sensitive command output, tokens, recovery codes, and secret values should not enter logs. Session recording can be valuable for infrastructure recovery, but recording a secrets console can capture material that should never be replayable. Use command or API audit where possible, redact known secret fields, tightly restrict recordings, and set retention deliberately.

Monitoring also detects normalization. I track:

• activations per quarter
• activation reason distribution
• time from activation to first alert
• time active
• extensions
• actions outside the requested scope
• rotation completion time
• session revocation completion time
• overdue review items
• attempted use while sealed
• repeat activation for the same ordinary-access failure

The desired count is not blindly zero. A scheduled test is legitimate use. A real activation may be necessary. The warning signal is unexplained frequency, long duration, routine change activity, or repeated use to compensate for a broken standard path.

Use a positive and negative acceptance matrix

A break-glass procedure is not ready until it works under the failure conditions it claims to cover and refuses the uses it claims to prohibit.

I add platform-specific cases for cached sessions, refresh tokens, personal access tokens, API keys, SSH certificates, recovery codes, remembered devices, local console access, and alternate administrative endpoints.

Each denial requires a side-effect check. A rejected role-creation request must not leave a partial role. A failed policy restore must not activate half a configuration. A denied secrets request must not expose values through an error, trace, or audit record.

Rehearse failure, not only login

A quarterly login test that leaves every dependency healthy proves little. I use a controlled non-production environment or a carefully bounded production exercise to simulate the named failure.

For an IdP recovery path:

1. verify the test scope and rollback path
2. make the ordinary federation path unavailable to the test client
3. retrieve emergency authentication material under the custody procedure
4. create the activation record
5. authenticate from the approved recovery device and network
6. complete a harmless representative administrative action
7. verify independent alert delivery and audit correlation
8. allow or force expiry
9. revoke sessions and tokens
10. rotate exposed material
11. prove the old material and session no longer work
12. restore sealed state and close the evidence record

For a secrets service, test unseal or recovery according to the platform’s documented model without copying production recovery material into a lab. For FreeIPA or another directory, test host console access, service dependencies, DNS assumptions, certificate validity, and the ability to restore ordinary administrator access. For a cloud tenant, test that the provider-native emergency identity does not depend on enterprise federation and that conditional restrictions do not accidentally block the recovery source.

A destructive production outage is not required to test independence. Network controls, test realms, temporary federation changes in a lab, and dependency injection can exercise the design safely. The evidence must state what was simulated and what was not.

Close every activation as a security event

Emergency access does not end when service returns.

Closure includes:

• disable the identity or remove emergency authority
• revoke active sessions, tokens, certificates, and leases
• rotate every credential or recovery component exposed by the procedure
• verify the previous material fails
• reconcile administrative changes against the approved recovery scope
• preserve independent audit evidence
• investigate failed authentication or out-of-scope attempts
• return custody packages to sealed state
• update diagrams, runbooks, contacts, and dependency records
• document why normal access failed
• create corrective work so the emergency path is not needed for the same routine condition again

OpenBao lease revocation can invalidate a leased secret or token, including broader revocation by prefix when the incident scope warrants it.[5] That is useful closure evidence for leased credentials. Static credentials still require rotation at their source and negative testing of the old value.

A review should compare the activation record, platform audit, independent telemetry, session evidence, configuration diff, and credential lifecycle. No single source is complete. If the emergency administrator could alter the target’s logs, target-local evidence cannot validate itself.

Put pressure on normalization

Controls drift toward convenience unless the design makes drift visible.

I use several pressure points:

• no ordinary licenses or application access on emergency identities
• prominent emergency naming and login banners
• alerting on every authentication attempt
• hard expiry on activated authority
• mandatory rotation and review after use
• quarterly reconciliation of account state, group membership, authenticators, and custody
• dashboards that distinguish test, real activation, and unexplained use
• a management metric for repeat activation causes
• prohibition on service integration or scheduled jobs using emergency identities
• explicit retirement when the protected platform is replaced

Access review must inspect effective permissions, not only the account’s direct roles. Nested groups, inherited tenant roles, local administrator mappings, API tokens, SSH keys, recovery codes, and application-specific permissions can preserve a path after the visible role is removed.

This is also where architectural documentation matters. In Cybersecurity Architect’s Handbook, Second Edition, I emphasize trust boundaries and operational control behavior. Break-glass access sits at several boundaries at once: identity, management network, secrets, endpoint trust, logging, and human approval. A diagram that shows only an emergency account misses most of the control.

The standard I use

I consider a break-glass path ready when I can support this bounded statement:

Under the tested failure condition, authorized custodians activated a separately authenticated recovery identity, completed the enumerated recovery action from the approved path, produced independent audit and alert evidence, lost authority at the defined deadline, revoked active sessions, rotated exposed credentials, and proved the previous access no longer worked. Prohibited actions in the acceptance matrix were denied without observed side effects.

That statement is stronger than “we have an emergency admin account.” It describes capability, containment, observability, and closure.

Break-glass access will always carry concentrated risk. Its purpose is to preserve recovery when ordinary controls fail, not to pretend that extraordinary authority is ordinary least privilege. The architecture succeeds when the path remains available during the tested control-plane failure, is difficult to use casually, generates independently monitored evidence under the tested conditions, and returns to sealed state after every activation.