Hardening - observability

Observability and incident readiness

Review the local structured log contract, Sentry adapter shape, uptime health endpoint, incident severity levels, response roles, and disabled production incident actions.

Otto Fixture BPPE Career School - observability hardening

School Administrator · Sam Director

Structured logs, Sentry adapter, uptime check, and incident runbook

Ticket 155 proves the PRD 34 observability shape, PRD 31 platform health signals, and SEC-R-040 incident response plan using local deterministic fixtures. It does not send production Sentry events, page an on-call rotation, post a status update, notify schools, start emergency access, or close a real postmortem.

Reviewed 2026-10-13

Log fields

Required structured log fields, including tenant_id and request_id.

Sentry checks

Adapter checks for context tags, DSN guard, and metadata redaction.

Uptime checks

Platform health checks represented by the local uptime endpoint.

Severity levels

Severity definitions from Sev1 through Sev4.

Missing fields

Missing required structured log fields in the local sample.

Structured log contract

tenant_id and request_id are mandatory

10 fields

timestamp

Required

Required structured log context for local observability proof.

Example

timestamp

level

Required

Required structured log context for local observability proof.

Example

level

message

Required

Required structured log context for local observability proof.

Example

message

tenant_id

Required

Tie every log, error, and uptime event to tenant context when tenant-scoped.

Example

tenant_bppe_fixture

request_id

Required

Join logs, Sentry envelopes, route responses, and support evidence.

Example

req_obs_155

module

Required

Required structured log context for local observability proof.

Example

module

route

Required

Required structured log context for local observability proof.

Example

route

event_name

Required

Required structured log context for local observability proof.

Example

event_name

outcome

Required

Required structured log context for local observability proof.

Example

outcome

metadata

Required

Required structured log context for local observability proof.

Example

{"probe":"web-runtime"}

Sample tenant_id

tenant_bppe_fixture

Sample request_id

req_obs_155

Redacted metadata

{"probe":"web-runtime","token":"[redacted]","ssn":"[redacted]","authorization":"[redacted]"}

Sentry adapter

Error events carry safe context only

missing

DSN configuration guard

Passing locally

Adapter resolves `SENTRY_DSN` without printing the DSN and stays proof-only when missing.

tenant_id and request_id tags

Passing locally

Sentry envelope tags include tenant_id, request_id, module, route, and outcome.

Sensitive metadata redaction

Passing locally

Token, SSN, authorization header, raw payload, card, bank, and IP-like keys are redacted before display.

Stacktrace redaction in local proof

Passing locally

Local Sentry envelopes replace stack output with `[redacted]` in review fixtures.

Release and environment labels

Passing locally

Adapter labels environment as local and release as local-proof until deployment variables exist.

Vendor send disabled

External later

No production Sentry event is sent from this local proof; staging DSN activation remains later.

Envelope tenant_id

tenant_bppe_fixture

Envelope request_id

req_obs_155

Stacktrace

[redacted]

Uptime checks

Local health endpoint and platform signals

/api/health

Web runtime

Passing locally

Target: HTTP 200 within 3 seconds
Evidence: Local `/api/health` response can be checked without auth.

Database connectivity

External later

Target: Postgres query succeeds
Evidence: Production database probe remains outside this local proof.

Background jobs

External later

Target: Outbox and automation workers heartbeat
Evidence: Worker uptime waits for staging worker deployment.

Email and text delivery

External later

Target: Provider health and delivery lag
Evidence: Email/SMS provider choice is still a later phase decision.

Backup status

External later

Target: Backup age and restore-drill artifact freshness
Evidence: Production backup provider status is not queried locally.

AI usage and cost

External later

Target: Per-school budget ledger freshness
Evidence: Production AI provider and ZDR activation are later gates.

Incident runbook

Customer notification target and response cadence

8 steps

Sev1 - data-risk emergency or critical outage

SEV1

Response target: 1-hour response, 24/7 paging during pilot
Notification target: Notify affected schools within 72 hours of confirming a data incident, target 24 to 48 hours.

Sev2 - major workflow blocked

SEV2

Response target: 4 business hours
Notification target: School status update if pilot workflow remains blocked.

Sev3 - standard product issue

SEV3

Response target: 1 business day
Notification target: Support case update cadence.

Sev4 - how-to or cosmetic issue

SEV4

Response target: 2 business days
Notification target: Support case update as needed.

Step 1 - Incident commander

Detect and declare

Runbook

Target: Severity assigned within 15 minutes of confirmation
Evidence: Create incident timeline with trigger source, request_id, tenant_id, suspected impact, and owner.

Step 2 - Engineering on call

Triage scope

Runbook

Target: Affected tenants, routes, and integrations identified
Evidence: Use structured logs, Sentry envelope tags, uptime checks, and audit logs to scope impact.

Step 3 - Engineering on call

Contain or rollback

Runbook

Target: Containment plan recorded before broad changes
Evidence: Feature flag, rollback, kill switch, or access lock action is tied to a reason and approver.

Step 4 - Incident commander

Notify internal roles

Runbook

Target: Owner, security/privacy, support, and customer communications paged
Evidence: Record who was paged, when, and which severity definition was used.

Step 5 - Security and privacy lead

Customer notification target

Runbook

Target: Within 72 hours of confirming a data incident, target 24 to 48 hours
Evidence: Counsel-reviewed school notice plan is captured with affected-school list.

Step 6 - Customer communications

Status updates every 60 minutes

Runbook

Target: Every 60 minutes until resolved
Evidence: Public status page and school updates continue until resolved.

Step 7 - Engineering on call

Resolution and monitoring

Runbook

Target: Resolution criteria and post-fix monitoring window recorded
Evidence: Uptime checks, error rates, job queues, delivery providers, and backup status are monitored.

Step 8 - Incident commander

Post-incident report

Runbook

Target: Within 10 business days
Evidence: Written report includes timeline, root cause, impact, remediation, owner follow-ups, and due dates.

Incident roles

Owners for response and communications

5 roles

Incident commander

Declares severity, assigns owners, and keeps timeline.

Engineering on call

Triage, containment, rollback, and technical updates.

Security and privacy lead

Assesses data impact, notification duties, evidence preservation, and counsel handoff.

Customer communications

Posts school updates, status page notes, and post-incident report drafts.

Owner approver

Approves emergency access, high-risk changes, and final external communications.

Production observability actions are disabled

Proof boundary

6 disabled