Monitoring & Alerting Runbook

​

1. Overview

​

Architecture

┌─────────────┐   errors, traces   ┌─────────────┐
│  React App  │ ──────────────────→ │   Sentry    │
│  (browser)  │   replay, logs     │  Dashboard  │
└─────────────┘                    └─────────────┘
      │
      │ page load, custom marks,
      │ API histograms
      ▼
┌─────────────────────┐
│  pf_health_metrics  │
│  (Supabase)         │
└─────────────────────┘

​

2. Sentry Configuration

​

PHI Scrubbing

• Truncates all event/exception messages to 500 characters
• Strips emails, phone numbers, SSNs, DOBs via regex
• Drops breadcrumb messages matching PHI patterns
• Only UUIDs are sent as user/org context — never names, emails, or clinical data

​

Source Maps

​

3. Alerting Thresholds

​

Error Rate

​

Performance (Web Vitals)

​

API Performance

​

4. Dashboards

​

Sentry Project

• Issues: Real-time error feed with stack traces and session replay
• Performance: Transaction duration, Web Vitals, throughput
• Replays: Session recordings for error context
• Logs: Structured log search (Sentry.logger.*)

​

Key Sentry Queries

# High-frequency errors in the last hour
is:unresolved times_seen:>10 firstSeen:-1h

# Errors on auth routes
transaction:/auth/* is:unresolved

# Clinical module errors
module:cl is:unresolved

​

Platform Health Metrics

• Page load timing (page_load, dom_ready)
• Custom marks (startMark/endMark)
• API response time histograms

​

5. Error Boundaries

​

6. Correlation IDs

• Set in the logger context for all subsequent logs
• Included in structured log entries
• Useful for tracing a user session across log entries

​

7. Escalation Procedure

1. P3 (Low): New non-critical issue appears in Sentry → assign to relevant core team in next standup
2. P2 (Medium): Error rate warning threshold → investigate within 4 hours
3. P1 (High): Error rate critical threshold or auth/billing errors → investigate within 1 hour
4. P0 (Critical): Application-wide crash or data integrity issue → page on-call immediately

​

8. Maintenance

​

Sentry Housekeeping

• Review and resolve/archive stale issues monthly
• Update ignoreErrors patterns when new non-actionable errors are identified
• Verify source map uploads after Vite/build tool upgrades
• Review sampling rates quarterly (adjust based on event volume and budget)

​

Performance Monitor

• performanceMonitor flushes metrics to pf_health_metrics every 30 seconds
• Metrics are sampled at 10% in production, 100% in development
• Stale metrics can be cleaned up via SQL on pf_health_metrics

​

9. Structured Logging

{
  "timestamp": "2025-01-07T10:00:00Z",
  "level": "info",
  "module": "hr",
  "action": "create_employee",
  "message": "Employee created successfully",
  "user_id": "uuid",
  "org_id": "uuid",
  "correlation_id": "uuid",
  "context": {
    "employee_id": "uuid"
  }
}

• timestamp — ISO 8601 timestamp
• level — Log level (debug, info, warn, error)
• module — Module/core name
• action — Action being performed
• message — Human-readable message
• user_id — User ID (stable UUID, not PHI)
• org_id — Organization ID
• site_id — Site ID (if applicable)
• correlation_id — Request correlation ID (see §6)

• Never log names, emails, SSNs, addresses
• Only log stable IDs (UUIDs)
• Sanitize error messages

​

10. Planned Integration: LogRocket

1. Create a LogRocket project.
2. Install the SDK:
   npm install logrocket
   
3. Initialize in src/main.tsx:
   import LogRocket from 'logrocket';
   
   LogRocket.init(import.meta.env.VITE_LOGROCKET_APP_ID, {
     shouldCaptureIP: false, // Privacy
     sanitizeInputs: true,
   });
   

​

11. Monitoring Checklist

​

Setup

• Error tracking configured (Sentry/LogRocket)
• Performance monitoring initialized
• Log aggregation configured
• Alerts configured
• Dashboard created
• On-call rotation established

​

Ongoing

• Daily error review
• Weekly performance review
• Monthly alert tuning
• Quarterly monitoring review
• Incident post-mortems completed

​

12. Troubleshooting

​

Logs Not Appearing

1. Check log level (may filter out)
2. Verify logger initialized
3. Check browser console filters
4. Verify structured format
5. Test with an explicit log call

​

Performance Monitoring Not Working

1. Verify initialization:
   performanceMonitor.init({ enablePerformanceTracking: true });
   
2. Check sample rate (may be too low)
3. Verify the web-vitals library is installed
4. Check the browser console for errors
5. Test in development (100% sampling)

​

Too Many Alerts

1. Increase alert thresholds
2. Reduce alert frequency
3. Group similar alerts
4. Use alert suppression
5. Review and tune alerts

​

Missing Alerts

1. Lower alert thresholds
2. Add more alert types
3. Improve monitoring coverage
4. Test alert delivery
5. Review alert configuration