Production Incident Simulator

Overview

A public engineering project built as a deliberately bounded, production-style simulation rather than a production service.

Problem / motivation

Reliability and incident-response skills are difficult to demonstrate with a simple happy-path application.

What it implements

• Connect a FastAPI service with PostgreSQL, Redis and Nginx in Docker Compose.
• Expose structured logs, correlation IDs, Prometheus metrics and a provisioned Grafana dashboard.
• Provide deterministic incident scenarios, resilience controls and repeatable recovery workflows.

Architecture / workflow

• Nginx fronts a FastAPI service that coordinates PostgreSQL and Redis within Docker Compose.
• Deterministic incident controls alter bounded failure conditions while the same request path remains observable.
• Structured logs, correlation IDs, Prometheus metrics and provisioned Grafana panels support investigation and recovery practice.

Key engineering decisions

Reliability / security considerations

• Failure modes are intentional and repeatable, keeping experiments comparable and locally contained.
• Healthy baselines and incident states use the same telemetry so changes can be traced rather than guessed.
• The lab models production-style boundaries without claiming that it is a production deployment.

Validation / testing

• The documented workflow starts from a healthy checkout and then introduces deterministic incidents.
• Provisioned dashboards expose baseline and incident-state changes for repeatable inspection.
• Recovery guidance and postmortem-oriented evidence keep the exercise reproducible.