Site Reliability Engineering (SRE) is a discipline for keeping software reliable using engineering rather than heroics. It gives you a framework for deciding how reliable something needs to be, and a calm process for when things break. If Cartara flagged something in your diff related to reliability, health checks, or error handling, this explains the mindset behind why those patterns matter.
The core idea: reliability has a budget
Perfect uptime is impossible and, past a certain point, not worth chasing. Going from 99.9% to 99.99% availability can cost more than the downtime it prevents. SRE reframes reliability as something you set a target for and spend deliberately, rather than an absolute to pursue at any cost.
This leads to the central insight: a little bit of unreliability is acceptable and even useful — it's the room you spend on shipping features and taking risks. The trick is making that trade-off explicit rather than implicit.
SLI, SLO, SLA — three acronyms that actually matter
These sound interchangeable but aren't:
SLI (Service Level Indicator) — a measurement of how the service is doing. For example: "the percentage of requests that succeed" or "the percentage of requests served in under 200ms."
SLO (Service Level Objective) — your internal target for an SLI. For example: "99.9% of requests succeed over a 30-day window." This is the number your team actually manages to.
SLA (Service Level Agreement) — a contractual promise to customers, usually with financial penalties if you miss it. SLAs are set looser than your SLOs on purpose, so you have a safety margin.
Rule of thumb: SLA (the promise) < SLO (your internal target) < what you actually aim to deliver. You want to breach your SLO and start worrying long before you'd ever breach a customer-facing SLA.
Error budgets: turning reliability into a decision rule
If your SLO is 99.9% success over 30 days, then 0.1% failure is allowed — that's your error budget. Roughly 43 minutes of downtime a month.
The error budget turns reliability into a shared, unemotional decision:
- Budget remaining? Ship freely, take risks, deploy that ambitious feature
- Budget exhausted? Slow down, freeze risky changes, and invest engineering time in stability until you're back in budget
This dissolves the classic tension between developers who want to ship and operations who want stability. Both sides agree on the budget upfront, and the budget makes the call — not personalities.
Incident management: a calm process beats panic
When something breaks, process beats improvisation. A basic incident flow:
- Detect — ideally your monitoring alerts you before customers do
- Declare — name it an incident, assign one person as incident commander (coordinating, not necessarily fixing), and open a shared communication channel
- Mitigate first, diagnose later — the priority is restoring service (roll back, failover, scale up), not finding root cause. Understanding why can wait until users are unblocked
- Communicate — keep stakeholders and, if needed, customers updated on a status page
- Resolve and record — once stable, capture the timeline while it's fresh
A simple severity scale (Sev1 = major outage affecting all users, Sev3 = minor degradation) helps everyone calibrate response intensity.
Blameless postmortems
After a significant incident, write a postmortem: what happened, the timeline, the impact, what caused it, and what you'll change so it can't recur.
The essential word is blameless. The goal is to fix the system, not punish a person. Human error is almost always a symptom of a system that made the error easy to make — a missing guardrail, a confusing interface, a test that didn't exist. If people fear blame, they hide problems and you learn nothing. Blameless postmortems are how reliability compounds over time.
A basic postmortem template:
- Summary: What happened and what was the user impact?
- Timeline: When was it detected, when was it resolved, what happened in between?
- Root cause: What technical or process failure allowed this?
- Action items: What specific changes will prevent or reduce future impact?
What You'll See in Your Code
Cartara often flags missing health check endpoints — essential for both load balancers and on-call engineers to verify service status:
```javascript
// Missing health check
// Load balancer has no way to know if this service is healthy
// Add a real health endpoint
app.get('/health', async (req, res) => {
const checks = {}
try {
await db.$queryRawSELECT 1
checks.database = 'ok'
} catch (e) {
checks.database = 'error'
}
const healthy = Object.values(checks).every(v => v === 'ok')
res.status(healthy ? 200 : 503).json({
status: healthy ? 'healthy' : 'degraded',
checks,
timestamp: new Date().toISOString()
})
})
```
Missing retry logic for critical operations:
```javascript
// Brittle: one failure takes down the flow
async function sendWelcomeEmail(userId) {
await emailService.send({ to: user.email, template: 'welcome' })
}
// More resilient: retry with backoff
async function sendWelcomeEmail(userId, attempt = 1) {
try {
await emailService.send({ to: user.email, template: 'welcome' })
} catch (error) {
if (attempt < 3) {
await sleep(1000 * attempt) // exponential backoff
return sendWelcomeEmail(userId, attempt + 1)
}
// Log and alert after max retries, don't crash the calling flow
logger.error({ error, userId, event: 'welcome_email_failed' })
}
}
```
Alerting philosophy
Bad alerting is its own outage. If every minor blip pages someone at 3am, people develop alert fatigue and start ignoring pages — including the real ones.
- Alert on symptoms, not causes — page when users are affected (error rate up, latency high), not on every internal metric that twitches
- Every page should be actionable — if there's nothing a human can do about it, make it a dashboard item or a ticket instead
- Tie alerts to SLOs — page when you're burning through your error budget fast, not on arbitrary thresholds
For small teams
You don't need a formal SRE org to apply this. High-value starter kit: pick one or two SLOs for your most important user journey (e.g., "the app loads and login works 99.5% of the time"), set up alerting on actual user impact, and write a short blameless postmortem after anything that scared you. The discipline matters more than the headcount.