In any system with more than one moving part, things will fail — a service goes down, a network call hangs, a dependency gets slow. Resilience is designing so that these inevitable failures stay contained and recoverable instead of cascading into a full outage.
If Cartara flagged this in your diff, you likely added timeout logic, retry handling, circuit breakers, or fallback behaviour to your code.
The Mindset: Failure Is Normal
The shift that separates fragile systems from resilient ones is assuming failure rather than hoping to avoid it. Every network call can hang or error. Every dependency can be unavailable. A resilient system treats these as ordinary events with planned responses.
The main enemy is the cascading failure: one slow service causes callers to pile up waiting on it, which exhausts their resources, which makes them slow, and the failure ripples outward until the whole system is down.
Timeouts
The most fundamental resilience tool, and the most often forgotten.
Never wait indefinitely on another service. A call with no timeout, to a dependency that has hung, will tie up a connection forever — and enough of those will exhaust your resources and bring you down because they are slow.
Set a timeout on every external call: database queries, API calls, anything over a network.
```ts
// ❌ No timeout — hangs forever if the service is down
const response = await fetch('https://api.example.com/data');
// ✅ With timeout — fails fast and lets you handle it
const response = await fetch('https://api.example.com/data', {
signal: AbortSignal.timeout(5000), // 5 second timeout
});
```
A missing timeout is the root cause behind a surprising share of production outages.
Retries with Backoff
When a call fails transiently (brief network blip, momentary overload), retrying often works. But naive immediate retries are dangerous — if a service is struggling and every client retries instantly, you create a retry storm that hammers the already-struggling service.
The disciplined approach:
```ts
async function fetchWithRetry(url: string, maxAttempts = 3) {
for (let attempt = 0; attempt < maxAttempts; attempt++) {
try {
return await fetch(url, { signal: AbortSignal.timeout(5000) });
} catch (error) {
if (attempt === maxAttempts - 1) throw error;
// Exponential backoff with jitter
const baseDelay = 1000 * Math.pow(2, attempt); // 1s, 2s, 4s
const jitter = Math.random() * 500;
await new Promise(r => setTimeout(r, baseDelay + jitter));
}
}
}
```
Key rules:
- Exponential backoff — wait progressively longer (1s, 2s, 4s) rather than retrying immediately
- Jitter — add randomness so many clients don't retry in synchronized waves
- Retry cap — give up after a few attempts rather than forever
- Only retry safe operations — retrying "charge a credit card" could double-charge. Use idempotency keys to make retries safe.
Circuit Breakers
If a dependency is clearly down, continuing to call it (even with backoff) wastes time and slows you down waiting for timeouts. A circuit breaker wraps calls to a dependency and watches the failure rate:
- Closed (normal) — calls flow through
- Open — after failures cross a threshold, the breaker trips and fails fast immediately without trying, for a cooldown period
- Half-open — after cooldown, a test call goes through; if it succeeds, the circuit closes again
This protects both you (no waiting on timeouts) and the struggling dependency (no extra load while it recovers).
Graceful Degradation
When something is broken, degrade gracefully rather than failing completely. Ask: what's the minimum useful experience if this component is unavailable?
- If recommendations are down, show the page without recommendations
- If search is slow, show recent items instead
- If the cache is cold, show a loading state rather than an error
```ts
async function getRecommendations(userId: string) {
try {
return await recommendationService.get(userId);
} catch {
// Degrade gracefully — return empty rather than failing the whole page
return [];
}
}
```
A partial experience the user can still use beats an error page.
What You'll See in Your Code
Resilience patterns appear in diffs as:
- Try/catch blocks around external API calls, with specific error handling
- Timeout signals added to fetch calls or database queries
- Retry logic with exponential backoff
- Fallback values returned when a service is unavailable
- Circuit breaker libraries like
opossum(Node.js) wrapping external service calls
Getting Started
You don't need a full resilience framework on day one. A few habits prevent the most common outages:
- Set a timeout on every external call — the single most impactful habit
- Retry transient failures with exponential backoff and jitter — most HTTP client libraries support this
- Degrade gracefully on key pages — so one broken dependency doesn't blank the screen
Add circuit breakers once you have multiple services where cascading failure is a real risk.