Observability Deep Dive

Observability is the ability to understand what's happening inside your system by looking at what it outputs. The basics — uptime monitoring, error tracking, logging — cover early-stage apps well. This goes a level deeper: the three pillars in detail, how they connect to each other, the OpenTelemetry standard, and how to alert on SLOs instead of raw thresholds. If Cartara flagged observability patterns in your diff, you likely have logs or metrics that aren't structured consistently, or a monitoring setup that will generate alert fatigue as you grow.

Monitoring vs observability

People use these interchangeably, but they're different ideas:

Monitoring watches for known failure modes. You decide in advance what to measure (CPU, error rate, latency) and alert when it crosses a threshold. It answers questions you knew to ask.

Observability is the ability to ask new questions about your system after the fact, without shipping new code. It's about handling unknown unknowns — the weird, novel failure you never anticipated.

The practical difference: monitoring tells you that something is wrong; observability lets you figure out why, even when "why" is something you've never seen before. You get there by emitting rich, detailed telemetry rather than just a handful of pre-chosen metrics.

The three pillars

Logs

Discrete, timestamped records of events. The most important practice beyond "write logs" is to make them structured:

```javascript
// Unstructured: hard to search, requires parsing
console.log(Payment failed for user ${userId}: ${error.message})

// Structured: fields are queryable, filterable, and alertable
logger.error({
event: 'payment_failed',
userId: userId,
amount: 29.99,
currency: 'USD',
errorCode: error.code,
requestId: req.id,
timestamp: new Date().toISOString()
})
```

Use severity levels deliberately — debug, info, warn, error — and make them mean something so you can filter noise from signal. Never log passwords, tokens, or sensitive personal data.

Metrics

Numeric measurements aggregated over time. Cheap to store, ideal for dashboards and alerts. Two frameworks for knowing what to measure:

RED (for request-driven services): Rate (requests per second), Errors (error rate), Duration (latency distribution)

USE (for infrastructure resources): Utilization, Saturation, Errors

The main trap with metrics is cardinality — don't use high-cardinality values as labels. A metric tagged by user_id (millions of unique values) explodes your storage and cost. Keep label values bounded: status code, endpoint name, region, not unique IDs.

Traces

A trace follows one request across every service and function it touches. It's made of spans — each span represents one unit of work (a database query, an API call, a function) with a start time, duration, and parent span. Together they form a tree showing exactly where a request spent its time.

```
Request: POST /api/orders (total: 850ms)
└── Auth middleware (12ms)
└── Validate input (3ms)
└── Query user (45ms) ← database span
└── Query inventory (580ms) ← this is where time is going
└── Create order (120ms) ← database span
└── Queue email job (90ms)
```

Traces are how you debug "this endpoint is slow sometimes" in a system with moving parts. They become essential when you have more than one service.

The glue: correlation IDs

The three pillars are far more powerful when connected. The mechanism is a correlation ID (or trace ID) — a unique identifier generated at the start of a request and attached to every log line, metric label, and span it produces. Propagated across service boundaries.

```javascript
// Generate a correlation ID at the start of every request
app.use((req, res, next) => {
req.correlationId = req.headers['x-correlation-id'] || crypto.randomUUID()
res.setHeader('x-correlation-id', req.correlationId)
// Bind to async context so it's available in all downstream calls
next()
})

// Use it in every log line
logger.info({
event: 'order_created',
orderId: order.id,
correlationId: req.correlationId // ← the glue
})
```

With correlation IDs: an alert fires → you find the failing requests by trace ID → you pull every log line for that exact request across all services. Without them, the three pillars are three disconnected haystacks.

OpenTelemetry: the standard

OpenTelemetry (OTel) is the now-standard, vendor-neutral framework for generating and exporting telemetry — logs, metrics, and traces — through one set of libraries and one wire format. It's a project of the CNCF (Cloud Native Computing Foundation) and is supported by every major observability vendor.

Why it matters: you instrument your code once against OpenTelemetry, then export to whatever backend you like (Grafana, Datadog, Honeycomb, etc.) and switch backends without re-instrumenting. It frees you from vendor lock-in at the instrumentation layer, which is the expensive layer to change.

```javascript
// OpenTelemetry setup in Node.js
import { NodeSDK } from '@opentelemetry/sdk-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { resourceFromAttributes } from '@opentelemetry/resources'
import { ATTR_SERVICE_NAME } from '@opentelemetry/semantic-conventions'

const sdk = new NodeSDK({
resource: resourceFromAttributes({
[ATTR_SERVICE_NAME]: 'my-api',
}),
traceExporter: new OTLPTraceExporter({
url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT, // can point to any backend
}),
})

sdk.start()
// Now your Express routes, DB queries, etc. are automatically instrumented
```

What You'll See in Your Code

Cartara commonly flags unstructured logs and missing correlation IDs:

```javascript
// Before: logs that are hard to query or correlate
console.log('Error processing order: ' + error.message)
console.log('User ' + userId + ' signed up')

// After: structured logs with correlation IDs
import pino from 'pino'
const logger = pino()

logger.error({ err: error, orderId, requestId: req.id }, 'Error processing order')
logger.info({ userId, requestId: req.id, event: 'user_signup' })
```

Alerting on SLOs instead of raw thresholds

The basics say "alert on symptoms, not causes." The deeper practice ties alerting to SLOs (see SRE and Incident Management):

Define an SLO (e.g., "99.9% of requests succeed over 30 days"), which implies an error budget (the allowed 0.1%). Then alert on burn rate — how fast you're consuming that budget — rather than on instantaneous spikes.

A brief blip that barely touches the budget shouldn't page anyone. A fast burn that will exhaust the month's budget in an hour should page immediately. Burn-rate alerting produces fewer, more meaningful pages and is the antidote to the alert fatigue that kills naive threshold-based setups.

Scaling the stack

Tools that grow with you:

Don't adopt this stack speculatively. Add each piece when you feel the pain it solves.

A maturity path

1. Errors + uptime — Sentry and an uptime check
2. Structured logs with correlation IDs you can search
3. Metrics with RED/USE dashboards for your key services
4. Tracing once you have multiple services
5. OpenTelemetry instrumentation unifying all three pillars
6. SLO-based, burn-rate alerting so pages are rare and meaningful

Most small apps live happily at step 1–2 for a long time. The point is knowing what the next step is when you outgrow the current one.

Related Concepts

• Monitoring and Observability Basics
• SRE and Incident Management
• DORA Metrics
• DevOps
• Database Scaling and Performance