Monitoring & Observability — DevOps

Observability is the ability to understand what's happening inside your system from the outside. If your system is a black box and it breaks at 3am, observability is how you find out why — without ssh-ing into servers. The three pillars: Metrics, Logs, Traces.

The Three Pillars of Observability

Three pillars: Metrics (pink), Logs (blue), Traces (yellow). Press Spike Traffic to trigger an alert. Press Trace a Request to see latency breakdown.

drag — scroll to zoom

Code example

Prometheus metrics instrumentation and structured JSON logging in Node.js.

METRICS + STRUCTURED LOGGING

// metrics.js — instrument your Node.js service
import { Registry, Counter, Histogram, Gauge } from 'prom-client';

const register = new Registry();

export const httpRequests = new Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status_code'],
  registers: [register],
});

export const httpDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration in seconds',
  labelNames: ['method', 'route'],
  buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5],
  registers: [register],
});

// Express middleware — wraps every route automatically
export function metricsMiddleware(req, res, next) {
  const end = httpDuration.startTimer({ method: req.method, route: req.path });
  res.on('finish', () => {
    httpRequests.inc({ method: req.method, route: req.path, status_code: res.statusCode });
    end();
  });
  next();
}

// Expose /metrics for Prometheus to scrape
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.send(await register.metrics());
});
// logger.js — structured JSON logging with trace correlation
import pino from 'pino';

export const logger = pino({
  level: process.env.LOG_LEVEL ?? 'info',
  base: { service: 'api', version: process.env.npm_package_version },
  timestamp: pino.stdTimeFunctions.isoTime,
});

// Usage in a route handler
async function handleGetUser(req, res) {
  const log = logger.child({
    traceId: req.headers['x-trace-id'],
    userId: req.params.id
  });

  log.info('fetching user');
  try {
    const user = await db.findById(req.params.id);
    if (!user) {
      log.warn({ statusCode: 404 }, 'user not found');
      return res.status(404).json({ error: 'not found' });
    }
    log.info({ durationMs: Date.now() - req.startTime }, 'user fetched');
    res.json(user);
  } catch (err) {
    log.error({ err, statusCode: 500 }, 'database error');
    res.status(500).json({ error: 'internal server error' });
  }
}
// Output: {"time":"...","service":"api","traceId":"abc","msg":"user fetched","durationMs":12}
      

Key concepts

📊

Metrics

counters — gauges — histograms

Numeric time-series data. Counters (total requests), Gauges (current memory), Histograms (latency distribution). Stored in Prometheus — cheap to store, fast to query, easy to alert on.

📝

Structured Logs

JSON — searchable — correlation

Log every event as a JSON object: timestamp, level, service, traceId, message. The traceId links a log line to the distributed trace it belongs to — essential for cross-service debugging.

🔗

Distributed Traces

spans — latency — service map

A trace is a tree of spans — one per service call. Each span records start/end time and parent span ID. Assembled into a waterfall, you see exactly which service added the most latency.

🚨

Alerting

threshold — SLO — runbook

Alert on symptoms, not causes: high error rate, not "CPU is high." Define SLOs (Service Level Objectives) and alert when burning your error budget. Every alert should link to a runbook.

→ Infrastructure as Code DevOps — Algo3DVerse Git Branching ?