# The Lumin agent

> What the agent does, what it collects, and how it fits into your stack.

Canonical: https://docs.getlumin.dev/agent/

`lumin-agent` is a single Go binary distributed as a Docker image. You run one per host (typically one VM, sometimes one per cluster node). It is the only thing in the Lumin instrumentation story besides the optional browser SDK — there is no application-side SDK to install for logs or metrics.

## What the agent collects

- **Host metrics.** CPU, memory, disk, network. Read from `/proc` and `/sys` via [`gopsutil`](https://github.com/shirou/gopsutil), reported every `LUMIN_INTERVAL` (default `15s`).
- **Container metrics.** Per-container CPU, memory, network, state, and restart count for every **running** container. No labels required. A stopped container drops out of the report; it does not appear as an `exited` row.
- **Container logs.** Opt-in per container via `lumin.logs=true`. The agent demultiplexes the Docker log stream into stdout/stderr lines, parses structured JSON lines (lines with a `level` field), and falls back to a best-effort level sniff for unstructured lines.
- **Prometheus `/metrics` scrapes.** Opt-in per container via `lumin.metrics=true` plus `lumin.metrics.port` and `lumin.metrics.framework`. The agent hits the endpoint, parses the exposition format, and maps known framework metrics onto Lumin's canonical `http.*` schema.

## Container alerting

The server derives two extra gauges from each snapshot, so alert rules can target container health directly:

- `container.mem.used_pct` — memory use as a percent of the limit. A container without a memory limit reports host RAM as the limit, so the percent then means share of host RAM.
- `container.up` — `1` while the state is `running`, `0` for `paused` or `restarting`.

Restarts use the `increase` aggregation on `container.restart_count`. It sums the positive changes over the window, so a container recreate does not produce a false spike. The `increase` aggregation is available in alert rules only.

Example rules (Alerts → New rule):

- `increase(container.restart_count) {container=api} > 1 over 5m` — the container restarted more than once.
- `avg(container.up) {container=api} < 1 over 5m` with **If no data: Fire** — the container is paused, restarting, or gone. The agent reports only running containers, so a stopped container sends no rows; the "If no data" option is what catches it.
- `max(container.mem.used_pct) {container=api} > 90 over 5m` — memory pressure.

A rule with **If no data: Fire** opens an incident as soon as its window is empty. This includes a rule on a metric that has never reported — check the metric name and filters when a new rule fires at once.

## Watching the agent itself

A log shipper that quietly stops shipping is worse than one that fails loudly, so the agent reports its own health as ordinary metrics. Every tick it sends these counters on service `lumin-agent`, labelled with the host:

| Metric | Meaning |
|---|---|
| `agent.logs.emitted` | Log lines handed to the ingest API. |
| `agent.logs.dropped` | Log lines dropped, because the internal queue was full or a send failed. |
| `agent.scrapes.ok` | Successful target scrapes. |
| `agent.scrapes.err` | Failed target scrapes — unreachable, wrong status, unparseable, or over the size cap. |
| `agent.scrape_points.emitted` | Metric points handed to the ingest API. |
| `agent.scrape_points.dropped` | Metric points dropped on a full queue or a failed send. |
| `agent.scrape_labels.capped` | Label values dropped by the cardinality guard. |
| `agent.transport.sent` | Successful requests to the ingest API. |
| `agent.transport.errors` | Failed requests to the ingest API. |
| `agent.transport.rejected` | Rows the ingest API accepted the request for but refused individually, e.g. a malformed field. |

All of them count up for the life of the process, so the `increase` aggregation reads them correctly. One rule below is the exception: it fires on missing data instead of on a threshold.

### The rules to create

Build these seven under Alerts → New rule. Alerts need a paid plan — see [Plans](https://docs.getlumin.dev/plans/).

| Rule | Condition | If no data | Severity |
|---|---|---|---|
| Agent stopped reporting | `avg(agent.transport.sent) < 0 over 5m` | **Fire** | critical |
| Agent is dropping log lines | `increase(agent.logs.dropped) > 0 over 15m` | Stay quiet | warning |
| Scrapes are failing | `increase(agent.scrapes.err) > 0 over 15m` | Stay quiet | warning |
| Scraped points are dropped | `increase(agent.scrape_points.dropped) > 0 over 15m` | Stay quiet | warning |
| Ingest refuses rows | `increase(agent.transport.rejected) > 0 over 15m` | Stay quiet | warning |
| Requests to the ingest API fail | `increase(agent.transport.errors) > 5 over 15m` | Stay quiet | warning |
| A label is capped | `increase(agent.scrape_labels.capped) > 0 over 1h` | Stay quiet | info |

What each one tells you, when it fires:

- **Agent is dropping log lines** — a container logs faster than the agent ships, or the ingest API refuses the requests. There is no spool and no replay, so every drop is a log line you will not get back.
- **Scrapes are failing** — a `lumin.metrics=true` container is not being read. Check its port and path labels first.
- **Scraped points are dropped** — the scrape queue is full, or the send failed. The chart for that container now has holes.
- **Ingest refuses rows** — the API accepted the request but rejected individual rows, for example a malformed field. The agent logs the first reason of each batch, so read its output for the cause.
- **A label is capped** — a metric produces too many distinct label values and the guard drops them. Reduce the cardinality at the source.

### Why the set looks the way it does

- **The first rule compares against a negative number on purpose.** `agent.transport.sent` never goes below zero, so the threshold never breaches while data arrives. The rule exists only to fire when the data stops. Watch this counter and not another: every agent sends it, and an agent without `DOCKER_HOST` sends only the three `agent.transport.*` counters.
- **It is also the only rule that finds a full outage.** The agent sends its own counters at the end of each tick, after the snapshot. A failed snapshot request ends the tick early, so the agent sends no counters at all. A broken connection hides its own error counts, and no threshold rule can see that state.
- **The error rule uses 5, not 0.** The agent makes up to four attempts on a failed snapshot and counts each one, so a single bad tick can add four. A failure on the log path also raises `agent.logs.dropped`, so the drop rule already reports it. This counter is the only one that reports a failed host and container snapshot.
- **Keep If no data: Stay quiet on the other six rules.** An agent without `DOCKER_HOST` never sends the log counters or the scrape counters. With **Fire**, those rules open an incident that never closes.
- **Do not alert on the success counters.** `agent.logs.emitted`, `agent.scrapes.ok`, and `agent.scrape_points.emitted` belong on a chart, not in a rule. A quiet application and a broken agent both produce zero, so a threshold cannot tell them apart.
- **One rule covers every agent.** `increase` adds the counters of all your hosts together. Add a `host=<hostname>` filter only if you want one incident per host.
- **Lumin evaluates every rule every 30 seconds.** An incident opens on the first breach inside the window; there is no delay step. The absence rule reports a stopped agent about five minutes after its last tick.

The counters are cumulative, so a tick lost to a failed request loses no data. The next tick that succeeds carries the running total. `lumin-agent --healthcheck` reports the same outage from inside the host, and marks the container unhealthy. Docker does not restart an unhealthy container, so keep the alert rule as well.

## What the agent does **not** do

- **It does not replay logs on restart.** When the agent starts, it begins tailing `from now`. A re-emit storm on every rollout would be worse than a few seconds of log loss.
- **It does not spool to disk.** Every internal queue has a fixed cap; queue-full means drop, not block. Drops increment the `agent.logs.dropped` counter, which the agent ships every tick — see [Watching the agent itself](#watching-the-agent-itself).
- **It does not touch `/var/run/docker.sock` directly.** A read-only socket bind-mount is not safe — see [Security model](https://docs.getlumin.dev/agent/security/). The agent only speaks to a sidecar that mediates a narrow allowlist of Docker API verbs.
- **It does not read application memory or attach debuggers.** Logs are read from the container's standard streams; metrics are read from an HTTP endpoint *you* expose. The agent is observation-only.

## How to install it

See **[Install](https://docs.getlumin.dev/agent/install/)** for the full Compose service and environment-variable reference. The five-minute version is in the [Quickstart](https://docs.getlumin.dev/quickstart/).

Compose is not required. The agent reads the Docker API and Docker labels, so `docker run`, Swarm, Portainer, or Coolify work as well — see [Run without Compose](https://docs.getlumin.dev/agent/install/#run-without-compose).

## How to opt services into observation

See **[Container labels](https://docs.getlumin.dev/agent/labels/)**. Every label the agent reads, with semantics and defaults.

## Why the socket proxy sidecar

See **[Security model](https://docs.getlumin.dev/agent/security/)**. The short version: a read-only socket bind-mount only protects the inode, not the Docker daemon's API surface, and the agent makes only five requests to the daemon: the ping, and the container list, inspect, stats, and logs. The proxy forwards those five and refuses every other path, every other method, and every other client.