The Lumin agent

lumin-agent is a single Go binary distributed as a Docker image. You run one per host (typically one VM, sometimes one per cluster node). It is the only thing in the Lumin instrumentation story besides the optional browser SDK — there is no application-side SDK to install for logs or metrics.

What the agent collects

• Host metrics. CPU, memory, disk, network. Read from /proc and /sys via gopsutil, reported every LUMIN_INTERVAL (default 15s).
• Container metrics. Per-container CPU, memory, network, state, and restart count for every running container. No labels required. A stopped container drops out of the report; it does not appear as an exited row.
• Container logs. Opt-in per container via lumin.logs=true. The agent demultiplexes the Docker log stream into stdout/stderr lines, parses structured JSON lines (lines with a level field), and falls back to a best-effort level sniff for unstructured lines.
• Prometheus /metrics scrapes. Opt-in per container via lumin.metrics=true plus lumin.metrics.port and lumin.metrics.framework. The agent hits the endpoint, parses the exposition format, and maps known framework metrics onto Lumin’s canonical http.* schema.

Container alerting

The server derives two extra gauges from each snapshot, so alert rules can target container health directly:

• container.mem.used_pct — memory use as a percent of the limit. A container without a memory limit reports host RAM as the limit, so the percent then means share of host RAM.
• container.up — 1 while the state is running, 0 for paused or restarting.

Restarts use the increase aggregation on container.restart_count. It sums the positive changes over the window, so a container recreate does not produce a false spike. The increase aggregation is available in alert rules only.

Example rules (Alerts → New rule):

• increase(container.restart_count) {container=api} > 1 over 5m — the container restarted more than once.
• avg(container.up) {container=api} < 1 over 5m with If no data: Fire — the container is paused, restarting, or gone. The agent reports only running containers, so a stopped container sends no rows; the “If no data” option is what catches it.
• max(container.mem.used_pct) {container=api} > 90 over 5m — memory pressure.

A rule with If no data: Fire opens an incident as soon as its window is empty. This includes a rule on a metric that has never reported — check the metric name and filters when a new rule fires at once.

Watching the agent itself

A log shipper that quietly stops shipping is worse than one that fails loudly, so the agent reports its own health as ordinary metrics. Every tick it sends these counters on service lumin-agent, labelled with the host:

Metric	Meaning
agent.logs.emitted	Log lines handed to the ingest API.
agent.logs.dropped	Log lines dropped, because the internal queue was full or a send failed.
agent.scrapes.ok	Successful target scrapes.
agent.scrapes.err	Failed target scrapes — unreachable, wrong status, unparseable, or over the size cap.
agent.scrape_points.emitted	Metric points handed to the ingest API.
agent.scrape_points.dropped	Metric points dropped on a full queue or a failed send.
agent.scrape_labels.capped	Label values dropped by the cardinality guard.
agent.transport.sent	Successful requests to the ingest API.
agent.transport.errors	Failed requests to the ingest API.
agent.transport.rejected	Rows the ingest API accepted the request for but refused individually, e.g. a malformed field.

All of them count up for the life of the process, so the increase aggregation reads them correctly. One rule below is the exception: it fires on missing data instead of on a threshold.

The rules to create

Build these seven under Alerts → New rule. Alerts need a paid plan — see Plans.

Rule	Condition	If no data	Severity
Agent stopped reporting	avg(agent.transport.sent) < 0 over 5m	Fire	critical
Agent is dropping log lines	increase(agent.logs.dropped) > 0 over 15m	Stay quiet	warning
Scrapes are failing	increase(agent.scrapes.err) > 0 over 15m	Stay quiet	warning
Scraped points are dropped	increase(agent.scrape_points.dropped) > 0 over 15m	Stay quiet	warning
Ingest refuses rows	increase(agent.transport.rejected) > 0 over 15m	Stay quiet	warning
Requests to the ingest API fail	increase(agent.transport.errors) > 5 over 15m	Stay quiet	warning
A label is capped	increase(agent.scrape_labels.capped) > 0 over 1h	Stay quiet	info

What each one tells you, when it fires:

• Agent is dropping log lines — a container logs faster than the agent ships, or the ingest API refuses the requests. There is no spool and no replay, so every drop is a log line you will not get back.
• Scrapes are failing — a lumin.metrics=true container is not being read. Check its port and path labels first.
• Scraped points are dropped — the scrape queue is full, or the send failed. The chart for that container now has holes.
• Ingest refuses rows — the API accepted the request but rejected individual rows, for example a malformed field. The agent logs the first reason of each batch, so read its output for the cause.
• A label is capped — a metric produces too many distinct label values and the guard drops them. Reduce the cardinality at the source.

Why the set looks the way it does

• The first rule compares against a negative number on purpose. agent.transport.sent never goes below zero, so the threshold never breaches while data arrives. The rule exists only to fire when the data stops. Watch this counter and not another: every agent sends it, and an agent without DOCKER_HOST sends only the three agent.transport.* counters.
• It is also the only rule that finds a full outage. The agent sends its own counters at the end of each tick, after the snapshot. A failed snapshot request ends the tick early, so the agent sends no counters at all. A broken connection hides its own error counts, and no threshold rule can see that state.
• The error rule uses 5, not 0. The agent makes up to four attempts on a failed snapshot and counts each one, so a single bad tick can add four. A failure on the log path also raises agent.logs.dropped, so the drop rule already reports it. This counter is the only one that reports a failed host and container snapshot.
• Keep If no data: Stay quiet on the other six rules. An agent without DOCKER_HOST never sends the log counters or the scrape counters. With Fire, those rules open an incident that never closes.
• Do not alert on the success counters. agent.logs.emitted, agent.scrapes.ok, and agent.scrape_points.emitted belong on a chart, not in a rule. A quiet application and a broken agent both produce zero, so a threshold cannot tell them apart.
• One rule covers every agent. increase adds the counters of all your hosts together. Add a host=<hostname> filter only if you want one incident per host.
• Lumin evaluates every rule every 30 seconds. An incident opens on the first breach inside the window; there is no delay step. The absence rule reports a stopped agent about five minutes after its last tick.

The counters are cumulative, so a tick lost to a failed request loses no data. The next tick that succeeds carries the running total. lumin-agent --healthcheck reports the same outage from inside the host, and marks the container unhealthy. Docker does not restart an unhealthy container, so keep the alert rule as well.

What the agent does not do

• It does not replay logs on restart. When the agent starts, it begins tailing from now. A re-emit storm on every rollout would be worse than a few seconds of log loss.
• It does not spool to disk. Every internal queue has a fixed cap; queue-full means drop, not block. Drops increment the agent.logs.dropped counter, which the agent ships every tick — see Watching the agent itself.
• It does not touch /var/run/docker.sock directly. A read-only socket bind-mount is not safe — see Security model. The agent only speaks to a sidecar that mediates a narrow allowlist of Docker API verbs.
• It does not read application memory or attach debuggers. Logs are read from the container’s standard streams; metrics are read from an HTTP endpoint you expose. The agent is observation-only.

How to install it

See Install for the full Compose service and environment-variable reference. The five-minute version is in the Quickstart.

Compose is not required. The agent reads the Docker API and Docker labels, so docker run, Swarm, Portainer, or Coolify work as well — see Run without Compose.

How to opt services into observation

See Container labels. Every label the agent reads, with semantics and defaults.

Why the socket proxy sidecar

See Security model. The short version: a read-only socket bind-mount only protects the inode, not the Docker daemon’s API surface, and the agent makes only five requests to the daemon: the ping, and the container list, inspect, stats, and logs. The proxy forwards those five and refuses every other path, every other method, and every other client.