A high-cardinality log field could grow mtail's memory without bound

mtail, the Go tool that turns application logs into Prometheus-style metrics, enforced its per-metric cardinality cap only at garbage-collection time. A burst of high-cardinality labels from log input could therefore push a metric past its limit and grow memory without bound between GC cycles. The fix, merged as jaqx0r/mtail PR #476, applies the cap where each series is created.

mtail extracts Prometheus-style metrics from application logs: you write a small program that runs over each log line and increments counters. Those counters are almost always labelled with fields pulled straight from the log, a request path, a status code, a User-Agent. Every distinct combination of label values is a new time series held in memory.

We found that mtail's per-metric cardinality cap, the limit that is supposed to stop that set of series from growing without bound, was not enforced where the series are actually created. A burst of distinct label values could push a metric far past its configured limit, and memory grew with it.

Feed a metric that is labelled by a user-controlled field, say a URL path or a User-Agent, a stream of distinct values, and the series count climbs unbounded until the next garbage-collection pass, which may be far too late.

Why was the cardinality cap checked on the wrong path?

The limit existed, but only Store.Gc enforced it, on a periodic sweep. The hot path that runs for every matching log line, Metric.GetDatum, created a new label series for every unseen combination without checking the cap at all:

// Metric.GetDatum, before the fix
if series not present {
    create new series      // no cardinality check here
}
// the limit was applied only later, in Store.Gc

Between two garbage-collection cycles, nothing bounded the number of series. A flood of unique label values, which is exactly what log fields like request paths, error messages, or spoofable headers look like under load or attack, creates unbounded data in that window. Because logs routinely carry values an outside user influences, adversarial log volume turns directly into adversarial memory use.

How was it fixed?

PR #476 (merged) moves the enforcement to where series are born. GetDatum now applies the cap at creation time with an evict-oldest policy: when a metric is already at its limit and a new label combination arrives, the oldest series is removed before the new one is added.

• It keeps GetDatum's contract of always returning a usable datum.
• It matches what Store.Gc already did, so the creation path and the sweep path can no longer disagree about the limit.
• The steady-state cache-hit path is untouched: the check only runs on a cache miss, so there is no cost on the common case.

The approach was green-lit in google/mtail #1006. The result is that the invariant “series per metric stays within the limit” now holds continuously, not just after the next sweep.

Why is this the kind of bug RedMirror finds?

A metric store is a state machine whose state is the number of series per metric, and the invariant is simple: that count stays at or below the configured limit at all times. The defect was that the limit was true eventually (at the next GC) but not continuously. RedMirror looks for a reachable sequence of inputs that drives a quantity past its bound before anything prunes it, and reports the run of inputs that gets there. A cap enforced on the wrong path is precisely the sort of “holds eventually, not always” gap that shows up when you model when each check actually runs.

Frequently asked questions