Skip to main content

​
Overview

The controller watches execve and shared-library mappings per container and reports the distinct set of files it sees, keyed by workload, container and image digest. A path is stored once per (workload, container, digest), not per event, so the inventory grows with what a workload runs rather than how often it runs it. Processes that were already running when the controller started are backfilled from /proc and marked source: "backfill"; a later runtime sighting upgrades them to ebpf. Joined with an image’s SBOM on digest and path, this separates packages an image ships from packages it actually loads. Each path also records where the file lived when it ran (origin), which is what drift detection uses to flag binaries the image did not ship.

​
Enabling it

Off by default until its per-node overhead is measured. Set the chart value controller.runtimeInventory.mode: Each (container, file) is reported once: the probe deduplicates in the kernel, so after warm-up a container costs one hash lookup per exec or executable mapping. Containers already running when the probe attached are backfilled once from /proc. Namespaces in controller.excludedNamespaces and pods annotated kguardian.dev/runtime-inventory: "off" are not recorded. An empty inventory means nothing was observed, not that nothing ran: the feature may be off on that node, the pod may be excluded, or the process may have run before the controller started and exited before the backfill.

​
Origin

When sightings of one path disagree, the most suspicious wins (in the order unknown < image < otherFs < deleted < writableLayer < memfd) and is never downgraded: one replica running the file from its writable layer is the finding, whatever the others did. deleted, writableLayer and memfd together are the “unshipped” origins. A file changed in place while the container runs (an image binary overwritten, which copies it up into the writable layer, or a file unlinked and then executed through an open descriptor) is reported again with its new origin: the probe’s dedup key includes the file’s state. Not observed: a mapping made executable later with mprotect(PROT_EXEC) (JIT compilers, some custom loaders), and code an interpreter reads as data (scripts, .pyc, jars). Only execve and executable mmap are seen. Rows not seen for RUNTIME_INVENTORY_RETENTION_DAYS (chart: broker.runtimeInventory.retentionDays, default 30) are pruned.

​
GET /workloads/{namespace}/{kind}/{name}/runtime

Requires the read scope when broker auth is on. Returns 200 with an empty containers list when nothing has been reported for the workload.
coverage lists every (container, digest) of the workload with a controller heartbeat in the window, with the answer of kg_runtime_coverage (see Coverage). An empty coverage means no controller is watching the workload, so an empty inventory says nothing about what runs. One containers item per (container, digest): a workload mid-rollout lists the same container once per image. imageDigest is "" when the controller could not resolve the digest. Entries are ordered by kind, then path. pathComplete: false means the kernel-side path walk was cut short and path is a suffix. truncated is true when more rows exist than limit.

​
GET /images/{digest}/runtime

Requires the read scope. The digest must be sha256:<64 hex> or sha512:<128 hex> (400 otherwise). Returns every path seen running from the image, aggregated across all workloads that run it.
When truncated is true, nextCursor is the after value for the next page. coverage counts the heartbeating containers running the image in the window; all zero means nobody is watching. workloads counts distinct workloads that ran the path. firstSeen and lastSeen span every workload, and origin is the most suspicious one seen in any of them.

​
POST /runtime/executables

Controller ingest; requires the ingest scope. The body is a JSON array of at most 5000 entries (larger batches are refused with 413 and nothing is written; the body limit is 8 MiB).
Entries are validated one by one: an entry with an unknown kind or source, an empty or over-4096-byte path, an invalid digest, or a missing namespace, container or workload is dropped and counted, and the rest of the batch is stored. An entry with no workload is keyed to its pod (kind: "Pod"). origin is optional: absent, or a value this broker does not know, is stored as unknown. Timestamps are UTC; a last_seen in the future is clamped to the broker’s database clock.
written counts rows inserted or changed. Re-reporting a known path only rewrites its row when last_seen moved by five minutes or more, or when the report adds something (an earlier first_seen, an eBPF sighting of a backfilled path, a complete path, a more suspicious origin), so a steady-state batch writes nothing.

​
Coverage

An inventory row proves a file ran. It cannot prove that a file did not run, because a node with the feature off writes nothing. For that, each controller heartbeats every container it tracks every 5 minutes, whether or not anything new ran, and the broker answers “was this watched continuously?” with the SQL function
observed_since is naive UTC, like every broker timestamp. covered is a negative claim (“this never ran”) that can end up in a VEX statement, so every doubt answers false. It is true only when all of these hold for the workload container and image:
  • at least one instance heartbeated within the window;
  • every such instance had the exec probe, and the library probe in mode full (exec mode cannot vouch for a library never being loaded);
  • no instance lost events in the window: the kernel could not queue them, the container hit its 4096-path cap, the broker dropped entries at ingest, or sightings arrived before the pod was known;
  • no instance still had entries waiting to reach the broker at its last heartbeat;
  • no path of the container and image is incomplete (a truncated path cannot be matched to the package that owns it), and no instance reported its inventory incomplete: the /proc backfill reads at most 4 MiB of a process’s maps, and a container cut short there stays uncovered for as long as it runs;
  • every such instance is still heartbeating on time (within 3 × interval + 60 s), or ended cleanly;
  • every such instance was either captured from its start with no gap since, or covered without a gap since before the window began. A container already running when the probe attached counts from its /proc backfill;
  • no live pod of the workload runs the container without a fresh heartbeat, for example on a node with the feature off;
  • coverage began at least window_hours ago.
Otherwise covered is false, and reason is the first failing check in this order: no_runtime_data, probes_missing, libraries_not_tracked, events_dropped, events_pending, incomplete_paths, capture_gap. A late heartbeat or a probe or mode change restarts the gap-free run at the heartbeat that ended it. A lost event keeps every window containing it uncovered. A full dedup map in the kernel loses nothing: it evicts its oldest entry, and that file is reported again on its next use.

​
POST /runtime/coverage

Controller heartbeats; requires the ingest scope. At most 5000 per post, counted while parsing (413 above that), 4 MiB body. Invalid entries are dropped and counted. A heartbeat older than the stored one is ignored, and a container’s row is frozen once a heartbeat with ended: true arrives. probe_attached_at is when the controller’s probes attached: a heartbeat whose probes attached after the previous one means they were detached in between (a controller restart, however short), and the gap-free run restarts there (absent from older controllers: no such check).
Rows not heartbeated for RUNTIME_INVENTORY_RETENTION_DAYS are pruned.

​
Capabilities

With controller.runtimeInventory.capabilities: true (and mode exec or full), the controller also counts every capability check a container makes, per capability and verdict, from an fexit program on cap_capable (or security_capable where cap_capable cannot be traced; override with RUNTIME_INVENTORY_CAP_HOOK). Counting happens in the kernel, one map entry per (container, capability, verdict, probed); the first check of each also sends an event naming the container. Heartbeats carry cap_probe and cap_hook, so the broker knows which windows had the probe on and on which hook. The security_capable fallback misses the checks commoncap makes directly (for example capset’s SETPCAP test), so a window on it is never full capability evidence. Classes:
  • used: an ordinary check that succeeded.
  • denied: an ordinary check that failed (the container asked without holding the capability).
  • probed: a CAP_OPT_NOAUDIT check that succeeded. Mostly the kernel asking whether a process is privileged: every root process’s memory admin-reserve check asks for SYS_ADMIN this way. Some gate real behaviour (a seccomp filter installed without no_new_privs, ptrace access to other processes), so probed capabilities are kept in the recommendation and named, never dropped without a person confirming. One exception: a probed-only SYS_ADMIN is left out (probedOmitted) unless the container is privileged, because the recommendation keeps or sets allowPrivilegeEscalation: false, which gives no_new_privs and removes the seccomp case; the memory reserve is a heuristic, not functionality.
Not counted:
  • Container runtime setup, decided by provenance, not by name (a container process that renames itself runc:[...] is still counted): a task whose real parent is outside every pod cgroup and that has either not exec’d since that parent created it (a runtime forking without exec, until it execs the container’s command) or exec’d exactly once, from a sealed copy of its executable (a memfd or unlinked file, or a file on a mount outside its mount namespace), and runs the same executable file as that parent or has a parent named runc* (runc init: runc execs it from a sealed copy of its own binary). The container’s command is exec’d from its rootfs, never a sealed copy, so another runtime installed as runc does not hide it; it and a process started by exec are counted. A task that has not exec’d is not setup in the host pid namespace under a parent that adopts orphans (a child subreaper such as the shim, or a pid namespace init): a hostPID container’s forked worker that outlived its parent lands there. runc 1.1 (one more re-exec before setup), a foreground crun (a subreaper) and exec into a hostPID container have their setup counted (extra checks, never hidden ones). When unsure, the check is counted. Tested in CI against real runc (overlay and memfd seals) and crun, also installed as runc.
  • Checks the kernel answers without the container’s capabilities: a task that moved into a user namespace below the container’s own (it holds every capability there), and a check against a descendant user namespace owned by the task’s euid (granted by ownership). A check against a descendant namespace owned by another uid falls back to the container’s own capability bits, so it is counted.
  • Capabilities used by a container before the probe attached: see the evidence rules below.
The evidence feeds the workload profile’s capabilities block and the capabilities part of its securityContext recommendation (see the profile contract, section 2.9). A recommendation is made only when kg_capability_coverage covers every current digest of the container for the evidence window (CAPABILITY_EVIDENCE_WINDOW_HOURS, default 168):
It is stricter than kg_runtime_coverage, which it builds on:
  • every instance in the window had the capability probe on (capabilities_not_tracked otherwise), on the cap_capable hook (capabilities_partial_hook otherwise);
  • every instance in the window was captured from its start with no gap since (capabilities_not_seen_since_start otherwise). Capabilities have no /proc backfill: a container already running when the probe attached (a new install, a controller restart) may have used a capability at startup that was never seen, so it becomes evidence only once it restarts under the probe.
A capability that was used or probed is never recommended for dropping. Capability rows are kept for at least the evidence window plus a day, whatever RUNTIME_INVENTORY_RETENTION_DAYS says, so a capability used inside the window is never pruned while the window still counts.

​
GET /workloads/{namespace}/{kind}/{name}/capabilities

Requires the read scope; charged to the read budget. Returns the profile’s capabilities block: per current container, the capabilities used (granted) and denied with counts and times, the evidence verdict and reason, the probed capabilities, the recommendation (drop: ["ALL"], add: the used and probed set, probedKept: the part kept only because of probes) when the evidence is sufficient, and unusedAdded: capabilities the container adds today that it never used or probed.

​
POST /runtime/capabilities

Controller ingest; requires the ingest scope. At most 5000 entries, counted while parsing (413 above that), 4 MiB body. count is the number of checks since the controller’s previous post of that entry; counts add up across replicas. Running containers are re-reported hourly with count: 0, which keeps them from retention: rows are pruned when no controller has reported them for RUNTIME_INVENTORY_RETENTION_DAYS, never because a capability was last used long ago.
capability is the Kubernetes spelling without CAP_ (upper case, digits and _, up to 32 characters). Invalid entries are dropped and counted in the response.