Runtime Inventory Endpoints

Overview

The controller watches execve and shared-library mappings per container and reports the distinct set of files it sees, keyed by workload, container and image digest. A path is stored once per (workload, container, digest), not per event, so the inventory grows with what a workload runs rather than how often it runs it. Processes that were already running when the controller started are backfilled from /proc and marked source: "backfill"; a later runtime sighting upgrades them to ebpf. Joined with an image’s SBOM on digest and path, this separates packages an image ships from packages it actually loads. Each path also records where the file lived when it ran (origin), which is what drift detection uses to flag binaries the image did not ship.

Enabling it

Off by default until its per-node overhead is measured. Set the chart value controller.runtimeInventory.mode:

Mode	What is captured
`off` (default)	Nothing; no probe is loaded
`exec`	Executed files (`tp_btf/sched_process_exec`)
`full`	Executed files plus files mapped executable, i.e. shared libraries and the dynamic loader (`fentry/security_mmap_file`). Falls back to `exec` on a kernel that cannot attach the fentry probe

Each (container, file) is reported once: the probe deduplicates in the kernel, so after warm-up a container costs one hash lookup per exec or executable mapping. Containers already running when the probe attached are backfilled once from /proc. Namespaces in controller.excludedNamespaces and pods annotated kguardian.dev/runtime-inventory: "off" are not recorded. An empty inventory means nothing was observed, not that nothing ran: the feature may be off on that node, the pod may be excluded, or the process may have run before the controller started and exited before the backfill.

Origin

`origin`	Meaning
`image`	In an image layer (overlayfs lower layer, never modified)
`otherFs`	Not on overlayfs: a volume (emptyDir, configMap, hostPath, PVC), or a root filesystem from a non-overlay snapshotter
`writableLayer`	In the container’s writable (overlayfs upper) layer: written or modified after the container started
`memfd`	An anonymous memory file (`memfd_create`), never on disk
`deleted`	Unlinked while running
`unknown`	Not determined: a `/proc` backfill (which cannot see layers; the broker stores every backfill row as `unknown` unless its path shows `deleted` or `memfd`), or overlayfs without its type in kernel BTF

When sightings of one path disagree, the most suspicious wins (in the order unknown < image < otherFs < deleted < writableLayer < memfd) and is never downgraded: one replica running the file from its writable layer is the finding, whatever the others did. deleted, writableLayer and memfd together are the “unshipped” origins. A file changed in place while the container runs (an image binary overwritten, which copies it up into the writable layer, or a file unlinked and then executed through an open descriptor) is reported again with its new origin: the probe’s dedup key includes the file’s state. Not observed: a mapping made executable later with mprotect(PROT_EXEC) (JIT compilers, some custom loaders), and code an interpreter reads as data (scripts, .pyc, jars). Only execve and executable mmap are seen. Rows not seen for RUNTIME_INVENTORY_RETENTION_DAYS (chart: broker.runtimeInventory.retentionDays, default 30) are pruned.

GET /workloads/{namespace}/{kind}/{name}/runtime

Requires the read scope when broker auth is on. Returns 200 with an empty containers list when nothing has been reported for the workload.

Query	Description	Default
`container`	Only this container	all
`kind`	`exec` or `lib`; anything else is `400`	both
`origin`	One origin, or `unshipped` for `deleted`, `writableLayer` and `memfd`; anything else is `400`	all
`limit`	Rows (paths) returned, capped at 5000	`1000`
`window_hours`	Coverage window, clamped to 1-720	`24`

{
  "namespace": "prod",
  "kind": "Deployment",
  "name": "web",
  "containers": [
    {
      "containerName": "app",
      "imageDigest": "sha256:0123…cdef",
      "entries": [
        {
          "path": "/usr/sbin/nginx",
          "kind": "exec",
          "source": "ebpf",
          "origin": "image",
          "pathComplete": true,
          "firstSeen": "2026-09-26T10:00:00",
          "lastSeen": "2026-09-27T08:15:00"
        }
      ]
    }
  ],
  "truncated": false,
  "coverage": [
    {
      "containerName": "app",
      "imageDigest": "sha256:0123…cdef",
      "instances": 3,
      "lastHeartbeat": "2026-09-27T08:15:00",
      "mode": "full",
      "librariesTracked": true,
      "covered": true,
      "observedSince": "2026-09-25T10:00:00",
      "reason": null,
      "windowHours": 24
    }
  ]
}

coverage lists every (container, digest) of the workload with a controller heartbeat in the window, with the answer of kg_runtime_coverage (see Coverage). An empty coverage means no controller is watching the workload, so an empty inventory says nothing about what runs. One containers item per (container, digest): a workload mid-rollout lists the same container once per image. imageDigest is "" when the controller could not resolve the digest. Entries are ordered by kind, then path. pathComplete: false means the kernel-side path walk was cut short and path is a suffix. truncated is true when more rows exist than limit.

GET /images/{digest}/runtime

Requires the read scope. The digest must be sha256:<64 hex> or sha512:<128 hex> (400 otherwise). Returns every path seen running from the image, aggregated across all workloads that run it.

Query	Description	Default
`kind`	`exec` or `lib`	both
`limit`	Rows returned, capped at 5000	`1000`
`after`	Resume after this `<kind>:<path>`: the previous page’s `nextCursor`	start
`window_hours`	Coverage window, clamped to 1-720	`24`

{
  "digest": "sha256:0123…cdef",
  "entries": [
    {
      "path": "/lib/x86_64-linux-gnu/libc.so.6",
      "kind": "lib",
      "pathComplete": true,
      "origin": "image",
      "firstSeen": "2026-09-20T10:00:00",
      "lastSeen": "2026-09-27T08:15:00",
      "workloads": 3
    }
  ],
  "truncated": false,
  "nextCursor": null,
  "coverage": {
    "workloads": 3,
    "instances": 7,
    "lastHeartbeat": "2026-09-27T08:15:00",
    "librariesTracked": true,
    "instancesWithDrops": 0,
    "windowHours": 24
  }
}

When truncated is true, nextCursor is the after value for the next page. coverage counts the heartbeating containers running the image in the window; all zero means nobody is watching. workloads counts distinct workloads that ran the path. firstSeen and lastSeen span every workload, and origin is the most suspicious one seen in any of them.

POST /runtime/executables

Controller ingest; requires the ingest scope. The body is a JSON array of at most 5000 entries (larger batches are refused with 413 and nothing is written; the body limit is 8 MiB).

[
  {
    "pod_namespace": "prod",
    "pod_name": "web-7d9f-abcde",
    "workload_kind": "Deployment",
    "workload_name": "web",
    "container_name": "app",
    "image_digest": "sha256:0123…cdef",
    "kind": "exec",
    "path": "/usr/sbin/nginx",
    "path_complete": true,
    "source": "ebpf",
    "origin": "image",
    "first_seen": "2026-09-26T10:00:00",
    "last_seen": "2026-09-26T10:05:00"
  }
]

Entries are validated one by one: an entry with an unknown kind or source, an empty or over-4096-byte path, an invalid digest, or a missing namespace, container or workload is dropped and counted, and the rest of the batch is stored. An entry with no workload is keyed to its pod (kind: "Pod"). origin is optional: absent, or a value this broker does not know, is stored as unknown. Timestamps are UTC; a last_seen in the future is clamped to the broker’s database clock.

{ "accepted": 1, "dropped": 0, "written": 1 }

written counts rows inserted or changed. Re-reporting a known path only rewrites its row when last_seen moved by five minutes or more, or when the report adds something (an earlier first_seen, an eBPF sighting of a backfilled path, a complete path, a more suspicious origin), so a steady-state batch writes nothing.

Coverage

An inventory row proves a file ran. It cannot prove that a file did not run, because a node with the feature off writes nothing. For that, each controller heartbeats every container it tracks every 5 minutes, whether or not anything new ran, and the broker answers “was this watched continuously?” with the SQL function

kg_runtime_coverage(cluster text, namespace text, kind text, name text,
                    container text, image_digest text, window_hours integer)
  RETURNS TABLE (covered boolean, observed_since timestamp, reason text)

observed_since is naive UTC, like every broker timestamp. covered is a negative claim (“this never ran”) that can end up in a VEX statement, so every doubt answers false. It is true only when all of these hold for the workload container and image:

at least one instance heartbeated within the window;
every such instance had the exec probe, and the library probe in mode full (exec mode cannot vouch for a library never being loaded);
no instance lost events in the window: the kernel could not queue them, the container hit its 4096-path cap, the broker dropped entries at ingest, or sightings arrived before the pod was known;
no instance still had entries waiting to reach the broker at its last heartbeat;
no path of the container and image is incomplete (a truncated path cannot be matched to the package that owns it), and no instance reported its inventory incomplete: the /proc backfill reads at most 4 MiB of a process’s maps, and a container cut short there stays uncovered for as long as it runs;
every such instance is still heartbeating on time (within 3 × interval + 60 s), or ended cleanly;
every such instance was either captured from its start with no gap since, or covered without a gap since before the window began. A container already running when the probe attached counts from its /proc backfill;
no live pod of the workload runs the container without a fresh heartbeat, for example on a node with the feature off;
coverage began at least window_hours ago.

Otherwise covered is false, and reason is the first failing check in this order: no_runtime_data, probes_missing, libraries_not_tracked, events_dropped, events_pending, incomplete_paths, capture_gap. A late heartbeat or a probe or mode change restarts the gap-free run at the heartbeat that ended it. A lost event keeps every window containing it uncovered. A full dedup map in the kernel loses nothing: it evicts its oldest entry, and that file is reported again on its next use.

POST /runtime/coverage

Controller heartbeats; requires the ingest scope. At most 5000 per post, counted while parsing (413 above that), 4 MiB body. Invalid entries are dropped and counted. A heartbeat older than the stored one is ignored, and a container’s row is frozen once a heartbeat with ended: true arrives.

[
  {
    "pod_namespace": "prod", "pod_name": "web-7d9f-abcde",
    "workload_kind": "Deployment", "workload_name": "web",
    "container_name": "app", "image_digest": "sha256:0123…cdef",
    "container_id": "5e1e2d3c…", "node_name": "node-1",
    "mode": "full", "exec_probe": true, "lib_probe": true,
    "start_mode": "start", "tracking_since": "2026-09-26T10:00:00",
    "events_dropped": 0, "unsent": 0, "incomplete": false, "ended": false,
    "heartbeat_at": "2026-09-27T08:15:00", "heartbeat_secs": 300
  }
]

Rows not heartbeated for RUNTIME_INVENTORY_RETENTION_DAYS are pruned.

​Overview

​Enabling it

​Origin

​GET /workloads/{namespace}/{kind}/{name}/runtime

​GET /images/{digest}/runtime

​POST /runtime/executables

​Coverage

​POST /runtime/coverage