cataloger sidecar, which
returns the image’s packages. The feature is off by default; with
nodeCatalog.enabled=false the chart renders exactly what it rendered
before the feature existed.
Enable it
It needs scoped Broker authentication and acatalog key
in the same Secret. The key is optional to the pods: without it the Broker
answers the catalog routes 503 and the Controller idles, nothing crash
loops.
-f <your values file> or --reset-then-reuse-values
(Helm 3.14 and later). Plain --reuse-values renders the new chart over the
previous chart’s defaults, so none of the nodeCatalog defaults below are
in the values; the chart then falls back to the same defaults, built in.
With --set, use --set-string for a value that looks like a number but
must stay a string (an image tag such as 1.2, a Secret key name such as
2026), or --set-json for an object such as nodeCatalog.worker.appArmorProfile.
Enabling restarts the Controller DaemonSet and the Broker once (new env
vars and the sidecar). Rotating the catalog key needs both restarted
again. It also needs Broker and Controller releases that include the node
catalog (Broker 1.20.0 and Controller 1.17.0 or later) and the
cataloger image; an older Broker answers 404 and the Controller idles.
The chart refuses to render when nodeCatalog.enabled=true and
broker.auth is off or in shared mode, when nodeCatalog.epoch is above
nodeCatalog.maxEpoch, when nodeCatalog.maxHoldSeconds does not outlast
nodeCatalog.scanTimeoutSeconds, and when the worker’s memory limit cannot
hold memoryLimit + tmpLimit + 128Mi. values.schema.json checks the
types and ranges below.
Check progress with GET /catalog/coverage and GET /catalog/status?node=<name>
on the Broker (read scope). With broker.metrics.prometheusRule.enabled
the chart also ships the kguardian-broker-node-catalog PrometheusRule:
token missing, lsm_denied skips, scan failures (timeout, oom,
error) above 20% of grants, grants with no completed SBOM in 6 hours
that no node skipped (silenced while grants=false), queued work with no
grant for 25 hours, and scans deferred by node pressure for 2 hours.
Kill switch: --set nodeCatalog.grants=false stops every new grant at
the Broker (a Broker restart, no Controller restart). Uploads under a live
lease still complete.
Vulnerability matching. The Broker stores node SBOMs; the supplychain
component matches them with Grype. That needs supplychain.enabled,
supplychain.grype.enabled (the matcher) and the node SBOM source,
supplychain.sources.node.enabled, which follows nodeCatalog.enabled
unless set to true or false. With the source on and the matcher off,
the component ignores it with a warning and NOTES says so. Findings come
back as source grype with node among their SBOM sources. It needs a
supplychain 0.2.0 or later; older ones ignore the setting. A name set
in supplychain.env (for example NODE_SBOM_INTERVAL) replaces the
chart’s own, and the chart refuses an interval of zero or a
nodeGroupMaxWait under 1m, which the component would not start with.
Helm values
The Controller’s other
NODE_CATALOG_* variables (below) keep their
defaults; the chart pins NODE_CATALOG_SOCKET to
/run/kguardian/catalog/worker.sock and sets POD_UID from the downward
API.
The sidecar
Thecataloger container in the Controller pod runs capability model (i)
from cataloger/README.md:
runAsUser: 0,runAsNonRoot: false,allowPrivilegeEscalation: false,readOnlyRootFilesystem: true,seccompProfile: RuntimeDefault; capabilitiesdrop: [ALL],add: [DAC_READ_SEARCH, SETUID, SETGID]. The uid-0 parent parses nothing; every scan runs in a fresh child as uid 2000000000 withDAC_READ_SEARCHalone, under seccomp.- Network: the Controller pod is
hostNetwork, so the sidecar shares the node’s network namespace. The uid-0 parent opens only its Unix socket, but nothing but its own code stops it opening others. The scan child, the only process that parses image content, has a seccomp filter that deniessocketandsocketpairof every family. - No token: the pod mounts the service-account token for the Controller,
and the sidecar gets an empty directory over
/var/run/secrets/kubernetes.io/serviceaccountinstead. No Broker token, no host mounts. /run/kguardian/catalog: an emptyDir shared with the Controller. The worker makes it0700and the socket0600, both owned by uid 0, and the Controller refuses anything else. An emptyDir is created owned by uid 0, andfsGrouponcontroller.podSecurityContextchanges only its group and mode, never the owner, so the check holds with or without it (the worker’sfchmodto0700also clears the set-group-ID bit kubelet adds)./tmp: a memory-backed emptyDir withsizeLimitequal totmpLimit. Its pages count against the sidecar’s memory limit, which is why the limit must holdmemoryLimit+tmpLimit+ 128Mi: the worker’s own limits then trip before a cgroup OOM kill, which on cgroup v2 takes the whole container (kubelet setsmemory.oom.group). Nodes with kubelet’ssingleProcessOOMKill: true(Kubernetes 1.32 and later) kill only the scan child.- Liveness runs
kguardian-cataloger ping, which the worker answers even while a scan runs; there is no readiness probe. The worker exits instead of idling when it cannot run safely (a badCATALOG_*setting, a capability set it cannot reduce, a uid-0 parent withoutSETUID/SETGID, a scan child that fails to start, a socket directory not owned by uid 0). Kubernetes then restarts it, and while it is not running the Controller pod reports NotReady, which also holds a DaemonSet rollout on that node. The Controller itself keeps capturing; cataloging on that node reportsworker_unavailable. - Memory: requests 512Mi, limits 1Gi. The request is reserved on every node. Under node memory pressure the kubelet evicts pods using more than their request first, and this is the Controller’s pod: a lower request saves memory per node but makes the pod an earlier eviction candidate while a large scan runs; a request equal to the limit avoids that at 1Gi per node.
container_t with s0-s0:c0.c1023 lets the
worker read another container’s files through the handle it is given: the
category range lifts MCS separation only, and container_t stays the
confined container domain. With the default per-pod label every read is
denied (lsm_denied). Never use super_t, spc_t or control_t; the
schema refuses them. Nodes without SELinux ignore the setting.
Cloud credentials. Workload-identity webhooks inject credentials into
every container of a pod whose service account carries an identity. If the
Controller’s service account is given one, the chart keeps it out of the
sidecar with pod annotations the webhooks honour:
eks.amazonaws.com/skip-containers: cataloger (the Amazon EKS pod identity
webhook, which serves IRSA) and azure.workload.identity/skip-containers: cataloger (Azure Workload Identity). Check your EKS Pod Identity add-on
version honours the same annotation before relying on it. GKE Workload
Identity injects nothing, but it does not apply to hostNetwork pods: they
use the node’s service account through the metadata server. The same holds
on every cloud: from the node’s network namespace the sidecar’s parent can
reach the instance metadata service (EC2, Azure, GCE) and the node’s
credentials, exactly as the Controller already can. The scan child cannot
(no sockets). Where that matters, restrict metadata access at the node.
Pod Security. The Controller pod is already privileged, so its
namespace must allow the privileged Pod Security level; the sidecar asks
for less than the Controller container does. The chart ships no
NetworkPolicy or admission policy that selects the Controller pod.
Environment variables
Controller (kguardian-controller DaemonSet)
With NODE_CATALOG unset or off, the Controller reads no other variable in
this table and starts nothing: no task, no socket, no Broker call.
Values over a maximum are clamped with a warning.
How the worker is verified
Anything in the pod that can write to the sharedemptyDir could plant a
socket there and receive container root handles, so the Controller pins
the socket to the worker before every connection
(cataloger/PROTOCOL.md section 1.1):
- The socket file. The socket is opened
O_PATH|O_NOFOLLOWand must be a socket owned by uid 0 with mode0600; its directory is openedO_PATH|O_NOFOLLOW|O_DIRECTORYand must be owned by uid 0 with mode0700. Symlinks are never followed. The connect goes through/proc/self/fd/<n>of thatO_PATHhandle, so the file that was checked is the one connected to. This proves the socket was created by a uid-0 process with theemptyDirmounted. - The peer’s uid. After connecting,
SO_PEERCREDmust report an allowed uid (0). - The peer’s container (Linux 6.5 and later).
SO_PEERPIDFDpins the worker process, and its cgroup must be a sibling of the Controller’s own container, which means the same pod.
How the container root is verified
For a granted digest, the Controller picks a running container with it and:- asks containerd for the task’s pid, and for the upperdir of the container’s
snapshot (
Containers.Get, thenSnapshots.Mounts); - opens
/proc/<pid>as a handle that dies with the process, checks the cgroup (pod and container) and start time, opensrootthrough it, and checks the identity again (pid reuse cannot swap the process); - requires the
/entry of the container’s mountinfo to be a whole overlay mount (root field/) with an upperdir, requires the root handle to be on that mount (statxSTATX_MNT_ID), and requires the upperdir to be the one containerd reports for the container’s snapshot. Anything else isunsupported_rootfs; lazily pulled roots (stargz, SOCI, nydus, FUSE) arelazy_snapshotter. If containerd cannot answer (unreachable, an error, a timeout), that is a fact about the node, not the image: the next replica is tried, and if none works the grant endspid_gone.
Drift
A container that changed its packages at runtime is not the image. The Controller looks at the four package databases (lib/apk/db/installed,
var/lib/dpkg/status, var/lib/rpm, usr/lib/sysimage/rpm) in the
container’s upperdir, resolved under the host root with RESOLVE_IN_ROOT,
one name at a time with no symlinks followed. These are drift, and the
next replica is tried:
- the database written, or deleted (an overlay whiteout);
- a parent directory deleted (a whiteout), replaced by a file or symlink,
or deleted and recreated (
trusted.overlay.opaqueoruser.overlay.opaqueset toyorx).
partial (drift_unknown). The same applies when the
Controller lacks CAP_SYS_ADMIN (it is privileged by default): without it
trusted.overlay.* reads as absent, so it logs once at startup and every
SBOM from the node is partial.
Language packages. Deleting a Python, Node or Ruby package does not
touch those databases. The Controller also looks for whiteouts or opaque
directories in the system-wide locations only:
usr/lib/python*/{site,dist}-packagesandusr/local/lib/python*/{site,dist}-packages;usr/lib/node_modulesandusr/local/lib/node_modules;usr/local/bundle/gems,var/lib/gems/*/gemsandusr/local/lib/ruby/gems/*/gems.
partial (lang_whiteout);
a directory that could not be read completely makes it partial too
(lang_whiteout_unknown). Deletions in application-local trees
(/app/node_modules, a virtualenv) are not detected.
Kernel requirements
Memory
One scan runs at a time. The worker’s response is at mostNODE_CATALOG_MAX_RESPONSE_BYTES (16 MiB by default). It is parsed into
bounded structures that discard over-long strings, invalid paths and
anything past the caps while parsing, and the raw response is freed as
soon as parsing ends. The upload moves components into one page at a time
(at most 7 MiB encoded). The worst-case extra memory is about the response
size plus 2.5 times it for the parsed package list: roughly 56 MiB at the
default, and roughly 220 MiB at the 64 MiB ceiling. Size the Controller’s
memory limit for that before raising it.
A node the Broker refuses
If the Broker answers a claim with 422 (for exampleNODE_CATALOG_EPOCH
above the Broker’s NODE_CATALOG_MAX_EPOCH), the Controller logs one
error naming both settings and stops claiming. It keeps sending an empty
offer (epoch 0) every NODE_CATALOG_IDLE_SECS, so the Broker still records
the node and its platform with no claims, and repeats a warn at most once
an hour. Once an hour it sends the empty offer under its real epoch
instead; when the Broker accepts that (its limit was raised), claiming
resumes without a restart. GET /catalog/status?node=<name> shows such
a node as seen recently, with no claims.
A 503 from the claim route (catalog not configured on the Broker) backs off
from 60 s, doubling, to at most 30 minutes, and is logged once.
SBOM sources and trust
The worker itself (its sandbox, protocol and Syft pin) is described incataloger/README.md.
Its SBOM is one source among several. The Broker keeps each source’s SBOM
separately, keyed (digest, source), and one never replaces another:
scanned means a scanner read the running image in the cluster. Only
verified may be presented as signed, and nothing produces it yet. The
default SBOM a read returns is Trivy’s if there is one, else the node’s,
else a registry one. The merge rules are in the
supplychain README.
A node SBOM supports installed_not_observed (the Background tier) only
when it is full, every node that ran the container reports the SBOM’s
platform, and the package’s file list was neither cut nor included
interpreted content. Otherwise the package stays unknown, with reason
sbom_incomplete, platform_mismatch or interpreted_content, and is
ranked as if loaded.
In the UI, #/images shows a node SBOM as “Cataloged on node
(linux/arm64)”, with “partial” or “OS packages only” when it is not full.
An image with no SBOM from any source and no vulnerability report, whose
catalog found no packages (no_packages_found) or failed (backing off),
reads “Not assessable” with the reason. It never reads as zero CVEs. A
pending image whose last node could not catalog it (a per-node reason such
as lsm_denied) reads “Node catalog: pending · last node: LSM denied”:
that reason skips one node for 24 hours, and other nodes may still catalog
it. The coverage banner is cluster-wide, and shows only while the Broker
has a catalog token.
Reading /catalog/coverage and /catalog/status
Both need the Broker’s read scope. A Broker without the node catalog
answers 404; the UI then hides its coverage banner.
GET /catalog/coverage is cluster-wide:
runningImagescounts the digests running now.trivycounts those with an SBOM or report from Trivy Operator,nodethose with a node SBOM; they overlap, andtrustedis the union.coverageRatioistrusted / runningImages.grantsEnabled: falsemeans the kill switch is on;tokenConfigured: falsemeans the Broker has noBROKER_TOKEN_CATALOG, so no node SBOM can be stored.byStatecounts claim rows.pendingincludes rows released by a node that could not catalog them (they wait for another node), andfailedrows back off before they are granted again.byReasoncounts rows that are not done, by the reason their last attempt ended. It says where to look: an LSM, a kernel, memory.platformscounts the nodes that have offered, by platform. A node missing here has never claimed: its Controller is off, idle, or refused.
GET /catalog/status?node=<name> is one node:
platform and seenAt are null for a node that never offered. claims
are the leases it holds now (at most 64 listed), cataloged the digests it
completed, byReason the digests it last released and why, and skipped
the digests it is skipped for over the next 24 hours.
Runbook
Kill switch.nodeCatalog.grants=false, as in Enable it.
Stored SBOMs stay, and the Controllers need no restart: they keep offering
and get nothing back. grantsEnabled in /catalog/coverage reads
false, and the stalled and not-claiming alerts are silenced. Set it back
to true to resume.
Rotating the catalog token. Write the new value under the Secret’s
catalog key, as in Enable it, then restart both the Broker
and the Controller DaemonSet; both read the token only at startup. Between
the two restarts, a Controller on the old token is refused and retries;
nothing is lost, since a lease that runs out is granted again.
Alerts (the kguardian-broker-node-catalog PrometheusRule):
Other reasons in
byReason that are normal churn, not faults: pid_gone,
drift and exited_before_catalog (the container went away or changed;
another instance is tried). Per-node reasons (sandboxed,
lazy_snapshotter, unsupported_rootfs, kernel_unsupported,
caps_unavailable, worker_unavailable, no_cataloger) skip that node
for 24 hours while others try; the image stays pending, and is retried on
that node after the 24 hours.
no_packages_found is final: the image has nothing the cataloger
recognises.