how-to
Kubernetes
Trivy Operator

How to run a scheduled vulnerability scan of a Kubernetes cluster without filling etcd

One CronJob, a read-only ServiceAccount, Trivy, and results in a bucket instead of etcd. The Dockerfile, the YAML and the script, with SBOMs built once per digest and re-matched daily.

StackRadar Team

October 18, 202612 min read

You want every image your cluster runs scanned every day, and you want the results somewhere you can query. The usual way to get that is an operator that scans in the cluster and saves each report as a Kubernetes object. We wrote about where that breaks: a report has to fit in one write to etcd, large images do not fit, and the failure is silent. This post is the other way to do it.

The short answer: one CronJob you own, running a scanner you already trust, with a read-only ServiceAccount, writing one file per image digest to a bucket or a database. Nothing is stored in the cluster. The whole thing is a Dockerfile, one YAML file and a 70-line script, all below. It builds each image's SBOM once and re-matches the stored SBOMs every day, so a run on a quiet cluster pulls nothing at all. The last section is honest about what you do not get for that price, and when a product is cheaper than keeping this running.

The design

The whole design: one CronJob, three steps, and results that leave the cluster instead of going into etcd.

Five decisions, each of which the manifests below make for you:

A CronJob, not an operator. The job runs, writes, and exits. There is no controller to upgrade, no CRDs to install, and no reconcile loop to debug. The cost is that coverage is daily, not at deploy time; more on that at the end.
Read-only RBAC. The job lists pods and nodes. It cannot create, update or delete anything, so a bug in the script cannot change the cluster.
Scan what the kubelet reports. The digest in status.containerStatuses[].imageID is what was pulled, which differs from the manifest's tag whenever a tag has moved. The KEV check uses the same trick.
Key everything by digest. A digest is immutable, so its SBOM is built once, stored, and never pulled again. Ten pods running one image are one scan.
Results leave the cluster. Object storage with a lifecycle rule, or a Postgres table. Either answers "what is new since yesterday" in one command, which a pile of CRDs does not.

The scanner image

No official image does the whole job. Trivy's image (aquasec/trivy) is Alpine with the binary and nothing else, no jq or curl. Grype's is built from scratch and has no shell at all. The kubectl image at registry.k8s.io/kubectl is distroless. So build one small image from the three, pinned to versions you chose:

Dockerfile

FROM alpine:3.22
RUN apk add --no-cache bash ca-certificates curl jq aws-cli
# The scanner and kubectl, copied from their official images: no package
# manager involved, and both pinned to a version you chose.
COPY --from=docker.io/aquasec/trivy:0.76.0 /usr/local/bin/trivy /usr/local/bin/trivy
COPY --from=registry.k8s.io/kubectl:v1.34.1 /bin/kubectl /usr/local/bin/kubectl
RUN adduser -D -u 10001 scanner
USER 10001
ENTRYPOINT ["/bin/bash"]

Build it for every architecture your nodes run (docker buildx build --platform linux/amd64,linux/arm64), because the job itself has to be schedulable on whichever node has room. Nothing in it is specific to your cluster, so one image serves all of them. For Grype, copy syft and grype in the same way and swap the two scanner lines in the script; the FAQ has the details.

The manifests

One file, applied once. Replace the image reference, the bucket, the cluster name and the two credentials.

image-scan.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: image-scan
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: image-scan
  namespace: image-scan
  # On EKS, GKE or AKS, the bucket credential is an annotation here
  # (IRSA, Workload Identity, workload identity federation), not a Secret.
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: image-scan
rules:
  # Everything the job reads. There is no create, update, patch or delete.
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["list"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: image-scan
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: image-scan
subjects:
  - kind: ServiceAccount
    name: image-scan
    namespace: image-scan
---
# One read-only registry credential, created by the registry's robot-account
# feature. The job uses this, not the workloads' own imagePullSecrets.
apiVersion: v1
kind: Secret
metadata:
  name: registry-readonly
  namespace: image-scan
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: <base64 of a docker config.json>
---
# Skip this Secret if the ServiceAccount carries a cloud identity.
apiVersion: v1
kind: Secret
metadata:
  name: bucket-credentials
  namespace: image-scan
stringData:
  AWS_ACCESS_KEY_ID: <key>
  AWS_SECRET_ACCESS_KEY: <secret>
---
# The vulnerability database and pulled layers, kept between runs so each
# run downloads only what changed.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: image-scan-cache
  namespace: image-scan
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 20Gi
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: image-scan
  namespace: image-scan
spec:
  schedule: "0 3 * * *"
  timeZone: "Etc/UTC"
  concurrencyPolicy: Forbid          # a slow run never overlaps the next one
  startingDeadlineSeconds: 3600      # a missed 03:00 still runs before 04:00
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 0                # the script handles per-image failures itself
      activeDeadlineSeconds: 14400   # four hours, then the run is killed
      ttlSecondsAfterFinished: 604800
      template:
        spec:
          serviceAccountName: image-scan
          restartPolicy: Never
          securityContext:
            runAsNonRoot: true
            runAsUser: 10001
            fsGroup: 10001           # makes the PVC writable by the scanner user
            seccompProfile:
              type: RuntimeDefault
          containers:
            - name: scan
              image: registry.example.com/platform/image-scan:2026-10-18
              command: ["/opt/scan/scan.sh"]
              env:
                - name: CLUSTER
                  value: "prod-eu-1"
                - name: BUCKET
                  value: "s3://acme-image-scans/prod-eu-1"
                # - name: AWS_ENDPOINT_URL         # MinIO or another S3-compatible store
                #   value: "https://minio.internal:9000"
                - name: TRIVY_CACHE_DIR
                  value: /cache
                - name: DOCKER_CONFIG
                  value: /docker-config
                - name: SLACK_WEBHOOK_URL
                  valueFrom:
                    secretKeyRef:
                      name: slack-webhook
                      key: url
                      optional: true
              envFrom:
                - secretRef:
                    name: bucket-credentials
                    optional: true
              resources:
                requests:
                  cpu: 500m
                  memory: 1Gi
                limits:
                  memory: 3Gi
              securityContext:
                allowPrivilegeEscalation: false
                readOnlyRootFilesystem: true
                capabilities:
                  drop: ["ALL"]
              volumeMounts:
                - name: script
                  mountPath: /opt/scan
                - name: cache
                  mountPath: /cache
                - name: docker-config
                  mountPath: /docker-config
                  readOnly: true
                - name: tmp
                  mountPath: /tmp
          volumes:
            - name: script
              configMap:
                name: scan-script
                defaultMode: 0555
            - name: cache
              persistentVolumeClaim:
                claimName: image-scan-cache
            - name: docker-config
              secret:
                secretName: registry-readonly
                items:
                  - key: .dockerconfigjson
                    path: config.json
            - name: tmp
              emptyDir:
                sizeLimit: 2Gi

Four choices in there are worth knowing about before you change them:

Registry credentials. The job mounts one read-only credential as its Docker config, and Trivy reads it the way docker pull would. The tempting alternative is to reuse the workloads' own imagePullSecrets, but that needs get on Secrets across the cluster, and a job that can read every pull secret is a job whose logs, ConfigMap and image become part of your credential boundary. If you must, scope the rule with resourceNames to the secrets you name. (StackRadar's scanner chart makes the same call: pull-secret resolution is off by default and only ever for a list of named secrets.)
The cache volume. Trivy keeps its vulnerability database and pulled layers under TRIVY_CACHE_DIR. On a PVC, each run downloads the database delta and nothing else it has seen. On an emptyDir it would download everything every night. Size it for the database plus the layers of your largest image.
Timeouts in two places. --timeout 15m per image, so one enormous image cannot eat the run, and activeDeadlineSeconds on the Job, so a hung registry cannot eat the night. concurrencyPolicy: Forbid means a slow run is skipped by the next schedule rather than doubled.
Memory. A scan of a large image unpacks its layers; a 1 GiB request and a 3 GiB limit is a safe start. Lower the limit only after watching a few runs.

The script

The script is mounted from a ConfigMap, so you can change it without rebuilding the image. It does five things: fetch the database once, list running containers, map each digest to its pods and architecture, build or fetch the SBOM per digest and scan it, then summarise.

scan.sh

#!/usr/bin/env bash
# scan.sh — scan every image this cluster is running, once per digest, and
# keep SBOMs and findings in a bucket. Needs kubectl, jq, trivy, aws, curl.
set -euo pipefail
export LC_ALL=C

: "${BUCKET:?s3://bucket/prefix for this cluster}"
: "${CLUSTER:?a name for this cluster}"
TODAY="$(date -u +%F)"
RUN="$(date -u +%Y-%m-%dT%H%MZ)"
WORK="$(mktemp -d)"

# 0. One database download per run; every scan below skips the update check.
trivy image -q --download-db-only

# 1. Every running container with the digest the kubelet reports, and its node.
#    status.*.imageID is what was pulled, digest included; spec.containers[].image
#    is only what the manifest asked for. Init containers and sidecars included.
kubectl get pods -A -o json | jq -c '
  .items[] | select(.status.phase == "Running")
  | .metadata.namespace as $ns | .metadata.name as $pod | .spec.nodeName as $node
  | ((.status.containerStatuses // []) + (.status.initContainerStatuses // []))[]
  | select(.imageID != "")
  | {ns: $ns, pod: $pod, node: $node, container: .name,
     digest: (.imageID | sub("^docker-pullable://"; ""))}' > "$WORK/containers.jsonl"

# 2. Each node's architecture, so a multi-arch image is scanned as the variant
#    its node runs, not as whatever the scanner's own CPU would pick.
kubectl get nodes -o json \
  | jq '[.items[] | {key: .metadata.name, value: .status.nodeInfo.architecture}] | from_entries' \
  > "$WORK/nodes.json"

# 3. The run manifest: one entry per digest, with every pod that runs it.
jq -c --slurpfile nodes "$WORK/nodes.json" '. + {arch: ($nodes[0][.node] // "amd64")}' \
  "$WORK/containers.jsonl" \
  | jq -s --arg run "$RUN" --arg cluster "$CLUSTER" '{
      run: $run, cluster: $cluster, failed: [],
      images: (group_by(.digest) | map({
        digest: .[0].digest,
        arch: (map(.arch) | unique),
        pods: map({ns, pod, container})
      }))}' > "$WORK/run.json"
echo "Running images: $(jq '.images | length' "$WORK/run.json")" >&2

# 4. Per digest: build the SBOM once, ever; match it against today's database every run.
while read -r digest arch; do
  key="${digest##*@}"; key="${key/:/-}"          # sha256-<hex>, safe as an object key
  sbom="$WORK/$key.cdx.json"
  if aws s3 cp --quiet "$BUCKET/sboms/$key.cdx.json" "$sbom" 2>/dev/null; then
    : # seen before: no pull
  elif trivy image -q --skip-db-update --platform "linux/$arch" --timeout 15m \
         --format cyclonedx --output "$sbom" "$digest" < /dev/null; then
    aws s3 cp --quiet "$sbom" "$BUCKET/sboms/$key.cdx.json"
  else
    echo "SCAN FAILED: $digest (registry credentials? architecture?)" >&2
    jq --arg d "$digest" '.failed += [$d]' "$WORK/run.json" > "$WORK/run.tmp" && mv "$WORK/run.tmp" "$WORK/run.json"
    continue
  fi
  trivy sbom -q --skip-db-update --format json --output "$WORK/$key.findings.json" "$sbom"
  aws s3 cp --quiet "$WORK/$key.findings.json" "$BUCKET/findings/$TODAY/$key.findings.json"
done < <(jq -r '.images[] | .digest as $d | .arch[] | "\($d) \(.)"' "$WORK/run.json")

aws s3 cp --quiet "$WORK/run.json" "$BUCKET/runs/$RUN.json"

# 5. One line for a human. Severity is Trivy's; "with a fix" is what you can act on today.
SUMMARY=$(jq -s --arg cluster "$CLUSTER" --slurpfile run "$WORK/run.json" '
  [.[] | .Results[]? | .Vulnerabilities[]?] as $v
  | { cluster: $cluster, images: ($run[0].images | length), failed: ($run[0].failed | length),
      critical: ($v | map(select(.Severity == "CRITICAL")) | length),
      critical_fixable: ($v | map(select(.Severity == "CRITICAL" and (.FixedVersion // "") != "")) | length) }' \
  "$WORK"/*.findings.json)
echo "$SUMMARY" >&2

if [ -n "${SLACK_WEBHOOK_URL:-}" ]; then
  curl -fsS -X POST -H 'Content-type: application/json' "$SLACK_WEBHOOK_URL" --data "$(jq -n --argjson s "$SUMMARY" \
    '{text: "Image scan \($s.cluster): \($s.images) images, \($s.critical) critical (\($s.critical_fixable) with a fix), \($s.failed) failed to scan"}')"
fi

Three details carry the design:

SBOM first, findings second. Step 4 asks the bucket for the digest's SBOM before it pulls anything. On a cluster that deployed nothing since yesterday, the run pulls nothing. The scan that produces findings is trivy sbom against the stored file, and it takes seconds per image.
The architecture comes from the node. For a multi-arch image the kubelet reports the manifest-list digest, and a scanner picks the variant for its own CPU unless told otherwise. The script scans each digest once per architecture it actually runs on.
Failures are recorded, not skipped. An image the job cannot pull is written to the run manifest's failed list and counted in the Slack line. A silent skip would look exactly like a clean image, which is the failure mode the operator post is about.

Create the ConfigMap, apply the file, and trigger a first run by hand:

bash

kubectl create configmap scan-script --namespace image-scan --from-file=scan.sh
kubectl apply -f image-scan.yaml
# Run it once now instead of waiting for 03:00, and watch:
kubectl create job --namespace image-scan --from=cronjob/image-scan image-scan-manual
kubectl logs --namespace image-scan -f job/image-scan-manual

The first run pulls every image and takes as long as your registry allows. Every run after that pulls only the digests that started since the last one. Expect the Slack line within a minute on a quiet cluster.

Where the results go

The bucket ends up with three prefixes:

bucket layout

s3://acme-image-scans/prod-eu-1/
  sboms/sha256-<hex>.cdx.json            # one per digest, written once, kept
  findings/2026-10-18/sha256-<hex>.findings.json   # one per digest per day
  runs/2026-10-18T0300Z.json             # digest -> pods, plus what failed

The SBOMs are small, one per digest, and they are the valuable part: with them, a CVE published tomorrow needs no new pull, only a new match. A new CVE should not need a new scan is the long version of that argument. Keep them. Findings are reproducible from the SBOMs, so let a lifecycle rule expire them:

lifecycle rule

aws s3api put-bucket-lifecycle-configuration --bucket acme-image-scans \
  --lifecycle-configuration '{"Rules": [
    {"ID": "findings-90d", "Status": "Enabled",
     "Filter": {"Prefix": "prod-eu-1/findings/"}, "Expiration": {"Days": 90}},
    {"ID": "runs-1y", "Status": "Enabled",
     "Filter": {"Prefix": "prod-eu-1/runs/"}, "Expiration": {"Days": 365}}
  ]}'

Pull once per digest, match every day. The second row is why the script keeps SBOMs and not only findings.

The question you will ask most often is "what is new since yesterday". With files in a bucket it is a diff of two sorted lists:

new critical findings since yesterday

# Critical findings that are in today's run and were not in yesterday's.
B=s3://acme-image-scans/prod-eu-1
for day in 2026-10-17 2026-10-18; do
  aws s3 sync --quiet "$B/findings/$day" "./$day"
  for f in "./$day"/*.findings.json; do
    jq -r --arg img "$(basename "$f" .findings.json)" \
      '.Results[]?.Vulnerabilities[]? | select(.Severity == "CRITICAL")
       | "\($img) \(.PkgName) \(.VulnerabilityID)"' "$f"
  done | sort -u > "$day.txt"
done
comm -13 2026-10-17.txt 2026-10-18.txt     # new today
comm -23 2026-10-17.txt 2026-10-18.txt     # gone since yesterday (fixed, or the image stopped)

If you would rather have a database, load each finding as a row and the same question becomes SQL. A table like this, filled by a few lines of jq at the end of the script, answers it across every cluster at once:

Postgres

CREATE TABLE finding (
  run_date   date        NOT NULL,
  cluster    text        NOT NULL,
  digest     text        NOT NULL,
  package    text        NOT NULL,
  installed  text        NOT NULL,
  cve        text        NOT NULL,
  severity   text        NOT NULL,
  fixed_in   text,
  PRIMARY KEY (run_date, cluster, digest, package, cve)
);

-- New critical findings since yesterday, across every cluster.
SELECT cluster, digest, package, cve, fixed_in
FROM finding WHERE run_date = CURRENT_DATE AND severity = 'CRITICAL'
EXCEPT
SELECT cluster, digest, package, cve, fixed_in
FROM finding WHERE run_date = CURRENT_DATE - 1 AND severity = 'CRITICAL';

Either way, the "gone since yesterday" list is as useful as the new one: it tells you a fix landed, or that an image stopped running. The run manifest says which, because it names the pods.

Alerting

The script already posts one line to Slack per run. Resist making it longer: a daily message with forty CVE IDs is a message nobody reads. The two numbers that earn a look are the count of critical findings with a fix available, because those are actionable today, and the count of images that failed to scan, because those are the ones you know nothing about. If you run Prometheus, push the same five numbers to a Pushgateway with one more curl and alert on failed > 0 and on critical_fixable rising. Anything subtler than that belongs in the diff above, run when someone asks.

What this does not give you

The setup above is adequate for a lot of teams, and we would rather you ran it than ran nothing. Be clear-eyed about what it leaves out, because each item is a thing you will eventually build or buy.

Coverage is daily. An image deployed at 09:00 is scanned at 03:00 the next day. If you deploy many times a day, that window is where a bad image lives unexamined.
Severity is the only ranking. Trivy's CRITICAL is a CVSS band. It does not know which of your findings are being exploited this week. Joining EPSS and KEV onto the findings is a few more lines, and how to prioritise container vulnerabilities with EPSS and KEV shows them.
One cluster per bucket prefix. Six clusters are six prefixes and a merge step you write yourself, with no way to ask "which clusters run this digest" without downloading everything.
No history page. The data is there, but every question is a jq invocation. After three months you will want a table with a column per day, and that is a small web app.
Pods, not owners. The run manifest maps a digest to pods. Turning that into the Deployment, the Helm release or the Argo CD application that someone has to change is an ownerReferences walk that is not in the script.
No base-image split. Most findings in most images are in the base layer and are fixed by a rebuild, not by anyone on the team that owns the service. The script does not separate them. Most of your container vulnerabilities are not yours to fix explains why that split matters.

When Trivy Operator is the better answer anyway: a small cluster whose images carry hundreds of findings rather than thousands, where kubectl get vulnerabilityreports and the Prometheus metrics are what you want, and where you also want its config audit, RBAC and exposed-secret reports, which this CronJob does not do. The etcd ceiling only bites at a scale you may never reach.

When a product is cheaper: when the person maintaining the script is the person whose time you are trying to save. Count the hours on the first bullet list above, then compare them with a licence. That is the honest comparison, and for a two-person platform team it often comes out in the product's favour.

What StackRadar does

We build StackRadar, so weigh this section accordingly. Its scanner is the loop above, packaged: a single Helm release whose agent watches the Kubernetes API for pod starts rather than waiting for a schedule, asks the server whether it already holds an SBOM for that digest before pulling anything, builds a CycloneDX SBOM with Syft for each new digest, and uploads it. A periodic sweep every six hours re-lists the cluster so nothing is missed between events. Its ClusterRole is read-only, and the chart installs no CRDs, so there is nothing in etcd to outgrow. The architecture reference lists every field that leaves the cluster.

The server side is the part this post told you to build or buy: findings are rows, not objects, so a report has no ceiling; the OSV mirror is refreshed hourly and every stored SBOM is re-matched when it changes; each finding carries its EPSS probability and KEV status, and the Radar Score ranks by them; and a digest is tied to the workload, the Helm release and the Argo CD application you would change to fix it, across every cluster in one view. The continuous scanning page covers the mechanics, and StackRadar vs Trivy Operator the trade-offs.

The server, dashboard and chart are open source under AGPL-3.0, so one cluster can self-host the whole thing for nothing but a Postgres. Several clusters in one dashboard, with email alerts and nothing to run but the scanner, is the cloud, which starts with a free trial.

Frequently asked questions

How do I run Trivy as a Kubernetes CronJob?

Build a small image that holds trivy, kubectl, jq and a bucket client, give it a ServiceAccount whose ClusterRole can only list pods and nodes, and run a script from a CronJob with concurrencyPolicy set to Forbid. The script lists the image digests the kubelets report, runs trivy image once per digest you have not seen before, and writes the output to object storage or a database rather than to a Kubernetes object. Keep the vulnerability database on a persistent volume so each run downloads only the delta.

Why not store scan results as Kubernetes resources?

Every Kubernetes object lives in etcd, and a write to etcd has to fit two limits: a 1.5 MiB default request limit in etcd itself and a 2 MiB client limit in the API server. An image with a few thousand findings produces a report larger than that, and the write fails silently. Findings also grow on their own as advisories are published, and the useful queries cut across every report. That is a database workload, and etcd is the control plane’s memory, not a database.

How often should I scan container images in Kubernetes?

Match the stored SBOMs against the vulnerability database daily, because that is how often the database changes in ways that matter. Pull and inventory an image only once per digest, since the image itself never changes. A daily CronJob gives you a window of up to a day between a deploy and its first scan; an event-driven scanner closes that window by scanning when the pod starts, which matters if you deploy many times a day.

Does a scheduled scan need to pull every image every day?

No. An image digest is immutable, so its package list is too. Build a CycloneDX SBOM the first time you see a digest, store it, and on every later run scan the stored SBOM instead of the image. Trivy and Grype both accept an SBOM as input. A daily run then pulls only the digests that started running since yesterday, which on most clusters is a handful.

What permissions does a cluster vulnerability scan need?

To list pods cluster-wide, and to list nodes if you want to scan multi-arch images as the variant each node runs. That is a ClusterRole with the list verb on pods and nodes, and nothing else. Registry access should come from one dedicated read-only credential rather than from reading every workload’s imagePullSecrets, which would need get on Secrets across the cluster.

Can I use Grype instead of Trivy in the CronJob?

Yes. Copy the syft and grype binaries into the same image, generate the SBOM with syft and scan it with grype, which reads a stored SBOM with the sbom: prefix. The official grype image is built from scratch with no shell, so it cannot run the script on its own. Grype keeps its database under the directory set by GRYPE_DB_CACHE_DIR, which plays the role TRIVY_CACHE_DIR plays here.

See it in your own cluster

Fourteen days free in the cloud, or free to self-host. Run one Helm command and expect your first findings within minutes. The whole product is open source, so you can audit what runs in your cluster.

Try StackRadar Cloud free Try the live demo

Not ready to install anything? The free CLI scans every image your cluster is running from your laptop, with no account: