Zero findings is not a clean image: two matching bugs and how we found them

A scanner that reports nothing for an image is saying one of two things. Either it looked at every package and found no advisory that applies, or it could not look. The output is the same in both cases: an empty list, usually with something green next to it.

On 6 September we checked our own matcher against two other scanners on the same eight images, and found it wrong in both directions at once. One image carried 140 findings that did not exist. Twenty-four other images had no operating-system findings at all, because we were not matching their packages against anything. This post is what the two bugs were, how we fixed them, what is still not fixed, and how to run the same check on whichever scanner you use.

The short answer: matching a package to an advisory depends on both sides spelling the package's name and version the same way, and they often do not. When the spellings differ, nothing errors. You get a wrong count. The test is to scan an old image of every distro you run and to compare two scanners package by package, and it takes about ten minutes.

The comparison

We run a public chart index that renders public Helm charts, scans the images they deploy, and publishes the findings. It uses the same matcher as the product, so it is also the largest test set we have. For the Harbor chart there were two other answers to compare with: the Trivy security report Artifact Hub publishes for the chart, and osv-scanner scan image run on each of its eight images.

Total counts are a poor comparison. Three scanners will never agree on a total, because they read different advisory sources and count aliases differently, and a difference of a few findings says nothing. So we compared per package: for each package in each image, how many findings does each scanner report? Two things stood out straight away. One package had fourteen times more findings from us than from the others. And across the index, some distributions had no findings on any operating-system package, on any image.

Bug one: 140 findings on one package

The image was goharbor/trivy-adapter-photon:v2.15.2. We reported 183 findings for it, and 140 of them were on one component: the Go standard library inside one binary. Trivy and OSV-Scanner each reported ten for the same binary.

The cause was the version string. The binary was built with GOEXPERIMENT=jsonv2, and Go records that in the version it stamps on the binary: go1.26.4 X:jsonv2. Syft writes it into the SBOM as go1.26.4-X:jsonv2.

Go advisories describe their affected ranges as semantic versions, so our matcher tried its semver comparator first. That comparator saw a string starting with the letter g, read no numbers from it, and treated the version as lower than every version it was compared with. Every advisory whose range starts at zero then counted as affecting this binary, back to ones fixed in Go 1.6. We had a Go-specific comparator as a fallback, which knows to drop the go prefix, but the fallback was only used when it was certain of its answer, and the : in the suffix made it uncertain. So the wrong answer stood.

Two details made this worse than a long list.

• One of the 140 was on the CISA KEV catalogue. It was CVE-2020-0601, fixed in Go 1.13.7 in January 2020, reported against a Go release from 2026, and it was the top finding on the chart's page. A false positive that is marked as exploited in the wild goes to the top of every list sorted by risk.
• The same bug hid three real findings. Three advisories that do apply to Go 1.26.4 have ranges that start at 1.26.0. A version that sorts below everything sorts below 1.26.0 too, so those were reported as not affected.

It was not one image either. 37 SBOMs in the index had a standard-library version with an experiment marker.

The fix has two parts. The marker is now removed before the version is compared: an experiment changes how the binary was built and not which release it is, so removing it loses nothing. And a version with no numbers in it is no longer given a place in the order. The comparison returns no answer, the finding is marked unconfirmed, and it is not counted. The first part fixes this spelling. The second is for the next spelling nobody has seen yet. The image now has 50 findings, ten of them on the standard library.

Bug two: three distributions with no findings at all

The second bug produced no suspicious number to notice, because its number was zero. Every image in the index built on Rocky Linux, Azure Linux or SUSE Linux Enterprise had no findings on its rpm packages. That was 24 images. One of them was a Rocky 8.9 image with 150 packages. Our advisory mirror held 143 Rocky advisory entries for 35 of those packages, and we reported none.

The cause was a name. We match a package to an advisory by its package URL, and for an rpm the URL includes a namespace that says which distribution the package belongs to. Syft takes that namespace from the image's os-release file. OSV.dev uses its own names:

Our matcher took the namespace from each side exactly as written and required them to be equal. That is the right rule when both sides agree, and it fails silently when they do not: the lookup returns no rows, and no rows is also what a patched package returns. Neither spelling is wrong. Nothing in the package URL specification says what the namespace of an rpm must be, so each producer chose one.

For Rocky and Azure Linux the fix was a table of three aliases, applied to the SBOM's side before the lookup. SUSE needed two more things.

SUSE: three problems behind one zero

With the alias in place, SUSE images still matched nothing. The package URLs in OSV.dev's SUSE and openSUSE records are malformed. A package URL separates the name from its qualifiers with a ?. These records use an &:

pkg:rpm/suse/curl&distro=SUSE%20Linux%20Enterprise%20Micro%205.5

A parser that follows the specification reads the package name there as curl&distro=SUSE Linux Enterprise Micro 5.5, and no package in any image has that name. We now repair the separator before parsing, and a migration repaired the 153,000 keys already stored.

The third problem was the release. An advisory for a distribution only applies to the release it was written for. Syft describes a SLES image as sles-15.7, from the VERSION_ID in os-release. OSV.dev calls the same release SUSE Linux Enterprise Server 15 SP7. Our release parser only understood numbers, so it could not tell which SUSE product or service pack an advisory was for. Had we switched on the alias alone, a SLES 15 SP7 package would have been compared with advisories for every SUSE product, and we would have traded a false negative for false positives. The parser now reads 15 SP7 as 15.7 and Micro 5.5 as 5.5.

All three fixes shipped on 6 September. The matcher runs behind the product as well as the index, so clusters running those three distributions had the same gap until that day. Stored SBOMs were matched again after the deploy, without a new scan.

What is still not fixed: distributions with no advisories

There is a third way to get zero, and we have not fixed it. Our advisory data comes from OSV.dev, and OSV.dev publishes nothing for Amazon Linux, Oracle Linux, CentOS, Fedora or Photon OS. An rpm package from one of those cannot match, because there is nothing to match it with. On 6 September that was 63 images in the index.

You can see it on the Harbor chart today. Harbor builds its images on Photon OS. The chart's page lists harbor-db and valkey-photon with zero findings. That zero means we have no advisories for their operating-system packages. It does not mean the packages are patched, and the page does not yet say which. Packages from language ecosystems in the same images are matched as normal, which is why the Go binaries in the other Harbor images do have findings.

This is a display bug as much as a data gap. An image whose operating-system packages cannot be matched should be reported as not measured, and not as zero. Until that is fixed, treat a StackRadar result for an image on one of those five distributions as covering its application dependencies only.

Other scanners differ here, and it is a fair reason to pick one. OSV-Scanner reads the same OSV data and has the same gap. Trivy keeps its own advisory feeds for Photon OS, Amazon Linux and Oracle Linux, and Grype for Amazon Linux and Oracle Linux. If your fleet runs on one of them, that matters more than any feature comparison.

Three different zeros

The three cases look the same in a report and mean different things.

Only the first one is good news. Some scanners log a warning for a distribution they do not support, but the report itself rarely separates the three, and ours does not. So the separation has to come from a test.

Test the scanner you run

None of these checks needs our product. They work on any scanner.

1. List the distributions you run

You cannot check coverage for a distribution you do not know you have. Third-party charts bring their own base images, and Harbor on Photon OS is a typical surprise. Syft reports the distribution of any image:

kubectl get pods -A -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}{end}{end}' \
  | sort -u \
  | while read -r image; do
      distro=$(syft -q "$image" -o syft-json | jq -r '.distro | "\(.id // "none") \(.versionID // "")"')
      echo "$distro  $image"
    done | sort

This pulls every image, so run it from a machine with registry access and some time. An image that prints none has no os-release file: a distroless or scratch image, where only language packages can be matched.

2. Scan an old image of each one

For each distribution in that list, scan a base image that is several years old. An old base image has hundreds of known vulnerabilities. If your scanner reports none, it is not looking. Here is the loop with Grype, and what it printed for us on 4 October:

for image in rockylinux:8.5 registry.suse.com/bci/bci-base:15.4 \
    photon:4.0-20220107 amazonlinux:2.0.20220121.0 \
    mcr.microsoft.com/azurelinux/base/core:3.0.20240727; do
  echo "$(grype -q "$image" -o json | jq '.matches | length')  $image"
done

# 1070  rockylinux:8.5
# 532   registry.suse.com/bci/bci-base:15.4
# 346   photon:4.0-20220107
# 325   amazonlinux:2.0.20220121.0
# 292   mcr.microsoft.com/azurelinux/base/core:3.0.20240727

# The same loop with another scanner:
#   trivy image -q -f json "$image" | jq '[.Results[]?.Vulnerabilities[]?] | length'
#   osv-scanner scan image "$image"
#   stackradar scan --image "$image"

A zero on any line is the finding. Pin a dated tag, as above. A tag such as core:3.0 is rebuilt with current packages, and for us it returned zero matches where the July 2024 build of the same image returned 292. That zero was honest, and it is useless as a test. Keep the loop, add the distributions from step one, and run it again when you upgrade the scanner. Coverage changes between releases in both directions.

3. Compare two scanners by package

Pick one busy image, scan it with two tools, and count findings per package in each. A different total is normal. What you are looking for is a package where one tool reports many and the other reports none. A whole class of packages at zero, such as every rpm or every Go module, is a false negative like our second bug. One package holding most of an image's findings is worth a look for the opposite reason.

4. Read the fixed versions

For a false positive, the quickest sign is a fix version far below the installed one. Our 140 findings said that Go 1.26.4 needed an upgrade to 1.6.4. Be careful with the reverse: a fix in the neighbouring release line is normal. An advisory fixed in both 1.25.13 and 1.26.6 is real for 1.26.4, even when the report shows only the lower number.

What we changed besides the two fixes

• A comparison with no answer is not counted. A version the comparator cannot order is unconfirmed. It is kept and shown as such, and it does not add to a count or a score.
• Every finding says how it was matched. Each one carries the identity it matched on and whether the version comparison was exact, fuzzy or unconfirmed. The matching reference describes the pipeline.
• The chart index is the regression test. Its images are matched again every day, in public. If the Harbor chart's numbers look wrong to you, they are there to be checked, and we would like to hear about it.

Check an image against ours

The free StackRadar CLI scans one image or a whole cluster from your laptop and needs no account, so it can be one of the two scanners in step three. The chart index shows our findings for the images of about 18,000 public charts if you would rather compare against something already scanned. For how the other scanners differ in what they cover, see open-source vulnerability scanners for Kubernetes.