[{"content":"Reading Time: 3 minutes\nYou know that feeling when your CISO forwards you an email with \u0026ldquo;FYI\u0026rdquo; in the subject line? Nine times out of ten, it\u0026rsquo;s about supply chain security. Log4Shell, SolarWinds, the XZ backdoor—every incident becomes another line item on your compliance checklist.\nThe regulatory landscape has shifted. Executive Order 14028, the EU Cyber Resilience Act, NIST\u0026rsquo;s SSDF framework—they\u0026rsquo;re all saying the same thing: you need to prove you know what\u0026rsquo;s in your software, where it came from, and whether it\u0026rsquo;s vulnerable.\nThis isn\u0026rsquo;t security theater. When the next critical CVE drops, you need to answer \u0026ldquo;which systems are affected?\u0026rdquo; in minutes, not days. When an auditor asks \u0026ldquo;how do you verify third-party dependencies?\u0026rdquo;, you need receipts.\nI\u0026rsquo;ve spent the past few months building supply chain security into a multi-cloud environment running Red Hat OpenShift, dealing with RHEL VMs, containers, and everything in between. This series documents what actually works—not theoretical best practices, but battle-tested tooling and architecture.\nSupply chain security is one of those areas where the gap between \u0026ldquo;what we should do\u0026rdquo; and \u0026ldquo;what we actually do\u0026rdquo; is enormous. The goal of this series is to close that gap—to give you actionable guidance that works in the real world.\nWhat This Series Covers Part 1: SBOM Tools for Software Architects Focus: How to generate, store, and search software bills of materials\nWe\u0026rsquo;ll compare the major SBOM generation tools (Syft, Trivy, cdxgen), vulnerability scanners (Grype, Clair, RHACS), and SBOM platforms (RHTPA, Dependency-Track, GUAC). You\u0026rsquo;ll learn when to use each tool, how Red Hat\u0026rsquo;s ecosystem fits together, and how to build SBOM search that actually works when you need it.\nKey decisions:\nSyft vs Trivy vs cdxgen for SBOM generation Where to store SBOMs (RHTPA, Dependency-Track, or S3) How to search SBOMs when CVEs hit Red Hat Insights for RHEL VMs vs container-focused platforms Part 2: Vulnerability Scanning That Doesn\u0026rsquo;t Lie Focus: Build-time vs runtime scanning, false positive management, and exploitability analysis\nNot all CVEs are created equal. A CVSS 9.8 vulnerability in a library you don\u0026rsquo;t actually call is noise. A CVSS 7.0 with public exploits targeting your exact version is a crisis. We\u0026rsquo;ll cover reachability analysis, VEX (Vulnerability Exploitability eXchange), EPSS scores, and how to build a scanning pipeline that prioritizes signal over noise.\nTopics:\nBuild-time scanners (Grype, Trivy, Clair) vs runtime (Falco, RHACS) Container vs VM scanning strategies VEX and CSAF for exploitability context Red Hat\u0026rsquo;s TSSC (Trusted Software Supply Chain) workflow Part 3: Signing, Attestation, and Provenance Focus: Proving your build process is trustworthy\nHow do you prove a container image came from your CI/CD pipeline and not an attacker\u0026rsquo;s laptop? SLSA provenance, in-toto attestations, and Sigstore. We\u0026rsquo;ll cover Cosign for signing, Tekton Chains for automated attestation, and how Red Hat\u0026rsquo;s Konflux integrates it all.\nTopics:\nCosign and keyless signing (Sigstore) SLSA provenance levels (what\u0026rsquo;s realistic to achieve) Tekton Chains for OpenShift Pipelines Red Hat\u0026rsquo;s Konflux vs DIY supply chain security Part 4: Policy Enforcement and Admission Control Focus: Blocking bad artifacts before they reach production\nYou\u0026rsquo;ve generated SBOMs, scanned for vulnerabilities, and signed your artifacts. Now what? Policy enforcement is where it all comes together. We\u0026rsquo;ll cover OPA/Gatekeeper, Kyverno, and Red Hat Advanced Cluster Security (RHACS) for Kubernetes admission control.\nTopics:\nOPA vs Kyverno for policy-as-code RHACS admission control on OpenShift Policy examples: \u0026ldquo;block CVSS \u0026gt; 7.0\u0026rdquo;, \u0026ldquo;require signatures\u0026rdquo;, \u0026ldquo;ban outdated base images\u0026rdquo; Handling policy exceptions without creating backdoors Part 5: Incident Response When Supply Chain Attacks Hit Focus: What to do when the next Log4Shell drops\nWhen a critical vulnerability hits (and it will), you need a runbook. We\u0026rsquo;ll cover the detection-to-remediation pipeline: how to search SBOMs for affected systems, prioritize patching, verify fixes, and communicate with stakeholders. Plus the stuff nobody talks about—like what to do when the vulnerable library is embedded in a third-party vendor product you can\u0026rsquo;t just patch.\nTopics:\nCVE response runbooks (detection, triage, remediation) Using SBOM platforms for blast radius analysis Coordinated disclosure and VEX documents Handling vendor dependencies (when you can\u0026rsquo;t patch yourself) A Note on Tooling Choices This series emphasizes Red Hat tooling where applicable (RHEL, OpenShift, RHTPA, RHACS, Clair) because that\u0026rsquo;s the environment I operate in. But I\u0026rsquo;m not a Red Hat evangelist—I\u0026rsquo;ll call out when open source alternatives are better, when vendor lock-in is a real concern, and when the \u0026ldquo;Red Hat way\u0026rdquo; doesn\u0026rsquo;t make sense.\nFor every Red Hat tool, I\u0026rsquo;ll provide:\nOpen source alternatives (Dependency-Track vs RHTPA, OPA vs RHACS) When to use the Red Hat option (on-prem, FIPS compliance, unified support) When to use the alternative (multi-cloud, vendor neutrality, cost) If you\u0026rsquo;re not on Red Hat\u0026rsquo;s stack, you can still apply 90% of the patterns—just swap in your vendor\u0026rsquo;s equivalent tools.\nWhat This Series Isn\u0026rsquo;t Not covered:\nBinary analysis and reverse engineering (Ghidra, Binwalk, radare2) Source code security scanning (SAST tools like SonarQube, Checkmarx) Secret scanning (gitleaks, TruffleHog) Dependency update automation (Renovate, Dependabot) License compliance deep-dives (ScanCode, FOSSology, ORT) Those are important, but they\u0026rsquo;re adjacent to supply chain security, not core to it. This series focuses on knowing what\u0026rsquo;s in your software and whether it\u0026rsquo;s vulnerable—the SBOM and vulnerability management stack.\nIf there\u0026rsquo;s demand, I\u0026rsquo;ll cover those topics in a follow-up series.\nHow to Follow Along Reading order:\nPart 0 (this post): Series overview Part 1: SBOM Tools for Architects (available now) Part 2-5: Publishing over the next 4 weeks Hands-on labs: Each post includes working examples you can run locally:\nSBOM generation with Syft/Trivy Vulnerability scanning with Grype SBOM platform setup (Dependency-Track or RHTPA) Policy enforcement with OPA or RHACS No SaaS required. Everything runs on your laptop or in a local Kubernetes cluster.\nUp next: SBOM Tools for Software Architects →\nQuestions or feedback? Open an issue at https://github.com/jkirklan/tuscon/issues\n","permalink":"https://blog.kubelet.org/blog/supply-chain-security-series-intro/","summary":"\u003cp\u003e\u003cstrong\u003eReading Time:\u003c/strong\u003e 3 minutes\u003c/p\u003e\n\u003chr\u003e\n\u003cp\u003eYou know that feeling when your CISO forwards you an email with \u0026ldquo;FYI\u0026rdquo; in the subject line? Nine times out of ten, it\u0026rsquo;s about supply chain security. Log4Shell, SolarWinds, the XZ backdoor—every incident becomes another line item on your compliance checklist.\u003c/p\u003e","title":"Building Secure Software Supply Chains: A Series for Architects"},{"content":"Reading Time: 8 minutes\nSBOMs aren\u0026rsquo;t optional anymore. Executive Order 14028, the EU Cyber Resilience Act, and your CISO\u0026rsquo;s latest email have made software bills of materials a non-negotiable part of your architecture.\nThe tooling has matured, but there are too many options. Picking the wrong stack costs months of rework. This post cuts through the noise and tells you what actually works.\nTL;DR: What to Use SBOM Generation:\nCompliance-critical: Syft (industry standard, Red Hat supported) All-in-one: Trivy (SBOM + vulnerability scanning) Application dependencies: cdxgen (npm/Maven/pip focus) Vulnerability Scanning:\nOpen source: Grype (pairs with Syft) or Trivy Red Hat/OpenShift: Clair (Quay built-in) or RHACS Enterprise: Snyk (best accuracy, high cost) SBOM Storage:\nOn-prem Red Hat: RHTPA (Red Hat Trusted Profile Analyzer) Open source: Dependency-Track (OWASP) Knowledge graph: GUAC (Google) RHEL VMs:\nRed Hat Insights (agent-based, RPM-focused) SBOM Generation Syft: Industry Standard Anchore\u0026rsquo;s Syft is the de facto standard. Fast, comprehensive, and Red Hat ships a supported version in RHEL 9.8+.\nUse Syft when:\nRegulatory compliance requires standalone SBOMs Feeding RHTPA, Dependency-Track, or GUAC You want separation between generation and scanning Known issue: Invalid PURLs for Java archives without Maven metadata (anchore/syft#4598). Fix: ensure META-INF/maven/ directories in JARs.\nExample:\n1# Generate CycloneDX SBOM 2syft registry.redhat.io/ubi9/ubi:latest -o cyclonedx-json \u0026gt; sbom.json 3 4# Sign it 5cosign sign-blob --key cosign.key sbom.json \u0026gt; sbom.json.sig 6 7# Upload 8curl -X POST https://rhtpa.company.com/api/v1/sbom \\ 9 -H \u0026#34;Content-Type: application/json\u0026#34; \\ 10 --data-binary @sbom.json Trivy: Swiss Army Knife Aqua\u0026rsquo;s Trivy does everything: SBOM generation, vulnerability scanning, misconfiguration detection, secret scanning.\nUse Trivy when:\nYou want one tool for multiple tasks Kubernetes/cloud-native environments Don\u0026rsquo;t need to separate SBOM from scanning Trade-off: SBOM generation is good but not as comprehensive as Syft for edge cases. Fine for 95% of use cases; use Syft for audited SBOMs.\n1# Scan + generate SBOM 2trivy image registry.redhat.io/rhtpa/rhtpa-server:2.2.4 3 4# Export SBOM 5trivy image --format cyclonedx registry.redhat.io/rhtpa/rhtpa-server:2.2.4 \u0026gt; sbom.json cdxgen: Developer\u0026rsquo;s Friend OWASP cdxgen excels at application dependencies. Understands build files better than filesystem scanners.\nUse cdxgen when:\nScanning application source code Need license compliance (SPDX) Want call graphs for reachability analysis 1# Scan Node.js with call graph 2cdxgen -t js -o sbom.json --include-formulation --evidence . Vulnerability Scanning Grype: Syft\u0026rsquo;s Partner Anchore\u0026rsquo;s Grype scans SBOMs for vulnerabilities. Matches against NVD, Red Hat Security Data, OSV, GitHub Security Advisories.\nWhy separate tools?\nGenerate SBOM once, scan repeatedly (new CVEs daily) Store for compliance, scan on-demand Different teams (DevOps generates, SecOps scans) 1# Generate with Syft 2syft image:tag -o cyclonedx-json \u0026gt; sbom.json 3 4# Scan months later 5grype sbom:./sbom.json Clair: Red Hat\u0026rsquo;s Scanner Clair is Red Hat\u0026rsquo;s container vulnerability scanner, built into Quay.\nUse Clair when:\nUsing Red Hat Quay as registry Scanning Red Hat images (UBI, RHEL, OpenShift) Want automatic scanning on push Limitation: Container-only. Won\u0026rsquo;t scan VMs, filesystems, or app dependencies.\nRHACS: OpenShift Admission Control Red Hat Advanced Cluster Security is admission control + runtime security for OpenShift.\nUse RHACS when:\nRunning OpenShift Container Platform Need policy enforcement at deploy time Want runtime threat detection Example policy:\n1apiVersion: platform.stackrox.io/v1alpha1 2kind: AdmissionControllerConfig 3spec: 4 enforce: true 5 policies: 6 - name: \u0026#34;Block Critical CVEs\u0026#34; 7 severity: CRITICAL_SEVERITY 8 enforcement: HARD SBOM Storage \u0026amp; Search When the next Log4Shell drops, you need to answer \u0026ldquo;which systems are affected?\u0026rdquo; in minutes.\nRHTPA: Red Hat\u0026rsquo;s Platform Red Hat Trusted Profile Analyzer is an on-prem SBOM platform. Natively supports CSAF (Red Hat\u0026rsquo;s advisory format) and VEX.\nUse RHTPA when:\nHeavy Red Hat investment (RHEL, OpenShift, Ansible) On-prem deployment needed (air-gapped, compliance) Want native CSAF/VEX support Known issue: Rejects SBOMs with invalid PURLs (HTTP 400). Don\u0026rsquo;t hack around it—fix the root cause (add Maven metadata to JARs).\n1# Upload SBOM 2curl -X POST https://rhtpa.company.com/api/v1/sbom \\ 3 -H \u0026#34;Content-Type: application/json\u0026#34; \\ 4 --data-binary @sbom.json 5 6# Query affected systems 7curl \u0026#34;https://rhtpa.company.com/api/v1/vulnerabilities/CVE-2024-1234/affected-systems\u0026#34; Dependency-Track: Open Source OWASP Dependency-Track is the most mature open source SBOM platform. Excellent UI, policy engine, 10+ vulnerability databases.\nUse Dependency-Track when:\nWant free, open source Need SQL-based search (easier than GraphQL) Managing thousands of SBOMs (containers + apps) GUAC: Knowledge Graph Google\u0026rsquo;s GUAC (Graph for Understanding Artifact Composition) is a supply chain knowledge graph. GraphQL queries over SBOMs, SLSA attestations, and vulnerabilities.\nUse GUAC when:\nNeed complex supply chain queries Correlating SBOMs with build provenance Team comfortable with GraphQL Trade-off: Bleeding-edge. Powerful but steeper learning curve.\nRed Hat\u0026rsquo;s Integrated Stack (TSSC) Red Hat\u0026rsquo;s Trusted Software Supply Chain integrates everything:\n1Code → OpenShift Pipelines (Tekton) 2 → Syft generates SBOM 3 → Tekton Chains generates SLSA provenance 4 → Cosign signs everything 5 → Red Hat Quay (push + Clair scan) 6 → RHTPA (consume SBOM, match vulnerabilities, generate VEX) 7 → RHACS (admission control, enforce policy) 8 → OpenShift (deploy with runtime monitoring) Why this matters:\nUnified support (single vendor) CSAF/VEX native (Red Hat publishes all advisories as CSAF 2.0) FIPS compliance (all components FIPS 140-3) Air-gapped support Trade-off: Vendor lock-in.\nContainers vs VMs Containers:\nGeneration: Syft, Trivy Scanning: Grype, Trivy, Clair, RHACS Why: Layer-based, package managers (apk, rpm, deb) VMs (RHEL):\nGeneration: Syft (filesystem scan), Red Hat Insights Scanning: Red Hat Insights, OpenSCAP, dnf security Why: RPM-focused, agent-based Don\u0026rsquo;t use Red Hat Insights for containers. It only tracks RPMs, won\u0026rsquo;t see npm/pip/Maven. Use RHTPA or Dependency-Track.\nBeyond Basic SBOM Generation Generating and scanning SBOMs is foundational, but you\u0026rsquo;ll eventually need:\nSBOM Attestation \u0026amp; Signing - Prove SBOMs came from your build pipeline, not an attacker. Tools: Cosign, in-toto, SLSA provenance. (Covered in Part 3: Signing \u0026amp; Attestation)\nVEX (Vulnerability Exploitability eXchange) - Document which CVEs actually affect you vs. false positives. Red Hat publishes all advisories as CSAF with VEX status. (Covered in Part 2: Vulnerability Scanning)\nSBOM Quality Validation - Validate NTIA minimum elements (author, timestamp, dependencies, component names/versions, hashes) before ingesting. Tool: ntia-conformance-checker. Tip: RHTPA rejects invalid PURLs—validate locally first.\nSBOM Diffing - Detect supply chain drift (unexpected dependencies, downgrades). Tool: Bomber. Use case: audit dependency changes before production.\nLong-Term Storage - Regulators may require 7-year retention. S3 Object Lock (WORM) + PostgreSQL metadata index, or let RHTPA/Dependency-Track handle it.\nBuild-Time vs Runtime Build-time (most tools here):\nAnalyze static artifacts (images, code, SBOMs) Catch known CVEs before deployment Runtime:\nFalco (eBPF threat detection) Sysdig Secure (commercial) Tetragon (Cilium eBPF) RHACS (OpenShift runtime) You need both. Build-time catches known CVEs. Runtime catches zero-days and attacks-in-progress.\nDecision Frameworks Scenario 1: Red Hat-Heavy On-Prem Stack:\nSBOM: Syft (Red Hat supported) Scanning: Clair (containers), Insights (VMs), RHACS (OpenShift) Platform: RHTPA (on-prem, CSAF native) CI/CD: OpenShift Pipelines or Konflux Policy: RHACS admission control Why: Maximum integration, single vendor support, FIPS compliance\nScenario 2: Kubernetes Multi-Cloud Stack:\nSBOM: Trivy + cdxgen (app dependencies) Scanning: Trivy or Grype Platform: Dependency-Track or GUAC CI/CD: GitLab CI, GitHub Actions, Tekton Policy: OPA/Gatekeeper or Kyverno Why: Vendor-neutral, cloud-native, cost-effective\nScenario 3: Enterprise Compliance Stack:\nSBOM: Syft (compliance-grade) Scanning: Snyk (highest accuracy) or Grype Platform: Dependency-Track or RHTPA Attestation: Cosign + in-toto (SLSA provenance) Storage: S3 Object Lock (WORM), 7-year retention Policy: OPA with compliance policies Why: Audit trail, immutable storage, provable compliance\nPractical Recommendations Starting from scratch:\nWeek 1: Deploy Syft, generate SBOMs for 10 critical services, store in S3\nWeek 2: Add Grype, scan SBOMs, send results to Slack\nWeek 3: Deploy Dependency-Track or RHTPA, upload SBOMs\nWeek 4: Integrate into CI/CD (auto-generate on every build)\nMonth 2: Add policy enforcement (block CVSS \u0026gt; 7.0)\nMonth 3: Advanced features (attestation, VEX, reachability)\nAlready generating SBOMs but can\u0026rsquo;t search them?\nDeploy Dependency-Track this week. SBOMs in S3 that nobody can search is compliance theater. When the next Log4Shell drops, you need to answer \u0026ldquo;which systems?\u0026rdquo; in under an hour.\nOn OpenShift but not using RHACS?\nRHACS admission control prevents vulnerable images from reaching production. Worth the licensing cost.\nThe Bottom Line SBOM tooling has matured—there\u0026rsquo;s no excuse not to have this in production. Regulatory pressure is accelerating.\nPick based on your ecosystem:\nRed Hat-heavy: Syft + RHTPA + RHACS Cloud-native: Trivy + Dependency-Track + OPA Enterprise compliance: Syft + Snyk + GUAC + S3 WORM Start small (Syft + Grype), prove value (find vulnerabilities before production), then expand (platform, policy, attestation).\nThe goal isn\u0026rsquo;t perfect security—it\u0026rsquo;s demonstrable due diligence. When your CISO asks \u0026ldquo;which systems are vulnerable to CVE-2026-XXXX?\u0026rdquo;, answer in minutes with data, not guesses.\nBuild it now, before the regulator asks.\nResources Related Case Studies:\nCase 04479158: RHTPA invalid PURL issue (Syft Maven bug) Case 04473326: Hummingbird container mirroring Case 04466330: CSAF metadata cross-product mismatch Reference:\nSBOM \u0026amp; Vulnerability Scanner Comparison (full technical deep-dive) CSAF/VEX Reference Tools:\nSyft: https://github.com/anchore/syft Grype: https://github.com/anchore/grype Trivy: https://github.com/aquasecurity/trivy cdxgen: https://github.com/CycloneDX/cdxgen Dependency-Track: https://dependencytrack.org GUAC: https://guac.sh RHTPA: https://docs.redhat.com/en/documentation/red_hat_trusted_profile_analyzer/ Next in series: Part 2 - Vulnerability Scanning That Doesn\u0026rsquo;t Lie (runtime vs build-time, VEX, EPSS, false positive management)\nPrevious: Series Introduction\nAuthor: James Kirkland\nLast Updated: 2026-06-26\nFeedback: https://github.com/jkirklan/tuscon/issues\n","permalink":"https://blog.kubelet.org/blog/sbom-tools-for-architects/","summary":"\u003cp\u003e\u003cstrong\u003eReading Time:\u003c/strong\u003e 8 minutes\u003c/p\u003e\n\u003chr\u003e\n\u003cp\u003eSBOMs aren\u0026rsquo;t optional anymore. Executive Order 14028, the EU Cyber Resilience Act, and your CISO\u0026rsquo;s latest email have made software bills of materials a non-negotiable part of your architecture.\u003c/p\u003e","title":"SBOM Tools for Software Architects"},{"content":"About Me I\u0026rsquo;m James Kirkland, Principal Account Solutions Architect at Red Hat, based in the Atlanta area. I\u0026rsquo;m a Red Hat Certified Engineer (RHCE) with expertise spanning enterprise Linux, IoT solutions, and emerging AI/ML technologies on OpenShift.\nI co-authored Linux Troubleshooting for System Administrators and Power Users (Prentice Hall, 2006) and have been a key member of the Eclipse IoT working group since 2014.\nWhat I Write About This blog covers technical deep-dives on:\nSupply Chain Security - SBOM tools, vulnerability scanning, attestation, and policy enforcement Software Architecture - Design decisions, trade-offs, and real-world implementation patterns Container \u0026amp; Kubernetes Security - OpenShift, Red Hat ecosystem tooling, and cloud-native architectures Infrastructure Automation - CI/CD pipelines, GitOps, and reproducible builds IoT \u0026amp; Edge Computing - Enterprise-scale IoT implementations and edge infrastructure Background I work extensively with Red Hat\u0026rsquo;s enterprise stack (RHEL, OpenShift, Ansible) and open source supply chain security tools. The content here is informed by production deployments, vendor evaluations, and real-world enterprise implementations.\nMy recent work includes AI/ML applications on OpenShift AI and building secure, compliant supply chain pipelines for enterprise customers.\nWhy \u0026ldquo;Kubelet Speaking\u0026rdquo;? The kubelet is the primary node agent in Kubernetes—it watches for work and makes it happen. This blog is about making things happen: practical guidance for architects who need to ship secure, compliant systems in the real world.\nConnect LinkedIn: linkedin.com/in/jameskirkland GitHub: github.com/jkirklan Current Series Supply Chain Security for Architects (5-part series):\nSBOM Tools for Architects - Generation, scanning, and storage Vulnerability Scanning - Coming soon Signing \u0026amp; Attestation - Coming soon Policy Enforcement - Coming soon Incident Response - Coming soon This blog is built with Hugo and hosted on Cloudflare Pages.\n","permalink":"https://blog.kubelet.org/about/","summary":"\u003ch2 id=\"about-me\"\u003eAbout Me\u003c/h2\u003e\n\u003cp\u003eI\u0026rsquo;m \u003cstrong\u003eJames Kirkland\u003c/strong\u003e, Principal Account Solutions Architect at Red Hat, based in the Atlanta area. I\u0026rsquo;m a Red Hat Certified Engineer (RHCE) with expertise spanning enterprise Linux, IoT solutions, and emerging AI/ML technologies on OpenShift.\u003c/p\u003e","title":"About"}]