Cilium and CI/CD protection: how a Kubernetes kernel-level open-source project secures its supply chain

Introduction

The past twelve months have been tough for the open source supply chain. Axios was compromised in npm — normal-looking releases contained a remote access trojan (RAT). The PyPI package LiteLLM was hijacked to steal environment variables. Typosquatting forks of Trivy were published, counting on people who make typos when typing go install. And the canonical example — the 2020 SolarWinds breach — remains a cautionary tale everyone refers back to: attackers infiltrated the build system and spread malware via regular Orion updates to around 18,000 organizations, including US federal agencies, NATO, and Microsoft. The malware lay dormant for months. The breach went unnoticed for most of the year.

Cilium operates in the kernel-level network path of millions of Kubernetes pods. If our supply chain were compromised, the blast radius would be significant. Strengthening protection against this scenario is ongoing work, and we want to document in detail exactly what we are doing. Most of what is described is not tied to Cilium: any open source project running CI/CD on GitHub Actions can apply these patterns. We also noted areas where we still fall short — maybe something will serve as a useful starting point for others.

In brief: pipeline protection levels

If you don't have time to read the full article, here's what Cilium is doing today to strengthen the supply chain, broken down by pipeline stage:

Below, each item is broken down in detail — including design decisions and things we intentionally chose not to do (for example, forking every third-party action into our own organization).

Who can run what in CI

The first question in any discussion of CI supply chains is: who can trigger a build and what code does it run? Many CI compromises start right here — by tricking the system into running attacker-controlled code with elevated privileges.

Restricting workflow triggers via Ariane

Ariane is an in-house GitHub bot we built to trigger (dispatch) CI workflows from PR comments. When a maintainer writes /test or /ci-eks in a pull request, Ariane verifies that the commenter is a member of the organization-members team, identifies which workflows to run (including dependencies — for example, tests that first need a fresh image build), and triggers them via workflow_dispatch.

The interesting part is the allow-list. Only verified organization members can trigger workflows, and the set of available workflows is manually listed in the config:

# .github/ariane-config.yaml
allowed-teams:
  - organization-members
triggers:
  /test\s*:
    workflows:
      - conformance-aws-cni.yaml
      - conformance-clustermesh.yaml
      - conformance-eks.yaml
    # ...and so on
    depends-on:
      - /build-images-dependency
  /ci-aks:
    workflows:
      - conformance-aks.yaml
    depends-on:
      - /build-images-dependency

A random external commenter posting /test in a PR will be ignored. They won’t trigger expensive cloud provider conformance test suites or burn through our CI minutes.

Separating trusted and untrusted code in CI

When someone opens a PR, we need to build their code, but we can’t trust it. This is the classic pull_request_target problem. We avoid pull_request_target where possible, but a few workflows still require it — so we wrap them with protective safeguards.

The image build workflow is a canonical example. It splits the checkout into two parts:

# .github/workflows/build-images-ci.yaml
- name: Checkout base or default branch (trusted)
  uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
  with:
    ref: ${{ github.base_ref || github.event.repository.default_branch }}
    persist-credentials: false
# ...trusted setup steps run here, including loading composite actions...
# Warning: since this is a privileged workflow, subsequent workflow job
# steps must take care not to execute untrusted code.
- name: Checkout pull request branch (NOT TRUSTED)
  uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
  with:
    persist-credentials: false
    ref: ${{ steps.tag.outputs.sha }}

The first checkout pulls the base branch — code that has already been reviewed and merged — so that composite actions, scripts, and Cosign signing logic can be loaded from a trusted source. Only after that does the workflow perform the PR head checkout, and this checkout is used purely as the build context for docker build. Nothing from the PR branch is executed as a script.

Security reports for this pattern come in regularly. Automated scanners and well-meaning researchers see "pull_request_target plus a second checkout" and flag this as a vulnerability. In typical cases, they are right. In our case, the workflow is intentionally designed to keep this pattern secure:

• No run steps execute scripts from an untrusted checkout. Every shell block after the second checkout is written inline in the YAML workflow (disk usage checks, file copying, digest output). Nothing is pulled from the PR branch.

• No composite actions are loaded from an untrusted checkout either. All composite actions (set-runtime-image, cosign, set-env-variables) come from a trusted checkout of the base branch or from the saved ../cilium-base-branch/ directory. We are also working on migrating these composite actions to a dedicated repository, so that running them will not require checking out the source code at all.

• Docker BuildKit does execute untrusted Dockerfiles — this is the entire point of building a CI image from a PR. BuildKit runs in isolation: it has no access to GitHub Actions environment variables, repository secrets, or the runner's Docker credential store. The build arguments we pass contain no secrets — only a link to the runtime image and the operator variant name.

• Untrusted data enters exactly one trusted action. The runtime-image*.txt file from the PR is fed into the trusted set-runtime-image action, which verifies that the image link starts with quay.io/cilium/, and removes line breaks — to prevent an attacker from smuggling a GITHUB_ENV injection. It is impossible to redirect the build to anything outside the Cilium namespace.

• Only CI credentials are in scope. Docker login uses QUAY_USERNAME_CI / QUAY_PASSWORD_CI, which can only push to the -ci development registry. No production credentials exist on the runner at all.

The worst possible outcome of a compromised PR build is a malicious CI image in the development registry. This is the same blast radius as any CI system that builds contributor code. We appreciate every report and read all of them carefully, but this pattern is intentional.

CODEOWNERS as a review gate

We rely heavily on CODEOWNERS to ensure that changes always reach the people most familiar with the relevant area. For CI configuration, this means that everything under .github/ is owned by @cilium/github-sec (our security-focused CI team) plus @cilium/ci-structure, and workflow auto-approve.yaml is owned by @cilium/cilium-maintainers:

# CODEOWNERS
/.github/                          @cilium/github-sec @cilium/ci-structure
/.github/ariane-config.yaml        @cilium/github-sec @cilium/ci-structure
/.github/renovate.json5            @cilium/github-sec @cilium/ci-structure
/.github/workflows/                @cilium/github-sec @cilium/ci-structure
/.github/workflows/auto-approve.yaml  @cilium/cilium-maintainers

No one can modify the CI pipeline without explicit review from the team responsible for its security.

Dependency locking

Once you control who runs builds, the next question is what code those builds pull in. A pinned workflow that pulls in a compromised dependency remains a compromised workflow.

Pinning GitHub Actions to SHA digests

The most beneficial thing any project can do here is stop trusting mutable tags.

Every uses: directive in our workflow files references actions by the full 40-character commit SHA, with a human-readable version in the trailing comment (pinning an action to the full commit SHA, SHA pinning):

- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2

If someone compromises the v6 tag in actions/checkout and force-pushes malicious code, our workflows will not pull it in. They are pinned to a specific commit. The same applies to every third-party action we use: docker/build-push-action, sigstore/cosign-installer, golangci/golangci-lint-action, and dozens more. Container images used directly in workflow steps are also pinned by their @sha256 digest, so even tools running inside CI are addressed by their content.

Pinning has one annoying blind spot: transitive dependencies. When we pin actions/checkout@de0fac2e..., we know exactly what code runs for that action. But if actions/checkout itself references another action by tag (uses: some-org/some-helper@v1), resolution happens at runtime and is invisible to us. An attacker who compromises a nested dependency can still reach our pipeline.

The fix is on the way: dependency locking at the workflow level was announced in the 2026 GitHub Actions security roadmap. A dependencies: section will be added to YAML workflows, which locks all direct and transitive action dependencies by commit SHA — similar to what go.mod + go.sum do for Go. We'll enable it as soon as it's available.

Automated updates with a trust boundary

Maintaining SHA pins manually would be tedious, so we don't do that. The Renovate configuration extends the helpers:pinGitHubActionDigests preset and sets pinDigests: true globally. When a new action version is released, Renovate opens a PR with the updated SHA. We stay up to date and never revert to a mutable reference.

Renovate runs as a self-hosted bot on an hourly schedule, using a dedicated GitHub App with granular permissions instead of a personal access token. vulnerabilityAlerts are enabled, so known CVEs in the dependency tree are immediately turned into PRs.

We recently added a cooldown period for Renovate to avoid picking up brand new releases the moment they are published. Given the current pace of supply chain attacks, a few days is usually enough for a compromised package to be detected and yanked:

# .github/renovate.json5
{
  // Dependency cooldown: skip versions published less than 5 days ago
  "matchUpdateTypes": ["major", "minor", "patch"],
  "minimumReleaseAge": "5 days"
},
{
  "matchPackageNames": [
    "actions/{/,}**",                  // GitHub's official actions
    "docker/{/,}**",                   // Official Docker actions
    "cilium/{/,}**",                   // Our own ecosystem
    "k8s.io/{/,}**",                   // Kubernetes official
    "sigs.k8s.io/{/,}**",              // Kubernetes SIGs
    "golang.org/x/{/,}**",             // Go experimental
    "github.com/golang/{/,}**",        // Go official org
    "github.com/prometheus/{/,}**",
    "github.com/hashicorp/{/,}**",
    "go.etcd.io/etcd/{/,}**",
    // ...trimmed
  ],
  "automerge": true,
  "automergeType": "pr",
  "groupName": "auto-merge-trusted-deps",
  "reviewers": ["ciliumbot"]
}

Updates from this allowlist are automatically merged after passing CI. Everything else requires a human review.

The workflow auto-approve adds another redundant layer of protection: it verifies that the PR was created by cilium-renovate[bot] and that the review request was actually initiated by the bot itself, not a human impersonating it:

if: ${{
  github.event.pull_request.user.login == 'cilium-renovate[bot]' &&
  (github.triggering_actor == 'cilium-renovate[bot]' ||
  github.triggering_actor == 'auto-committer[bot]')
  }}

If these conditions are not met, auto-approval does not occur.

Vendoring Go modules

All Go dependencies are vendored and committed to the repository. CI checks that there are no discrepancies between go.mod, go.sum and vendor/. Builds are reproducible and do not communicate with external module proxies during the build process, so a tampered module on a proxy will never reach us. We also run license checks (go run ./tools/licensecheck) to keep dependencies with undesirable licenses out of the dependency tree.

Should you fork actions into your own organization

In theory, yes. If we forked every third-party action into cilium/ and pinned it to the SHA of our fork, upstream compromises would never reach us at all. Some projects with strict security requirements do exactly this.

We have decided not to do this — primarily because the operational overhead is real, and the security gain is smaller than it appears at first glance:

• Support burden. We use dozens of third-party actions. Synchronizing forks with upstream security patches becomes an ongoing concern, and an outdated fork with unpatched vulnerabilities is itself a security risk.

• Missed improvements. Upstream actions regularly fix bugs and ship security features. Forks add friction and make it harder to adopt these changes.

• Renovate complexity. The update pipeline would have to track upstream releases, open PRs against every fork, and then update the consuming workflows. The chain of work doubles.

SHA pinning gives us a truly important immutability guarantee: a specific commit is a specific commit, regardless of which organization it is hosted in. Combined with Renovate, which suggests updates as new versions are released, we get a security benefit without the operational overhead. If a major action provider were compromised repeatedly, forking only high-risk actions would be a reasonable escalation, but we have not reached that point yet.

The Same Tradeoff — for Go Dependencies

The question "should we fork this?" also applies to our Go dependency tree. Cilium pulls in hundreds of Go modules: Kubernetes client libraries, gRPC, etcd, Prometheus, and others. Forking and maintaining all of them is unrealistic.

Go has a slightly better starting position than npm or PyPI, because import paths explicitly include the source (github.com/stretchr/testify) — this completely eliminates the class of dependency confusion attacks. Typosquatting, however, is still a real threat. Research by Michael Henriksen found typosquatted Go packages in the wild, including a fork of urfave/cli registered as utfave (one rearranged letter) that sent the host name, OS, and architecture to a remote server. Replacing this callback with a reverse shell would be a one-line change.

And typosquatting is not the worst case. SolarWinds showed that a legitimate, widely trusted vendor can have its build pipeline compromised — and then it spreads malware via regular updates. The same can happen to any Go module: an attacker gains access to a maintainer's account, publishes a malicious release, the proxy caches it, and anyone who runs go get pulls it in. That's why we vendor: this moves the trust decision from build time, where it is invisible, to review time, where a human can see the diff.

Vendoring is the core protection here. A typosquatted import path shows up as a diff in vendor/ during code review, instead of being silently resolved via the module proxy. It doesn't catch a typo the moment it is introduced (relying on the reviewer to spot an unfamiliar path in a PR), but combined with CODEOWNERS gating, it works well so far.

We are also careful about which dependencies we adopt. The Renovate config includes an explicit list of disabled dependencies that we manage manually — either because they require coordinated updates (like sigs.k8s.io/gateway-api alongside conformance tests), or because we maintain a fork with project-specific patches (like github.com/cilium/dns), or because the dependency is a project we develop ourselves and want to update intentionally (like github.com/cilium/ebpf — this is not a fork, but a standalone Go library maintained under the Cilium organization). Changes to vendor/ are reviewed by the dedicated @cilium/vendor team via the same CODEOWNERS mechanism as above.

There is a Go proverb worth quoting: "A little copying is better than a little dependency". We take this very seriously — not just as a stylistic preference. We periodically audit third-party libraries and actively prune the dependency tree. If a dependency exists only for a small utility function, we replace it with a few lines of code copied inline. Every removed dependency is one that can never be compromised, the vendor tree becomes smaller, and reviewing future dependency changes is simplified. The benefits accumulate over time.

Catching errors with static analysis

Even with correct policies, mistakes happen. A well-meaning contributor might add a workflow without permissions set, or use ubuntu-latest instead of a pinned runner. We use static analysis to catch such issues before review.

Where a workflow requires write access (for release signing, OIDC for Cosign), only the specific required scope is declared — for example, id-token: write or contents: write. Where write access is not needed, we set permissions: read-all or permissions: {} to explicitly disable broader default permissions. We do not rely on memory to enforce this. CodeQL runs on every push and pull request with the actions/missing-workflow-permissions rule enabled, and any workflow run fails if a modified workflow file does not explicitly specify permissions.

On top of that, actionlint statically checks every workflow file for syntax errors, unsafe patterns, and incorrect configurations. The same lint pipeline also enforces mandatory adherence to project conventions: every job and step has a name, no job uses the floating runner tag ubuntu-latest (we pin to ubuntu-24.04), and there is no trailing whitespace in workflow files.

One class of vulnerability is worth highlighting: GitHub Actions expression injection. The ${{ }} syntax in YAML workflows is a text substitution that occurs before bash even sees the string. If an attacker controls the substituted value (PR title, branch name), they can inject arbitrary shell commands via ;, $(...) or backticks. Bash has no idea where the value originated. The fix is to first assign the value to an environment variable and reference it as "$MY_VAR" in the run: block, so bash treats it as a single variable regardless of its contents. The GitHub security team notified us of this issue some time ago, and we fixed every instance. This is a subtle bug — easy to introduce and hard to catch during review, which is why static analysis is so important: both actionlint and CodeQL flag the use of ${{ }} in run: blocks where untrusted input flows in.

Credential Protection

We operate under the assumption that any individual component can fail. If a CI workflow is compromised anyway, the question becomes: what can an attacker actually access? The answer should be: nothing of value.

Secure Defaults

By default, our GITHUB_TOKEN is restricted to minimal read permissions for contents and packages. Workflows that need more access must explicitly opt in to additional permissions — so a workflow where permissions were not explicitly specified does not end up with broad org-wide write access.

Isolation of CI and Production Credentials

We maintain two separate sets of registry credentials under separate protected GitHub environments:

• CI credentials may push to our development image registry (quay.io/cilium/*-ci) and are available to CI builds. Even if the CI workflow is somehow compromised, these credentials will not be able to push to production image tags.

• Production credentials are behind the release environment, which requires explicit approval from a maintainer before a workflow run can access them. Neither a fork, nor a feature branch, nor a CI build can obtain these secrets. Only tag-triggered release builds approved by a maintainer.

In the worst case, if CI is compromised, an attacker may publish a malicious -ci image. They will not be able to publish to quay.io/cilium/cilium:v1.x.x or docker.io/cilium/cilium:v1.x.x. The credentials are simply not available on the runner.

Every invocation of actions/checkout also sets persist-credentials: false, so that GITHUB_TOKEN is never left in the runner's git config, where a later step could pick it up.

Signing and Attesting what we ship

The previous sections are about how to prevent bad things from getting into the pipeline. This one is about giving consumers the ability to verify what comes out of it.

Every container image that we release (cilium, operator-*, hubble-relay, clustermesh-apiserver) is signed using Sigstore Cosign via keyless OIDC. There are no long-lived signing keys to be stolen.

A reusable composite action handles the signing pipeline:

# .github/actions/cosign/action.yaml
- name: Install Cosign
  uses: sigstore/cosign-installer@cad07c2e89fa2edd6e2d7bab4c1aa38e53f76003 # v4.1.1
- name: Generate SBOM
  uses: anchore/sbom-action@e22c389904149dbc22b58101806040fa8d37a610 # v0.24.0
  with:
    artifact-name: sbom_${{ inputs.sbom_name }}.spdx.json
    output-file: ./sbom_${{ inputs.sbom_name }}.spdx.json
    image: ${{ inputs.image_tag }}
- name: Sign Container Image
  shell: bash
  run: cosign sign -y "${{ inputs.image }}"
- name: Attach SBOM Attestation
  shell: bash
  run: |
    cosign attest -y \
      --predicate "./sbom_${{ inputs.sbom_name }}.spdx.json" \
      --type spdxjson \
      "${{ inputs.image }}"

This is executed for every release build of an image and for our Helm charts OCI artifacts. Instructions on how to verify this are available in the Cilium documentation.

Release builds also run inside protected environments (release, release-tool, release-helm), so production registry credentials are limited by environment protection rules. You cannot run a release build from a fork or feature branch.

Cilium Security Team

If you've ever reported a security issue to the project (via GitHub security advisories or [email protected]), you've already interacted with the Cilium Security Team. In addition to primary processing of vulnerability reports, the team handles the operational side of supply chain security:

• Audit and rotation of credentials and permissions across the GitHub organization.

• Incident investigations and audits — when necessary.

• Monitoring patterns in our security обращения and industry developments — to suggest mitigations and controls in areas where protection is weaker.

Additional layers

A few smaller things worth mentioning:

• Tag immutability. Once a GitHub release is published, tags and attached assets cannot be changed. Configuration is on the Settings → Releases page of the repository.

• Mandatory DCO sign-off. Every commit must carry the Signed-off-by line. Our maintainers-little-helper configuration blocks merges with the dont-merge/needs-sign-off label until sign-off is present.

• Third-party security audits. We have been audited by ADA Logics and maintain a published threat model.

What we're still working on

We audited the .github/ directory against current best practices (OpenSSF Scorecard, SLSA, StepSecurity recommendations) and found several real gaps. Larger ones:

• No SLSA provenance. Every invocation of docker/build-push-action sets provenance: false. We sign images with Cosign, but we do not generate SLSA build provenance attestations. Consumers can verify who signed the image, but not how it was built. Adopting slsa-framework/slsa-github-generator (or at minimum enabling native BuildKit provenance) is on the list.

• No dependency review during PRs. We rely on vulnerabilityAlerts Renovate to flag known vulnerable dependencies, but this is reactive. Enabling actions/dependency-review-action would catch malicious or vulnerable new dependencies before they are merged.

• No govulncheck in CI. We run fuzzing and linters, but we do not yet run the official Go vulnerability scanner, which checks whether our code actually invokes vulnerable functions, rather than just whether a vulnerable package appears in go.sum.

• 68 internal references to @main. A number of conformance and scale test workflows reference cilium/cilium/.github/actions/set-commit-status@main — this is a mutable branch reference. The risk is lower than with a third-party tag, but this is inconsistent with our SHA pinning policy. The plan is to move all composite actions from cilium/cilium to a dedicated repository, which will eliminate the need for @main.

Several smaller items from the same audit:

• No OpenSSF Scorecard workflow for continuous supply chain health monitoring.

• Our SECURITY-INSIGHTS.yml expired in January 2025 and has not been updated. (We actually noticed this while writing this post.)

• No go mod verify step to validate the integrity of the vendor directory against go.sum checksums.

If any of these looks like a good first issue and you want to submit a PR — we will take it.

2026 GitHub Actions Roadmap and Our Workarounds

In April 2026, GitHub released the Actions security roadmap — platform-level changes across three tiers: ecosystem, attack surface, and infrastructure. Reading it felt like validation of the issues we have been working around for years, and a real signal that the platform is finally catching up to what major open source projects need. Here is how it aligns with what we are doing today.

Dependency Locking: SHA Pinning as First-Class

We pin every action by SHA and rely on Renovate to keep pins up to date, but a blind spot remains for transitive references. The planned GitHub dependencies section in YAML workflows will block all direct and transitive dependencies by commit SHA, with hash verification before execution starts. This closes the gap.

Policy-Governed Execution: Centralizing What We Currently Enforce Per-File

We restrict who can run workflows (Ariane allow-list), which events are allowed (per-workflow configuration), and who can approve releases (protected environments). Right now, all of this is hardcoded across dozens of YAML files plus a custom bot, and auditing the full picture means reading every single file.

The upcoming GitHub workflow execution protections built on rulesets will let us define these controls centrally, at the organization level: which actors can run workflows, which events are allowed, and which repositories the rules apply to. We will be able to ban pull_request_target across the entire organization, except for workflows where we have intentionally designed a secure two-phase checkout, instead of relying on code reviews and CODEOWNERS to enforce this.

Scoped Secrets: Closing the Implicit Inheritance Gap

Isolation of CI and production credentials is one of our strongest controls, but within a single environment, secrets are still scoped quite broadly: any workflow running in that environment can access them.

Scoped secrets will let us bind credentials to specific workflow paths, branches, or even individual reusable workflows. Release credentials can be restricted not just to the release environment, but to the specific release.yaml workflow file — so any new workflow added to that environment (either accidentally or by an attacker) will not inherit the credentials. This is a meaningful step beyond what protected environments alone provide.

The roadmap also separates secret management from write access to the repository. Today, anyone with write access to the repository can manage its secrets. GitHub plans to move secret management to a dedicated custom role — this aligns with the principle of least privilege, which we already apply to workflow permissions, but we cannot yet apply it to secret administration.

Native egress firewall

GitHub's planned native egress firewall will restrict outbound network access from GitHub-hosted runners. It operates at L7 outside the runner VM, so it remains unchanged even if an attacker gains root access inside the runner. Organizations will define allowed domains, IP ranges, and HTTP methods; everything else is blocked.

For Cilium, this is less critical than the rest. Our most security-critical workflows (release builds, image signing) already run with credential isolation and least-privilege permissions — this limits what a compromised step could do even with unrestricted network access. Building a precise egress allow list for a project that interacts with container registries, Go module proxies, cloud APIs, and Sigstore is a significant piece of work. Public preview is expected in 6–9 months, at which point we will evaluate it.

Actions Data Stream: CI observability

Our workflows generate logs, but we have no centralized telemetry for them. If a workflow starts behaving abnormally (allowing unexpected dependencies, running longer than usual, making unusual network calls), we would have to notice it manually.

Actions Data Stream will deliver near-real-time execution telemetry to external systems (S3, Azure Event Hub) — including workflow execution details, dependency permission patterns, and ultimately network activity. For an open source project with hundreds of workflow runs per day, this is a blind spot that is worth addressing.

Conclusion

Supply chain security is largely the practice of repeatedly asking "what if the thing I trust is compromised?" and adding a layer that limits the blast radius when that happens.

We have attempted to build layered defense:

• Access controls, so that only trusted people can run builds;

• Pinned digests, so that a compromised tag doesn't affect us;

• Minimal privilege permissions, so that a rogue action can't steal secrets;

• Credential isolation, so that CI never touches production;

• Signatures, so that users can verify what they're running.

None of this makes us invulnerable. But security through obscurity doesn't really exist, and the converse is also true: the more open source projects openly share their defenses, the higher the collective bar for attackers. We've shown ours - including parts that aren't great yet. If you're running CI/CD for an open source project and have solved something we haven't, open an issue, write a post, or write to us on Slack. The open source supply chain is only as strong as its weakest project, and the only way to strengthen it is together.