On October 21, 2026, at 11:39 UTC, a poisoned litellm==1.107.0 reached PyPI, and seven minutes later the same code shipped in the official LiteLLM images on Docker Hub and GHCR (LiteLLM). The wheels stayed up for 1 hour 43 minutes and the Docker Hub images for 2 hours 33 minutes. On a LiteLLM proxy with a database, the payload dumped the tables where the gateway stores its upstream provider keys.

LiteLLM is downloaded about 23 million times a week (pypistats), and many teams run it as their LLM gateway: the one service that holds API keys for every model provider and sees every prompt and completion. This post walks through how the release was built and what the payload took, then lists the controls we would put around any LLM gateway in production.

Status on November 1, 2026: LiteLLM has published three posts on the incident, the last one on October 30. Version 1.107.0 is gone from PyPI, Docker Hub and GHCR, the newest stable release on PyPI is 1.106.6, and LiteLLM images are signed with a new cosign key since October 24.

Key takeaways

  • 1 h 43 min on PyPI, 2 h 33 min on Docker Hub. Unpinned installs and floating tags such as latest and main-stable pulled the malicious build in that window.
  • The payload hooked every Python start, even when nothing imported litellm. On a proxy with a database it dumped LiteLLM_CredentialsTable and LiteLLM_ProxyModelTable, where the gateway keeps provider keys.
  • 136 days passed between an admin key landing in a public repository on May 27 and its first use on October 10. Nobody had revoked it. Eleven days later the attacker shipped 1.107.0 through the real LiteLLM release pipeline.
  • The images carried a valid LiteLLM signature, and the wheels went out under the real Trusted Publishing identity. In our reading, a waiting period before adopting new releases and digest pinning would have kept 1.107.0 out, and an egress allowlist would have kept the stolen data in.
  • Wiz saw stolen AWS credentials tested about 4.5 hours after exfiltration, with Amazon Bedrock calls it describes as consistent with LLMjacking.

What was published, and for how long

LiteLLM lists all seven litellm==1.107.0 wheels on PyPI as affected, live from 11:39:51 to 13:23 UTC. On Docker Hub (litellm/litellm) the tags v1.107.0, latest, main-latest and main-stable pointed to the compromised image index sha256:1bbcb1443bc9… from 11:46:56 until 14:20. On GHCR the same tags of litellm, litellm-database and litellm-non_root were unavailable by 13:51.

The artifacts were built from an attacker controlled repository, Berri-AI/litellm, whose name differs from the real BerriAI/litellm by one hyphen. According to issue #47826, none of the malicious commits are on a branch or tag of the real repository, and LiteLLM is skipping the 1.107.0 version number.

How long each channel served 1.107.0 on October 21, 2026, and what happened around it. Times in UTC. Removal times as reported by LiteLLM; GHCR is an upper bound. Sources: LiteLLM incident posts, GitHub issue #47826, Wiz.
How long each channel served 1.107.0 on October 21, 2026, and what happened around it. Times in UTC. Removal times as reported by LiteLLM; GHCR is an upper bound. Sources: LiteLLM incident posts, GitHub issue #47826, Wiz.

Lidor Machluf of Upwind Security opened issue #47826 at 12:06:41 UTC, 27 minutes after the first wheel, and LiteLLM began incident response at 13:02. Wiz estimates from ClickPy data that the malicious version was downloaded about 49,000 times (Wiz).

What the payload did

Each wheel carried three extra files under litellm-1.107.0.data/purelib/, which pip installs at the root of site-packages. Python processes .pth files and sitecustomize.py on every interpreter start. cook.pth contains one line, import cook, so the stealer in cook.py ran whenever any Python process started in that environment, pip included. Deleting the litellm package directory does not remove them, and pip uninstall would start Python and run the payload once more (Datadog).

According to the analysis in issue #47826, the stealer calls itself privcollect and collects:

  • AWS role credentials from IMDSv2 and ECS task credentials, which it validates with STS GetCallerIdentity, plus GCP and Azure tokens
  • the environment variables of every readable process, through /proc/*/environ
  • when DATABASE_URL is set, the contents of LiteLLM_CredentialsTable and LiteLLM_ProxyModelTable, where a proxy keeps its upstream provider keys
  • SSH, GPG and TLS private keys, Kubernetes service account tokens, .env files, shell history, AI coding tool credentials and browser passwords

The data was encrypted and posted to www.litellm.dev/healthcheck, a lookalike domain registered on October 13. The other hook, sitecustomize.py, started a background process that checked a GitHub dead drop every hour for a domain to download Python code from and run. The dead drop never held a live domain, and GitHub had taken the repository down by 13:30 UTC (Wiz).

Between 16:20 and 17:01 UTC, Wiz saw the attacker test AWS instance role credentials taken from CI runners and Kubernetes nodes, calling sts:GetCallerIdentity, then bedrock:ListFoundationModels and ListModels. Wiz calls the pattern "consistent with LLMjacking".

Why a gateway host loses more than a typical dependency

What the 1.107.0 stealer could read on a host running the LiteLLM proxy, based on the published payload analyses.
What the 1.107.0 stealer could read on a host running the LiteLLM proxy, based on the published payload analyses.

An LLM gateway is built to concentrate access. Besides the provider keys, it holds the master key that controls the virtual keys it issues to teams. On a proxy host the stealer covered the two places where those secrets usually live, the database and the process environment.

The LiteLLM docs have operators set LITELLM_SALT_KEY, which encrypts the provider keys stored in the database, as an environment variable (LiteLLM docs). Our reading is that on a typical proxy host the stealer collected the encrypted keys and the salt key together. For a gateway, a poisoned release therefore means rotating the keys of every provider account it could reach.

LiteLLM reviewed what the attacker did in the other internal systems it reached and found "attempts to identify and exfiltrate additional keys and tokens, with a primary focus on upstream inference API tokens" (LiteLLM). A separate critical advisory from October 27, GHSA-w4c7-34wj-hqm8, went after the same secrets: an unauthenticated request could make an Enterprise licensed proxy send /proc/self/environ to a URL chosen by the attacker.

How a signed release came out of the real pipeline

According to the root cause analysis LiteLLM published on October 23, on May 27 a config file committed to the public LiteLLM-Labs/lite-harness repository contained an administrative key for LiteLLM Agent Platform, an internal experimental service that files pull requests. A second commit deleted the file about three hours later, but the original commit stayed in public history and nobody revoked the key.

On October 10 the attacker used that key against the Agent Platform API, and one response returned the GitHub CLI OAuth token of one of the two organization owners. Within the hour, throwaway workflows in BerriAI/litellm and BerriAI/litellm-auto posted repository secrets to webhook.site, among them the GitHub CLI tokens of two more maintainers.

The path from a leaked key to a signed release, per the LiteLLM root cause analysis of October 23, 2026. Times in UTC.
The path from a leaked key to a signed release, per the LiteLLM root cause analysis of October 23, 2026. Times in UTC.

On October 21 at 10:08 UTC, branch protection rejected a change that let the private release workflow build from another repository. Five seconds later the owner token overrode the protection and pushed it. The attacker started a staging run from the fork at 10:46, and at 11:11 a second stolen token approved the production gate, which had been relaxed on Monday, October 19, so that any two engineers could trigger and approve a release.

LiteLLM had rebuilt its pipeline after an earlier compromise in March (versions 1.82.7 and 1.82.8), when a Trivy scanner running unpinned in CI was compromised and read static release credentials. The new pipeline used Trusted Publishing, cosign signing, immutable tags and pinned GitHub Actions. In October it produced artifacts that LiteLLM describes as "Validly signed, valid Trusted Publishing provenance", and it concludes that "no pipeline controls can defend against an attacker who is an organization owner, because the owner owns the pipeline" (LiteLLM).

LiteLLM changed its explanation two days in. The first post suspected the GitHub tokens came from logs of an internal LiteLLM deployment exposed to the internet. The October 23 record replaces that: "The request logs show the first token came from a different service".

If you ran 1.107.0

Do not start with pip install, because any Python start in the affected environment runs the payload again. Delete the files from a shell, kill the background process, then rebuild images and virtual environments from scratch on a pinned version.

bash
# Run from a shell, not from Python: any interpreter start in a suspect
# environment executes cook.pth and sitecustomize.py again.
find / \( -path /proc -o -path /sys \) -prune -o \( -name 'cook.pth' -o -name 'cook.py' \
  -o -name 'litellm-1.107.0.dist-info' -o -path '*-packages/sitecustomize.py' \) -print 2>/dev/null
# compare any sitecustomize.py hit with the hashes Wiz published
ls -la /tmp/.pc_once_72129118c59f5636c0bb /tmp/remote_code_loader.lock 2>/dev/null
# in containers, check the container /tmp; also look for PRIVCOLLECT_ACTIVE=1 and /tmp/.pc_sq_* or /tmp/.pc_e_*
docker images --digests | grep -i litellm
# compromised indexes: litellm (GHCR and docker.io/litellm/litellm) sha256:1bbcb1443bc9...,
# litellm-database sha256:c00f9d92c427..., litellm-non_root sha256:7b00e91cd2fe...

LiteLLM is explicit that a clean result proves little: "Finding one confirms exposure, but not finding one does not prove the environment is clean." If the host had outbound internet, assume the data left. Rotate the instance or task IAM role, every provider key in the proxy database, the master key and virtual keys, the DATABASE_URL password and every secret in any process environment. Then check CloudTrail for GetCallerIdentity calls from the instance role around install time, and for use of those credentials from outside your infrastructure.

How we would run an LLM gateway in production

These are the controls we would put around any LLM gateway, including the ones in front of the RAG and document AI pipelines we build.

1. Pin by hash and digest, and let new releases age. The poisoned wheels were on PyPI for 1 h 43 min, so a rule that ignores anything uploaded in the last few days would have skipped them, and a digest pin would have ignored the repointed latest tag. uv supports the waiting period with exclude-newer, and pip with --uploaded-prior-to P7D together with --require-hashes.

toml
# pyproject.toml of the gateway service (tested with uv 0.13.6)
[project]
name = "llm-gateway"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = ["litellm>=1.106,<2"]   # uv.lock pins the exact version and the sha256 of every file

[tool.uv]
exclude-newer = "7 days"   # never resolve a file uploaded to PyPI in the last 7 days
# For a reviewed security fix, shorten the window for one package only:
# exclude-newer-package = { litellm = "1 day" }   # then: uv lock --upgrade-package litellm
# Deploy with `uv sync --locked`, which fails if uv.lock is out of date.
bash
# Pull by tag once (pulling runs nothing), read the digest, verify that digest
# with the cosign key LiteLLM rotated to on October 23, then deploy only by digest.
docker pull ghcr.io/berriai/litellm:v1.106.6
docker inspect --format='{{index .RepoDigests 0}}' ghcr.io/berriai/litellm:v1.106.6
cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/e82a3c646908ba0a6ee0beddc4310d7a6cac4065/cosign.pub \
  ghcr.io/berriai/litellm@sha256:<digest>
# compose or Helm values: image: ghcr.io/berriai/litellm@sha256:<digest>

A waiting period has a cost. The fix for GHSA-w4c7-34wj-hqm8 shipped in 1.106.5 on October 27, and on November 1 the configuration above still resolved 1.106.4, which lacks it. Override the window for such a fix in a reviewed change, one package at a time, as in the commented line above.

2. Verify against the source, beyond the signature. The compromised images carried a valid signature from the cosign key LiteLLM used at the time, rotated on October 23 to a key with the SHA-256 fingerprint e3fe3e08…62c2c0. A signature tells you which pipeline produced an artifact, and this pipeline had been pointed at the fork.

The SLSA provenance that BuildKit attached to every 1.107.0 image named the source as https://github.com/Berri-AI/litellm at 20535d8f, while 1.106.2 from the same releaser named BerriAI/litellm (issue #47826). Datadog writes that the provenance "faithfully recorded the malicious fork as the source", so a policy that checks the source repository in image provenance would have rejected 1.107.0. Current LiteLLM files on PyPI carry no attestation to check, so for Python dependencies a simpler test works: refuse wheels that add code to interpreter start.

python
# check_wheels.py: fail the build if a wheel would run code at interpreter start.
# Run on downloaded wheels, before anything is installed:
#   pip download -r requirements.txt --require-hashes --no-deps --only-binary=:all: -d wheels/
#   python3 -I check_wheels.py wheels/*.whl
import sys, zipfile

HOOKS = {"sitecustomize.py", "usercustomize.py"}
ALLOWED = {"distutils-precedence.pth"}  # shipped by setuptools; review each addition

failed = False
for wheel in sys.argv[1:]:
    for name in zipfile.ZipFile(wheel).namelist():
        base = name.rsplit("/", 1)[-1]
        if (base.endswith(".pth") and base not in ALLOWED) or base in HOOKS:
            print(f"{wheel}: {name}")
            failed = True
sys.exit(1 if failed else 0)

We ran it on the clean litellm 1.106.6 wheel, which passes, and on a test wheel with the 1.107.0 file layout, which it rejects for cook.pth and sitecustomize.py.

3. Give the gateway an egress allowlist and keep it away from instance metadata. A gateway needs its provider endpoints, its database and its logging sink. The stealer posted to www.litellm.dev and the loader polled raw.githubusercontent.com. The October 27 advisory recommends "limiting the proxy's outbound traffic to known provider endpoints", and that blocks both.

By default LiteLLM fetches its model cost map from raw.githubusercontent.com at startup (LiteLLM docs), so set LITELLM_LOCAL_MODEL_COST_MAP=True before you leave GitHub off the list. Block the instance metadata endpoint for the gateway container too, since the stealer took instance role credentials from IMDS. If the gateway gets its own cloud identity from a local endpoint, as with ECS task roles or GKE Workload Identity, that endpoint has to stay reachable, and the narrow permissions in point 4 are what limit the loss.

4. Hold scoped provider keys with spend caps, and prefer short lived cloud credentials. Use one key per gateway and environment, created in a provider project or workspace with a spend limit, and never an organization admin key. For cloud hosted models, give the gateway a workload identity limited to bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream (or their equivalents) on named models. It cannot list foundation models, one of the first calls Wiz saw, although sts:GetCallerIdentity answers for any credential.

The October 27 advisory also recommends loading the master key from a secret manager instead of the process environment, and setting a separate LITELLM_SALT_KEY. A master key loaded that way stays out of /proc/*/environ, which the 1.107.0 stealer read for every process it could reach.

5. Write the rotation runbook before you need it. List every credential the gateway host can reach, who rotates each one and how long it takes, and rehearse it. Rotate from a clean machine, as Datadog advises, and rebuild the host instead of cleaning it. The root cause here was a key flagged for revocation in May and never revoked, and LiteLLM now commits to revocation tickets that close only once revocation is confirmed.

6. Monitor the gateway like a privileged identity. Alert on denied egress, on new .pth or sitecustomize.py files in a running image, and on cloud API calls by the gateway role that are not model invocations. Compare usage per provider key at the provider with the spend the gateway logs. A gap that caching and pricing differences do not explain points to use from somewhere else.

7. Keep personal and agent tokens out of automation. The chain started with an internal agent service that held the GitHub CLI token of an organization owner. LiteLLM now requires that "Automations never authenticate as a person", and states that "Everything on a developer's machine should be assumed readable by the agents running on it". Apply the same rule to whatever calls your gateway: each service gets its own identity and only the virtual key it needs.

What is still unknown on November 1: how many running gateways installed 1.107.0, and whether harvested provider keys were used beyond the AWS tests Wiz reported. On PyPI we found PEP 740 attestations only for the 1.83.x releases and none for the current 1.106.x files (our check of the PyPI Simple API).

Our RAG at scale write up describes a system over more than 50,000 documents in regulated industries. In a pipeline like that, the gateway sees retrieved document text in every prompt, so its keys and egress need the same review as the document store. If you are putting a gateway in front of a RAG or document AI pipeline, our AI solutions team designs and reviews that layer.

Frequently Asked Questions

LLM gateway review for RAG and document AI teams

We check how your gateway is pinned, what it can reach, which provider keys it holds and how fast you can rotate them.

Book a free technical validation

Sources

Accessed November 1, 2026. Interval arithmetic (136 days, 1 h 43 min, 27 minutes) and the PyPI attestation check are ours.