Model scanning

What it detects

The core entry point is scanModelFile(bytes, filename) (@evalguard/core). It detects the format by magic bytes (falling back to the filename extension) and dispatches to a format-specific scanner. Every path returns the same ModelScanResult — a verdict, a list of findings, and byte/timing stats — and never throws.

pickle (.pkl .pickle)     REDUCE — the classic RCE opcode (calls a
                          top-of-stack callable)
                          GLOBAL / STACK_GLOBAL importing a dangerous
                          module: os, posix, subprocess, socket, ctypes,
                          builtins, marshal, importlib, shutil, pickle…
                          GLOBAL of a code-exec name: eval, exec, system,
                          popen, __import__, compile…
                          BUILD / INST / OBJ / NEWOBJ / NEWOBJ_EX —
                          __setstate__ / __reduce__ run-on-load surface

pytorch (.pt .pth .bin)   ZIP container (PK\x03\x04). Every *.pkl/*.pickle
                          entry is extracted (STORED + DEFLATE inflate) and
                          pickle-scanned. A compressed pickle the old code
                          couldn't see is now decompressed first.
                          Zip-bomb bounds: ≤512 entries, ≤64 MiB/entry,
                          ≤256 MiB inflated total.

safetensors (.safetensors)  Header-length out-of-bounds / oversized
                          (>100 MB) — DoS vector; header not valid UTF-8
                          JSON; tensor data_offsets out of bounds or
                          overlapping (data smuggling); shape·dtype byte
                          size ≠ declared offsets; pickle-opcode signatures
                          or oversized blobs hidden in __metadata__.

gguf / onnx               Format detected → single info finding. Deep
                          structural scan is roadmap; verdict = unknown.

unknown                   Couldn't identify from magic bytes or extension
                          → info finding asking for a recognized filename.

Findings roll up to a verdict: safe (nothing dangerous), suspicious (a high-severity finding — e.g. an unscannable entry, or a STACK_GLOBAL), malicious (a critical finding — a dangerous-module import or a REDUCE on a shell/exec callable), or unknown (format not yet deep-scanned).

Running a scan (URL / HuggingFace / S3)

POST /api/v1/security/model-scan fetches a public model file and scans it. The fetch is SSRF-guarded — the URL is validated with the shared isPublicUrl helper and pulled with safeFetch, so every redirect hop is re-checked against private / link-local / cloud-metadata addresses. Default cap is 2 GB per fetch.

curl -X POST https://evalguard.ai/api/v1/security/model-scan \
  -H "Authorization: Bearer $EVALGUARD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "projectId": "<uuid>",
    "source":    "huggingface",          # huggingface | url | s3
    "sourceRef": "meta-llama/Llama-2-7b-hf:pytorch_model.bin",
    "maxBytes":  2147483648              # optional, ≤ 2 GB default
  }'

# 201 Created
# {
#   "scanId":  "<uuid>",
#   "format":  "pytorch",
#   "verdict": "safe",                   # safe|suspicious|malicious|unknown
#   "findings": [ { "severity": "...", "category": "...", "detail": "..." } ],
#   "stats":   { "bytesScanned": 13476283, "durationMs": 412 },
#   "sha256":  "…"
# }

sourceRef accepts owner/repo:filename or a blob/<rev> HuggingFace path for huggingface, a full https:// URL for url, and a pre-signed https:// URL for s3. The scan row and each finding are written to model_scans / model_scan_findings (with a SHA-256 of the exact bytes). List history with GET /api/v1/security/model-scan?projectId=<uuid>.

Scanning an upload

For a local artifact, POST /api/v1/security/model-scan/upload takes a multipart/form-data body. Files are scanned in-memory and never written to disk or retained. Soft cap is 500 MB — larger files should be hosted and scanned via the URL endpoint above. The projectId must be sent both as the form field and the ?projectId= query param (the two must match — this closes a cross-tenant write on the multipart path).

curl -X POST "https://evalguard.ai/api/v1/security/model-scan/upload?projectId=<uuid>" \
  -H "Authorization: Bearer $EVALGUARD_API_KEY" \
  -F "projectId=<uuid>" \
  -F "file=@./checkpoint.pkl"

CLI (offline)

evalguard model-scan runs a self-contained heuristic scanner fully offline — no API key, nothing leaves the machine. It takes a file or a directory (recursing into .pkl .pickle .pt .pth .bin .onnx .safetensors) and exits non-zero on any critical finding, so it drops straight into a CI gate.

# Scan a single file
evalguard model-scan ./checkpoint.pkl

# Recurse a directory, only report high+ severity
evalguard model-scan ./models/ --severity high

# JSON output for a pipeline
evalguard model-scan ./checkpoint.pkl --format json
#   [ { "file": "…", "findings": [...],
#       "hasCritical": true, "hasHigh": true } ]
#   exit 1 when any critical finding is present

The CLI catches dangerous pickle opcodes (REDUCE / NEWOBJ / GLOBAL / STACK_GLOBAL / INST / BUILD), suspicious imports (os.system, subprocess, eval, exec, socket, urllib, ctypes, marshal.loads…), obfuscation hints (base64, zlib.decompress, hex escapes, inline exec( / eval(), and ONNX custom-operator markers. It reads up to the first 50 MB of each file.

Model lineage / provenance

Beyond the bytes, POST /api/v1/security/model-scan/lineage verifies a supply-chain chain (base → fine-tune → deployment) you already hold as an AI-SBOM. It's stateless compute over the caller-supplied chain — nothing is fetched or persisted. Backed by verifyLineage in @evalguard/core (security/model-audit/provenance.ts), it checks:

{
  "chain": [                             # ordered root → leaf, 1..64 nodes
    { "modelId": "meta-llama/Llama-2-7b", "format": "safetensors",
      "manifestDigest": "sha256:…", "license": "llama2",
      "origin": { "country": "US" } },
    { "modelId": "acme/llama-2-7b-ft", "format": "safetensors",
      "manifestDigest": "sha256:…",
      "parentModelId": "meta-llama/Llama-2-7b",
      "parentManifestDigest": "sha256:…",   # must equal the parent node's
      "signature": { "algorithm": "ed25519", "signature": "…" } }
  ],
  "options": {
    "requireSignature": true,
    "allowedLicenses": ["apache-2.0", "llama2"],
    "highRiskOriginCountries": ["CN", "RU"],
    "trustedManifestDigests": ["sha256:…"]   # leaf must be in this set
  }
}

# → { "trusted": false, "chainDepth": 2,
#     "findings": [ { "type": "lineage_parent_mismatch",
#                     "severity": "critical", ... } ],
#     "summary": { "critical": 1, "high": 0, "medium": 0, "low": 0 } }

The checks: parent-hash integrity — a node's parentManifestDigest must equal its chain parent's manifestDigest; a mismatch is critical (the declared base model is not the one in the chain); signature presence (when requireSignature), license policy, origin risk, and an untrusted leaf when a trusted-digest allowlist is supplied. trusted is false if any critical/high finding fires. The same module also exposes computeModelManifest (SHA-256 content-addressing across many weight files) and Ed25519 sign / verify over that digest, with the key operation delegated to a caller-supplied signer so no private key material touches the library.

Attestation (CycloneDX-ML)

GET /api/v1/security/model-scan/<scanId>/attestation emits a CycloneDX 1.6 machine-learning BOM for a scan — a machine-learning-model component with the SHA-256 hash, the findings mapped to vulnerabilities, and a declarations.attestationsblock. It's generated on first fetch and cached on model_scans.attestation, so repeat pulls (by SOC 2 auditors, release-pipeline gates) are cheap and deterministic. Tools like dependency-track and Anchore ingest it as-is.

Promotion gate

POST /api/v1/security/model-scan/<scanId>/promote gates a scanned model into an environment on its verdict. Only verdict = "safe" promotes; suspicious / malicious returns 403 PROMOTE_BLOCKED unless you pass override: true with a reason (≥ 8 chars). Every call — promoted, blocked, or overridden — writes a model_promotionsaudit row (who / when / why) and flips the scan's gate_status. Admin role required.

curl -X POST https://evalguard.ai/api/v1/security/model-scan/<scanId>/promote \
  -H "Authorization: Bearer $EVALGUARD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "toEnv": "prod", "fromEnv": "staging" }'

# safe      → { "decision": "promoted", "gateStatus": "promoted" }
# blocked   → 403 PROMOTE_BLOCKED (verdict + critical/high counts)
# override  → { "decision": "override" }   # needs override:true + reason

Interpreting results

A ModelScanResult gives you the verdict plus every finding with a severity, a category (namespaced by scanner — pickle.REDUCE, safetensors.tensor, pytorch.no_pickle…), a human detail, and — where known — a byte offset or tensor field. Wire it into CI by branching on the verdict or the persisted severity counts:

critical > 0   →  block. A dangerous-module import or a REDUCE on a
                  shell/exec callable — do NOT load this artifact.
high     > 0   →  review. STACK_GLOBAL, an unscannable zip entry, or a
                  SafeTensors shape/offset mismatch.
verdict = unknown  →  format not yet deep-scanned (gguf/onnx) — treat as
                  "not cleared", not "clean".
verdict = safe →  scanned, nothing dangerous found (still a heuristic —
                  see the limits below).

If the persistence migration hasn't been applied yet, the scan still runs and the response carries migrationPending: true alongside the full result — the analysis never depends on the database.

Where the scanner stops

Honest by design — these limits are a trust signal, not something to hide:

• Static parse only.The artifact is never deserialized. That's far safer, but it means a payload that only manifests down a specific runtime path can be missed.
• Heuristic, not formal. A skilled adversary can construct a pickle that avoids the flagged opcodes. We surface obfuscation hints — the final call is yours.
• GGUF / ONNX are roadmap. Both formats are detected and classified, but deep structural scanning returns an info finding and a verdict of unknowntoday — don't read that as a clean result.
• Size caps. 2 GB per URL fetch, 500 MB per upload, and the CLI reads the first 50 MB of each file. Stream larger artifacts and scan by URL / S3.