Traces and evals. Plus security and guardrails.
LangSmith does tracing and dataset-driven evals well. EvalGuard imports both — your run traces and your eval datasets — and adds red-team scans (300+ plugins), an LLM firewall, and a SOC 2 evidence engine on the same data. No sign-up needed to run your first eval.
Honest positioning
Where LangSmith stops, EvalGuard keeps going
LangSmith is a solid observability and eval tool — especially if you live in LangChain. EvalGuard overlaps on tracing and evals, then extends into the security, firewall, and compliance work you'd otherwise buy separately.
| Capability | LangSmith | EvalGuard |
|---|---|---|
| Tracing & observability | Yes | Yes — OTel trace ingest + cost attribution |
| Dataset-driven evals | Yes | Yes — 200+ scorers (LLM-as-judge, pairwise, rubric) |
| LangChain / LangGraph integration | First-party, native | Yes — LangChain middleware + SDK |
| Prompt playground & versioning | Yes | Yes — Prompt IDE + optimizer across 90+ providers |
| Human annotation / feedback | Yes | Yes — annotation queues with multi-reviewer assignment |
| Red-team & security scans | Not in their documented feature set | Yes — 300+ attack plugins |
| LLM firewall / guardrails | Records guardrail results; no blocking documented | Yes — real-time input + output firewall |
| SOC 2 evidence automation | Not a documented product feature | Yes — live evidence engine + audit log |
| Managed BYOK gateway | Not in their documented feature set | Yes — 15 proxied providers, similarity response cache |
| Self-hosting | Yes | Yes — self-host available |
LangSmith claims on this page were checked against their published documentation — docs.smith.langchain.com + langsmith-sdk @ ea8caba3 (2026-08-07) — on 2026-08-10. Capabilities move; check their current docs before you decide. “SOC 2 evidence automation” describes a product capability, not any vendor’s own certification status.
LangSmith is closed-source. Every “Yes” above is a symbol in their public SDK; the rows that say “not documented” mean exactly that and nothing more — a client library cannot prove what a hosted product does not do.
Migration paths
LangSmith is two things. So the move is two paths.
LangSmith owns your export end-to-end — we never touch your LangSmith account. Convert what you download with the EvalGuard CLI.
Observability — bring your run traces
Export your runs from LangSmith (the SDK client.runs.query() — or client.list_runs() on older versions, which LangSmith has deprecated — or a project download), then convert them to neutral-shape spans.
Datasets & evals — convert and run
Export a LangSmith dataset, convert it to a runnable EvalGuard config, then run it — keyless on your machine, or on the EvalGuard cloud.
# UI download, or the SDK: client.list_examples(dataset_name=...) → JSONnpx @evalguard/cli import:langsmith langsmith-dataset.json -o evalguard.config.jsonnpx @evalguard/cli eval:local evalguard.config.json --provider echoThe echo provider runs the whole eval loop with no API key — perfect for validating the config before you spend a token. Swap in --provider openai (with your key) for real model calls, or run on the cloud for shared dashboards and run history:
Want a hand with the migration?
If you have a large trace history or a dataset with custom evaluators, send us your LangSmith export and we'll help you map it and validate the first run. Free.