Traces and evals. Plus security and guardrails.
LangSmith does tracing and dataset-driven evals well. EvalGuard imports both — your run traces and your eval datasets — and adds red-team scans (300+ plugins), an LLM firewall, and a SOC 2 evidence engine on the same data. No sign-up needed to run your first eval.
Honest positioning
Where LangSmith stops, EvalGuard keeps going
LangSmith is a solid observability and eval tool — especially if you live in LangChain. EvalGuard overlaps on tracing and evals, then extends into the security, firewall, and compliance work you'd otherwise buy separately.
| Capability | LangSmith | EvalGuard |
|---|---|---|
| Tracing & observability | Yes | Yes — OTel trace ingest + cost attribution |
| Dataset-driven evals | Yes | Yes — 200+ scorers (LLM-as-judge, pairwise, rubric) |
| LangChain / LangGraph integration | First-party, native | Yes — LangChain middleware + SDK |
| Prompt playground & versioning | Yes | Yes — Prompt IDE + optimizer across 90+ providers |
| Human annotation / feedback | Yes | Yes — annotation queues + Krippendorff's alpha |
| Red-team & security scans | Not offered | Yes — 300+ attack plugins with threat-feed sync |
| LLM firewall / guardrails | Not offered | Yes — real-time input + output firewall |
| SOC 2 evidence engine | Not offered | Yes — live evidence engine + audit log |
| Managed BYOK gateway | Not offered | Yes — 90+ providers, semantic cache |
| Self-hosting | Enterprise plan | Yes — self-host available |
Comparison reflects each product's core offering; check current LangSmith plans for the latest on enterprise self-hosting.
Migration paths
LangSmith is two things. So the move is two paths.
LangSmith owns your export end-to-end — we never touch your LangSmith account. Convert what you download with the EvalGuard CLI.
Observability — bring your run traces
Export your runs from LangSmith (the SDK client.list_runs(), or a project download), then convert them to neutral-shape spans.
Datasets & evals — convert and run
Export a LangSmith dataset, convert it to a runnable EvalGuard config, then run it — keyless on your machine, or on the EvalGuard cloud.
# UI download, or the SDK: client.list_examples(dataset_name=...) → JSONnpx @evalguard/cli import:langsmith langsmith-dataset.json -o evalguard.config.jsonnpx @evalguard/cli eval:local evalguard.config.json --provider echoThe echo provider runs the whole eval loop with no API key — perfect for validating the config before you spend a token. Swap in --provider openai (with your key) for real model calls, or run on the cloud for shared dashboards and run history:
Want a hand with the migration?
If you have a large trace history or a dataset with custom evaluators, send us your LangSmith export and we'll help you map it and validate the first run. Free.