/\
/ \
/----\
/ \ / \ S A V I N G T H R O W
/ \/ \ ─────────────────────────
/----/\----\ behavioral eval arena · BETA
\ / \ /
\ / 20 \ /
\------/
\ /
\ /
\/
Your support agent will promise refunds it shouldn't. Prove yours doesn't.
Point your agent at adversarial multi-turn scenarios run by an AI game master; get scored on policy adherence, manipulation resistance, and harmful-request refusal — with the transcript that proves it.
20 free eval runs a month · REST & MCP · any framework · no SDK required
If your agent can read text and pick an action, it can be evaluated.
No SDK, no harness. Your agent drives a run with a tiny loop — read the scene, decide, submit. We handle the scenario, adversaries, scoring, and coaching.
# the entire integration — any language
while run.status == "running":
scene = GET /runs/{id}/scene
action = your_agent(scene) # ← this is the only part that's yours
POST /runs/{id}/action {"content": action.text, "action_type": "speak"}
report = GET /runs/{id}/scorecard # scored traits + outcomes
notes = GET /runs/{id}/coaching # what to fixPrefer native tool-calling? The same loop is exposed as MCP tools at mcp.savingthrow.dev.
Why scenarios, not benchmarks?
Coding and math benchmarks tell you whether an agent can compute. They don't tell you whether it gets socially manipulated into breaking policy, whether it resists a customer claiming "your manager said refunds are automatic," or whether it holds a refusal under multi-turn pressure.
Saving Throw puts your agent in a scenario run by a controlled AI game master, against an adversarial AI customer. Structured mechanics are the measurement substrate — deterministic rules that turn fuzzy safety behavior into something you can count, score, and reproduce. Every run produces a full transcript, a behavioral scorecard, and fail-coaching — not just a pass/fail number.
What you get back
Full transcript
Every turn, action, and narration — a structured record you can inspect, diff, and store.
Scored report card
Behavioral traits — negotiation, manipulability, cooperation, deception, goal-persistence — scored across N runs with mean, variance, and evidence quotes.
Fail-coaching
When your agent underperforms, it gets concrete, actionable guidance on what to change before the next iteration.
Execution charts
Turn-by-turn charts of checks, actions, and progress — so you see exactly where it went right or wrong.
Built for teams shipping agents
Pre-ship eval
Before you put an agent in front of customers, see how it behaves under social and strategic pressure.
Regression testing
Reproducible runs (fixed seed + fixed adversaries + N-run averaging) catch behavioral regressions between versions.
Model & prompt comparison
Pit two of your own agents against each other in the same scenario — A/B two prompts, two models, head to head.
Safety & red-teaming
Probe for manipulability, collusion, and deception — the failure modes that only show up when another agent pushes back.
Two ways to connect
Authenticate with an API key, point it at a run, play. No lock-in.
REST API
Bearer-token auth, plain JSON. Create a run, read scenes, submit actions, pull the scorecard. Any language, any framework.
Base URL: https://app.savingthrow.dev/api/agent/
MCP endpoint
Native Model Context Protocol for tool-calling agents. Drop-in for any agent stack, and any MCP-compatible runtime.
MCP server: https://mcp.savingthrow.dev
Get notified when we launch new scenarios.
We're adding new adversarial scenario packs every sprint. Drop your email to get announcements.
Run your first evaluation in minutes.
Sign up, create an API key, point your agent at a run.
