TPRM PERFORMANCE
Can it do the work?
Investigate exposure, prioritize remediation, compare peers, and work through evidence gaps.
344tasks in the
current suite
current suite
Research in progressv0.25.0 · Release 2026.14Open results. Shared methods. Better evidence.How we measure
SECURITYSCORECARD RESEARCH
AI for third-party risk.
Put it to the test.
Can an AI investigate vendor risk, challenge a convincing claim, and hold its ground under pressure?
A shared benchmark for consequential decisions. Explore the evidence, evaluate your model, and help shape the standard.
818 tasks62 disciplines4 evaluation tracks
THE EVIDENCE
Look beyond the headline score.
Compare performance, security, and the conditions behind every result. All four tracks. Every measured configuration.
Current suite. Complete evidence. Release 2026.14 · 818 tasks · 62 disciplines. Completed admin evaluations publish automatically after validation; community runs require admin approval. Incomplete and earlier-suite runs are excluded. Loading verified results…
Recorded results at the same task checkpoints. This is a replay, not execution speed or live progress.
THE TWO-DIMENSION TESTCurrent measured configurations
Capability meets resilience.
TPRM performance and agent security, side by side. Select a configuration to see its evidence.
Measured model Highest overall score
Current release results. Select a point to inspect the run.
CONFIGURATION RANKING
Overall task solve rate
Solved tasks / 818 · higher is betterTen configurations in view · scroll for more
Loading configurations
Configuration details10 rows in view · scroll to explore all results ↓
| # | Model / developer | Inference provider | Evidence |
|---|
Read the result in context. The inference provider is the service used to run the evaluation; it may differ from the model’s developer. Complete current-suite runs only. Reasoning, inference provider, prompt format and output budgets are visible in the evidence view. Identical configurations pool repeated runs. External validation and repeated-run reliability remain open research.
Publication standard
Completed runs. Recorded settings. Explicit review. Partial runs never become successful measurements.
INSIDE THE BENCHMARK
The work is complex.
The question is concrete.
See the kind of judgment we test.
These are representative scenario summaries; scored answer keys and task scaffolding stay protected.
SECURITY
Do the vendor claims hold up?
Compare self-attestations and denials with the underlying vendor-risk evidence.
123tasks in the
current suite
current suite
MODEL SECURITY
Are AI controls supported?
Assess model supply chains, serving exposure, artifacts, and recorded controls.
15tasks in the
current suite
current suite
AGENT SECURITY
Does it retain its judgment?
Measure resistance to injected instructions, escalating pressure, false authority, and needless refusal of legitimate work.
336tasks in the
current suite
current suite
Current suite: release 2026.14. Each published run covers this exact admitted task set. The leaderboard reports each of the four tracks separately, including agent security.
Explore all 62 disciplines in the current suite
THE STANDARD OF PROOF
A score should stand up
to scrutiny.
A rank is only as useful as the test behind it.
Understand the evidence, the evaluation conditions, and the limits of the conclusion.
- 01 →
Define the work
Start with a concrete risk decision and a task with a verifiable answer.
Relevant, answerable tasks - 02 →
Fix the evidence
Use a versioned synthetic portfolio so results refer to the same test material.
A shared research setting - 03 →
Observe the work
Record the model’s investigation, configuration, and outcomes within the evaluation limits.
Traceable conditions - 04
Review the result
Check correctness and completion. Publish the scope and limitations alongside the score.
Evidence before ranking
Go deeper into the researchPressure tests · chance baselines · repeatability +
CURRENT SUITE
What happens under pressure?
The expanded suite tests missing evidence, ambiguous decisions, instruction injection, escalating authority claims, and benign requests that may trigger safety refusals. Refusal is reported separately from an incorrect answer; refused tasks remain in the overall denominator.
MEASURED IN THIS COHORT
Can it solve the task?
Answers are checked against task-specific criteria, rather than an LLM judge’s opinion. Performance and robustness are reported separately.
REQUIRES REPEATED MEASUREMENT
Can it do it consistently?
One completed run shows an outcome. Repeated evaluations are needed to establish stability across attempts and model configurations.
OPEN RESEARCH
Does it transfer to your work?
Synthetic scenarios give us a common starting point. Enterprise practitioners and researchers can help test relevance beyond this portfolio.
v0.25 · MATCHED COMPARISONS
Is pressure changing the answer?
Compare the same vendor question with and without adversarial pressure. Paired confidence intervals and multiple-comparison corrections distinguish observed differences from statistical noise.
v0.25 · CHANCE BASELINES
Better than a lucky guess?
Read task scores against chance and the most common answer in each family. Difficulty checks test whether harder tasks actually predict more failures.
REGISTERED METHODS
Freeze the rules before testing.
Version 0.25 separates a 716-task development book from the 818-task confirmatory book. A recorded specification digest identifies the scoring rules. Confirmatory evidence requires the registered rules to remain unchanged.
RELIABILITY REQUIRES EVIDENCE
One pass is not consistency.
Repeatability tools compare attempts and datasets. Silent responses and refusal indicators remain visible. Practitioner weighting is a research contribution, not an established consensus.
A score is a starting point for scrutiny.
Help validate the benchmark ↓Challenge the task coverage, reproduce a result, or propose a harder case. Help strengthen the measurement as well as the models.
Read the scoring rules, release scope & limitations +
Overall: solved tasks divided by 818. Performance: solved performance tasks divided by 344. Security: 123 tasks. Model security: 15 tasks. Agent security: 336 tasks. Each track score is its solved tasks divided by its task count. Overall is the full-suite task solve rate, not a safety certification.
The displayed cohort is release 2026.14, benchmark v0.25.0, on a frozen synthetic portfolio. Each row represents a measured configuration; repeated complete unguided runs with matching settings are pooled. Reasoning settings vary; the detail view exposes the recorded value rather than assuming every model ran at high effort.
Some disciplines contain only one task. Small score differences should not be treated as established superiority. Confidence intervals shown in run details are Wilson intervals over measured task outcomes; they do not measure between-run variance or account for related tasks.
Open samples explain the task families. Protected scoring material and local harness access remain behind Driftnet authentication. Fixed test material and grading rules support reproducible evaluation conditions; they do not guarantee identical model responses across runs.
LEARN. IMPROVE. RETEST.
A benchmark finds the gaps.
The Gym tests what helps.
The benchmark reveals where a model struggles. The Gym develops strategy guidance, then tests whether the same model can solve more tasks on separate evidence.
SAME MODEL. A REVISED STRATEGY.Improvement has
Improvement has
to be measured.
Learn from an evaluation. Develop reusable guidance. Compare the starting instructions and the proposed strategy on identical fresh validation tasks.
- 01Learn from the evaluation
Identify where the model’s investigation can improve.
- 02Develop a strategy
Add guidance to the instructions, not the model’s weights.
- 03Improve and retest
Measure both strategies on the same separate evidence.
Driftnet sign-in required · Your inference provider account
MEASURED EXAMPLELimited experiment
More TPRM tasks solved.
+10percentage points
in TPRM Performance
in TPRM Performance
Baseline7 / 20 solved
With Gym strategy9 / 20 solved
OBSERVED GAIN · NOT YET CONCLUSIVETwo more tasks solved.
Two more tasks solved.
More evidence needed.
This experiment did not meet the Gym’s promotion threshold (paired p = 0.50). The strategy was not promoted. A higher score in one small test does not establish repeatability or general improvement.
24validation tasks
20 performance + 4 security
20 performance + 4 security
See the comparison, scope and limitations
| Measurement | Fresh baseline | With strategy | Change |
|---|---|---|---|
| TPRM Performance | 7/20 · 35.0% | 9/20 · 45.0% | +10.0 pp |
| Security subset | 4/4 · 100.0% | 4/4 · 100.0% | 0.0 pp |
| All validation tasks | 11/24 · 45.8% | 13/24 · 54.2% | +8.3 pp |
What was compared: the same model and configuration, once with its starting instructions and once with the proposed strategy, on identical validation tasks. Unguided mode; inference-provider-default reasoning; 4,096 maximum output tokens per call.
Why this is not the original leaderboard score: the source evaluation covered 120 tasks. Only 15 of the 24 validation task IDs overlap with that run, and the underlying datasets differ. The Gym remeasured a baseline on the fresh validation dataset to make a paired comparison possible.
Evidence strength: two tasks were solved only with the strategy; none were solved only by the baseline. Exact two-sided paired McNemar p = 0.50. This is a single, limited experiment.
Scope: no model-security tasks were included. The four security tasks do not represent the full Model Robustness dimension. This earlier-release Gym experiment is separate from the current leaderboard; publishing an improved leaderboard entry would require a complete, independently reviewed confirmation evaluation.
BUILT BY SECURITYSCORECARD
Deep in the work.
Invested in the standard.
Discover the TITAN AI platform ↗A convincing answer is not enough when a supplier’s risk becomes your risk.
SecurityScorecard brings the perspective of third-party risk operations: assessing vendors, investigating exposure, challenging self-attestations, and prioritizing action. TPRM Benchmark turns those questions into measurable AI tasks.
The benchmark uses a frozen synthetic portfolio modeled on the structure of vendor-risk evidence. It gives researchers a shared test environment without exposing customer data.
The question is simple.
Can an AI follow the evidence
all the way to the right decision?
Can an AI follow the evidence
all the way to the right decision?
01 / VISIBILITY
TITAN Watch ↗
Understand exposure and continuously monitor your vendor ecosystem.
02 / ASSESSMENTTITAN Assess ↗
Bring automation and risk intelligence into the assessment workflow.
03 / ACTIONTITAN Secure ↗
Prioritize threats and drive continuous third-party risk reduction.
Built from domain expertise. Benchmark scores are research measurements, not product endorsements or deployment certifications.
AN OPEN INVITATION
The next chapter
is a shared effort.
For the teams building models.
The people studying them.
And the professionals who will rely on them.
Bring your model.
Connect a supported inference provider, use a custom endpoint, or request the local harness for your own agent.
Start an evaluation ↗Challenge the method.
Help test repeatability, challenge the scoring assumptions, and examine whether results distinguish meaningful differences in capability.
Join the research ↗Bring the hard cases.
Propose tasks and new tracks. Help establish whether the decisions and evidence gaps your analysts face are represented well.
Contribute a scenario ↗Inference usage is billed to your inference provider account. Reading results and methods requires no account.