Inspect
Evaluation framework from the UK AI Security Institute and Meridian Labs for testing language models across reasoning, safety, and tool use.
Technical Architecture & Overview
Inspect is an open-source framework developed by the UK AI Security Institute and Meridian Labs for designing, running, and scoring LLM evaluations. It supports safety evaluations, multi-turn agentic scenarios, and benchmark scoring across multiple model providers.
Targeted Technical Use Cases
Running safety and security evaluations of foundation models, including agentic and cybersecurity benchmarks.
Evaluation & Trade-offs
Core Strengths
- +Developed by the UK AI Security Institute for frontier AI evaluations.
- +Supports multi-turn agentic evaluations and complex scoring pipelines.
- +Works with OpenAI, Anthropic, Google, Mistral, Hugging Face, and local model providers.
Trade-Offs & Limitations
- -Each evaluation runs over a dataset of labelled samples, which must be supplied or taken from a pre-built evaluation.
- -Custom tasks, solvers, and scorers are written as Python code against the inspect_ai API.
Defensive Security Application
Conducting pre-deployment safety and security evaluations of foundation models.
Frequently Asked Questions
What is Inspect?→
Inspect is an open-source framework developed by the UK AI Security Institute and Meridian Labs for designing, running, and scoring LLM evaluations. It supports safety evaluations, multi-turn agentic scenarios, and benchmark scoring across multiple model providers.
What is Inspect used for?→
Running safety and security evaluations of foundation models, including agentic and cybersecurity benchmarks.
What are the strengths of Inspect?→
- +Developed by the UK AI Security Institute for frontier AI evaluations.
- +Supports multi-turn agentic evaluations and complex scoring pipelines.
- +Works with OpenAI, Anthropic, Google, Mistral, Hugging Face, and local model providers.
What are the limitations of Inspect?→
- +Each evaluation runs over a dataset of labelled samples, which must be supplied or taken from a pre-built evaluation.
- +Custom tasks, solvers, and scorers are written as Python code against the inspect_ai API.
How is Inspect used defensively?→
Conducting pre-deployment safety and security evaluations of foundation models.