garak
Generative AI vulnerability scanner and red-teaming tool that probes language models for prompt injections and data leaks.
Technical Architecture & Overview
Garak is an open-source LLM vulnerability scanner backed by NVIDIA and actively maintained by NVIDIA and the community. It tests Large Language Models against known attack vectors like prompt injections, jailbreaks, data leakage, toxicity generation, and hallucination prompts.
Targeted Technical Use Cases
Automated baseline vulnerability testing of foundation models and private LLM endpoints.
Evaluation & Trade-offs
Core Strengths
- +Broad library of pre-built attack probes and evaluation detectors.
- +Works with models from OpenAI, Hugging Face, Replicate, AWS Bedrock, LiteLLM, Cohere, Groq, and local endpoints.
- +Writes a JSONL report for every run plus a hit log of attempts that yielded a vulnerability.
Trade-Offs & Limitations
- -Can incur high API costs during deep probe runs against commercial models.
- -Probe coverage is limited to the installed release; the PyPI package is updated periodically, and newer changes ship first on GitHub.
Defensive Security Application
Evaluating model guardrails and testing for prompt leakage before deploying public AI assistants.
Frequently Asked Questions
What is garak?→
Garak is an open-source LLM vulnerability scanner backed by NVIDIA and actively maintained by NVIDIA and the community. It tests Large Language Models against known attack vectors like prompt injections, jailbreaks, data leakage, toxicity generation, and hallucination prompts.
What is garak used for?→
Automated baseline vulnerability testing of foundation models and private LLM endpoints.
What are the strengths of garak?→
- +Broad library of pre-built attack probes and evaluation detectors.
- +Works with models from OpenAI, Hugging Face, Replicate, AWS Bedrock, LiteLLM, Cohere, Groq, and local endpoints.
- +Writes a JSONL report for every run plus a hit log of attempts that yielded a vulnerability.
What are the limitations of garak?→
- +Can incur high API costs during deep probe runs against commercial models.
- +Probe coverage is limited to the installed release; the PyPI package is updated periodically, and newer changes ship first on GitHub.
How is garak used defensively?→
Evaluating model guardrails and testing for prompt leakage before deploying public AI assistants.