garak
Generative AI vulnerability scanner and red-teaming tool that probes language models for prompt injections and data leaks.
Technical Architecture & Overview
Garak is an open-source LLM vulnerability scanner backed by NVIDIA and actively maintained by NVIDIA and the community. It tests Large Language Models against known attack vectors like prompt injections, jailbreaks, data leakage, toxicity generation, and hallucination prompts.
Targeted Technical Use Cases
Automated baseline vulnerability testing of foundation models and fine-tuned LLM endpoints in CI/CD pipelines.
Evaluation & Trade-offs
Core Strengths
- +Broad library of pre-built attack probes and evaluation detectors.
- +Works with models from OpenAI, Hugging Face, Replicate, AWS Bedrock, LiteLLM, Cohere, Groq, and local endpoints.
- +Generates detailed hit logs, JSONL reports, and compliance summaries.
Trade-Offs & Limitations
- -Can incur high API costs during deep probe runs against commercial models.
- -Requires regular updates as new jailbreak techniques emerge.
Defensive Security Application
Evaluating model guardrails and testing for prompt leakage before deploying public AI assistants.
Frequently Asked Questions
What is garak?→
Garak is an open-source LLM vulnerability scanner backed by NVIDIA and actively maintained by NVIDIA and the community. It tests Large Language Models against known attack vectors like prompt injections, jailbreaks, data leakage, toxicity generation, and hallucination prompts.
What is garak used for?→
Automated baseline vulnerability testing of foundation models and fine-tuned LLM endpoints in CI/CD pipelines.
What are the strengths of garak?→
- +Broad library of pre-built attack probes and evaluation detectors.
- +Works with models from OpenAI, Hugging Face, Replicate, AWS Bedrock, LiteLLM, Cohere, Groq, and local endpoints.
- +Generates detailed hit logs, JSONL reports, and compliance summaries.
What are the limitations of garak?→
- +Can incur high API costs during deep probe runs against commercial models.
- +Requires regular updates as new jailbreak techniques emerge.
How is garak used defensively?→
Evaluating model guardrails and testing for prompt leakage before deploying public AI assistants.