Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪

LLM Reasoning and Generation Benchmark. Evaluate LLMs in complex scenarios systematically.

Prompture is an API-first library for requesting structured JSON output from LLMs (or any structure), validating it against a schema, and running comparative tests between models.

Test, compare, and optimize your AI prompts in minutes

The prompt engineering, prompt management, and prompt evaluation tool for TypeScript, JavaScript, and NodeJS.

Community Plugin for Genkit to use Promptfoo

prompt-evaluator is an open-source toolkit for evaluating, testing, and comparing LLM prompts. It provides a GUI-driven workflow for running prompt tests, tracking token usage, visualizing results, and ensuring reliability across models like OpenAI, Claude, and Gemini.

LLM Prompt Test helps you test Large Language Models (LLMs) prompts to ensure they consistently meet your expectations.

Test Claude Projects without copy-pasting. Local workbench for prompt engineering, agent testing, and workflow iteration. Direct Claude.ai access via cookie auth, 20+ prompt templates, web fetch/search tools, file uploads. Stop switching tabs to test your prompts.

curl for prompts. Run .prompt files against any LLM (Anthropic, OpenAI, Ollama) from the terminal. Treat prompts as code — version them, review them in PRs, and test them in CI.

Open-source prompt optimization tool — improve your LLM prompts with AI-powered suggestions and explanations.

OWASP LLM Top 10 vulnerability scanner CLI — test your AI endpoints for prompt injection, jailbreaks, data leakage & more. Fast red-teaming tool with pass/fail reports + fix recommendations. 🛡️

An open-source testing framework for AI agents. Simulate LLM and tool calls to test edge cases, failure paths, and agent logic without live API calls.

An open-source AI prompt engineering playground with live code execution. Test OpenAI & Claude prompts, execute JavaScript, and iterate in real-time.

AI agent that helps you create, test, and iterate on LLM prompts. Saves versioned artifacts, generates test samples, runs evaluations, and provides detailed performance analysis.

Sample project demonstrates how to use Promptfoo, a test framework for evaluating the output of generative AI models

Visual prompt engineering platform for creating, testing, and versioning LLM prompts across multiple providers (OpenAI, Anthropic, Mistral, Gemini).

PromptGuard is a pragmatic, opinionated framework for establishing continuous integration for LLM behavior. It operates on a simple, verifiable principle: run the same prompts across multiple model configurations, compare outputs against defined expectations, and flag semantic regressions.

A collection of prompts that I use on a day-to-day basis for work and leisure.