Hey everyone!

I’m one of the co-founders of Addepto, and I’m excited to share ContextCheck—a new open-source framework we’ve developed for testing and evaluating LLMs, RAGs, and chatbots.

ContextCheck offers tools to:

• Automatically generate queries and request completions
• Detect regressions and assess hallucinations
• Perform penetration testing
• Ensure the robustness and reliability of AI systems

It’s fully configurable via YAML and integrates seamlessly into CI pipelines for automated testing.

We built ContextCheck during the development of our AI-powered Knowledge Base Assistant to solve the challenges we faced with testing and validating Large Language Models. It’s a tool designed by developers for developers to tackle real-world issues.

We’d love for you to try it out, contribute, and share your feedback!

Github repo