Giskard Unveils Phare: A New Benchmark for Evaluating AI Models

Giskard has introduced Phare, a new open and independent benchmark designed to evaluate large language models (LLMs) on key security dimensions such as hallucination, factual accuracy, bias, and potential for harm. This announcement was made during the Paris AI Summit, with Google DeepMind collaborating as a research partner. The initiative aims to provide open measurements to assess the trustworthiness of generative AI models in real-world applications announced on their website.

Phare, which stands for "Potential Harm Assessment & Risk Evaluation," is designed to evaluate language models across multiple languages, initially including English, French, and Spanish. The benchmark will incorporate diverse linguistic and cultural contexts to ensure comprehensive assessments. The initial scope covers leading models from top AI labs such as OpenAI, Anthropic, Google DeepMind, Meta, Mistral, Alibaba, and DeepSeek.

The benchmark consists of modular test components focusing on four fundamental safety categories: hallucination, bias and fairness, intentional abuse by users, and harmful content generation. Giskard maintains full autonomy in determining the benchmark design, ensuring independence from model developers. The results from these assessments will be tracked on a public leaderboard, with future modules expanding to cover more languages and additional security aspects.

This collaborative effort is part of a broader initiative to improve AI security and robustness, encouraging practical developments in AI safety. Giskard plans to open-source a representative set of samples for each benchmarking module, enabling independent verification and private model testing.