Patronus AI Introduces Multimodal LLM-as-a-Judge for Image Evaluation

Patronus AI has launched the industry's first Multimodal LLM-as-a-Judge (MLLM-as-a-Judge), a tool designed to enhance the evaluation of image-to-text applications, announced in a press release. This new capability allows developers to score and optimize multimodal AI systems by assessing various criteria such as text presence, grid structure, spatial orientation, and object identification.

The Judge-Image tool, powered by Google Gemini, offers several out-of-box evaluation criteria, including caption hallucination detection, object description verification, and object location accuracy. It also tests OCR extraction accuracy for tabular data, AI-generated brand asset accuracy, and scene description validity.

Etsy, a leading e-commerce platform, is already utilizing Patronus AI's MLLM-as-a-Judge to reduce AI hallucinations in product image captions, thereby optimizing their multimodal AI system. Patronus AI plans to expand its evaluation capabilities to include audio and vision features in future releases.