Automate LLM Evaluation at Scale with MLflow make_judge()
Plus LangChain’s PIIMiddleware for data safety
Grab your coffee. Here are this week’s highlights.
📅 Today’s Picks
Automate LLM Evaluation at Scale with MLflow make_judge()
Problem
When you ship LLM features without evaluating them, models might hallucinate, violate safety guidelines, or return incorrectly formatted responses.
Manual review doesn’t scale. Reviewers might miss subtle issues when evaluating thousands of outputs, and scoring standards often vary between people.
Solution
MLflow make_judge() applies the same evaluation standards to every output, whether you’re checking 10 or 10,000 responses.
Key capabilities:
Define evaluation criteria once, reuse everywhere
Automatic rationale explaining each judgment
Built-in judges for safety, toxicity, and hallucination detection
Typed outputs that never return unexpected formats
🧪 Run code | ⭐ View GitHub
⭐ Worth Revisiting
LangChain v1.0: Auto-Protect Sensitive Data with PIIMiddleware
Problem
User messages often contain sensitive information like emails and phone numbers.
Logging or storing this data without protection creates compliance and security risks.
Solution
LangChain v1.0 introduces PIIMiddleware to automatically protect sensitive data before model processing.
PIIMiddleware supports multiple protection modes:
5 built-in detectors (email, credit card, IP, MAC, URL)
Custom regex for any PII pattern
Replace with [REDACTED], mask as ****1234, or block entirely
☕️ Weekly Finds
litellm [LLM] - Python SDK and Proxy Server (AI Gateway) to call 100+ LLM APIs in OpenAI format with cost tracking, guardrails, and logging.
parlant [LLM] - LLM agents built for control with behavioral guidelines, ensuring predictable and consistent agent behavior.
GLiNER2 [ML] - Unified schema-based information extraction for NER, text classification, and structured data parsing in one pass.
Before You Go
🔍 Explore More on CodeCut
Tool Selector - Discover 70+ Python tools for AI and data science
Production Ready Data Science - A practical book for taking projects from prototype to production
⭐ Rate Your Experience
How would you rate your newsletter experience? Share your feedback →