🎬 How to build, evaluate, and refine prompts with AI — Latitude
Video Details
Channel: Latitude
URL: https://www.youtube.com/watch?v=G-0Kq9Dt-8c
Relevance: ⭐⭐⭐⭐
Summary
Tutorial on systematic prompt evaluation and refinement using the Latitude platform: defining evaluation criteria (accuracy, format compliance, hallucination rate), running A/B tests between prompt variants, tracking prompt performance over time, and using AI to suggest prompt improvements based on failure cases.
PUMA Relevance
The systematic prompt evaluation workflow is applicable to PUMA’s experiment design. While PUMA uses Promptfoo rather than Latitude, the evaluation criteria (accuracy = F1-macro, format compliance = Pydantic validation pass rate, hallucination rate = cases where output contradicts the issue text) are identical. The A/B testing between strategies mirrors PUMA’s 4-strategy comparison.