Building a Model Evaluation Practice for Production AI
- WHO IT IS FOR: This session is designed specifically for AI practitioners, machine learning engineers, and software developers.
- WHAT IT IS ABOUT: Learn how to operationalize LLM evaluation using golden datasets, hallucination metrics, and cost-per-answer analysis.
- FORMAT: This event features an educational technical session followed by a live model comparison demo and professional networking.
Transition your Large Language Models from experimentation to production with confidence. This technical session provides a comprehensive framework for building a robust model evaluation practice, focusing on the critical metrics needed to ensure reliability in real-world applications. You will dive deep into methodologies for using golden datasets to track hallucination rates and optimize cost-per-answer metrics.
Attendees will explore practical implementation strategies, including how to establish production evaluation gates using industry-leading tools like Ragas, LangSmith, and Amazon Bedrock. The session concludes with an interactive side-by-side model comparison demo, offering a hands-on look at how different models perform under scrutiny. Wrap up the evening by connecting with fellow AI professionals in New York during our scheduled networking segment.