AI Agent SF Meetup #4 - Agent Evaluations

Our 4th AI AGENT MEETUP IS SPONSORED BY BEEAI. The SF AI AGENT MEETUP IS A NEW & growing community for AI AGENT DEVELOPERS, Engineers, UX, Ops, and Applied Researchers exploring and leading the evolution of AI AGENTS. Whether you're building autonomous systems, experimenting with LLM-powered assistants, or integrating AI AGENTS INTO REAL-world applications, this MEETUP IS THE PLACE TO DISCOVER, share insights, and collaborate with your peers. Join us for talks, demos, discussions and networking with like-minded innovators shaping the next generation of AI. ? Agenda 5:00 pm – Checkin & Networking 5:30 pm – Introductions & Welcome 5:45 pm – Lightning TalkNatural Language Interaction Protocol (NLIP) by Ranjan Sinha, IBM Fellow 6:00 pm - Talks:"Best Practices for Monitoring AGENT APPLICATIONS IN PRODUCTION" by Claire Longo, Lead AI Researcher @ Comet Prompt Learning: Automatically Optimizing Production AGENTS USING TARGETED FEEDBACK BY PRIYAN JINDAL, AI Engineer @ Arize Open RAG Eval by Ofer Mendelevitch @ Vectara 7:30 pm – Open Networking w/ Pizza and Sodas 8:30 pm – Close Talk Details: Talk #1 - "Best Practices for Monitoring AGENT APPLICATIONS IN PRODUCTION" by Claire Longo, Lead AI Researcher at Comet - Building AI AGENTS REQUIRES A SHIFT FROM MONOLITHIC TO MODULAR ARCHITECTURES, especially when dealing with the non-deterministic nature of LLMs. Traditional monitoring falls short as AGENTS CAN SILENTLY ESCALATE COSTS, hallucinate, and generate biased outputs. In this talk, Claire will share practical strategies for AI AGENT OBSERVABILITY, including cost tracking, eval metrics, and automated AGENT OPTIMIZATION. She will also introduce Comet Opik, an open-source tool designed to optimize and monitor AI AGENTS IN REAL TIME. Talk #2 - Prompt Learning: Automatically Optimizing Production AGENTS USING TARGETED FEEDBACK BY PRIYAN JINDAL, AI Engineer @ Arize - This talk introduces TextPRO, a lightweight and cost-efficient framework developed at Arize for continuously improving AI AGENTS IN PRODUCTION. By simulating realistic failure cases using latent task rules, TextPRO benchmarks the adaptability of various prompt optimization techniques using structured feedback like explanations, annotations, and rules. The talk highlights how even minimal feedback can drive significant improvements in AGENT BEHAVIOR, outperforming traditional feedback-free methods. Practical takeaways include strategies for building self-correcting AGENTS THAT ADAPT TO EVOLVING USER EXPECTATIONS AND TASK REQUIREMENTS. We look forward to building San Francisco's AI AGENT COMMUNITY TOGETHER! ? Food and drink, engaging conversation, and incredible company will all be provided! This AI Alliance event is sponsored by BeeAI. Why join? This is your chance to dive deep into the world of AI AGENTS, connect with a new and emerging developer community. This event is designed for experienced developers, and those who are curious about AGENTS. Please be advised: Unfortunately, space is very limited at these community events and we can not always accept everyone we would like to. We appreciate your application and are looking forward to seeing you at a future event! By coming to this event you agree to be part of group photographs which are used for community building and promoting future events. Code of Conduct: We expect our members to treat each other like family, so please click here to read our Code of Conduct. If you see someone violating it, please speak with your event organizer or reach out to dnielsen@thealliance.ai. The AI Alliance is a non-profit, grassroots community of 150+ companies, startups, researchers, and individuals. Through world-class events, online events, and workshops, members leverage their diverse perspectives and innovative minds to foster meaningful relationships, solve challenging problems, and define the future of AI.