This repository contains lesson materials and code examples for Unit 11 of the Agentic AI Developer Certification Program by Ready Tensor.
This is a work-in-progress repository. We are actively adding code examples and documentationto this repository. Stay tuned for updates!
- How to monitor agentic AI systems in production using logs, traces, and tools like LangSmith
- Techniques for debugging agent failures, hallucinations, and non-deterministic behavior
- Best practices for handling sensitive data and ensuring compliance with frameworks like GDPR and HIPAA
- Key considerations for choosing between hosted models and API-based LLMs, including tradeoffs in latency, control, cost, and reliability
- How to approach real-world agentic AI development through high-level case studies focused on system design, architecture, and planning decisions
-
Lesson 1a: Monitoring and Observability 101 Understand the difference between monitoring and observation - and building blocks such as metrics, logs, and traces.
-
Lesson 1b: What to Monitor Learn how to track failures like hallucinations, bad routing, and tool misuse. We’ll map common failure modes and their causes to the right metrics and traces.
-
Lesson 1c: Choosing Monitoring Tools Compare popular observability tools for agentic systems, including LangSmith, Langfuse, OpenTelemetry, and others. You’ll learn what to look for and how to choose based on your stack and scale.
-
Lesson 1d: Diagnosing Root Causes Go a level deeper to understand how to debug agentic AI systems when they misbehave.
-
Lesson 2: Data Privacy and Compliance Learn how to design with privacy and compliance in mind. We’ll cover the implications of GDPR, HIPAA, and other frameworks when agents handle sensitive data.
-
Lesson 3: Proprietary vs Open Weight LLMs Explore when to use proprietary APIs versus open-weight models, and how to weigh tradeoffs like performance, control, compliance, and cost.
- Lesson 4: Real-World Development Blueprints Walk through real-world agentic AI development blueprints, starting from project planning through development and deployment.
This project is licensed under the CC BY-NC-SA 4.0 License – see the LICENSE file for details.
Ready Tensor, Inc.
- Email: contact at readytensor dot com
- Issues & Contributions: Open an issue or PR on this repo
- Website: https://readytensor.ai