Skip to content

feat(examples): migrate AI examples to OpenTelemetry instrumentation#482

Merged
richardsolomou merged 10 commits intomasterfrom
feat/otel-examples
Apr 15, 2026
Merged

feat(examples): migrate AI examples to OpenTelemetry instrumentation#482
richardsolomou merged 10 commits intomasterfrom
feat/otel-examples

Conversation

@richardsolomou
Copy link
Copy Markdown
Member

@richardsolomou richardsolomou commented Apr 7, 2026

Problem

Our AI examples use PostHog's direct SDK wrappers (posthog.ai.openai, posthog.ai.anthropic, etc.) for tracking LLM calls. We want to silently deprecate these in favor of standard OpenTelemetry auto-instrumentation, which is more portable and follows industry conventions.

Changes

Migrates Python AI examples from PostHog wrappers to OpenTelemetry auto-instrumentation:

  • OpenAI-compatible providers (Groq, DeepSeek, Mistral, xAI, Together AI, Ollama, Cohere, Hugging Face, Perplexity, Cerebras, Fireworks AI, OpenRouter, Helicone, Vercel AI Gateway, Portkey) → opentelemetry-instrumentation-openai-v2
  • OpenAI (all files), Azure OpenAI, Instructor, Autogen, Mirascope, Semantic Kernel, smolagentsopentelemetry-instrumentation-openai-v2
  • Anthropic (chat, streaming, extended thinking) → opentelemetry-instrumentation-anthropic
  • LangChain, LangGraphopentelemetry-instrumentation-langchain
  • LlamaIndexopentelemetry-instrumentation-llamaindex
  • Geminiopentelemetry-instrumentation-google-generativeai

All OTel-based examples set resource attributes to demonstrate the full feature set:

resource = Resource(
    attributes={
        SERVICE_NAME: "example-groq-app",
        "posthog.distinct_id": "example-user",
        "foo": "bar",
        "conversation_id": "abc-123",
    }
)

These map to distinct_id and custom event properties via PostHog's OTLP ingestion endpoint.

Kept as-is: CrewAI (uses LiteLLM callbacks, internally manages its own TracerProvider), LiteLLM/DSPy (use LiteLLM's built-in PostHog callback), OpenAI Agents (uses dedicated posthog.ai.openai_agents.instrument()), Pydantic AI (already OTel via Agent.instrument_all()), AWS Bedrock (already OTel via opentelemetry-instrumentation-botocore).

Key implementation details:

  • Uses SimpleSpanProcessor instead of BatchSpanProcessor so spans export immediately without needing provider.shutdown()
  • # noqa: E402 on intentional late imports after Instrumentor().instrument() calls
  • Azure OpenAI example uses the generic gpt-4o model name instead of a deployment-specific one

How did you test this code?

Manually ran each example against real provider API keys via llm-analytics-apps/run-examples.sh to verify:

  1. Each script runs successfully end-to-end
  2. Traces arrive at PostHog as $ai_generation events
  3. Resource attributes (posthog.distinct_id, foo, conversation_id) flow through as event properties
  4. distinct_id is correctly set on each event

All examples passing ruff format and ruff check.

This is an agent-authored PR — I haven't manually tested each provider end-to-end beyond spot checks, though all examples follow the same pattern and the migration was verified on several providers.

Publish to changelog?

No

Docs update

The onboarding docs will be updated separately in PostHog/posthog#53668 and PostHog/posthog.com#16236.

🤖 LLM context

Co-authored with Claude Code. Related PRs:

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 7, 2026

posthog-python Compliance Report

Date: 2026-04-15 12:04:06 UTC
Duration: 159998ms

⚠️ Some Tests Failed

29/30 tests passed, 1 failed


Capture Tests

29/29 tests passed

View Details
Test Status Duration
Format Validation.Event Has Required Fields 517ms
Format Validation.Event Has Uuid 1508ms
Format Validation.Event Has Lib Properties 1507ms
Format Validation.Distinct Id Is String 1508ms
Format Validation.Token Is Present 1507ms
Format Validation.Custom Properties Preserved 1507ms
Format Validation.Event Has Timestamp 1508ms
Retry Behavior.Retries On 503 9520ms
Retry Behavior.Does Not Retry On 400 3506ms
Retry Behavior.Does Not Retry On 401 3508ms
Retry Behavior.Respects Retry After Header 9512ms
Retry Behavior.Implements Backoff 23532ms
Retry Behavior.Retries On 500 7507ms
Retry Behavior.Retries On 502 7512ms
Retry Behavior.Retries On 504 7513ms
Retry Behavior.Max Retries Respected 23518ms
Deduplication.Generates Unique Uuids 1509ms
Deduplication.Preserves Uuid On Retry 7514ms
Deduplication.Preserves Uuid And Timestamp On Retry 14523ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry 7504ms
Deduplication.No Duplicate Events In Batch 1508ms
Deduplication.Different Events Have Different Uuids 1509ms
Compression.Sends Gzip When Enabled 1507ms
Batch Format.Uses Proper Batch Structure 1507ms
Batch Format.Flush With No Events Sends Nothing 1006ms
Batch Format.Multiple Events Batched Together 1505ms
Error Handling.Does Not Retry On 403 3509ms
Error Handling.Does Not Retry On 413 3508ms
Error Handling.Retries On 408 7515ms

Feature_Flags Tests

⚠️ 0/1 tests passed, 1 failed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id 500ms

Failures

request_payload.request_with_person_properties_device_id

404, message='NOT FOUND', url='http://sdk-adapter:8080/get_feature_flag'

@socket-security
Copy link
Copy Markdown

socket-security bot commented Apr 7, 2026

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedanthropic@​0.87.096100100100100
Addedposthog@​7.10.399100100100100
Addedopentelemetry-sdk@​1.39.199100100100100
Addedopentelemetry-sdk@​1.40.099100100100100
Addedrequests@​2.33.199100100100100
Addedopentelemetry-sdk@​1.34.1100100100100100
Addedopentelemetry-instrumentation-anthropic@​0.57.0100100100100100

View full report

@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps bot commented Apr 8, 2026

Vulnerabilities

No security concerns identified. API keys are sourced exclusively from environment variables (os.environ[...]) and are not hardcoded. The OTLP exporter sends keys in an HTTP Authorization header over HTTPS to the PostHog endpoint, which is the expected pattern.

Prompt To Fix All With AI
This is a comment left during a code review.
Path: examples/example-ai-azure-openai/chat.py
Line: 28

Comment:
**Unrecognized Azure model name**

`grok-4-20-non-reasoning` does not correspond to any model ID in Azure AI Foundry's catalog. The available xAI Grok names there are `grok-4`, `grok-4-fast-non-reasoning`, `grok-4-fast-reasoning`, `grok-4.1-fast-non-reasoning`, etc. This looks like a typo or an accidental change — using this value in an example will mislead anyone who tries to replicate it and will fail against any standard deployment. This change also appears unrelated to the OTel migration that is the purpose of this PR.

```suggestion
    model="gpt-4o",
```

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: examples/example-ai-openai/transcription.py
Line: 31-34

Comment:
**Inconsistent `provider.shutdown()` on early exit path**

This `provider.shutdown()` before `sys.exit(0)` is the only remaining shutdown call across all 28 migrated examples. The PR description explicitly states that `SimpleSpanProcessor` is used so spans export immediately without needing `provider.shutdown()`. Remove it for consistency.

```suggestion
if not os.path.exists(audio_path):
    print(f"Skipping: audio file not found at '{audio_path}'")
    print("Set AUDIO_PATH to a valid audio file (mp3, wav, m4a, etc.)")
    sys.exit(0)
```

How can I resolve this? If you propose a fix, please make it concise.

Reviews (1): Last reviewed commit: "style(examples): add noqa: E402 for inte..." | Re-trigger Greptile

Copy link
Copy Markdown
Member

@andrewm4894 andrewm4894 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think the otel examples should be as full featured as possible and show stuff like setting distinct id and any custom foo-bar properties - am i missing that or something?

richardsolomou added a commit that referenced this pull request Apr 9, 2026
Aligns with the endpoint used by the OTel examples in PR #482.

Generated-By: PostHog Code
Task-Id: 1ba1f07a-1453-4162-90a8-665958c5fe46
@richardsolomou richardsolomou changed the base branch from master to feat/otel-integration April 14, 2026 08:52
Copy link
Copy Markdown
Member Author

richardsolomou commented Apr 14, 2026

richardsolomou added a commit that referenced this pull request Apr 14, 2026
Aligns with the endpoint used by the OTel examples in PR #482.

Generated-By: PostHog Code
Task-Id: 1ba1f07a-1453-4162-90a8-665958c5fe46
@richardsolomou richardsolomou force-pushed the feat/otel-integration branch from ee77800 to 8818397 Compare April 14, 2026 09:40
richardsolomou added a commit that referenced this pull request Apr 14, 2026
Aligns with the endpoint used by the OTel examples in PR #482.

Generated-By: PostHog Code
Task-Id: 1ba1f07a-1453-4162-90a8-665958c5fe46
@richardsolomou richardsolomou force-pushed the feat/otel-integration branch from 8818397 to 284aba6 Compare April 14, 2026 11:22
Copy link
Copy Markdown
Member Author

richardsolomou commented Apr 15, 2026

Merge activity

  • Apr 15, 11:59 AM UTC: A user started a stack merge that includes this pull request via Graphite.
  • Apr 15, 12:01 PM UTC: Graphite rebased this pull request as part of a merge.
  • Apr 15, 12:02 PM UTC: @richardsolomou merged this pull request with Graphite.

@richardsolomou richardsolomou changed the base branch from feat/otel-integration to graphite-base/482 April 15, 2026 11:59
richardsolomou added a commit that referenced this pull request Apr 15, 2026
## Problem

Users instrumenting their AI/LLM applications with OpenTelemetry have no way to route AI spans to PostHog without writing custom glue code. This is the Python equivalent of PostHog/posthog-js#3358.

## Changes

Adds a `posthog.ai.otel` module with two integration patterns:

- **`PostHogSpanProcessor`** (recommended) — self-contained `SpanProcessor` that wraps `BatchSpanProcessor` + `OTLPSpanExporter`, filtering to only forward AI spans to PostHog's OTLP endpoint at `/i/v0/ai/otel`
- **`PostHogTraceExporter`** — `SpanExporter` that filters AI spans before forwarding, for setups that only accept an exporter (e.g. passed to `SimpleSpanProcessor`)
- **`is_ai_span()`** — shared predicate matching `gen_ai.*`, `llm.*`, `ai.*`, `traceloop.*` prefixes on span names and attribute keys

Also adds `posthog[otel]` optional dependency group for `opentelemetry-sdk` and `opentelemetry-exporter-otlp-proto-http`.

### Usage

```python
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.resources import Resource, SERVICE_NAME
from posthog.ai.otel import PostHogSpanProcessor

resource = Resource(
    attributes={
        SERVICE_NAME: "my-app",
        "posthog.distinct_id": "user-123",
    }
)
provider = TracerProvider(resource=resource)
provider.add_span_processor(
    PostHogSpanProcessor(api_key="phc_...")
)
trace.set_tracer_provider(provider)
```

### Consistency with #482

Uses the same OTLP endpoint (`/i/v0/ai/otel`) and auth header format (`Authorization: Bearer {api_key}`) as the migrated examples in #482.

## How did you test this code?

31 unit tests covering span filtering, processor forwarding/dropping, exporter filtering, endpoint configuration, and lifecycle delegation.
@richardsolomou richardsolomou changed the base branch from graphite-base/482 to master April 15, 2026 11:59
@richardsolomou richardsolomou requested a review from a team as a code owner April 15, 2026 11:59
richardsolomou added a commit to PostHog/posthog that referenced this pull request Apr 15, 2026
…tion (#53668)

## Problem

The in-app LLM analytics onboarding docs currently show PostHog SDK wrappers (`posthog.ai.openai`, `@posthog/ai`) as the primary integration method. We're moving toward standard OpenTelemetry auto-instrumentation as the recommended approach, keeping the wrappers available as a last resort for users who need them.

## Changes

Migrates 28 provider onboarding guides from PostHog SDK wrapper approach to OpenTelemetry auto-instrumentation. Each migrated doc follows a consistent 3-step pattern:

1. **Install dependencies** — OTel SDK + provider-specific instrumentation + provider SDK
2. **Set up OpenTelemetry tracing** — `TracerProvider` with `OTLPSpanExporter` (Python) or `PostHogTraceExporter` (Node)
3. **Call the provider** — Native SDK usage, no `posthog_` parameters

User identification uses the `posthog.distinct_id` resource attribute; custom properties like `foo` and `conversation_id` demonstrate the passthrough behavior.

**Providers migrated (21 original + 7 more):**
- OpenAI-compatible: openai, groq, deepseek, mistral, cohere, fireworks-ai, xai, hugging-face, openrouter, together-ai, ollama, cerebras, perplexity
- AI gateways: portkey, helicone, vercel-ai-gateway
- Azure OpenAI, Anthropic
- Frameworks: langchain, langgraph, llamaindex
- Vercel AI (SimpleSpanProcessor update)
- **Newly migrated:** autogen, instructor, mirascope, semantic-kernel, smolagents, pydantic-ai (already OTel-based in examples — docs just hadn't caught up)
- **Mastra** (migrated to `@mastra/posthog` exporter — Mastra's native integration since there's no mature OTel path yet)

**Kept as-is:** google (no Node.js OTel instrumentation for `@google/genai`), aws-bedrock (already OTel), crewai, litellm, dspy, openai-agents, manual

**Migration callout:** Every migrated doc has a callout at the top linking to the full Node.js and Python examples on GitHub (current `main`/`master`), plus legacy wrapper examples pinned to the last commit before the migration. This helps both new users (who see the recommended OTel path) and existing users (who can find the old wrapper examples they're already using).

**Manual capture doc:** Split into multiple steps so it can have a useful table of contents (Capture LLM events manually → Event properties → Verify traces and generations).

**Removed:** "No proxy" callouts from all migrated docs — they made sense when wrapping SDK clients but don't apply to standard OTel tracing.

## How did you test this code?

Agent-authored PR. Manually verified:
- TypeScript compiles (`pnpm --filter=@posthog/frontend format` runs cleanly)
- Each migrated doc follows the same pattern as other OTel docs
- Example code in the docs matches what's in the real example repos (PostHog/posthog-js#3349 and PostHog/posthog-python#482)

No runtime tests since these are in-app onboarding content (static TSX returning step definitions).

## Publish to changelog?

No

## Docs update

This IS the docs update. Companion PR: PostHog/posthog.com#16236 (updates TOC frontmatter in posthog.com).

## 🤖 LLM context

Co-authored with Claude Code. Related PRs:
- PostHog/posthog-js#3349 (Node.js OTel examples)
- PostHog/posthog-python#482 (Python OTel examples)
- PostHog/posthog.com#16236 (docs.posthog.com TOC updates)
Switch from PostHog direct SDK wrappers to OpenTelemetry auto-instrumentation
for all AI provider examples where OTel instrumentations are available.

Uses opentelemetry-instrumentation-openai-v2 for OpenAI-compatible providers,
opentelemetry-instrumentation-anthropic for Anthropic, opentelemetry-instrumentation-google-generativeai
for Gemini, opentelemetry-instrumentation-langchain for LangChain/LangGraph,
opentelemetry-instrumentation-llamaindex for LlamaIndex, and opentelemetry-instrumentation-crewai for CrewAI.
Switch from BatchSpanProcessor to SimpleSpanProcessor so spans are
exported immediately. This removes the need for provider.shutdown()
which could throw export errors on exit.
CrewAI manages its own TracerProvider internally, which conflicts
with setting one externally. LiteLLM callbacks remain the correct
integration approach for CrewAI.
…rter

Adopt the new PostHogSpanProcessor API from #494 across all OTEL examples.
Drops the manual SimpleSpanProcessor + OTLPSpanExporter wiring in favor of
the higher-level processor that handles batching and AI span filtering.

Each example now depends on posthog[otel] via a uv path source so it
resolves against the workspace checkout while #494 is unreleased.
@richardsolomou richardsolomou merged commit b738fd9 into master Apr 15, 2026
25 checks passed
@richardsolomou richardsolomou deleted the feat/otel-examples branch April 15, 2026 12:02
pauldambra pushed a commit to PostHog/posthog that referenced this pull request Apr 15, 2026
…tion (#53668)

## Problem

The in-app LLM analytics onboarding docs currently show PostHog SDK wrappers (`posthog.ai.openai`, `@posthog/ai`) as the primary integration method. We're moving toward standard OpenTelemetry auto-instrumentation as the recommended approach, keeping the wrappers available as a last resort for users who need them.

## Changes

Migrates 28 provider onboarding guides from PostHog SDK wrapper approach to OpenTelemetry auto-instrumentation. Each migrated doc follows a consistent 3-step pattern:

1. **Install dependencies** — OTel SDK + provider-specific instrumentation + provider SDK
2. **Set up OpenTelemetry tracing** — `TracerProvider` with `OTLPSpanExporter` (Python) or `PostHogTraceExporter` (Node)
3. **Call the provider** — Native SDK usage, no `posthog_` parameters

User identification uses the `posthog.distinct_id` resource attribute; custom properties like `foo` and `conversation_id` demonstrate the passthrough behavior.

**Providers migrated (21 original + 7 more):**
- OpenAI-compatible: openai, groq, deepseek, mistral, cohere, fireworks-ai, xai, hugging-face, openrouter, together-ai, ollama, cerebras, perplexity
- AI gateways: portkey, helicone, vercel-ai-gateway
- Azure OpenAI, Anthropic
- Frameworks: langchain, langgraph, llamaindex
- Vercel AI (SimpleSpanProcessor update)
- **Newly migrated:** autogen, instructor, mirascope, semantic-kernel, smolagents, pydantic-ai (already OTel-based in examples — docs just hadn't caught up)
- **Mastra** (migrated to `@mastra/posthog` exporter — Mastra's native integration since there's no mature OTel path yet)

**Kept as-is:** google (no Node.js OTel instrumentation for `@google/genai`), aws-bedrock (already OTel), crewai, litellm, dspy, openai-agents, manual

**Migration callout:** Every migrated doc has a callout at the top linking to the full Node.js and Python examples on GitHub (current `main`/`master`), plus legacy wrapper examples pinned to the last commit before the migration. This helps both new users (who see the recommended OTel path) and existing users (who can find the old wrapper examples they're already using).

**Manual capture doc:** Split into multiple steps so it can have a useful table of contents (Capture LLM events manually → Event properties → Verify traces and generations).

**Removed:** "No proxy" callouts from all migrated docs — they made sense when wrapping SDK clients but don't apply to standard OTel tracing.

## How did you test this code?

Agent-authored PR. Manually verified:
- TypeScript compiles (`pnpm --filter=@posthog/frontend format` runs cleanly)
- Each migrated doc follows the same pattern as other OTel docs
- Example code in the docs matches what's in the real example repos (PostHog/posthog-js#3349 and PostHog/posthog-python#482)

No runtime tests since these are in-app onboarding content (static TSX returning step definitions).

## Publish to changelog?

No

## Docs update

This IS the docs update. Companion PR: PostHog/posthog.com#16236 (updates TOC frontmatter in posthog.com).

## 🤖 LLM context

Co-authored with Claude Code. Related PRs:
- PostHog/posthog-js#3349 (Node.js OTel examples)
- PostHog/posthog-python#482 (Python OTel examples)
- PostHog/posthog.com#16236 (docs.posthog.com TOC updates)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants