[AI-6669] NiFi - Add log pipeline and log integration assets by philjlee · Pull Request #23302 · DataDog/integrations-core

philjlee · 2026-04-13T20:51:35Z

Jira: AI-6669
Epic: AI-6662
Parent PR: #23110 (AI-6668 - NiFi Agent Integration)

Why

The NiFi Agent integration (PR #23110) includes metrics, events, and a basic log pipeline. This PR completes the log integration by adding test samples for CI validation, an enhanced pipeline for all NiFi log types, saved views, and required metadata.

Summary

Enhanced log pipeline (nifi.yaml): Restructured into nested sub-pipelines — one for application logs (nifi-app.log, nifi-user.log, nifi-bootstrap.log) and one for HTTP request logs (nifi-request.log, Combined Log Format). Request log parsing uses the same grok patterns as the Apache/Nginx integrations.
Test samples (nifi_tests.yaml): 5 test cases covering INFO/ERROR/WARN app logs and GET 200/POST 404 request logs, with log_type:request tag routing.
Log configuration (spec.yaml, conf.yaml.example): Added all 4 NiFi log file paths with log_type:request tag on the request log entry.
Saved views: Error logs view (source:nifi status:(error OR warn)) and access logs view (source:nifi log_type:request).
Dataflows (dataflows.yaml): Declares nifi-metrics and nifi-logs data types.
Manifest: Added logs.source and saved view references.
README: Added Log Collection section with configuration examples.

Test plan

ddev validate config -s nifi passes
ddev validate models -s nifi passes
ddev test -fs nifi passes (formatting)
CI validates log pipeline YAML + test samples
Logs pipeline team review on assets/logs/ files

codecov · 2026-04-13T20:57:13Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.05%. Comparing base (a10f0fd) to head (2ada410).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

datadog-prod-us1-5 · 2026-04-13T20:57:24Z

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 98.20% (+11.07%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 2ada410 | Docs | Datadog PR Page | Give us feedback!}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fe87384749

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-04-13T20:57:51Z

+        if self.instance.get('username') and self.instance.get('password'):
+            self.http.options['auth'] = None


Preserve configured HTTP auth when credentials are supplied

This condition unconditionally disables RequestsWrapper auth whenever username and password are set, and the API client then forces token auth via /access/token. In deployments that use auth_type credentials for a reverse proxy (for example basic/digest in front of NiFi), the check will stop sending configured HTTP auth and fail before scraping metrics if the token endpoint is not available through that proxy. The token-auth path should be gated by an explicit mode instead of keying only on presence of credentials.

Useful? React with 👍 / 👎.

Rebased onto master now that the AI-6668 implementation is merged. This PR completes the integration tile by adding assets the implementation PR deliberately left out of scope. Adds: - `manifest.json` with metrics, events, logs, dashboard, saved views - `assets/dashboards/nifi_overview.json` overview dashboard - `assets/dataflows.yaml` declaring nifi-metrics, nifi-events, nifi-logs - `assets/logs/nifi.yaml` pipeline with nested sub-pipelines for app logs and Combined Log Format request logs - `assets/logs/nifi_tests.yaml` with 5 pipeline test cases - `assets/saved_views/` for error logs and HTTP access logs - `images/IMAGES_README.md` media gallery guide - Log config entries in `spec.yaml` and `conf.yaml.example` for all four NiFi log files (app, user, bootstrap, request) - README log collection section with multiline stack trace guidance Drops the earlier `service_checks.json` and its manifest reference — the check only emits the `nifi.can_connect` gauge, not a service check, so the README already documents connectivity there.

cit-pr-commenter-54b7da · 2026-04-20T18:20:29Z

Monitor Template Quality Assessment

52 monitors analyzed across 9 integrations.

29 monitors have missing sections
Most common missing sections: WHY, IMPACT, HOW_TO_TROUBLESHOOT

Monitors with missing sections

Integration	Monitor	Missing Sections	Suggested Links
kubernetes	[Kubernetes] Monitor Kubernetes Deployments Replica Pods	IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS	- Logs

Infrastructure
Metrics Explorer (kubernetes_state.deployment.replicas_desired)
Metrics Explorer (kubernetes_state.deployment.replicas_available) |
| kubernetes | [Kubernetes] Monitor Unschedulable Kubernetes Nodes | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Logs
Infrastructure
Metrics Explorer (kubernetes_state.node.status) |
| kubernetes | [Kubernetes] Pod {{pod_name.name}} is CrashloopBackOff on namespace {{kube_namespace.name}} | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Logs
Infrastructure
Metrics Explorer (kubernetes_state.container.status_report.count.waiting) |
| kubernetes | [Kubernetes] Pod {{pod_name.name}} is ImagePullBackOff on namespace {{kube_namespace.name}} | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Logs
Infrastructure
Metrics Explorer (kubernetes_state.container.status_report.count.waiting) |
| kubernetes | [Kubernetes] Pod {{pod_name.name}} is OOMKilled on namespace {{kube_namespace.name}} | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Logs
Infrastructure
Metrics Explorer (kubernetes.containers.state.terminated) |
| kubernetes | [Kubernetes] Monitor Kubernetes Failed Pods in Namespaces | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Logs
Infrastructure
Metrics Explorer (kubernetes_state.pod.status_phase) |
| kubernetes | [Kubernetes] Monitor Kubernetes Pods Restarting | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Logs
Infrastructure
Metrics Explorer (kubernetes.containers.restarts) |
| kubernetes | [Kubernetes] Monitor Kubernetes Statefulset Replicas | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Logs
Infrastructure
Metrics Explorer (kubernetes_state.statefulset.replicas_desired)
Metrics Explorer (kubernetes_state.statefulset.replicas_ready) |
| nginx | [NGINX] 4xx Errors higher than usual | WHY | - Logs
Infrastructure
Metrics Explorer (nginx.upstream.peers.responses.4xx) |
| nginx | [NGINX] 5xx Errors higher than usual | WHY | - Logs
Infrastructure
Metrics Explorer (nginx.upstream.peers.responses.5xx) |
| postgres | [Postgres] Number of connections is approaching connection limit on {{host.name}} | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Metrics Explorer (postgresql.percent_usage_connections) |
| postgres | [Postgres] Replication delay is abnormally high on {{host.name}} | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Metrics Explorer (postgresql.replication_delay) |
| redis | [Redis] High memory consumption | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Metrics Explorer (redis.mem.used)
Metrics Explorer (redis.mem.maxmemory) |
| amazon-rds | [AWS] RDS CPU utilization is high | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Logs
Infrastructure
Metrics Explorer (aws.rds.cpuutilization) |
| amazon-rds | [AWS] RDS Anomaly in database connections | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Logs
Infrastructure
Metrics Explorer (aws.rds.database_connections) |
| amazon-rds | [AWS] RDS Storage utilization is high | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | - Logs
Infrastructure
Metrics Explorer (aws.rds.free_storage_space)
Metrics Explorer (aws.rds.total_storage_space) |
| amazon-sqs | Queue {{queuename.name}} has a high age of oldest message | WHY | - Logs
Infrastructure
Metrics Explorer (aws.sqs.approximate_age_of_oldest_message) |
| apm | Abnormal change in p75 latency for $service.value on $env | WHATS_HAPPENING, IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | no metrics found in query |
| apm | Watchdog Anomaly: {{event.title}} for {{service}} | WHATS_HAPPENING, IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | Monitor type is not supported for message generation: event-v2 alert |
| apm | New issues found for $service on $env | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | no metrics found in query |
| apm | High p95 latency for $service on $env | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | no metrics found in query |
| apm | Abnormal change in Apdex for $service.value on $env | WHATS_HAPPENING, IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | no metrics found in query |
| apm | Low traffic on $service on $env | IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | no metrics found in query |
| apm | New issues found for {{service}} on {{env}} | WHATS_HAPPENING, IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | Monitor type is not supported for message generation: error-tracking alert |
| apm | No data for $service.value on $env | WHATS_HAPPENING, IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | no metrics found in query |
| apm | Increased error rate for $service.value in $env | WHATS_HAPPENING, IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | no metrics found in query |
| real-user-monitoring | View Error Rate is high for view {{[view.name].name}} | WHY | Unhandled exception while parsing message: parsing metric query failed: Rule 'or_op' didn't match at '.name} by {view.name' (line 1, column 53). |
| system | Network traffic (sent) is high on host {{host.name}} | WHY | - Logs
Infrastructure
Metrics Explorer (system.net.bytes_sent) |
| system | Host $host.value not reporting data | WHATS_HAPPENING, IMPACT, WHY, HOW_TO_TROUBLESHOOT, RELATED_LINKS | Monitor type is not supported for message generation: service check |

- Fix GC widget query to use `sum:...as_rate()` for correct rate semantics on the monotonic_count metric (was `avg:` which misrepresents the rate). - Expand the overview dashboard to cover cluster health, bytes queued, processor status breakdown, and top connections by backpressure, so the dashboard reflects the full scope the check advertises. - Fix the error-logs saved view query: include `warning` status so 4xx HTTP request logs (tagged `status:warning` by the pipeline) surface too.

Review from janine-c is dismissed. Related teams and files:

documentation
- nifi/assets/dashboards/nifi_overview.json
- nifi/assets/saved_views/nifi_errors.json

- Add missing standard attribute facets (URL host/port/scheme, useragent browser/device/os families) and fix sort order - Update request log test expectations to match actual pipeline user-agent parser output - Add changelog entry for PR #23302

Resolve dependency lockfile conflicts by taking master's versions.

These assets belong in Developer Platform, not integrations-core.

dd-octo-sts · 2026-04-28T18:12:57Z

Validation Report

All 20 validations passed.

Show details

Validation	Description	Status
`agent-reqs`	Verify check versions match the Agent requirements file	✅
`ci`	Validate CI configuration and Codecov settings	✅
`codeowners`	Validate every integration has a CODEOWNERS entry	✅
`config`	Validate default configuration files against spec.yaml	✅
`dep`	Verify dependency pins are consistent and Agent-compatible	✅
`http`	Validate integrations use the HTTP wrapper correctly	✅
`imports`	Validate check imports do not use deprecated modules	✅
`integration-style`	Validate check code style conventions	✅
`jmx-metrics`	Validate JMX metrics definition files and config	✅
`labeler`	Validate PR labeler config matches integration directories	✅
`legacy-signature`	Validate no integration uses the legacy Agent check signature	✅
`license-headers`	Validate Python files have proper license headers	✅
`licenses`	Validate third-party license attribution list	✅
`metadata`	Validate metadata.csv metric definitions	✅
`models`	Validate configuration data models match spec.yaml	✅
`openmetrics`	Validate OpenMetrics integrations disable the metric limit	✅
`package`	Validate Python package metadata and naming	✅
`readmes`	Validate README files have required sections	✅
`saved-views`	Validate saved view JSON file structure and fields	✅
`version`	Validate version consistency between package and changelog	✅

View full run

…#23531) * Revert "[AI-6669] NiFi - Add log pipeline and log integration assets (#23302)" This reverts commit 2dd1b78. * Add changelog entry for NiFi log pipeline revert * Revert "Add changelog entry for NiFi log pipeline revert" This reverts commit 3643b91. fccf488

…DataDog#23531) * Revert "[AI-6669] NiFi - Add log pipeline and log integration assets (DataDog#23302)" This reverts commit 2dd1b78. * Add changelog entry for NiFi log pipeline revert * Revert "Add changelog entry for NiFi log pipeline revert" This reverts commit 3643b91.

philjlee requested review from a team as code owners April 13, 2026 20:51

temporal-github-worker-1 Bot added agent/review-requested docs/review-requested ecosystems/review-requested product/review-requested labels Apr 13, 2026

dd-octo-sts Bot added documentation changelog/no-changelog dev/testing dev/tooling release integration/nifi team/agent-integrations team/documentation team/logs-integrations-reviewers team/saas-integrations labels Apr 13, 2026

chatgpt-codex-connector Bot reviewed Apr 13, 2026

View reviewed changes

janine-c previously approved these changes Apr 14, 2026

View reviewed changes

temporal-github-worker-1 Bot added docs/approved and removed docs/review-requested labels Apr 14, 2026

philjlee force-pushed the philip.lee/AI-6669 branch from a263bc6 to 0630394 Compare April 20, 2026 18:00

temporal-github-worker-1 Bot removed the agent/review-requested label Apr 20, 2026

dd-octo-sts Bot removed the changelog/no-changelog label Apr 20, 2026

dd-octo-sts Bot removed dev/testing dev/tooling release labels Apr 20, 2026

Update dependency resolution

6739774

datadog-agent-integrations-bot Bot requested a review from a team as a code owner April 20, 2026 19:01

dd-octo-sts Bot added the team/agent-build label Apr 20, 2026

temporal-github-worker-1 Bot added docs/review-requested and removed docs/approved labels Apr 20, 2026

aiuto approved these changes Apr 23, 2026

View reviewed changes

philjlee added 2 commits April 28, 2026 14:06

Merge origin/master into philip.lee/AI-6669

3f53185

Resolve dependency lockfile conflicts by taking master's versions.

gh-worker-ownership-write-b05516 Bot removed the request for review from a team April 28, 2026 18:07

Remove dashboard, manifest, and images from repo

2ada410

These assets belong in Developer Platform, not integrations-core.

steveny91 approved these changes Apr 28, 2026

View reviewed changes

temporal-github-worker-1 Bot added the agent/approved label Apr 28, 2026

philjlee added this pull request to the merge queue Apr 28, 2026

Merged via the queue into master with commit 2dd1b78 Apr 28, 2026
43 of 44 checks passed

philjlee deleted the philip.lee/AI-6669 branch April 28, 2026 18:20

dd-octo-sts Bot added this to the 7.79.0 milestone Apr 28, 2026

philjlee mentioned this pull request Apr 29, 2026

Revert "[AI-6669] NiFi - Add log pipeline and log integration assets" #23531

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AI-6669] NiFi - Add log pipeline and log integration assets#23302

[AI-6669] NiFi - Add log pipeline and log integration assets#23302
philjlee merged 6 commits intomasterfrom
philip.lee/AI-6669

philjlee commented Apr 13, 2026 •

edited by atlassian Bot

Loading

Uh oh!

codecov Bot commented Apr 13, 2026 •

edited

Loading

Uh oh!

datadog-prod-us1-5 Bot commented Apr 13, 2026 •

edited by datadog-datadog-prod-us1 Bot

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Apr 13, 2026

Uh oh!

cit-pr-commenter-54b7da Bot commented Apr 20, 2026 •

edited

Loading

Uh oh!

dd-octo-sts Bot commented Apr 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		if self.instance.get('username') and self.instance.get('password'):
		self.http.options['auth'] = None

Conversation

philjlee commented Apr 13, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

Summary

Test plan

Uh oh!

codecov Bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

datadog-prod-us1-5 Bot commented Apr 13, 2026 • edited by datadog-datadog-prod-us1 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

cit-pr-commenter-54b7da Bot commented Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Monitor Template Quality Assessment

Uh oh!

dd-octo-sts Bot commented Apr 28, 2026

Validation Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

philjlee commented Apr 13, 2026 •

edited by atlassian Bot

Loading

codecov Bot commented Apr 13, 2026 •

edited

Loading

datadog-prod-us1-5 Bot commented Apr 13, 2026 •

edited by datadog-datadog-prod-us1 Bot

Loading

cit-pr-commenter-54b7da Bot commented Apr 20, 2026 •

edited

Loading