Dynatrace Integration with AI Agents: Automating Problem Remediation

Dynatrace solved detection years ago, and Davis AI got diagnosis most of the way there: a problem card arrives with a root-cause candidate and an impact analysis already attached. What remains stubbornly manual is the last mile. Someone still reads the problem, confirms the candidate, checks what changed outside Dynatrace's view, and files the ticket that eventually becomes a fix. Your time-to-detect is measured in seconds; your time-to-remediate is measured in whoever is on call.

This final part of the series is about closing that gap with a Dynatrace integration that treats Davis problems — not raw alerts — as the unit of work. Part one covered why the Problems feed and DQL are the right surfaces to build on. Part two pushed the native tools to their ceiling: problem notifications, the Problems API v2, and AutomationEngine workflows that execute predefined steps well but investigate nothing.

Here we add the layer that investigates: CloudThinker agents that subscribe to your Problems feed, enrich each problem with DQL context and with the things Dynatrace cannot see, validate the Davis root-cause candidate, and — under autonomy levels you control — remediate and close the problem with an evidence trail. To position this precisely: it is not a Dynatrace replacement. OneAgent keeps collecting, Davis keeps diagnosing. CloudThinker is the action layer on top.

Connection: one scoped token, read-only, about five minutes

CloudThinker connects to Dynatrace with an access token you create yourself, so the permission boundary is visible before anything runs. The read-only baseline:

• problems.read — the Problems feed and per-problem detail, including the Davis root-cause candidate
• entities.read — the Smartscape topology behind each affected entity
• metrics.read, events.read — metric context and the event stream (deployments, config change events)
• Grail query scopes (storage:logs:read, storage:events:read, storage:buckets:read) if you want the agents running DQL against logs and events

That is the whole grant at connection time. problems.write — needed later for posting comments and closing problems — is a separate, deliberate step, not part of the default. You generate the token, paste it with your environment URL into CloudThinker, and the first pass over your open problems starts. Budget about five minutes; the Dynatrace connection guide lists the exact scopes so your security review reads the same list you do.

What happens when a problem opens

Agents watch the Problems feed continuously — the same GET /api/v2/problems surface you polled by hand in part two. When a new problem opens, the sequence looks like what a good on-call engineer does, compressed from half an hour into two or three minutes:

1. Take the problem, not the symptoms. The agent pulls the full problem record: affected entities, impact analysis, and the Davis root-cause candidate. This matters — building on problems instead of raw metric alerts means the deduplication and causal correlation Davis already did is preserved, not redone worse.

2. Enrich with DQL. For the affected entities and the problem window, the agent queries Grail for the evidence a human would open next — error logs, event bursts, related failure rates:

fetch logs
| filter dt.entity.service == "SERVICE-9A3F2C7D41B8E650"
| filter loglevel == "ERROR"
| summarize errors = count(), by: { bin(timestamp, 5m) }
| sort timestamp desc

3. Add what Dynatrace cannot see. This is the step no native workflow can do, because it leaves Dynatrace entirely. If you have also connected your cloud accounts and repositories, the agent checks the surrounding state: the merge that deployed twelve minutes before the problem opened, the security group edit, the autoscaling event, the database parameter change, the quota that quietly hit its ceiling. Davis correlates brilliantly within its own telemetry; most real root causes involve at least one fact outside it.

4. Validate the Davis candidate. The agent's job is not to out-diagnose Davis — it is to confirm or challenge the candidate with independent evidence. "Davis points at the billing-api process; the error logs, the 14:02 deploy event, and the connection-pool saturation metric agree" is a different artifact from a forwarded notification. When the evidence disagrees, the agent says so and shows both sides.

5. Propose the fix. The output is a finding in your chat channel: what opened, what the evidence shows, whether the root-cause candidate held up, and a specific remediation with rollback notes — waiting at whatever autonomy level you configured.

Autonomy is graduated, and escalation survives it

Every action class carries a per-environment autonomy level:

• Notify — investigate and report, nothing more. The default for everything.
• Suggest — propose the specific remediation, with impact and rollback, and wait.
• Approve — stage the action; execute only after a named human approves in chat.
• Autonomous — execute and report. Earned, per action class, for reversible and well-understood fixes — restarting a crash-looping non-prod pod, say — after the agent has been consistently right at Approve.

The part that matters for a Dynatrace shop: your escalation path is untouched. If the investigation concludes this is a genuine customer-facing outage, the page goes out exactly as it does today — it just arrives with the investigation already attached instead of a bare problem link. The agent never decides an incident is not worth waking someone for.

Closing the problem with evidence, not silence

Part two showed the mechanics: POST /api/v2/problems/{problemId}/close accepts a closing message. Native workflows can call it, but they cannot write anything worth reading. Agents close the loop properly — once a remediation is applied and verified, the problem gets closed with a comment that doubles as the incident record:

curl -X POST \
  "https://{env-id}.live.dynatrace.com/api/v2/problems/{problemId}/close" \
  -H "Authorization: Api-Token $DT_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"message": "Root cause confirmed: connection pool exhaustion in billing-api after 14:02 deploy. Pool max raised 20 to 50 (change approved by j.nguyen). Error rate returned to baseline within 4 minutes. Evidence and timeline in CloudThinker audit log."}'

Six months later, when the same problem pattern reappears, the history is in Dynatrace where the next engineer will actually find it — not in a Slack thread that scrolled away. Every investigation, proposal, approval, and executed change also lands in CloudThinker's own audit trail with its evidence.

What the first week typically surfaces

Beyond live problem response, the first connection reviews your problem history. Numbers are illustrative — a composite mid-market environment, a few hundred hosts, a few dozen services. Yours will differ.

Notice the shape: only one row is a new detection. The rest are your existing Dynatrace signal being ignored, delayed, or drowned in predictable noise — which is exactly the argument that the bottleneck was never detection.

Prompts worth trying first

CloudThinker is conversational; you ask in plain language and the agent cites the underlying Dynatrace data so you can verify rather than trust:

"A response-time degradation problem just opened on checkout-service. Pull the Davis root-cause candidate, run a DQL error-log query for the affected window, check for deploys or cloud-side changes in the last two hours, and post a summary. Notify only."

"Which problems recurred more than five times in the last 30 days without anyone acting on them? For each, tell me whether the right fix is a threshold change, a maintenance window, or an actual defect."

"Sample five problems Davis closed automatically last week and verify the root-cause candidate against logs and events. Flag any where the evidence disagrees."

Run the first one during your next real problem and compare it against what your on-call assembles by hand. That comparison, not this article, is the evaluation.

The boundaries, stated plainly

An agent with a token to your production observability platform should raise questions. The answers:

• Read-only by default. The baseline token cannot modify anything in Dynatrace or your infrastructure. Acting requires write scopes you grant explicitly, plus an autonomy level above Notify for that action class.
• No unapproved changes. Below Autonomous, every change waits for a named human. Autonomous is opt-in per action class and per environment, never a global switch.
• No swallowed escalations. Paging behavior is yours; the agent adds evidence to it, never subtracts urgency from it.
• No unauditable actions. Everything is logged with evidence, approver, and timestamp — in CloudThinker and, via the closing comment, in Dynatrace itself.

Detection was never the bottleneck

Across this series the pattern held: Dynatrace detects in seconds, Davis diagnoses in minutes, and remediation waits on a human queue measured in tens of minutes to days. Native notifications and workflows automate the predictable slice. Agents automate the rest — the investigation, the cross-system correlation, the approval-gated fix, the evidence-bearing close. Teams that put an action layer on an existing Dynatrace setup typically cut acknowledgment-to-remediation from the better part of an hour to minutes, with problem noise dropping as the recurring offenders finally get fixed instead of re-detected.

Try CloudThinker free — 100 premium credits, no card required — and follow the Dynatrace connection guide to see your own problem history reviewed within the hour.