Sign inBook a Demo

How To

Dynatrace Automation with Native Tools: The DIY Integration Guide

The most common Dynatrace integration stops at a Slack webhook — this guide builds the rest with native tools only. Part two of our Dynatrace series covers problem notifications with full JSON payload templating, the Problems API v2 (list, enrich, comment, close with evidence), DQL triage queries against Grail plus the programmatic query API, Workflows (AutomationEngine) for standard reactions, and alerting profiles and maintenance windows for noise control. Every call is copy-pasteable — and we close with the honest ceiling: workflows only execute steps you predicted, and investigation beyond Dynatrace's view stays manual.

ST
·
dynatraceobservabilityapmdqlautomationsre
Cover Image for Dynatrace Automation with Native Tools: The DIY Integration Guide

Dynatrace Automation with Native Tools: The DIY Integration Guide

The most common Dynatrace integration in production today is a problem-notification webhook pointed at Slack. A Davis problem opens, a message lands in #alerts, and the integration project is declared finished. Detection got dramatically better; response didn't change at all — an engineer still reads the message, opens the problem, runs the same three queries, and decides what to do.

Dynatrace ships considerably more automation surface than that webhook. This guide is the DIY build-out: notifications with real payloads, the Problems API v2, DQL for triage context, Workflows for standard reactions, and alerting profiles plus maintenance windows for noise control. All native, nothing to buy. In part one we argued the Problems feed — not raw alerts — is the right surface to automate against; everything below builds on that.

We'll go tool by tool, then finish with an honest accounting of where the ceiling is.

Tool 1: Problem notifications — push problems out with full context

Don't poll for problems when Dynatrace will push them. Under Settings → Integration → Problem notifications → Add notification, the Custom integration type gives you a webhook with a templated JSON payload (docs).

The default Slack-style payload is where most teams stop. Instead, send your receiving endpoint something it can act on:

{
  "problemId": "{ProblemID}",
  "pid": "{PID}",
  "title": "{ProblemTitle}",
  "state": "{State}",
  "severity": "{ProblemSeverity}",
  "impact": "{ProblemImpact}",
  "tags": "{Tags}",
  "impactedEntities": {ImpactedEntities},
  "url": "{ProblemURL}",
  "details": {ProblemDetailsJSONv2}
}

How to read it. {PID} is the ID you'll use against the Problems API; {State} fires as OPEN and again as RESOLVED, so your receiver must be idempotent and handle both. {ProblemDetailsJSONv2} embeds the full problem object — evidence, impacted entities, root-cause candidate — so a downstream script often doesn't need a second API call at all. Bind the notification to an alerting profile (Tool 5) or you'll forward every problem in the environment, including the ones you deliberately deprioritized.

Tool 2: Problems API v2 — list, enrich, comment, close

For anything the webhook can't carry, the Problems API v2 is the workhorse. Create an access token with the problems.read and problems.write scopes (Access tokens app), and export your environment ID:

export DT_ENV="abc12345"        # your environment ID
export DT_TOKEN="dt0c01.XXXX..."

List everything currently open:

curl -s "https://$DT_ENV.live.dynatrace.com/api/v2/problems?problemSelector=status(\"open\")&pageSize=100" \
  -H "Authorization: Api-Token $DT_TOKEN"

Pull one problem with the fields that matter for triage:

curl -s "https://$DT_ENV.live.dynatrace.com/api/v2/problems/$PROBLEM_ID?fields=evidenceDetails,impactAnalysis,recentComments" \
  -H "Authorization: Api-Token $DT_TOKEN"

How to read it. rootCauseEntity is Davis's root-cause candidate — the thing to verify first, not blindly trust. evidenceDetails lists the events and metric anomalies Davis correlated into this problem; impactAnalysis estimates affected users per impacted entity, which is your severity tiebreaker when two problems are open at once.

Comment your findings back onto the problem so the timeline lives in Dynatrace, not in a Slack thread:

curl -s -X POST "https://$DT_ENV.live.dynatrace.com/api/v2/problems/$PROBLEM_ID/comments" \
  -H "Authorization: Api-Token $DT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"message": "Error spike correlates with deploy 2026-07-12.3 on checkout-service.", "context": "triage-bot"}'

And close with an evidence comment when it's done:

curl -s -X POST "https://$DT_ENV.live.dynatrace.com/api/v2/problems/$PROBLEM_ID/close" \
  -H "Authorization: Api-Token $DT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"message": "Rolled back to 2026-07-12.2; error rate at baseline for 30 minutes."}'

A problem closed with a one-line cause is worth more than a dashboard of green — it's the dataset your next retro actually uses.

Tool 3: DQL — the triage queries you run every time

Every incident triage starts with the same questions: what's in the logs, when did it start, does it match a deploy. DQL against Grail answers them in one place — run interactively in a Notebook, or programmatically.

Error logs for the impacted service, bucketed to spot the inflection point:

fetch logs, from: -30m
| filter dt.entity.service == "SERVICE-6C8C6A2E12D2C0A1"
| filter loglevel == "ERROR"
| summarize errors = count(), by: { bin(timestamp, 5m), log.source }
| sort errors desc

Problems themselves are queryable in Grail too, which turns "what's been noisy this week" into a query instead of console archaeology:

fetch dt.davis.problems, from: -7d
| filter event.status == "ACTIVE"
| fields display_id, event.name, event.category, root_cause_entity_name

To run DQL from a script, use the Grail query API with a platform token (scope storage:logs:read for log queries):

curl -s -X POST "https://$DT_ENV.apps.dynatrace.com/platform/storage/query/v1/query:execute" \
  -H "Authorization: Bearer $DT_PLATFORM_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query": "fetch logs, from: -30m | filter loglevel == \"ERROR\" | summarize errors = count(), by: { log.source } | sort errors desc"}'

How to read it. The query API is asynchronous by design — a long-running query returns a request token you poll for results, so script accordingly. And note the entity ID in the first query: your automation gets it from the webhook payload or the Problems API, which is exactly why Tools 1–3 compose.

Tool 4: Workflows — automate the standard reactions

Dynatrace Workflows (the AutomationEngine) closes the loop for reactions you can define in advance. A workflow starts from a Davis problem trigger — filtered by problem category, tags, or affected entity type — and runs a graph of actions: Execute DQL Query, HTTP Request, Run JavaScript, Slack, Jira, ServiceNow.

A triage workflow worth building this week:

  1. Trigger: Davis problem trigger, filtered to problems tagged team:payments in the ERROR category.
  2. Execute DQL Query: the error-log query from Tool 3, with the affected entity injected from the trigger event.
  3. Run JavaScript: shape the query result into a five-line summary.
  4. Slack: post the summary — not just the problem title — to the team channel.
  5. HTTP Request: POST the same summary to /api/v2/problems/{id}/comments so the enrichment is on the problem record.

That single workflow removes the first ten minutes of every incident for that team: the on-call opens Slack and sees what's failing and since when, not just that something is failing. Build one per problem category, not one grand workflow with twenty branches — filters are cheap, debugging a mega-workflow is not.

Tool 5: Alerting profiles and maintenance windows — cut the noise first

Automation amplifies whatever you feed it, so tune the feed. Two native controls:

Alerting profiles (Settings → Alerting → Problem alerting profiles) decide which problems reach which notification. The underused feature is the delay: only forward a Slowdown problem if it's still open after 15–30 minutes, which silently drops the self-healing blips that make on-call hate the pager. Combine severity rules with tag filters so the payments profile never pages the data team.

Maintenance windows (Settings → General → Maintenance windows, or the Settings API schema builtin:alerting.maintenance-window) suppress alerting — optionally detection — during planned work. Create them from your deploy pipeline via the Settings API rather than by hand, or the 2 a.m. migration will page exactly the person who scheduled it.

The ceiling: what native automation won't do

Run everything above and you'll have a genuinely good setup — better than most. You should also be clear about its structural limits:

Workflows execute steps you predicted. Every workflow encodes a failure mode you already imagined, with a fixed response. The incident that matters is usually the one you didn't enumerate — and for it, your automation posts context while a human starts from scratch. Coverage grows one hand-built workflow at a time, and each one is code you now maintain.

Investigation stops at Dynatrace's horizon. Davis correlates what Dynatrace observes. The root cause frequently lives outside that view: an IAM policy change, a cloud provider capacity event, a config flag flipped an hour earlier, a database parameter nobody instrumented. Cross-checking those means a person with five other consoles open.

Judgment is still manual. Nothing here validates the root-cause candidate, chooses between rollback and scale-up, or decides a problem is safe to close. The webhook moved the notification; the human still moves the incident.

What comes next

The gap left over is investigation and action — and that's what an agent layer is for. CloudThinker agents connect to Dynatrace read-only, subscribe to the Problems feed, enrich each problem with DQL plus the cloud-side context Dynatrace can't see, validate the Davis root-cause candidate, and propose or apply fixes under autonomy levels you control. That's part three of this series.

Or see it against your own environment now: Try CloudThinker free — 100 premium credits, no card required.