Automating AppDynamics Health Rule Responses with Native Tools

Alert & Respond gets you further than most teams realize — right up to the point where the remediation script attached to your restart policy becomes its own unmaintained service, running on a node someone rebuilt in March, doing something nobody remembers approving. Before you get there, it's worth knowing exactly how far the native stack goes.

In part one of this series we covered why AppDynamics health rules generate so much noise out of the box: untuned default baselines, violation storms during deploys, and node-level granularity mistakes. This article is the fix — tuning health rules and wiring up automated responses using only what ships with the Controller: baselines, schedules, action suppression, policies, actions, HTTP request templates, and the Events API. Exact steps, real API calls, nothing to buy.

We'll work in layers, from "fire less often" to "do something when you fire."

Layer 1: Tune the health rules themselves

Most violation noise is a configuration problem, not an alerting-tool problem. Open Alert & Respond → Health Rules, select your application, and edit the rules doing the most damage (part one's triage list). Three settings matter most (health rules docs).

Point baseline conditions at a seasonal baseline

A condition like "average response time is greater than 3 standard deviations of the default baseline" compares against a baseline that treats all history equally. If your traffic has weekly shape — Monday-morning login spikes, Friday batch jobs — that comparison is wrong twice a week, and the rule fires on schedule.

Fix: create a seasonal baseline under Configuration → Baselines → Create Baseline, choose Weekly Trend (or Daily, if your pattern repeats every 24 hours) with a rolling time range, then edit the health rule condition to evaluate against that baseline instead of the default. Now Monday 9 a.m. is compared with previous Mondays at 9 a.m., not with the whole week's average.

Widen the evaluation window on flappy rules

Each health rule evaluates on a cadence using a trailing data window — the default is evaluating with the last 30 minutes of data. For a rule that flaps (violates, resolves, violates again within the hour), the cheapest fix is lengthening this window in the rule's evaluation settings so a two-minute latency blip can't drag the whole window over the threshold. For rules protecting genuinely fast-moving SLOs, shorten it instead and accept more sensitivity — just do it deliberately, per rule.

Add a load floor and respect warm-up

Baselines are unreliable at low load: 3 errors in 10 calls is a 30% error rate. Add a second condition ANDed to the primary one — calls per minute greater than 50 is the pattern AppDynamics' own default rules use — so the rule can only violate when there's enough traffic to mean anything.

For node-level rules, JVM warm-up after a deploy or restart looks exactly like a regression: cold caches, JIT compilation, connection pools filling. Configure node health rules to skip recently started nodes (in the rule's affected-entities criteria you can restrict evaluation to nodes that have been reporting for a minimum time), or handle deploy windows with suppression — next layer.

Layer 2: Schedules and action suppression — kill the known windows

Two different tools, often confused:

Health rule schedules stop evaluation during windows where the rule is meaningless. Create one under Alert & Respond → Schedules (e.g., "Business hours: Mon–Fri 08:00–20:00", in the application's time zone), then attach it in the health rule's settings. The nightly-batch CPU rule stops evaluating at night; the checkout-latency rule only runs when checkout traffic exists. Violations are never recorded, so dashboards stay clean too.

Action suppression stops actions while still recording violations — right for maintenance and deploy windows where you want the history but not the pages. Configure under Alert & Respond → Actions → Action Suppression: name it, set the time window, scope it to specific applications, tiers, or nodes, and optionally disable agent reporting entirely for the window.

The part most teams miss: action suppression has an API, which means your CI pipeline can open a suppression window as a deploy step instead of a human remembering to click one:

curl -s -X POST \
  "https://<controller-host>/controller/alerting/rest/v1/applications/<application-id>/action-suppressions" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "deploy-checkout-service",
    "disableAgentReporting": false,
    "suppressionScheduleType": "ONE_TIME",
    "timezone": "Australia/Sydney",
    "startTime": "2026-07-13T02:00:00",
    "endTime": "2026-07-13T02:30:00",
    "affects": { "affectedInfoType": "APPLICATION" }
  }'

Thirty minutes of suppression scoped to the service being deployed eliminates the violation storm from part one without blinding you to everything else.

Layer 3: Policies and actions — the built-in reaction layer

A policy binds trigger events to actions: Alert & Respond → Policies → Create Policy, select the triggering events (typically Health Rule Violation Started – Critical and – Warning, plus Upgraded), filter to specific health rules — a policy scoped to "any violation anywhere" recreates the noise you just removed — then attach actions (policies docs).

Actions come in escalating strength:

Notifications. Email and SMS to individuals or groups. Fine as a floor; if this is your only action type, AppDynamics is a paging system with extra steps.

Diagnostic actions. The underused ones. Start a diagnostic session collects full transaction snapshots — with call graphs — for the affected business transactions for the next N minutes, so the evidence exists by the time a human looks. Take a thread dump captures N dumps at an interval from the affected nodes. Attach a diagnostic session to every latency-related critical policy; it costs nothing when nothing is wrong.

Remediation actions. Run a script or executable on problem nodes executes a script through the Machine Agent on the violating node. The script must already exist on that node under the Machine Agent's local-scripts directory:

# On each app node, alongside the Machine Agent
cat > /opt/appdynamics/machine-agent/local-scripts/recycle-app-pool.sh <<'EOF'
#!/bin/bash
systemctl restart checkout-service
echo "checkout-service restarted at $(date -u)" >> /var/log/appd-remediation.log
EOF
chmod +x /opt/appdynamics/machine-agent/local-scripts/recycle-app-pool.sh

In the action definition you reference the script path, set a timeout, and — use this — check require approval before execution, which emails a designated approver a link to authorize the run. Script output is captured as an event on the Controller, so there's an audit trail (remediation docs).

Remediation scripts are where DIY automation peaks and where it starts to rot. The script runs on the node, maintained by whoever wrote it, versioned nowhere, tested when it was written. Keep them to dumb, idempotent moves — restart a service, clear a temp directory — and keep them in your config management, not hand-edited on hosts.

Layer 4: HTTP request actions — webhooks with templating

To push violations into Slack, PagerDuty, Jira, or your own endpoint, create an HTTP Request Template (Alert & Respond → HTTP Request Templates), then an action that uses it. Templates support Velocity variables resolved from the triggering event (HTTP request templates docs):

{
  "summary": "${latestEvent.displayName}",
  "severity": "${latestEvent.severity}",
  "application": "${latestEvent.application.name}",
  "message": "${latestEvent.eventMessage}",
  "controller_link": "${latestEvent.deepLink}"
}

Set the method to POST, add auth headers, set the payload MIME type to application/json, and use the built-in Test button with a real past event before trusting it. The deepLink variable matters most in practice — it's the difference between a Slack message someone reads and one someone clicks.

Layer 5: The Events API — pull instead of push

Everything above is push. For dashboards, dedup logic, or feeding violations into your own tooling, poll the Controller. Create an API Client under Settings → Administration → API Clients, then exchange its secret for a token:

TOKEN=$(curl -s -X POST "https://<controller-host>/controller/api/oauth/access_token" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "grant_type=client_credentials&client_id=<client-name>@<account-name>&client_secret=<client-secret>" \
  | jq -r .access_token)

Open health-rule violations for an application over the last hour:

curl -s -H "Authorization: Bearer $TOKEN" \
  "https://<controller-host>/controller/rest/applications/<application-name>/problems/healthrule-violations?time-range-type=BEFORE_NOW&duration-in-mins=60&output=JSON"

Each violation includes the health rule name, severity, affected entity, start time, and a deep link. The broader Events API covers policy events too:

curl -s -H "Authorization: Bearer $TOKEN" \
  "https://<controller-host>/controller/rest/applications/<application-name>/events?time-range-type=BEFORE_NOW&duration-in-mins=1440&event-types=POLICY_OPEN_CRITICAL,POLICY_OPEN_WARNING,POLICY_CLOSE_CRITICAL,POLICY_CLOSE_WARNING&severities=WARN,ERROR&output=JSON"

And the alerting configuration itself is exportable — useful for diffing and version-controlling your rules:

curl -s -H "Authorization: Bearer $TOKEN" \
  "https://<controller-host>/controller/alerting/rest/v1/applications/<application-id>/health-rules" | jq .

Budget a day to do all five layers properly on one busy application. Done well, it cuts violation volume dramatically and means every critical page arrives with snapshots already collected.

The ceiling

Now the honest part. Everything above shares one shape: predefined trigger, predefined step. A policy can restart the service every time the error-rate rule fires — but it restarts the service whether the cause is a bad deploy, an exhausted connection pool, or a downstream API returning 500s. It cannot tell which.

The diagnostic sessions you wired up collect transaction snapshots with full call graphs — and then nobody reads them until an engineer opens the Controller, drills through snapshots, cross-references the release that went out 20 minutes earlier, and forms a hypothesis. The native stack automates reaction. Investigation — reading the evidence, correlating it with what changed, naming a cause — remains a human, per-incident, 30-to-90-minute job. And every remediation script is a small liability with a shelf life.

What comes next

The investigation step is what part three of this series automates: CloudThinker agents connect to AppDynamics read-only, pick up each health rule violation, read the transaction snapshots and error details a human would, correlate with recent releases and infrastructure state, and propose remediation with evidence — under approval gates you control. Try CloudThinker free — 100 premium credits, no card required.