Prompt #2: CloudWatch Agent Setup for a Java App Load Test - in One Prompt

Hello All,

Today's prompt comes straight from a real performance-testing session. We had a load test planned against a legacy Java application (JBoss with a Tanuki wrapper, running on Windows Server on EC2), and the default CloudWatch metrics were nowhere near enough — 5-minute CPU averages tell you nothing when your app falls over in 3 minutes. 🤦‍♂️

So instead of hand-crafting a CloudWatch agent config (that JSON gets big, fast), I handed my AI agent one prompt and let it do the heavy lifting: agent config, log shipping, SSM Parameter Store, dashboard — the works.

Sounds interesting? Let's dive in.

---

What Does This Prompt Do?

It turns your LLM agent (Claude Code, Cursor, anything with AWS CLI access) into a performance-test observability engineer. In one session it will:

• Enable EC2 detailed monitoring (spoiler: it's probably disabled on your instance)

• Build a full CloudWatch agent config for Windows with a 10-second collection interval — CPU per-core, memory, disk IOPS, TCP connections, context switches, and Java process-level counters (working set, private bytes, threads, handles)

• Ship your app server log, GC log, and Windows event logs to CloudWatch Logs

• Store the config in SSM Parameter Store and apply it via SSM Run Command — no RDP needed

• Build a CloudWatch dashboard organised by failure domain: Overview, Java process, Disk I/O, TCP, System contention

The goal: when the load test ends, you should be able to answer "was it resources or was it software?" in five minutes, not five days.

---

When Would You Use This?

• A performance/load test is coming up and you need deep visibility before it starts

• You're running Java on Windows EC2 (though the prompt adapts easily to Linux)

• Your app "crashes" under load and nobody knows if it's CPU, memory, heap, or something else entirely

---

The Prompt 📋

You are my AWS performance-test observability engineer. I'm about to run a
load test against a Java application (JBoss/Tomcat-style app server) running
on a Windows EC2 instance, and I need deep metrics in place before the test.

First, ask me for: instance id, region, AWS profile, the app server home
directory, and the paths to the app server log and GC log.

Use the AWS CLI, and SSM Run Command for anything that runs on the instance.
Show me every command before you execute it, and wait for my confirmation
on anything that changes state.

Tasks:
1. Enable EC2 detailed monitoring on the instance (it is usually disabled).
2. Build a CloudWatch agent config for Windows with a 10-second collection
   interval under a custom namespace "JavaPerfTest", collecting:
   - CPU: total, user, kernel, and per-core
   - Memory: % used, available MB, committed bytes, cache, pages/sec
   - Java process (java.exe): % processor time, working set, private bytes,
     virtual bytes, thread count, handle count, I/O bytes/sec
   - Disk: read/write throughput, IOPS, and queue length per volume
   - Network: bytes and packets in/out, errors
   - TCP: established/active/passive connections, connection failures,
     resets, retransmitted segments
   - System: context switches/sec, processor queue length, total threads
3. Ship logs to CloudWatch Logs under a /perftest/ prefix: the app server
   log, the GC log, and Windows Application + System event logs filtered
   to errors and warnings.
4. Store the agent config in SSM Parameter Store and apply it with the
   AmazonCloudWatch-ManageAgent SSM document.
5. Verify metrics are actually flowing into the custom namespace, then
   build a CloudWatch dashboard with sections: Overview (CPU/memory/network),
   Java Process, Disk I/O, TCP Connections, and System Contention.

After the load test, help me correlate the data. Answer in this order:
- Resources first: did CPU, memory, or JVM heap hit a ceiling?
- If resources look fine, go hunting in software: thread pools, blocked
  threads, connector limits, and upstream dependency failures in the logs.

---

Why This Matters: A War Story ⚔️

Here's what happened when we actually ran the 100-user test. The app went completely unresponsive — everyone's first reaction was "it crashed, the instance must be too small."

But the dashboard told a different story:

Resource	Peak During Test	Verdict
CPU	~74% max	✅ Fine
Memory	~23% committed	✅ Oversized, actually
JVM Heap	~2.7 GB used of 15 GB	✅ Plenty of room

The instance was innocent. The logs we shipped to CloudWatch had the real answer:

Maximum number of threads (200) created for connector — all AJP connector threads exhausted.

The cause chain: an upstream BI service started dropping SSL connections under load → logout threads blocked waiting on those failed handshakes (no timeout configured) → 100 users logging out at once ate all 200 AJP threads → thread starvation → app appears dead while the JVM is perfectly healthy.

Without process-level metrics and shipped logs, we'd have thrown money at a bigger instance and hit the exact same wall. Instead, the fix list was: add HTTP timeouts to the upstream calls, make logout async, and bump maxThreads for headroom.

---

The Catch (Gotchas to Keep in Mind)

• 10-second custom metrics cost real money. This config is for a test window, not forever. Dial the interval back (or remove the config) once the performance activity is done.

• Detailed monitoring is off by default. Basic 5-minute EC2 metrics will hide short spikes completely — enable 1-minute detailed monitoring first.

• Old JVMs need process counters, not just heap dashboards. On legacy stacks (this one was JDK 1.7!), Windows performance counters for java.exe are sometimes your only reliable window into the process.

• Ship the GC log. It's the cheapest flight recorder you'll ever get, and correlating GC pauses with response-time spikes is gold.

---

Key Takeaways

• Set up observability before the load test, not after the post-mortem starts

• Collect at the process level, not just the host level — the JVM can be healthy while the app is dead

• "Crashed under load" is a symptom, not a diagnosis — resources first, then thread pools, then upstream dependencies

Paste the prompt, answer the five questions it asks, and walk into your load test with eyes wide open. 😎

More prompts coming — keep an eye on the Prompts tab. Happy prompting!