Fast detection and rollback: a safer alternative to full regression testing

Oracle and the OpenJDK update projects ship scheduled security updates four times a year, in January, April, July and October, usually around the third Tuesday of the month. Many enterprises need most of a quarter to get a JVM update through regression testing across their applications. Some never finish before the next one arrives.

For business-critical systems, thorough testing is worth the time. For the long tail of internal tools, batch jobs and low-traffic services, it usually isn't. For that tier there's a better approach than either testing everything or skipping tests.

The argument

Think about the expected cost of a bad patch reaching production:

expected cost ≈ probability of a problem × impact per minute × minutes until it's fixed

Full regression testing attacks the first term, making a problem less likely to ship. Fast detection and fast rollback attack the last term, making any problem that does ship short-lived.

For a same-major JVM security update, such as moving between two 17.0.x releases, the chance of a functional regression is already low. For a low-criticality application the impact per minute is also low. If you can detect a problem in minutes and reverse it in minutes, the expected cost of skipping a multi-week regression cycle is small, and usually much smaller than leaving known CVEs open for another quarter.

That's the core claim: when mean time to detect (MTTD) and mean time to recover (MTTR) are low, the value of full pre-deployment regression testing drops sharply. It holds only if both numbers really are low, so most of the work goes into making them low.

Progressive rollout

Don't patch everything at once. Roll out in stages, with automated checks between each one:

1. Canary (about 1%): one instance or a small slice of traffic. Hold long enough to see steady-state behavior, not just startup
2. Partial (about 10%): enough traffic that low-frequency errors show up
3. Full (100%): complete the rollout, then keep watching through a soak period

Each step is gated. If a gate fails, the rollout stops and rolls back automatically, with no meeting required.

For small applications running on one or two instances, "1%" becomes "one instance first, then the rest." The principle is the same: expose the change to real load in a limited way before exposing all of it.

What to watch on a JVM change

Generic health checks, like "is the process up?", miss most JVM-specific regressions. Compare the patched instances against the unpatched baseline on these signals:

• Error rate: HTTP 5xx, exception counts, failed jobs
• Latency: p50 and p99, since JVM changes often affect the tail first
• GC behavior: pause time, pause frequency and collection counts
• Memory: heap after GC, metaspace and native memory where you can measure it
• Threads: thread counts and blocked threads
• Warm-up: time to reach steady-state latency after start
• TLS: handshake failures to downstream systems, which point to protocol or cipher changes
• Agent health: whether APM and security agents attached and are reporting

Example gates might look like this. The thresholds are illustrative only and should be tuned to each application's normal variance:

Comparing against a live unpatched baseline, rather than fixed numbers, removes most false alarms caused by normal traffic swings.

Smoke tests and synthetic transactions

Replace the full regression suite with a small set of checks that exercise what matters:

• Smoke tests run right after each instance starts: does the app start, connect to its database and queue, and serve its health endpoint?
• Synthetic transactions run continuously: scripted logins, searches, report generations, or whatever the app's two or three core paths are
• Integration probes hit each external dependency, which is especially useful for catching TLS problems

These run in minutes, can be kept current easily, and exercise the code paths a JVM change is most likely to disturb.

Making rollback fast

Fast rollback is often the actual gap. In many enterprise estates, particularly outside containers, reverting a JVM means a manual change, a ticket and a wait. Fix that first. Without it, none of the above is safe.

In containerized environments

• Treat images as immutable and version them. Rolling back means redeploying the previous image tag
• Keep the previous image available and tested until the soak period ends
• Have the deployment tooling perform the rollback automatically when a gate fails

On VMs and bare metal

• Install side by side. Put the new JDK in its own versioned directory and switch the application to it using a symlink, environment variable or service configuration. Don't overwrite the old one
• Don't uninstall the previous runtime until the soak period has passed
• Script the switch in both directions, and make sure the application restarts cleanly with either runtime
• Rehearse it. A rollback nobody has run is a hypothesis, not a procedure

# Illustrative side-by-side layout on a Linux host
/opt/java/jdk-17.0.15/
/opt/java/jdk-17.0.16/
/opt/java/current -> /opt/java/jdk-17.0.16   # rollback = repoint and restart

Where this doesn't apply

• Business-critical applications: keep rigorous testing and make it faster with risk-based test selection and shadow traffic. See Critical Patch Assurance
• Major version upgrades: removed APIs and changed defaults need targeted assessment before any rollout
• Apps without observability: if you can't see errors and latency, you can't gate on them

Getting it approved

The technical case is only half the work. A Change Advisory Board also has to accept the process. Our guide to getting faster JVM patching past the CAB covers classification, the standard change model and the evidence to keep.

What to do next

Rapid Patch is our service built on this model, including fixing rollback where it's missing. The full method is described in our approach.