From 20.04 to 24.04 Without Losing Sleep: A Real-World GCP VM Migration

A practitioner’s account of moving a production Docker host to a newer OS — what worked, what bit, and what we’d do again.

---

Why this had to happen

Ubuntu 20.04 LTS hit end of standard support in April 2025. After that, the only way to keep getting security updates was Canonical’s Extended Security Maintenance (ESM) program — a stopgap, not a strategy. Running an internet-facing host on an unsupported OS isn’t just risky; for anyone working under frameworks like CMMC, NIST 800-171, or SOC 2, it’s an unhealthy mark on your configuration management posture and a finding waiting to happen at your next assessment.

So we had a choice: in-place release upgrade, or rebuild fresh on 24.04 LTS.

We chose rebuild, and we’d choose it again. Here’s why.

Why rebuild, not do-release-upgrade

In-place upgrades preserve everything — including the cruft. Years of one-off package installs, abandoned PPAs, deprecated config files, orphaned Docker volumes from experiments that didn’t pan out. They also tend to leave behind subtle filesystem residue: directories owned by users that no longer exist, config files referencing tools that were removed three releases ago, kernel modules that linger in /lib/modules long past their useful life.

A clean rebuild on a new VM gives you:

• A reset to actual Ubuntu defaults, with only the packages you consciously chose to install
• A chance to right-size — bump RAM, swap to faster disk types, increase storage with no migration headache
• A safety net — the old VM keeps running until you’re confident in the new one
• Documented configuration — by definition, you can’t restore what you didn’t capture, so you end up with an explicit inventory of what was actually on the box
• A compliance win — for CMMC §3.4 (Configuration Management), a deliberately-built baseline beats an accreted one every time

The trade-off is more upfront work. But “more upfront work” trades cleanly against “less mystery later.” And this playbook is reusable — when 24.04 reaches EOL in 2034, future-you will thank present-you.

The high-level approach

We ran the old and new VMs in parallel for nearly a week. The only step with user-visible downtime was the final IP cutover — about 90 seconds while we moved a static external IP from one VM to the other.

Seven phases, in this order:

1. Discovery and backup on the old VM (zero impact)
2. Build the new VM with a temporary ephemeral IP (zero impact)
3. Provision the OS layer on the new VM (zero impact)
4. Restore data to the new VM and bring up the application stack (zero impact)
5. Cutover — swap the static IP from old to new (90 seconds of downtime)
6. Validation soak — 48-72 hours with the old VM stopped but kept (zero impact)
7. Decommission the old VM, retain snapshots for 30 days (zero impact)

Total active engineering time: about six hours spread across a few days. Total user-visible disruption: under two minutes.

What “discovery” actually means

This is the phase most people rush through, and it’s where 90% of migration failures originate.

A production host that’s been alive for years accumulates state in several layers, each of which needs to be captured separately:

The cloud layer. What machine type, disk size, disk type, network, firewall rules, service account scopes, and reserved IPs does the VM use? Is the external IP actually reserved, or is it ephemeral and at risk of being released the moment you delete the instance?

The OS layer. What did someone apt install over the years? Which third-party APT repos are configured? What’s installed via snap, pip, npm globally? What lives in /usr/local/bin or /opt? Are there custom systemd units, sysctl tweaks, cron jobs in /etc/cron.d/? What timezone, what locale, what network config?

The application layer. Where do all the Docker compose files actually live? (Spoiler: in a long-lived host, almost certainly not in one tidy directory.) What named volumes exist? What external networks are referenced? Where do .envfiles containing database passwords live?

The data layer. Database contents need a logical dump (e.g. mysqldump --single-transaction), not just a filesystem snapshot. Bind-mounted application data needs a tarball. User-uploaded files need to be accounted for.

The user layer. Which Linux users have real shells? What are their UIDs and GIDs? (Critical — see below.) What’s in their dotfiles? What’s in their ~/.ssh/authorized_keys?

Skipping any one of these turns the new VM into an unfinished house. You’ll notice the missing parts at the worst possible moment.

The lessons that hurt

Here are the issues that genuinely cost us time — the things that aren’t in any vendor’s getting-started guide but absolutely should be.

Service account scopes silently break GCS uploads. Older VMs were often created with devstorage.read_only as their default storage scope. That’s enough to read from GCS but not to write. So the moment you try to gsutil cp your backup tarballs to a staging bucket, you get cryptic AccessDeniedException: 403 Provided scope(s) are not authorized errors mid-upload — some files transfer, others fail, and you can’t tell what made it. Solution: run gcloud auth login on the old VM to use your user credentials, or set the new VM’s scope to cloud-platform from the start.

UID and GID parity matters more than you’d think. Bind-mounted files in tar archives carry their original UID. If your admin user is UID 1001 on the old VM and gets created as UID 1000 on the new one, every bind-mounted config file and data directory becomes invisible to the very containers that need to read them. Permission denied errors will haunt you until you fix this — and the fix is usermod -u plus a sweeping chown, which is much harder after the new user has started accumulating files.

Samba’s user database is not in /etc/samba/. This caught us cleanly. We backed up /etc/samba/ thinking we had the share definitions and the users in one go. We did not. Samba stores its user password database in /var/lib/samba/private/passdb.tdb. Without it, you have a Samba configuration with zero registered users — and CIFS error 13 (“Permission denied”) looks identical whether the user doesn’t exist, the password is wrong, or the share ACL denies access. The only way to tell is to journalctl -u smbd -f on the server side while a client attempts to mount and read the server’s rejection reason.

Tailscale IPs aren’t transferable. Each node gets a 100.x address bound to its identity. There’s no “move IP from old node to new node” operation. We hadn’t realized how many of our automation scripts had hardcoded the old Tailscale IP. The fix — and this is the right long-term answer regardless — is to use MagicDNS hostnames (vmname.your-tailnet.ts.net) so nothing depends on the IP.

Cloud Shell auth expires unpredictably. Cloud Shell looks like it should “just work” with gcloud since it’s already authenticated against your Google account. In practice, sessions time out, and certain commands that touch IAM-bound resources will fail with auth errors even though gcloud auth list shows you’re logged in. Re-run gcloud auth loginwhenever you get unexpected 403s in Cloud Shell.

Crontab editor errors from copy-paste. Pasting cron entries from a styled document (Slack, web pages, formatted notes) introduces invisible characters — smart quotes, non-breaking spaces, CRLF line endings — that the cron parser rejects with the famously unhelpful bad minute error. Diagnose with cat -A to see what your eyes can’t. Better yet, install cron entries via ( crontab -l; echo "<entry>" ) | crontab - rather than the interactive editor.

Nginx Proxy Manager’s admin UI is HTTP-only. Modern browsers aggressively assume HTTPS, and the “can’t establish a secure connection” error makes it look like a real TLS problem. It’s not — the NPM admin UI legitimately serves plain HTTP on port 8181. Use http://, not https://. (And consider locking the admin port behind Tailscale anyway.)

TLS SNI mismatch when testing by IP. Hitting a reverse proxy directly by IP with HTTPS triggers a tlsv1 unrecognized name error because there’s no Server Name Indication in the request, so the proxy can’t pick a certificate. This is actually a good sign — it means TLS is working and the proxy is correctly rejecting unknown hostnames. To test pre-cutover, use curl --resolve hostname:443:ip to send the proper SNI without DNS pointing the hostname at the new VM yet.

Long-lived Docker hosts accumulate orphan volumes. Look at docker volume ls on any server that’s been running for years and you’ll find volumes from old experiments, typos sitting next to the correct names, and data from apps you no longer run. Migration is the rare opportunity to leave the cruft behind. Build an explicit allowlist of volumes to migrate — don’t just restore everything from vol-*.tar.gz.

The cutover

Five lines of gcloud, in this order:

1. Stop the old VM
2. Detach the static IP from the old VM
3. Verify the IP shows as free
4. Detach the new VM’s temporary ephemeral IP
5. Attach the static IP to the new VM

Total time: under 90 seconds. The static IP doesn’t change, so DNS doesn’t need to update. Any existing client connection that was open during the swap drops once, retries, and hits the new VM. No DNS propagation, no TTL games.

One nuance worth flagging: access config names differ between old and new VMs. Console-created VMs from a few years ago often have an access config named External NAT (with a space, capitalized). VMs created via newer gcloudcommands have external-nat (kebab-case). Guess wrong and you get Invalid value for field. Always confirm with:gcloud compute instances describe $VM --zone=$ZONE \ --format="value(networkInterfaces[0].accessConfigs[0].name)"

The soak

This is the part most engineers underweight. After cutover, the old VM stays stopped (not deleted) for 48-72 hours. The point is to surface issues that don’t appear in basic smoke-testing — daily cron jobs firing at off-hours, certificate renewals, log rotation, backup runs, application sessions that span shifts.

Validation checklist for the soak:

• The site loads end-to-end (log in, do something real, log out)
• Database records match expected counts
• TLS certificates valid and auto-renewal still running
• All containers in expected state
• File shares mountable from external clients
• VPN/private-network connectivity working from peers
• Cron jobs fire at their scheduled times
• No new error entries in journalctl -p err
• Disk and memory within expected bounds
• Backup runs complete successfully overnight

If anything’s wrong, you have a one-command rollback: move the static IP back, start the old VM, and you’re serving from the original within minutes. Once the soak window passes cleanly, the old VM gets deleted (boot disk retained for 30 days as final-final insurance).

What we did during the soak that we wish we’d done years earlier

Migration is the natural moment to set up the operational hygiene that should have been there all along:

Automated disk snapshots. GCP resource policies handle daily/weekly schedules natively — no scripts, no cron, no maintenance. Critical flag: --on-source-disk-delete=keep-auto-snapshots, so accidental disk deletion doesn’t take your snapshots with it.

Logical database backups to a separate bucket. Disk snapshots are crash-consistent, which is fine for filesystems but ambiguous for databases. A nightly mysqldump --single-transaction piped through gzip and uploaded to a versioned, lifecycle-managed GCS bucket gives you a real recovery path. Test the restore path at least once — a backup you’ve never restored from is a wish, not a backup.

Alerting on backup failure. Two alerts: one that fires when the backup log contains ERROR, and one that fires when no Backup OK line has appeared in 26+ hours. The absence alert is the more important one — loud failures get noticed; silent ones don’t.

Firewall cleanup. Long-lived projects accumulate firewall rules. Remove auto-generated rules for protocols you don’t use (default-allow-rdp if there are no Windows hosts). Consolidate overlapping rules. Tighten admin UIs to known IPs and the tailnet CIDR only. Never expose SMB (139/445) or proxy admin ports to the public internet.

Deletion protection. A single flag on the VM prevents the kind of late-night accident that ruins a quarter.

What we’d do differently

Two things, both about timing.

Capture user dotfiles earlier and more carefully. Shell configs, oh-my-zsh, .gitconfig, .tmux.conf, custom prompt setups, aliases — none of this carries forward from a fresh Ubuntu image. But these directories also frequently contain secrets (SSH private keys, cloud credentials, API tokens). The right pattern is to tar dotfiles aggressively excluding secret paths, then transfer the secrets separately via gcloud compute scp rather than letting them sit in a GCS bucket even briefly.

Test more application connectivity from inside Docker networks. The “is NPM talking to WordPress?” test failed in an interesting way: the NPM container image doesn’t ship wget. Slim images often lack common diagnostic tools. The right test isn’t docker exec npm wget ... but docker exec npm curl ... or, more reliably, a throwaway alpine container attached to the same network: docker run --rm --network web_proxy_bridge curlimages/curl -I http://wordpress.

The TL;DR

Migrate from EOL operating systems on a schedule, not under pressure. Rebuild rather than upgrade in place. Spend more time on discovery than feels necessary. Run old and new in parallel until the new is validated. Cut over with one command in a controlled window. Soak for two to three days before decommissioning. Use the migration as the forcing function for the backup, snapshot, and alerting hygiene you’ve been meaning to set up.

And document the gotchas while they’re fresh. The second migration always goes better than the first — provided you wrote down what bit you the first time.

---

Need help with your own migration?

If you’re staring down an EOL operating system on a production VM and don’t have the bandwidth to plan this carefully, we can help. CyberCloud AI runs migrations like this for small and mid-sized businesses that need senior-level execution without a full-time hire. We’ve done the discovery, scripting, testing, and post-migration hardening work — your engineers can keep shipping features while we handle the underlying infrastructure refresh.

A typical engagement is two to three weeks: one week of discovery and planning, one week of execution and soak, optional ongoing managed support afterward. We deliver a documented baseline, a reusable playbook tailored to your environment, and a hardened post-migration state that maps cleanly to NIST 800-171 and CMMC requirements.

If you’re a defense contractor working toward CMMC Level 2 certification, this kind of infrastructure work is exactly the kind of practical evidence assessors look for under §3.4 (Configuration Management), §3.7 (Maintenance), §3.13 (System & Communications Protection), and §3.14 (System & Information Integrity). We’ve also built compliance packages designed for small DIB companies — practical, audit-ready, and priced for businesses that don’t have a compliance department.

Explore our CMMC compliance packages: https://www.cybercloudai.tech/cmmc-packets

Get in touch if you’d like to talk about your environment — we offer a no-cost initial assessment to scope what a migration or compliance engagement would look like for your specific stack.

---

This post is based on an actual migration completed in May 2026. All infrastructure details, hostnames, and identifiers have been generalized to protect operational security.