I just wanted to check the Proxmox version

Some context first. The server in Zwolle is a self-built Proxmox box: an i7-11700K, 128GB of RAM, a GTX 1060 with creatively licensed NVENC firmware for Plex, and a ZFS pool called data. That pool is six 6TB WD Reds in three mirrors. Sounds responsible, right? Mirrors! Redundancy! We thought about this!

Here's the catch. Four of those six drives are WD60EFAX. If that model number means nothing to you: congratulations on your blood pressure. The EFAX is the drive from WD's 2020 scandal where they quietly started shipping SMR drives in their NAS lineup without telling anyone. More on that later, because I have OPINIONS.

My dad and mom function as the datacenter's remote hands (sensors and actuators, if you will). Physical access is a 25 minute drive. Monitoring was, and I say this with love for past me, nothing.

All I wanted to do was check the Proxmox version. Proxmox 9 was out, I felt like upgrading, so I SSH'd in for what should have been a 30 second visit. And since I was there anyway, a quick health check:

root@pve:~# zpool status
 pool: data
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.

Ah.

One of the EFAX drives sat there FAULTED with 60 read errors and 69 checksum errors. Its neighbour in mirror-0 had 7 read errors of its own, just to keep things interesting. Two out of six drives misbehaving, both SMR, both in a pool nobody was watching.

Time for some journal archaeology. The kernel logged the first medium errors on June 27. ZFS's event daemon faulted the drive on July 12, right after a scheduled scrub (the scrub also repaired 1M of data, so the rot was actively rotting). I found it on July 28.

That means the pool ran degraded for SIXTEEN DAYS. Sixteen days of one mirror having exactly zero redundancy left, in a house where the closest thing to a NOC is my mother walking past the utility room. If the other drive in that mirror had decided to join its buddy, that would have been the pool. The Plex library, the VMs, everything.

Nobody knew. Not me, not my dad, not ZFS's email daemon (not configured), not the Discord webhook (did not exist yet). ZFS did everything right and then told absolutely no one, because I never gave it a phone.

So the drive must be dying, right? 60 read errors! Let's look at the SMART data:

 5 Reallocated_Sector_Ct   -   0
196 Reallocated_Event_Count -   0
197 Current_Pending_Sector  -   0
198 Offline_Uncorrectable   -   0

Zero. Zero everything. The drive is, by its own account, in perfect health.

This is the famous EFAX party trick. The drive isn't physically dying. It's an SMR drive, which means writes sometimes trigger a big internal reshuffling of shingled zones, during which the drive just... stops responding. For a while. ZFS sends a command, hears nothing, counts it as an error, and after enough of those it yeets the drive out of the pool. The drive, meanwhile, genuinely believes it did nothing wrong. It was just thinking!

So you end up with a drive that is simultaneously fine and unusable. Schrödinger's Red.

SMR is a few percent cheaper to manufacture per terabyte. That's the whole reason it exists. WD took that saving, put SMR into the ONE product line explicitly marketed for NAS and RAID workloads (where SMR performs worst), didn't put it on the spec sheet, and waited for the community to figure it out via benchmark forensics. Which it did, in 2020, loudly.

The consequences for WD: a US class action settlement worth a rounding error, some relabeling (SMR Reds became "Red", CMR became "Red Plus"), and that's it. No recall. No extended warranty. And if you bought your drives in Europe, like my dad did: helemaal niks. The settlement covered American buyers only. Our four EFAX drives fall in the "jammer dan" category.

The structural problem is that the hard drive market is three companies (WD, Seagate, Toshiba) and all three pulled the same silent SMR move in the same period. There is no fourth option to punish them with. The same oligopoly is currently jacking up drive prices for AI datacenter demand, which is relevant later in this story.

Since the SMART was clean and the errors were timeout theatre, the drive got one (1) second chance: zpool clear, back into the pool, resilver. ZFS immediately set expectations:

627M resilvered, 0.01% done, 3 days 23:08:30 to go

Four days. FOUR DAYS of one mirror running without a net, on the exact drive type that's famous for getting kicked during resilvers. I mentally prepared for a long week.

It finished in 19 minutes. 69.5GB, zero errors. The four day estimate was ZFS extrapolating from a cold start, like your car's range indicator after 200 meters of uphill. It only needed to catch up on what the drive missed while it was benched, not rebuild 6TB.

Both problem drives have shown clean counters since. They're not forgiven, they're on probation. There's a difference.

While I was in a testing mood anyway, I ran a self-test on the Samsung 970 EVO boot NVMe. Fun fact: smartctl can't do NVMe self-tests, you need nvme-cli for that. Result:

Operation Result : 0x7  (failed: media/data integrity)
Self Test Result : failing LBA 0x13a4cc80

Of course. OF COURSE. The boot drive too.

A manual dd read of that sector came back clean, meaning the controller already remapped it. The drive also carries 8 lifetime Media and Data Integrity Errors, which is not an emergency but is now A Pattern To Watch. This server is held together by remapped sectors and hope.

The actual lesson of the 16 days isn't "SMR bad" (although: SMR bad). It's that a ZFS pool without alerting is a pool that fails silently until it fails loudly.

So the server got a phone. ZED (the ZFS event daemon) now fires every state change into a Discord webhook, and a systemd timer posts a full disk health report every morning at 09:00. The channel is called "Road to Bankruptcy", named after where this hobby is headed financially.

Every future non-zero error counter is now news within a day instead of within a fiscal quarter. It's genuinely embarrassing how little work this was compared to how long I didn't do it.

The correct fix is replacing the EFAX drives with proper CMR enterprise drives. Recertified Seagate Exos or WD Ultrastar, boring and bulletproof. The problem: I picked the worst possible year to need hard drives. AI datacenters are hoovering up every drive on the planet and prices are ridiculous (a NEW 6TB Red Plus is around €309 right now, which is offensive).

So the plan is patience. The pool is healthy, the monitoring is loud, both suspects are behaving, and there's a refurb price alert running. The moment a sane deal on recertified enterprise drives appears, two of them are coming to Zwolle.

Oh, and ZFS helpfully suggested I could run zpool upgrade now that everything's healthy. No. Pool feature upgrades are a one way door and the features on offer solve problems I don't have. The pool stays as it is, importable anywhere, boring forever.

That's the goal, by the way. Boring. For about three weeks now, zpool status has been the most uninteresting output on that machine. Long may it last.