Goodbye, my drives
-
I just so stumbled across this drive failure. Again. Twice. Both disks are damage in a mirror.
pool: rpool2 state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Mon Mar 11 13:30:37 2024 2.33T / 2.75T scanned, 62.8G / 553G issued at 578K/s 29.1G resilvered, 11.37% done, no estimated completion time remove: Removal of vdev 6 copied 2.19G in 0h1m, completed on Mon Feb 13 00:13:43 2023 104K memory used for removed device mappings config: NAME STATE READ WRITE CKSUM rpool2 DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 ata-TOSHIBA_MG07ACA12TE_89W0A0DQFBXG-part2 ONLINE 0 0 0 ata-TOSHIBA_MG07ACA12TE_5020A0Z2FDUG-part2 REMOVED 0 0 0 special nvme-Samsung_SSD_970_EVO_Plus_1TB_S4EWNM0NC24223X-part2 ONLINE 0 0 0 logs nvme-Samsung_SSD_970_EVO_Plus_1TB_S4EWNM0NC24223X-part1 ONLINE 0 0 0 cache nvme-Samsung_SSD_970_EVO_Plus_1TB_S4EWNM0NC24223X-part3 ONLINE 0 0 0 errors: 15352787 data errors, use '-v' for a list
The first drive actually have a series of read and checksum error if ever running in a long scrub, and will fail to operate after an hour or so. But I was still able to read the data out despite there are "10 million errors" (scary numbers).
So far, the data in the zfs pool is still...in tact? I have iSCSI services running with ZFS (zvol to be exact) and I did a logical level NTFS check disk for all of them, so far none of them reported any error, even my game stream videos footages disk which are very susceptible to corruption error (which often turns into random white noise in byte stream), instead I got perfectly fine video, and not a single bit rot is to be seen.
Still, while I'm not sure why there is such a failure, I have already ordered an online backup server for an emergency data export (That's $180 yearly for 4TB, and my pockets aren't deep to be honest) by using
zfs send
andzfs recv
. After that, I'm going to destroy the ZFS pool and move the disk to my real NAS I built a few months ago which is still under stress test period.Bon voyage, my former ZFS dev drives, and hope your new home in the NAS cage would be the best you ever had.