Rescuing 10 Million Files from a Dying HDD with Nearly 100,000 Hours on the Clock

How This Started#

I recently started running a home server.

While looking into things, I learned that an open-source app called Immich is apparently an excellent replacement for Google Photos.

The D drive of my everyday PC is stuffed with old pictures I took ages ago. I went to check where they were, but File Explorer felt really sluggish. Opening a folder took over a second, and opening an image took over five seconds.

Something is definitely wrong. I asked Professor Gemini (the AI), and it said the HDD was most likely failing.

I installed CrystalDiskInfo to check, and the situation was pretty grim.

Reallocated Sectors Count: 8
Current Pending Sector Count: 1271
Uncorrectable Sector Count: 214
Power-On Hours: 99824 hours (= 11.4 years!!)

According to Professor Gemini, “it could become completely inaccessible at any moment, and in fact data may already be getting lost.” That’s not good.

The Situation#

The D drive I want to rescue holds 970 GB of data accumulated over about 15 years, all jumbled together. It even includes data carried over from the old PC I replaced 7 years ago. Frankly, I don’t get many chances to look back at it, but I really don’t want to lose the records of all my past projects. It’s also packed with ripped music, doujin books, anime, movies, treasures (wink wink), and so on.

On top of the D drive, I have a 2 TB external HDD (1 TB used) full of DTM (desktop music) sound sources and more treasures. I want to merge that into the new HDD too.

That means the new HDD needs at least 2.5 TB, and since I expect the data to keep growing, I’d like double that: 5 TB.

Hunting for an HDD#

Luckily, I’d been thinking that 1 TB obviously wouldn’t be enough once I put Immich on there, so I was already mentally prepared to buy another HDD. And the Amazon Prime sale was on!

After lots of consulting with Master Gemini, I ended up buying “[Amazon.co.jp Exclusive] Seagate Internal HDD 8TB for NAS, 24/7/365, 3-Year Warranty, 3-Year Data Recovery Service, CMR, IronWolf, Authorized Domestic Distributor Product ST8000VN002”. It cost 28,800 yen (about $190).

Amazon | Seagate IronWolf 内蔵HDD 8TB NAS用 ST8000VN002 | シーゲイト(SEAGATE) | 内蔵ハードディスク通販

Seagate IronWolf 内蔵HDD 8TB NAS用 ST8000VN002が内蔵ハードディスクストアでいつでもお買い得。当日お急ぎ便対象商品は、当日お届け可能です。アマゾン配送商品は、通常配送無料（一部除く）。

www.amazon.co.jp

↑ Not an affiliate link, so rest easy.

Because of the sale, I ended up buying even more capacity than I’d originally planned.

The 10-Million-File Rescue Plan#

Since I was at it, I decided to organize the files into a reasonably good shape before moving them over.

Here’s how I sorted things. I’d organize them to fit the open-source software I’d probably be using.

To rescue
- Photos and videos I shot myself
  - To be viewed from Immich
- Ripped music and videos
  - To be viewed from Jellyfin
- Treasures
  - To be viewed from Stash
- Personal projects, writing, and so on
  - Still figuring out what to do with these
To delete
- Files that can easily be downloaded from the internet again, and files that aren’t very important
  - Applications and the like
  - But files like images and music that might disappear from the internet at any time get rescued

To reduce the load on the dying HDD, I came up with the following procedure.

Copy the entire contents of the dying HDD (0.9 TB) to the external HDD (1 TB/2 TB)
- For this I used FastCopy, as recommended by Grand Master Gemini.
Organize the files on the external HDD
- I went through the folders one by one, sorting each into either “move to one of the 4 rescue folders” or “delete”.
- This took 15 hours, straight through the night…
- I gradually got so delirious that I forgot the sorting rules, so I took a nap in the middle.
Connect the newly bought 8 TB HDD and the external HDD with the organized data to the server machine, and mount them on lumine
rsync the 1 TB HDD currently connected to the server machine → 8 TB HDD
rsync the external HDD → 8 TB HDD ← YOU ARE HERE
Repoint the mount that Jekyll and the like currently read from, from the 1 TB HDD to the 8 TB HDD
Organize the transferred files
Set up a backup of the Immich data on the 8 TB HDD → 1 TB HDD
Remove the dying HDD from my everyday PC
Remove the 2 TB external HDD

I’m writing this article while waiting for the external HDD → 8 TB HDD rsync to finish. It should take about 10 hours…

And then it finished.

So from here on I’ll take working notes (I should have been doing that from the start).

Transfer Complete#

The transfer from the external HDD to the 8 TB HDD. It finished in about five and a half hours.

1
sent 1.48T bytes  received 19.50M bytes  85.79M bytes/sec
2
total size is 1.48T  speedup is 1.00
3
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7]

But some error showed up. I ran it again, and two files with absurdly long file names had failed. The Windows file system allows them, but they were too long for Linux, which is apparently what caused the error. It’s probably fine, so I’m ignoring it.

After it printed sending incremental file list:, I got stuck waiting for nearly 10 minutes.

Can I Watch It on Jellyfin?#

I put the transferred files under jellyfin/ in a reasonably sensible way.

I’d mixed music files in with images and videos, so I used the following command to copy them over while preserving the folder structure. Thank you, Gemini the Almighty.

1
sudo rsync -av --progress \
2
--include='*/' \
3
--include='*.mp3' \
4
--include='*.flac' \
5
--include='*.m4a' \
6
--include='*.aac' \
7
--include='*.ogg' \
8
--include='*.wav' \
9
--exclude='*' \
10
/mnt/data/media/jellyfin/video/jellyfin-fromD/ /mnt/data/media/jellyfin/music/

Success confirmed! Anything with album data is properly reflected.

And Jellyfin is really easy to use as a music player too. No ads, no pushy recommendations, just truly wonderful.

Excavation Notes#

While organizing, I dug up a bunch of interesting stuff, so here are some notes.

The OST of Irisu Syndrome#

It must have been more than 10 years since I played it. Such nostalgia.

And when I listened to it again, I realized the music is outrageously good.

Irisu Syndrome OST- About 10 Hours Looking at the Ceiling

Quite possibly one of the best puzzle games you won't play because you don't want to look like a weeaboo. I used the filename since I keep forgetting the act...

www.youtube.com

There was a 10-hour loop, so I listened to it while sorting through the drive. Listening to this track while tracing the records of everything my past self did on every PC I ever used was a pretty great experience: melancholic without being clingy.

About 10 Hours Looking at the Ceiling

YouTube でお気に入りの動画や音楽を楽しみ、オリジナルのコンテンツをアップロードして友だちや家族、世界中の人たちと共有しましょう。

www.youtube.com

It seems it’s recently been getting discovered overseas too. The trigger appears to be the following video, which sampled it into breakcore.

Emray - About 10 Hours of making Breakcore

https://emray.bandcamp.com/track/about-10-hours-of-making-breakcorehttps://emray.bandcamp.com/https://soundcloud.com/emraybc

www.youtube.com

A Rulebook for a Board Game I Made Long Ago#

In my student days I was insanely into board games. I started a small circle, and we even made our own games there. The name: The Money Worship Game (working title).

Rules for My Homemade Board Game “Mammonism Game (Working Title)” (拝金主義ゲーム(仮))

It was fairly well received among my friends, and there was talk of putting it out at a convention, but I couldn’t find the time and it got shelved.

You can play it with the 5-colors x 20-chips sets they sell at 100-yen shops, plus some random cards from something like MtG to make 2 types x 9 proxies. Please give it a try if you like.

The crux of the game is how well you can keep a poker face when you see “poop”.

Please prepare the cards like this.

Slides from (Maybe) My Middle School Graduation#

In a folder called USB遺物 (USB Relics), there were photos from my middle school days. The image quality was oddly good, and some had me in them and some didn’t.

By the way, this is a cropped part of an image from the 職場体験 (work experience) folder. You can feel the era.

And this folder also had several music files with inexplicable names.

TUW.mp3
- The Unsung War - Ace combat 5
TLoGM.mp3
- The Liberation of Gracemeria - Ace combat 6
EMFIV.mp3
- Egao no Mahou (Instrumental Version) - Strike Witches 2

I wondered where on earth they came from, and when I opened the pptx file in the same folder, I (probably) solved the mystery.

Right before graduating from middle school, there was some event where a slideshow looking back on the three years was going to be shown. At the time, I was the student council vice president and was treated as the computer teacher, so I ended up making the slides. The images in USB遺物 were materials I’d borrowed from teachers and parents to make the slides.

…is what must have happened.

And I suspect these music files were renamed so oddly in order to play songs I liked while keeping the teachers from figuring out where they came from.

The bad otaku habit didn’t stop there, but invaded the slides themselves. The slideshow follows three years of events in chronological order, but the slides serving as the title of each event are weird.

Given where the music came from, this is probably a screenshot from Ace Combat 6. Probably Gracemeria. I think it’s the demo before the mission starts. A few frames later, a fighter jet should come out from the bottom left.

This must be a clip from Strike Witches season 2, episode 6.

Showing Strike Witches clips and music in a gymnasium filled with the whole grade plus parents must have been a blast. Actually, the fun memories are coming back to me.

If anything, I think I remember planning with a friend who was into Strike Witches back then, strategizing about how to sneak it in.

It’s undeniably cringe history from my chuunibyou days, but somehow I felt nostalgic and warm inside. I don’t yet have the courage to look at my high school cringe folder.

Final Thoughts#

I plan to remove the dying D drive from my PC tomorrow and store it away so I don’t shorten its life any further.

Sorting through a drive that holds more than half my life was deeply moving. From the image files, of course, but also the music, MIDI files, games I made long ago along with their project folders, freeware (which I ended up deleting this time), and so on, memories came flooding back. It felt like a journey following my digital footprints.

Honestly, about 98% of it is stuff that isn’t worth keeping, but I’m sure I’d be pretty crushed if this data disappeared. Next, I have to think about a backup strategy.

Don’t forget to give your HDDs regular health checkups, everyone.