Bit rot and images...

Discussion in 'Photography and Cameras' started by AllanMarcus, Feb 20, 2023.

  1. AllanMarcus

    AllanMarcus Friend

    Pyrate
    Joined:
    Oct 23, 2015
    Likes Received:
    2,969
    Trophy Points:
    113
    Location:
    Los Alamos, NM
    Home Page:
    I saw some bit rot on one of my bit torrent files (easily recoverable) but then started to worry a little about bit rot on my images, which go back more than 40 years. Sure, I have a robust backup plan, but underlying changes to the file might not be detected until it's too late.

    To mitigate, I wrote a script to calculate the checksums of all the files in my photos directory and store the results in a SQLite database. The script, which I plan run monthly, will store the most recently calculated checksum and compare, thus letting me know if any of the files in that dir changed. Since I use Lightroom to manage the photo library, files in the dir never change; only new files are added.

    To any computer science folks out there, how does this approach sound? Here is the script:

    https://github.com/odinian/checksum.py-for-Bit-Rot-detection
     
    Last edited: Feb 20, 2023
  2. Kernel Kurtz

    Kernel Kurtz Friend

    Pyrate Contributor
    Joined:
    May 19, 2018
    Likes Received:
    1,677
    Trophy Points:
    93
    Location:
    Winnipeg, Canada
    I've been playing around trying to learn some Python and have written some simple scripts using GPIO on an Rpi, but your script is well beyond my capability. Looks nice though.

    That said, file integrity checkers are pretty advanced these days. I've been using Samhain on my Linux and BSD boxes for a long, long time (I've also used Tripwire/AIDE in the past), not just to check for corruption of data files, but also to monitor any changes to the system itself. It can watch for a lot more than just checksums, and also emphasizes protecting the integrity of it's own database, so it really is more of a system security tool than just a way to check for bit-rot, though in its simplest form could be configured for that.

    https://www.la-samhna.de/samhain/index.html

    Your script is a nice easy alternative for a user who just wants to check their music or image files on an occasional (or even regular as a cron/scheduler) basis. I'll give it a try and let you know what I think.
     
  3. AllanMarcus

    AllanMarcus Friend

    Pyrate
    Joined:
    Oct 23, 2015
    Likes Received:
    2,969
    Trophy Points:
    113
    Location:
    Los Alamos, NM
    Home Page:
    Thanks! I looks at FIMs, and only one that is free and seems decent on Windows is WinFIM.NET, which I might try.
     
  4. Thad E Ginathom

    Thad E Ginathom Friend

    Pyrate
    Joined:
    Sep 27, 2015
    Likes Received:
    14,130
    Trophy Points:
    113
    Location:
    India
    Hmmmmm....

    Some of the most valuable data on my computer, Beit pics, music, whatever, is old data, and it may not be accessed for months or years.

    I am strict about backups, but only a two-copy system, so any data corruption would be copied to the backups quite quickly.

    I have great faith in the Linux filesystem (ext4). I have much more trouble with external NTFS devices. But still... Food for thought here!
     
    • Like Like x 1
    • Agreed, ditto, +1 Agreed, ditto, +1 x 1
    • List
  5. SoupRKnowva

    SoupRKnowva Official SBAF South Korean Ambassador

    Pyrate Contributor
    Joined:
    Sep 26, 2015
    Likes Received:
    4,249
    Trophy Points:
    93
    Location:
    Austin, TX
    ext4 won't protect you from bit rot. what you really need is ZFS if you want to be sure you're data is ok

    edit: BTRFS maybe or may not also have protections against bit rot, but im less familiar with it
     
  6. Thad E Ginathom

    Thad E Ginathom Friend

    Pyrate
    Joined:
    Sep 27, 2015
    Likes Received:
    14,130
    Trophy Points:
    113
    Location:
    India
    I'm reading bit rot as corruption. And saying that my impression is that ext4 is robust. I know it won't necessarily save me from the results thereof.

    Don't know if SSD is going to prove otherwise, but the bits on my HDDs might not fail unless or until the disk does.

    I am somewhat techie, but in a very ex-techie sorry of way. I might go as far as checking out the monitoring system mentioned in a previous post.
     
  7. Kernel Kurtz

    Kernel Kurtz Friend

    Pyrate Contributor
    Joined:
    May 19, 2018
    Likes Received:
    1,677
    Trophy Points:
    93
    Location:
    Winnipeg, Canada
    I use BTRFS on my NAS. It does detect data corruption on read and can silently repair it if it is part of a RAID volume. I have not used ZFS myself, but it also has an excellent reputation in this regard.

    A really good look at this subject on Ars here;

    https://arstechnica.com/information...-and-atomic-cows-inside-next-gen-filesystems/

    And a deeper dive in to BTRFS for geeks on its wiki page.

    https://btrfs.wiki.kernel.org/index.php/Main_Page
     
  8. AllanMarcus

    AllanMarcus Friend

    Pyrate
    Joined:
    Oct 23, 2015
    Likes Received:
    2,969
    Trophy Points:
    113
    Location:
    Los Alamos, NM
    Home Page:
    Thanks. I'm not interested in running a RAID anymore. From what I understand, even mdadm RAID volumes will detect and rebuild corruption on disk scans, but for 8TB drives, that takes days. Disks are too large for rebuilds and scans.

    BUT....I might repurpose some smaller disk and create a ZFS mirrors, but that would have to be under linux. I'm thinking of testing Manjaro anyways, so I might combine the adventure into one fun trip! I would be running that solely for the purpose of keeping archive copies of pictures, but it seems like a good idea. If I decide to switch from Ubuntu to Manjaro, I will like switch from ext4 to ZFS as well.

    @Thad E Ginathom , run my script and get a baseline. Then run it monthly and see if you have issues. You can even just run it quarterly, as long as you have good backups to restore from if you see issues.
     
  9. GoodEnoughGear

    GoodEnoughGear Evil Dr. Shultz‎

    Pyrate
    Joined:
    Oct 25, 2015
    Likes Received:
    3,070
    Trophy Points:
    113
    Location:
    Cape Town, South Africa
  10. AllanMarcus

    AllanMarcus Friend

    Pyrate
    Joined:
    Oct 23, 2015
    Likes Received:
    2,969
    Trophy Points:
    113
    Location:
    Los Alamos, NM
    Home Page:
    too many poor reviews and a dead technology. I wouldn't waste money on them.
    Back up locally. Back up to the cloud. Keep a checksum database to ensure your local files are OK.

    BTW, If you have Amazon Prime, you have free, unlimited cloud backup of images. I use that has a quaternary backup for my images. I rarely do any video. Primary - local, secondary-local, tertiary - Google Drive, quaternary - Amazon Photos)

    Backblaze for $7/month, unlimited storage, also seems like a good option.
     

Share This Page