Cloud Storage

I back up to a QNAP RAID array and I have a program that runs periodically to sync he RAID contents to an encrypted AWS S3 bucket. That said, that's a bit more complicated to set up than the other end user focused solutions mentioned above. I also periodically clone my hard drives to external drives using Carbon Copy Cloner. Even paranoids have enemies

Rob

 

Anything that is on, or connected to, your system or network, is not really a backup. It's just another copy, which can come in handy, and even looks like a backup if you can restore one failed disk or filesystem from it.

But it can equally be taken out by the same power surge, lightening flash, burglar, flood, etc, as your main storage.

Of course, if lightening strikes my house, the disconnected external drive sitting on my desk is probably not going to be very useful --- but the one I sent to my friend's house, some distance away, is only going to be missing my last week of work. A one-week cycle. For personal stuff, I accept that level of risk.

 

I agree, but "a capable adversary" is rarely part of a realistic threat model. Most people are (or should be) more concerned about estranged spouses, disgruntled employees, business competitors, the attorney for the plaintiff, etc.

Those actors don't need zero-day vulnerabilities to get to the files; they may be able to misuse access they already have, or do a little burglary or social engineering, or legally compel the storage provider to give them access. Client-side encryption seems an excellent way to protect against them.

 

I use tarsnap for backups. Local deduplication, compression, and encryption, stored in multiple AZ's at Amazon S3.

https://www.tarsnap.com

 

I like this solution a lot! Is there a similar service with a GUI?

 

Open drive.google.com
click on the gear icon, select settings
You will see storage at the top. If it says something like 2.51GB of 15GB used, then you have a limit. If if says 11.37TB used, you got unlimited storage.

 

Most of the people I meet in real life who have been infected with malware sadly did not need all that particularly capable an adversary.

 

Not where I work .

 

There's a GUI for it, but I haven't used it. It's described here, you might need to build it or have a friend build it for you. Installation instructions are in the same repository here: https://github.com/Tarsnap/tarsnap-gui/blob/master/INSTALL

 

I do too. Being somewhat innumerate, though, I'm having trouble working out what my real cost for a complete system backup of 1.6TB (today) would cost me, from their picabyte costings! **

I really really wonder how long that first backup would take. And nobody can answer that except me as they are not sitting at the end of my network.

I may be innumerate, but I am not Linux illiterate: I don't mind a command-line/scripts based system at all. That's how I do my USB-hdd backups.

And I think I could have disarmed the bomb. (I follow xkcd, but the tarsnap site made me aware of this particular strip)


** The site says 25c per GB/month, which sounds cheap, but 1000GB would then cost $250 per month? Sounds like it can't be right.

Especially as they also say

 

The important bit is "non-unique". After deduplication, Alice is only storing around 16GB, so ~$4/month.

 

Not where I worked either, but sadly most of the people I meet in real life who have been infected with malware also don't have dedicated security teams.

 

Well, it looked like something I would feel at home with, but now I'm thoroughly confused so I'll forget it for now.

 

You can run tarsnap in a dry-run mode that helps you determine what your cost will be.

 

Thanks to @fastfwd for clearing up some confusion by PM. No, sadly, tarsnap is not, currently, for me.

Among other things, he reminded me that my jpeg photo library, which grows quite rapidly, will not compress much further. Same is true of my music library, which is almost all FLAC.

 

Do keep in mind is that tarsnap doesn't do file-level deduplication. It does variable-length block-level deduping across all known files in its store. That's quite a lot different than file-level compression, which wouldn't benefit much.

More here: https://www.tarsnap.com/deduplication-examples.html

Note, I myself haven't yet investigated the cost of putting my FLAC library into tarsnap, but I'll start that endeavour this weekend. For those, it's fairly easy to manually copy to a solid state drive and put into offsite storage. Besides, those aren't truly critical…I can always repurchase them. Photos and documents are different.

 

Very interesting.

I'm guessing that, whilst (working on the assumption that they are efficient compression methods) a single FLAC or JPG (for example) file cannot be usefully compressed further, a collection of a hundred of them might have repeating byte patterns.

It is all fascinating stuff, even though it does my head in (well, not nearly as much as cryptology... or digital music!). I'm tempted to do some "dry run" tests, or, if I can, perhaps write to a local-device archive.

 

Not of any significant size, unfortunately. The smallest block size I've seen used by deduplication systems is 4KB, and duplicated chunks of even that size are unlikely in a music library.

Starwind Software has a deduplication analysis tool you can download, although you'll need to register with a corporate email address. If you want to try a local experiment, OpenDedup is free and runs on Windows and Linux.

 

I ran the dry-run across my music files, and as I suspected a library of FLACs don't really compress well.

du -sh reports 415G as the size. And the stats from tarsnap are:



That final size works out to 403G, which would be $100/month, yeah not worth it!

I had already been through this exercise in my mind, and finally came to the conclusion that I'd use tarsnap to backup documents & photos but not music, but had never determined what the cost would be to use it for music.

I might just create my own Amazon AWS account and put my music in S3. At the Glacier tier, it's $0.004/GB/month ($1.66 for the above), and Deep Archive (12-hr restore time) it's only $0.00099/GB/month ($0.41/month for the above).

Pricing info for S3 (choose your region).

Edited:
- put an image of preformatted text (dunno why bbcode preformatted doesn't work).
- sorry about the largesse…Macs are all hi-DPI; doubling its PPI setting to 288 didn't seem to help.


Edit (final thoughts):

For me it doesn't make sense to spend monthly to backup my music files. I'd rather just buy a couple 1TB thumb drives and copy the files there, and keep one in the safe deposit box at the bank. These files aren't "secret" in any way, so don't need encryption, and they don't change over time so no version control is needed.

A 1-2TB thumb drive or M.2 NVME in a small metal box (an Altoids tin) should be best. Guards against accidental loss. Keeping a separate file with the SHA256 hashes of all the music files can let me know if they've degraded due to cosmic rays.

 

lol data rot.