Music industry’s 1990s hard drives, like all HDDs, are dying
Music industry’s 1990s hard drives, like all HDDs, are dying
The music industry traded tape for hard drives and got a hard-earned lesson.
My father told me he wanted to make USB flash drives of all the scanned and digitized family photos and other assorted letters and mementos. He planned to distribute them to all family members hoping that at least one set would survive. When I explained that they ought to be recipes to new media every N number of years or risk deteriorating or becoming unreadable (like a floppy disk when you have no floppy drive), he was genuinely shocked. He lost interest in the project that he’d thought was so bullet proof.
I explained that they ought to be recipes to new media every N number of years or risk deteriorating or becoming unreadable
This is important, and for some media, it should be more often than that.
People forget that flash memory uses electrical charge to store data. It's not durable. If left unpowered for too long, that data will get corrupted. A failure might not even be visible without examining every bit of every file.
Keep backups. Include recovery data (e.g. PAR2). Store them on multiple media. Keep them well-maintained (e.g. give flash drives power). Mind their environment. Copy them to new storage devices before the old ones become obsolete.
It's funny that with all our technology, paper is still the most durable storage medium (under normal conditions) that doesn't cost an arm and a leg.
It's funny that with all our technology, paper is still the most durable storage medium (under normal conditions) that doesn't cost an arm and a leg.
Sophistication often creates fragility. The human mind marvels at sophistication naturally; appreciation for resilience usually only comes after that fragile thing has broken. Of course it's too late by then.
All them young whipper snappers will continue to learn these life lessons the hard way, it seems.
If I had a cent every time an artist on patron had their computer die on them and lost works in progress or all their old stuff... I'd afford a few coffees.
Maybe you should have asked for a dollar everytime that happened. Youd be rich I tell ya!
Tape backups, baby.
No, I don't have a library of those. I don't even have a tape drive.
This is my next step, i just wish tape drives werent so expensive.
Anything important, I write on clay tablets.
People forget that flash memory uses electrical charge to store data. It's not durable. If left unpowered for too long, that data will get corrupted.
Yeah, but the link in the article, strict checks and no data loss over 52 weeks. Not neccessarily in USB sticks though. And sure, backups.
That number is a single manufacturer's performance target. It is not a guarantee of results. You might be able to get Intel to replace an SSD if it corrupts data in under 52 weeks (assuming you notice it) but your data will still be gone.
Hardware performance can and does vary by manufacturer, model, and production run. Even the nominally identical cores within a single CPU have slightly different operating limits. YMMV.
Note also: the 52 week target you quoted is halved for every 5° rise in temperature.
This is where "piracy" is actually the industry's saving grace. Decades or centuries later, will record labels exist and be well-managed (and flush with cash) enough to preserve archival copies of their artists catalogs? Probably not.
Will obscure weirdos exist all around the world on Usenet, IRC, or seeding torrents? Possibly.
What is really being discussed here is archiving of master recordings and session files. The publically avaliable releases themselves aren't really in jeopardy. Orthough piracy probably does provide an extra layer of security to more obscure releases.
I thought I read somewhere that when they were making one of the Toy Story movies, there was some catastrophic data loss that nearly tanked the whole production. But then one of the animators came back from maternity and said wait, I think I have most of it synced to my home server? And the next thing you know, John Lasseter himself is barrelling down the highway to her place and it turned out yeah, she did have it.
Oh, fuck. Prince's Vault.... God I hope the estate has a plan to preserve all of that.....
Many films and tv programs survive only thanks to a total stranger keeping their own copy. For a long term survival of any media it has to be copied and distributed far and wide.
There is no "write and forget" solution. There never has been.
Do you think we have ORIGINALS or Greek or roman written texts? No, we have only those that have been copied over and over in the course of the centuries. Historians knows too well. And 90% of anything ever written by humans in all history has been lost, all that was written on more durable media than ours.
The future will hold only those memories of us that our descendants will take the time to copy over and over. Nothing that we will do today to preserve our media will last 1000 years in any case.
(Will we as a specie survive 1000 more years?)
Still, it our duty to preserve for the future as much as we can. If today's historians are any guide, the most important bits will be those less valuable today: the ones nobody will care to actually preserve.
Citing Alessandro Barbero, a top notch Italian current historian, he would kill no know what a common passant had for breakfast in the tenth century. We know nothing about that, while we know a tiny little more about kings.
Do you think we have ORIGINALS or Greek or roman written texts?
In some cases we do. For example, the documents at Pompeii, those at Qumran, and other rare instances where documents have been preserved from the time they're written. It's also true that we have far more copies than originals, and that we've lost most of the works of many ancient authors.
There is: mdiscs. Allegedly 1000 years durability even in Blu-ray format. Should be good enough for most important things. The best tapes AFAIK 30- 100 years
Do you think we have ORIGINALS or Greek or roman written texts?
We do have originals of some much older texts, though (cuneiform on clay that was fired after impression seems to be a pretty good archival medium, overall). We'd probably have a lot more original Greek and Roman documents if they hadn't been destroyed in wars and other disasters, or recycled for various purposes. There's a big survival rate difference between documents that receive basic care throughout their lives—no rough handling handling, minimal direct sunlight exposure, and some degree of temperature and humidity control in the storage area—and those left to fend for themselves. That's why old documents in surprisingly good condition sometimes turn up in caves, which tend to have constant temperature and humidity levels.
(But, yeah, current electronic media doesn't have much chance, with select optical disk media stored under carefully chosen conditions offering the best chance for your files being retrievable decades later, if you can find a drive to read them on.)
BRB, etching all my MP3s into clay tablets
Yeah, I was going to say that we know that Ea-nasir's copper was shitty.
Obviously not everything from 1750 BC survived, but we do know that certain mediums are more likely to stand the test of time than others. Something physical with the writing carved in? That will probably last. Something with pigment on vellum, that won't be quite as good, but you can store a lot more information per kg. Something involving bits? That won't last even a quarter century. Something involving bits stored using magnetism and retrieved using mechanical motion? Good luck keeping that for even a decade.
But, the thing we've shown will 100% stand the test of time is keeping the information flowing, though at the cost of some degradation. In the past, this was one generation telling stories to the next. When that happens, not only does the information get passed on, the language used is subtly updated in time with the evolution of the language. You don't need to learn Akkadian cuneiform to read it, it's available in whatever the modern language is. Similarly, if digital files keep getting passed around, it doesn't matter if the original came on a floppy disk, and floppy disk readers are now gone. The file exists, stored in whatever medium is current. But, you get degradation with this process too. Music might be turned into mp3s with some data getting lost. Photos might be resized, cropped, recompressed, etc.
If I wanted something to be preserved exactly as-is for centuries, I'd carve it into a non-precious metal (so nobody melted it down). If I wanted something to be easily accessible for centuries, I'd try to share it as widely as possible to keep it "in motion" and in a format that was constantly up to date.
Citing Alessandro Barbero
Updoot just for this
But we do have originals of many other texts. Like vast, vast amounts of cuneiform texts because they were usually pressed into a clay tablet and then baked.
All but four Mayan paper codices were burned, but the Mayans loved carving their stories into rocks and making those rocks part of their cities' architecture, so we still have a lot of their textual information. Same with the Egyptians- they wrote a lot on papyrus, and most of that is lost, but they also carved and painted all over tombs.
The secret is not to keep copying the texts. That introduces errors and those errors can compound. The secret is to preserve the texts in a medium which will only degrade over exceptionally long periods of time (compared to a human life, anyway).
If you had a device which could carve stored data into stone and make it retrievable again, you could potentially preserve that data for thousands of years.
Sad little human. I have written my treatises into the warp and weft of reality itself. I have twisted my curiosity into the folds of your DNA and stamped my waxing madness into the ragged edges of the telomeres that mark your days as numbered. I have made of the stars a celestial QR code that burns across the skies of every planet, that burns across the eyes of every ape who stares into the night and asks "why?". I announced The Work with a bang of gas and light and awe and set time itself into motion so my scripture could expand eternally into the infinite, benighted expanse.
Yeah if you're looking for long term it needs to be archival media. Many people think the flash drive will hold it forever but they are potentially the most fickle.
But what actually is "archival"?
Like, what technology normal person has access to counts at least as enthusiast level archival?
Magnetic tape, optical media, flash, HDD all rot away, potentially within frighteningly short timeframes and often with subtle bitrot.
Probably M Disk as others have said for consumer use, but Microsoft is working on storing data in glass that could last for potentially 100,000 years +. Not that you’d ever likely have that in your home. Although, maybe by 2100 we will, who knows. https://www.microsoft.com/en-us/research/project/project-silica/
Hard drives offer the best price/capacity ratio, but they need to be powered periodically (at least once or twice per year). As with any other storage medium, include parity data and have multiple backups to avoid data loss.
Tape is too expensive.
Optical media can also be pretty good as long as you get discs made from inorganic materials and store them properly. M-disc is supposed to last like 100 years. The biggest problem is that they are on the path to obsolescence and optical drives may stop being manufactured. Also, it's a good idea to check on the condition of the discs periodically and redo any that shows signs of degradation (probably a good idea to replace non-M discs every 10 years regardless).
But regardless of the media, there is no archival method that doesn't require active maintenance, like periodically checking the data, ensuring you have multiple backups, refreshing any aged media.
There is also this thing called electron detrapping
Im really hoping, waiting, for a good dense long-term storage medium. It doesn't have to be fast, but large, cheap, and durable. I want a way to backup my plex library, or even, daily backups of documents and project files, and I don't want to think about them ever again.
It does has to be fast enough that you can transfer the files to a different disk within your lifetime.
Tape is cheap and durable if you store it properly. Except the tape drive is expensive af.
Microsoft is working on glass storage. A glass plate can last 10,000 years according to Microsoft. Hopefully that tech will get miniaturized and available to consumers within our lifetimes.
Punch cards? Stored correctly there's no reason they couldn't last many human lifetimes. But... Yeah it'll take a while to encode everything.
I would have thought that with modern technology we could come up with something like punch cards but more space/time efficient. Physical storage of data - only one write cycle of course, but extremely durable. Even just the same system as punch cards but using tiny, tiny holes very close together on a roll of paper. Could be punched or read by a machine at high speed (compared to a regular punch card, presumably still Ber slow compared to flash media).
Do optical disks degrade if protected from the elements? A stack of Blu-ray disks could store quite a bit.
Depends on the disk technologies.
CDs are prone to flaking, otherwise most disk are suspectible of oxidation (disk rot) if stored improperly. M-Disk is a long-life variant of both DVDs and Blu-Rays, although more expensive. However, write-once disks are very ransomware-resilient, and I recommend to add a write-once media to any proper backup setup.
M-Disk is rated to last like
1001000 years. They are also working on a 125 Terabyte CD. Optical storage is the way to go.It certainly would make planning the dang home lab easier. Im in a small place! I don't have room for all the stuff i wanna play with!
I think you're looking for stone tablets.
Archive discs are that but not exactly cheap.
To me, this is just another story of the music industry's technical incompetence.
Even in the 1990s, everyone would have known that hard drives were not a long-term archival storage solution. This is like crumpling up a piece of paper, tossing it in the corner, then being upset decades later when your "archival solution" had issues.
The piece of paper is basically much more likely to survive in a corner barring a fire. I have crumpled pieces of paper from 20 years ago. My PATA hard drive... I don't even have a computer with that connector
A bunch of paper tossed into a corner could get wet, mouldy, get munched on by rats, etc. But, I know what you mean. Spinning plates full of magnetized bits with a connector technology that only lasts a decade at most is hardly going to be reliable, even if stored under ideal conditions.
Not to mention bit rot
I'm sure they considered anyone telling them they needed to spend money to be a pain in the ass the same way companies don't follow the recommendations of their IT departments.
The problem isn't even the hard drives, it's how they are managing them. There's not many digital data storage solutions around that you can dump into a closet for a few decades and then still read.
You have to regularly test your hard drives, so that when one fails you can take your other copy of the data and put it on a new drive.
It's not a bad start though, and how hard would it be for the people who have surviving copies to copy it to "the next best new thing" in 10 years? The problem is of course that sequence would need to be continued on, like a tradition, which is doable as well.
Honestly in a use case like family photos, redoing it every x amount of time is probably a good idea anyway so new ones can be added.
how music is so complicated to archive now
??
That's why I back up my data on stone tablets in Cunieform.
Seriously though, if you wanted data to last for centuries, what would be your best bet? Would it be some sort of 3D-printed mechanical storage? At least plastics are generally not biodegradable, though they are photodegradable, so I guess you'd want to stick your archive in a dry cave somewhere?
Or what about this idea of encoding the data in the DNA of some microbe and cutting it loose? What could possibly go wrong?
Archival grade m-discs, apparently. I had the same question. https://www.howtogeek.com/858426/whats-the-best-way-to-store-data-for-decades-or-centuries/
It's a shame their capacity lags so far behind current hard drives. And not many drives for these discs are still made, so what are the chances of them becoming unreadable just because no one has equipment to read them?
Cool! I can see how optical media could, in theory, be very long-lasting as long as you don't use materials that oxidize or otherwise degrade over time.
Well, nothing lasts forever. I'd say distributing them on something that lasts 10+ years is better than doing noting. Otherwise they just get lost, buried in the attic or the next harddrive crash takes them.
In my life I've had several HDD's and SSD's fail on me. None of that shit lasts forever and you would honestly be surprised how often they fail. For huge servers like data centers the average failure rate is between 1% and 2% per year, and sometimes even higher. They literally have to replace hard drives on a daily basis because of failure.
Disk failure is one of the reasons I have copies of all of my important stuff saved to every hard drive on my computer, along with an external just in case. One time my system drive failed and I couldn't save anything because my operating system stopped working. Had to reinstall the OS and do a data recovery. Ended up with a bunch of photos that were corrupted, distorted, or missing half the photo.
I just replaced a drive today that has been running 24/7 for the last 5.18 years. It hadn’t failed yet but was in a predictive failure state due to the amount of bad sectors it has accumulated. 3 other disks in the same giant raid array are showing some bad sectors but not enough to be over the ‘replace me now’ threshold.
Buy Spinrite. It's not perfect but it's the best thing available for drive maintenance and recovery. I have used it for over 10 years. If the drive is dying it'll take forever, but I've recovered data that was nearly gone due to sector loss. It goes down to the bit level BTW. Someday Steve will release v7 .. someday.
This is the first time I've seen anyone else talk about SpinRite, known about it a long time too
If your data is really important, you should send it to a reputable data recovery service. Using the drive any more (even with a tool like SpinRite) risks further damage.
These people haven't heard of zfs then?
Zfs is just software raid, not an archival /backup solution. Sure, you can hold data on a zfs array for long term, but not without active maintenance (powering the drives periodically, replacing old drives, doing some kind of data refresh / scrubbing) and backups.
My system is to duplicate to fresh media once in a while. It's more hands on, but it's the only option I have. My NAS will be cloned to new drives in the next few years.
You might want to look at snapraid. I've recently overhauled my own NAS and love it. It is snapshot based (so not perfect safety) but it is highly configurable and provides parity and scrubbing for corruption even with a JBOD array.
I'll check that out. Thanks.
That's the way to do it. Just keep bringing data forward even for media that sits in a drawer.
I have a feeling USB drives will be readable for a long time to come, considering that we still use the standard almoat everywhere, nearly 28 years after its introduction.
That said, copying the data from old archives into new formats is always a good idea
Edit: I was envisioning actual external hard disk or solid state drives accessible using a USB connection. Thumb drives and other ultra-portable data formats are notorious for poor data integrity over time.
I have a feeling USB drives will be readable for a long time to come
The drives may still be useful, but the data will long be gone. I have a flash drive from 06 that was last used in like 08 09 and most of it's data was gone by 2019. The drive itself is fine. But what was on it has slowly faded away.
Note though that usb Sticks (if you mean that with USB drives) usually get the worst, cheapest quality flash. It's a lottery, and expect nothing in no-name sticks.
* quality in yield, ssd > sd-card/emmc > flash sticks
I've got several flash drives from 2 different brands that are less than 3 years old that are already dying. SanDisk and Lexar, both large brands and they're already losing reliability. Real shame that the older stuff works longer. I really only need them for making Linux bootable drives and they struggle with that.
That shouldnt be all doom like.
Easiest thing is to make 2-3 copies of usbs and label the date the usb began its use, put a rough estimate on when the next generation should repeat the whole process until the pictures become irrelevent.
USB sticks and SSDs are no good for long-term storage. The data on them degrades rapidly if they're not powered up. Spinning disks last longer. So your process would be better done with those.
Thats even better i didnt know that
I still have a dvdr and good quality discs stored properly can last decades. Not perfect and harder and harder to find readers, but for long term backups its an affordable solution.
My family data vault has three dvdrs still in the box just in case 40 years from now you can't find a drive.
That said, we will probably be moved beyond SATA then so.....