Vint Cerf Warns of Forgotten Century From Bit Rot
One of the pioneers of the Internet warns that we're in danger of losing entire generations' worth of history because of digitization.
One of the pioneers of the Internet warns that we’re in danger of losing entire generations’ worth of history because of digitization.
The Guardian (“Google boss warns of ‘forgotten century’ with email and photos at risk“):
Piles of digitised material – from blogs, tweets, pictures and videos, to official documents such as court rulings and emails – may be lost forever because the programs needed to view them will become defunct, Google’s vice-president has warned.
Humanity’s first steps into the digital world could be lost to future historians, Vint Cerf told the American Association for the Advancement of Science’s annual meeting in San Jose, California, warning that we faced a “forgotten generation, or even a forgotten century” through what he called “bit rot”, where old computer files become useless junk.
Cerf called for the development of “digital vellum” to preserve old software and hardware so that out-of-date files could be recovered no matter how old they are.
“When you think about the quantity of documentation from our daily lives that is captured in digital form, like our interactions by email, people’s tweets, and all of the world wide web, it’s clear that we stand to lose an awful lot of our history,” he said.
“We don’t want our digital lives to fade away. If we want to preserve them, we need to make sure that the digital objects we create today can still be rendered far into the future,” he added.
The warning highlights an irony at the heart of modern technology, where music, photos, letters and other documents are digitised in the hope of ensuring their long-term survival. But while researchers are making progress in storing digital files for centuries, the programs and hardware needed to make sense of the files are continually falling out of use.
“We are nonchalantly throwing all of our data into what could become an information black hole without realising it. We digitise things because we think we will preserve them, but what we don’t understand is that unless we take other steps, those digital versions may not be any better, and may even be worse, than the artefacts that we digitised,” Cerf told the Guardian. “If there are photos you really care about, print them out.”
Cerf is, needless to say, much more versed in the vagaries of technology than I am. But I can’t offhand think of any widespread photo, video, or music file type that has become unviewable and can’t imagine them ever becoming unviewable. Even if my software or operating system were to go away, the files themselves would still be there.
It’s true that floppy discs are all but unreadable, Ditto ZipDrive and various other short-lived storage media. And even CD-ROMs, which were supposed to last forever, degrade in relatively short order. But most of us are backing stuff up on both hard drives—a terabyte is now commonplace and dirt cheap—and the cloud.
More problematic is the fact that various wildly popular sites and programs go out of business on a regular basis. The old BlogSpot version of OTB went away a long time ago (although, thankfully, I ported all the posts over to this domain at the same time I moved the site here). All of the stuff that was on my old Google Reader archive is lost to me. It wouldn’t shock me if that were to happen with Facebook and Twitter at some point.
I suppose none of this would matter to future historians or archaeologists. For them, hard copies would be much better, since they’d be much less likely to have the ability to access people’s digital lives. Then again, we’re preserving a whole lot more of our lives than ever since photos, emails, video, etc. are essentially free to produce.
There’s two different problems being brought up here: format obsolescence (I can’t read this VHS tape because I don’t have a VCR anymore) and bit rot (I can plug this USB drive in, but the data it used to contain is gone). Each requires different solutions.
“And even CD-ROMs, which were supposed to last forever, degrade in relatively short order. But most of us are backing stuff up on both hard drives”
Hard drives have an even shorter functional life than CD-ROMs. I lost all of my college writings and other files to hard drive crashes over the years. And that was back in the days before “the cloud” when there wasn’t an easy or cheap way to back things up anywhere. I literally have no record of my life between 1996 and 2004, aside from a few pictures.
Cerf is right.
@Ben: Sorry to hear that. I’ve had motherboard failures and hard drive issues but never something so bad I couldn’t recover data. I back up my most important stuff to an external hard drive and the cloud. But there’s always risk of data loss–whether of the type Cerf describes or others hacking into the stuff on the cloud.
@James Joyner:
Well sure, now I back everything up in triplicate, but back in the last 90s, I was in college and it never occurred to me. It definitely made me realize that as virtual as “the Internet” feels, it still all depends on physical media that has a shelf life on it.
Think about this: I have boxes of photos made from mostly 35 mm film and the negatives are there also. These are anywhere from 10- 30 years old and in perfect condition. I can take the negatives to most photo shops and get perfect copies. I used to have a lot of old home movies (8 mm, Kodak camera) that dated to the ’50’s and were in great condition, but I got rid of them when VHS came out. The old Kodak movie camera is still around here somewhere. There are glass negatives from the 1800’s around in old storage and warehouses various places around the country , long forgotten, but still usable for prints. Most movie theaters have switched to digital from 35 and 70 mm film. The question is what will happen to all that film? Long ago the movie studios discovered that a lot of their films had deteriorated beyond saving so a lot of classics were lost forever. Kodak had an alternate film base that did not deteriorate. Maybe someone who is more familiar with all this can throw in on it. It seems to me that this preservation challenge could be an opportunity for Kodak. I hate that they are out of the camera field. I always found Kodak to be reliable.
I remember the good old days of being in the classroom and the old 16mm projectors were used, clattering and jumping away as we watched documentaries, Disney nature movies, and “Victory At Sea”. Video tape and DVDs just don’t have the same feel and character.
I have multiple eternal hard drive backups of pictures, videos, and other important files, but not good multiple locations. I’m reluctant to use the cloud due to security issues, but I’m not sure anyone would find anything very exciting about my family pictures, so I’m starting to lean that way for videos and photos.
A couple of years ago I purchased a VCR/DVD combo for dubbing and converted my VHS home videos and 8mm DAT video’s to DVD then to computer .ISO files. I also had some Super 8mm film tapes (from 1985-1988) converted to digital before they were lost.
@Ben: There was an easy and “cheap” way. Buy two hard drives or run a RAID. I have stuff I did in the early 90s still
@rodney dill: Our old 8mm films still showed like new, but in a hasty, thoughtless move I tossed them and the projector. Even worse, I did not copy them to video. My mistake.
I still have floppy discs in boxes.
Things that children have no idea what I am talking about: film, develop, negatives, floppy disc, rewind. You can probably name a few.
I know a man who still uses a stereo system with a turntable and 8 track player. I think it is one of those Realistic models in the brushed metal look of the ’80’s.
@Matt: It all depends on what you’re willing to do to preserve your data. I’d have to have four simultaneous hard drive failures to lose mine, and even then I would still have most of it up to about 18 months ago.
But I’m insane. (Or have lived through one too many scares, if you prefer.)
The real threat for me at this point is if the house burns down, because I don’t have a separate site yet. But that was just as vulnerable when everything was physical.
Depends what you mean by “widespread”.
I know a guy who used to work in the computer center at NIH. Every so often, he’d get a call from some poor researcher who had inherited a pile of data tapes, or discs, or what have you. Digital data, in an undocumented format, containing the results of some lengthy and expensive experiment or clinical trial. If my guy couldn’t figure out both how the data was encoded and how to interpret the resulting columns of numbers, then those data (and thus those experiments) were lost.
Stormy Dragon is right — being able to get the bits off the medium and being able to interpret the bits are separate problems. My guy occasionally had to build custom hardware to read the obsolete tape/disc/whatever. There, you get into all kinds of lower-level issues about how bits were encoded, spaced, compressed, etc.
@Trumwill: I keep one master copy of all my machines in a highly fire resistant safe for that reason. Short of dropping thermite on it my data is safe.
Just think of what would happen if some sort of electron blast hit the country, either from the sun or some country like North Korea or Iran. All the medical, financial, security, educational, statistical, and government data would be wiped out as everything electrical and computerized would be cooked. The good thing would be that all of the IRS data would be lost. It would be like everyone starting over clean ! No more back taxes ! But that would be the least of the problems because it would take a few years to rebuild the power grid.
Backing up six ways from Sunday is well and good but how about three generations down the road? I am working with family documents 130 yrs old that luckily were preserved. I greatly doubt that 130 years from now anybody in my family will be working with my digitized stuff. They won’t even know it exists, assuming it really does. I had my own darkroom, took over 30,000 Kute Kid pictures. Bound the best of them into 32 100 print albums by binding the 8×10 prints and giving each kid a digtized copy. The albums will last. The negatives and proof sheets will last. The digitized copies won’t. .
One thing to remember is that almost all stuff has generally been lost. For every box of family letters kept for 50 or 100 years, there are dozens of boxes lost, or letters never saved.
Microsoft is guilty of not having built-in software to allow opening archived Outlook Express emails (.eml files).
You might think that, since it was a MS program that created that file, their newest-&-greatest version of Outlook MIGHT have the built-in capability of opening them, OR their operating system (7, 8, 10) would have that feature.
(YES, I know that no version of Outlook will open those Outlook Express archived messages, but at a certain point, don’t you think that one of their software engineers should say, ‘Let’s build in a reader for old message types.’)
When I consider that I can walk into the British Library and ask for a law treatise from 1560…..
There’s a reason I’m a Luddite.
Let’s face it, much of what individuals hold on to is not of any historic importance. Two generations from now, or maybe one, nobody is going to care about my photos, emails, or anything else I did. If I’m lucky somebody will remember my name for a while. Get over yourselves.