Safe archive solutions for the future "Digital Dark Age"
"Digital Information lasts Forever, or Five Years, Whichever comes First."
Jeff Rothenberg Download this document as PDF:  Synopsis
This paper addresses the advantages and disadvantages of digital techniques for the archiving industry. Clearly digital technology offers unprecedented accessibility to information via Internet. The ability to automate the indexing of subject matter also improves access substantially. However, computers and digital media are far from stable as media. There have been many catastrophic instances of digital media becoming lost or inaccessible. Digital technology continues to advance at a phenomenal rate making digital hardware and formats obsolete in comparatively short times, often as short as a few years. This puts a heavy load on archives to maintain access to their growing digital resource despite advances and changes in technology.
Two media have stood the test of time. One is paper and the other photographic film. Film is ideal for archives since it’s a very compact storage medium suited both to documents (as proved with microfilm) and images. The high resolution digital film recorder is a relatively simple machine which can be easily interfaced to digital computer systems to place standardized images on film. Suitably stored archival film can be accessed again at will and by using a companion film scanner the heritage of our civilization can be preserved and saved from the threat of a “Digital Dark Age” where generations of irreplaceable digital documents could become lost after as short a time as 10 or 20 years.
The Digital Future of Archives
For years now, enormous efforts have been made to digitalise the contents of the world’s archives.
These culturally, scientifically or socio-politically important documents, images or photographs are now emerging into the digital age.
Along with thousands of other documents, valuable historical manuscripts and incunabula get scanned, reworked on the screen to increase legibility,
then catalogued and stored. Libraries, museums, national and regional archives, and both public and private institutions are all part of this process. All these efforts have three main goals: the long-term digital preservation of documents easier access to them their distribution by electronic means (networks, internet).
Large amounts of human and financial resources are being devoted to these tasks and new technical methods are being developed. The possibilities range from high performance book scanners which can automatically turn the pages to robotic systems for changing the storage media (tape storage banks or optical media) in order to permit rapid, automatic access to a rapidly growing mass of data. Naturally, this digitalisation of the archives proceeds step by step, usually beginning with the catalogues and indices. The actual contents of the archives then follow. Digitalisation of the catalogues is especially important for libraries and archives because it makes it possible for outsiders to search through their holdings via internet. |
Solutions for Archive Digitalisation
Digitally archiving documents requires a number of different tools, all of which are indispensable: Acquisition devices such as scanners or cameras used for paper documents or visual media such as negatives or slides. Processing the data and converting into an appropriate file format. Relevant for these processes are such graphic formats as TIFF or JPEG or special document exchange formats such as Adobe PDF. In future, the XML format will possibly have increasing importance, but probably only for new documents already produced in this format. Storage of data on appropriate storage media such as tape storage banks, magnetic disk storage (fixed disks) or optical storage media (CD-ROM, DVD etc.). There are two basic types of data storage. One is ongoing data storage while the documents are being processed. Here, appropriate backup systems are used, generally speaking tape storage banks. For permanent storage, the data is then put onto media which can remain stable over long periods of time, especially optical storage media. Making name and subject indices for the stored material. Distribution of the electronic contents, either physically or passing them on through such media as CD-ROMS, networks or the internet.
All of these tools are now very highly advanced and new, even more efficient solutions are constantly being found for individual elements of the process. All in all, the workflows for electronic archiving have shown themselves to be highly effective and efficient. Advantages of Digitally Archiving Documents Very old, historically important documents can be processed on the computer so as to make them more legible the safe, long-term storage of the documents in digital form the distribution possibilities through electronic communication media digitalised documents can be easily transported and distributed without loss the ’democritising‘ of information Digitalised documents offer the possibility of access to texts and using automatic computer tools.
There are numerous advantages of digitally archiving paper (or parchment) documents as well as negatives, slides or photographic prints. One of these advantages is that very old, historically important documents can be processed on the computer so as to make them more legible and easier to use. Another is that when printed documents are digitalised using OCR (Optical Character Recognition) they can be converted into texts capable of computer editing. This makes subsequent processing (e.g. automatic index systems) possible.
But, naturally, there are two other aspects which have priority:
- the safe, long-term storage of the documents in digital form
- the distribution possibilities through electronic communication media so that any document, no matter where its location is available all over the world, 24 hours a day
Safe storage in digital form offers libraries, archives etc. a whole series of interesting advantages. Digital representations of original documents are unchangeable and durable, i.e. they don’t change over time and remain in their original digitalised state This is an enormous advantage when one compares them to any of the storage media previously available: paper, film, clay tablets, stone etc. Pigments and inks on paper, paint on stone or the colour elements in film emulsion can all become discoloured or fainter with time. In addition the material base of the document can be subject to pollutants or other materials in the environment which can result in decay or disintegration. Especially in the last 10 to 20 years one of the most important topics in archive preservation has been the acidic decomposition of paper. But even apparently durable materials such as stone are increasingly endangered by environmental pollution.
Digital information as such remains uneffected by such factors and even a digital code which is hard to read still retains its unchanged information content that is, as long as it is legible at all.
An essential advantage of digitalised documents is that they are light and can be easily transported and distributed without loss. Whereas for example an analog fax message is clearly “worse” than the original, and contains not only less information but also additional extraneous information, a file which is sent through the internet always retains its complete store of information, without loss or disturbance. Even if several hundred people receive such a message, they will receive an identical representation of the original. If, on the other hand, one would send off a hundred thousand photocopies or photographic prints, each would have tiny flaws and would in some way differ from the original. and, moreover, each in a different way.
The idea of “democritising” information has at its heart the theoretical possibility of making all the important documents of human history available for use by all mankind.
Finally, there is a third basic advantage. Digitalised documents offer the possibility of access to texts and using automatic computer tools for word counts or indices for linguistic purposes. This has enormous scholarly potential and can be used, for example, to simplify the attribution of particular texts to particular authors. The Digital Dark Age
But despite all the advantages, digitalisation also has its disadvantages and problems. The worst of these possibilities has come to be known as the “Digital Dark Age”, a scenario in which humankind would lose its entire cultural heritage, to the extent that it was stored digitally.
Here are three impressive examples to illustrate the problem:
NASA is no longer able to read data from the Landsat satellite group that was acquired in the 60’s and 70’s. In 2002 as a result of the death of an employee the password for the catalogue data bank of Norway’s Ivar Aasen Centre of Language and Culture was lost. Only after a worldwide discussion on the internet, an expert from Sweden was found who could break the password and make the catalogue and its 14,000 entries accessible again. Researchers at the University of Southern California wanted to analyse the images from the Viking Mars Probe (1976). But they were unable to do so because the data were stored on old magnetic tapes and in a format that is unknown today. Instead, students had to go through the laborious (and quality-reducing) process of reconstructing the data from print-outs of the original. But let’s take a closer look at the situation and try to imagine ways to avoid the horrors of the Digital Dark Age.
Digital Data Archiving Gets Neglected
In business as in private life the long-term storage is neglected. Documents of considerable historical and sociological importance will be lost. Even in professional archiving institutions the digital archives all have a astonishingly short ‚shelf-life‘.
It becomes necessary after 10-15 years to transfer the entire digital archive from one storage system to another with increasing costs.
Today, many documents are written electronically and then get stored without a print-out ever being made. Often no attention at all is paid to durable, long-term storage. Especially in private firms there is a tendency to simply assume that business documents have been safely stored. And this safety is also usually present, based on continual back-ups stored on tape. But this is only a very short-term storage, intended to prevent the economic damage resulting from the loss of current operative data. Long-term archiving is another matter altogether. Oftentimes, business correspondence isn’t included even in this limited type of storage because it exists only in the e-mail system. As a result, it is estimated that in Great Britain alone more than 60,000 business documents get lost every year simply because they haven’t been archived.
In private life this is even more true because long-term storage is rare or non-existent. And even though these private documents may be of considerable historical or sociological importance, safe storage of them is neglected because most people are unaware of their potential value. As a result, the enormous store of cultural information we have from the past in the form of diaries and letters has practically dried up since the advent of e-mail. Most of today’s letters get automatically deleted.
The development of digital photography has made things even worse. Today, many image documents of historical and cultural value are no longer recorded on analog film material. Instead, primarily in order to reduce costs and increase speed, they are recorded digitally. Through advances in camera technology, these digital photographs are of very high quality. But these digital data also require reliable long-term storage. This applies not only to photographers, but to commercial or institutional archives as well.
One would naturally assume safe long-term archiving is a matter-of-course for libraries, museums and public archives. Unfortunately, this isn’t the case. Digital archives all have an astonishingly short “shelf-life”. This means that while traditional archives based on paper can survive for centuries, the average life expectancy of a digital archive is only 10-15 years.
If one wants to keep the archive materials accessible, it then becomes necessary after 10- 15 years to transfer the entire digital archive from one storage system to another. This naturally means that every time there will be increased costs, not only for the new technology but for the transfer itself.
When one considers these costs, then the costs for shelving and space in traditional paper and film archives don’t seem quite so expensive after all.
Digital Media with an Expiry Date
Life expectancy of storage media at 20° C and 40% relative humidity
Table: No matter what the storage medium, digital storage media have a life span which ranges from only five years to a maximum of 15. In contrast to this, analog media (especially if they have been conceived for long-term archiving) can last for 100 years and more. This is without taking into consideration that such media as film material, if stored under the right conditions (i.e. monitored refrigeration) can remain usable for several centuries.
Source: National media Lab, in „Digital Information lasts Forever – or Five Years, Whichever comes First.“ Josef Rothenberg, October 2nd, 2001 Digital storage media age and at a much faster rate than analog ones. CD-ROMs have been and still are one of the most popular storage media, yet they are among the least durable media we have. Egyptian papyrus or documents of medieval Europe survived for centuries. Quite apart from the continuous and unavoidable operating costs resulting from having to transfer data to new systems, digital archiving has a built-in flaw that inevitably produces problems: Digital storage media age and at a much faster rate than analog ones Let’s take a look at the usual storage media: magnetic media, whether on tape or hard-disk optical storage media magneto-optic storage media holographic storage media And then let’s compare their typical life span with that of analog storage media such as acid-free paper and silver or pigment based film material. Even though the chart given below is based on very conservative estimates of the life span of analog archiving media, it makes one thing abundantly clear: Digital storage media have a very short life span of only 10-15 years. When one considers the time frame within which archives have to think, these digital media can, at best, be only a temporary solution.
In addition to the technological reasons why digital media have to be transferred to new systems every year, there are physical reasons as well. These media simply have a very short life span. This is another reason why digital systems not only run the risk of data being lost altogether, they will also be the cause of an explosion of archiving costs in the coming decades and centuries. CD-ROMs have been and still are one of the most popular storage media, yet they are among the least durable media we have. Under normal storage conditions, they last for a maximum of 10 years and even under ideal storage conditions they survive for at most 50 years.
Even the holographic storage media which are being developed today and are intended to serve the needs of image archiving will only be able to offer a life span of 50 years. If one compares this with the durability of Egyptian papyrus or documents of medieval Europe such as the Domesday Book (1085), we can see that when it comes to long-term safe storage digital and analog systems are worlds apart. Missing Storage Systems and Computers
Storage media develops as fast as processors.
The replacement of computer generations is significant in a number of ways:
Graphic: Already in 1965 Dr. Gordon Moore predicted the exponential growth of the integration density of microchips and formulated the thesis named after him which says that processor capacity (and with it the capacity of computers in general double every few years. Developments since 1965 have proven him right.
Source: www.intel.com/research/silicon/mooreslaw.htm - when a system vanishes, data formats are lost
- when a system vanishes, connections and system buses are lost
- whenever a new system comes onto the market, this increases the pressure for change and the elimination of older systems.
But even if the storage media are able to survive over a long period of time, that’s still no guarantee that the data can still be used. And this despite the fact that developments in the area of data storage technology have not been quite so rapid as with computers themselves. According to “Moore’s law”, processor capacity doubles every few years, something that has been happening for some 40 years now. As a result, every three years a new generation of computers comes into being that has so surpassed the previous one that the older systems have to be gradually replaced. The causes for this are the steadily increasing demands of the software systems used as well as the enormous increase in data flow.
What does this mean for the long-term archiving of data? This replacement of computer generations is significant in a number of ways:
- when a system vanishes, data formats are lost
- when a system vanishes, connections and system buses are lost
- whenever a new system comes onto the market, this increases the pressure for change and the elimination of older systems.
In the past four decades, there have been more than enough examples for the consequences of all this. Magnetic tape in the old inch formats or 8-inch floppy disks are hardly used at all today. This is true of storage systems like the Shugart-Bus hard disks as well as for magnetic change media. Who today is still able to use such things as Syquest 44 MB and 88 MB or Bernouilli Drive media?
In principle, there is of course always some way or other of saving the data from older formats or storage media. But the costs of this are astronomical. In fact they are so high that they are beyond the budgets of the institutions responsible for archiving. Think of the huge sums the US government had to spend in order to reconstruct the e-mail correspondence relating to the Iran-Contra affair. What library or museum or archive has that kind of money to spend? In 1993 the Center for Electronic Records in the National Archives and Records Administration (NARA) did nothing else for months except to find the relevant data stored on 5,700 back-up tapes and 150 hard disks.
Digital File Formats That Everyone Has Forgotten
Not only the physical durability of the media themselves and the usability of the storage media is limited, but also the file formats used for the information also change roughly every ten years.
Practically speaking, this means that every ten years when the inevitable transfer of data takes place, a certain proportion of the data won’t be converted and in time will become unusable. This is of course the very opposite of what archives, museums and libraries were established for in the first place.
So far we have established two sets of problems which limit the life span of digital data to ca. 10 years: the physical durability of the media themselves and the usability of the storage media. There is, however, a third aspect which shouldn’t be forgotten. The file formats used for this information also change roughly every ten years. We’ve already mentioned one example, namely the no longer usable image data from the NASA’s Viking Mars Probe.
To stay with this particular aspect of the problem for a moment, no one really has any idea just how many imaging formats actually exist. The software graphic converter can deal with some 175 different formats and still covers only a small part of the total picture. Especially in academic contexts, but in industry and defence as well, there dozens or even hundreds of patented formats for special uses.
Even when such a widely used graphic format as TIFF (Tagged Image File Format) is employed, that doesn’t necessarily mean that every “TIFF-compatible” file can necessarily be opened. Even TIFF has a variety of different versions. And if people want to work with such scientific special formats as FITS, they will have to rely more or less entirely on a particular software application. But they can never be sure whether this application can still be used when (every three to five years) their computer system gets replaced.
Here too one has no other choice but to convert the entire data bank into a new format whenever there is a new version or a change in the system. This is so costly and labor intensive that it’s simply impossible to do it for all the data and often only the “really important” data can be converted. Practically speaking, this means that every ten years when the inevitable transfer of data takes place, a certain proportion of the data won’t be converted and in time will become unusable. This is of course the very opposite of what archives, museums and libraries were established for in the first place.
In addition, this transfer of data also results in data being lost. Just one example: If the existing data have been stored in a data format like the JPEG which is particularly compact yet offers very fast access, then a transfer to the JPEG2000 (which is in preparation) means that the data will have to be compressed yet once again. The result is that information is lost and new artefacts are added. In other words, every transfer means a distortion of digital data.
Global Networks Have Failed as a Universal Archive
During the internet euphoria of the early 1990’s people liked to think of the global internet as the kind of universally accessible repository of human knowledge they had been dreaming of a sort of universal archive.
Regarding the millions and millions of documents in the internet: . How does one go about searching through the internet to find really relevant data? In short: The internet is simply unable to solve the problems resulting from the regular changes in file formats and the consequences of outmoded or lacking software.During the internet euphoria of the early 1990’s people liked to think of the global internet as the kind of universally accessible repository of human knowledge they had been dreaming of a sort of universal archive.
In the meantime, there is a certain degree of disenchantment as British experience (see above) has shown. Moreover, the very success of the internet has also proven to be its greatest flaw. Whereas in the year 2000 there were roughly 414 million documents on line, there are now more than a trillion. In other words, an increase of 150% in four years. And for the 12 months until 2005 an increase of 20% is predicted.
Apart from the fact that there is a continuous turnover of the data which naturally results in the same sort of problems that any archive has to deal with when trying to store data for long periods of time, there is an additional problem: How does one go about searching through the internet to find really relevant data? Today, we can assume that at least 80% of the information found on the internet is either superfluous, nonsensical or simply wrong. Picking out what one really needs from this huge mass of data is becoming more and more difficult. Taskforces all over the world are looking for new ways of searching for, and above all finding, documents. These efforts have proven impractical, however, because, by the time they are ready for use, the exponential growth of the world-wide net has made them obsolete.
In short: The internet is simply unable to solve the problems resulting from the regular changes in file formats and the consequences of outmoded or lacking software.
Analog Storage Media: Easy Use and Long-Term Stability
As we mentioned before, the oldest documents of human history are archived on such “storage media” as parchment or papyrus. Quite simply, they’ve already proven that they are a durable and reliable means of storing data. Digital storage media can have a life span of at most 50 years (and even that only under optimum conditions) and in addition, they can only be used if there is also a drive which can interpret the file format. Analog media on the other hand can offer centuries of safe storage.
Paper: the "Classical" Archival Medium
Today we know that properly stored paper of archival standards (i.e. acid-free) can last for many centuries. Even so, paper still has a few disadvantages:
- it requires a relatively large amount of storage space
- it interacts with pigments and printer’s ink in a way that can reduce the durability of color-printed paper
- access is best done digitally.
Today we know that properly stored paper of archival standards (i.e. acid-free) can last for many centuries. Even so, paper still has a few disadvantages:
- it requires a relatively large amount of storage space
- it interacts with pigments and printer’s ink in a way that can reduce the durability of color-printed paper
- access is best done digitally.
A disadvantage, however, is that paper is best suited as a vehicle for only one data format, namely paper documents. Any important document which didn’t originate on paper (e.g. photographs, paintings, drawings etc.) is ill-suited to being archived on paper because of the information loss which would be entailed.
In contrast to this, it is no problem at all to reproduce digital text data on paper, particularly since modern monochrome laser printers can offer a certain degree of durability. Unfortunately, this issue still requires more detailed research. Even so, due to paper’s leading roll as a long-term storage medium, one can assume at least several decades of safe storage for such print-outs. The renowned research laboratory Wilhelm Research has analyzed certain combinations of paper and ink for ink-jet printers and certified that they offer “long-term stability”. More specifically, this means a life-span of 70 to 100 years under proper storage conditions.
Archiving Film Materials: Centuries of Stability
Graphic: Film materials are available in a wide variety of formats. In addition to the usual photographs, there are a number of variations on offer, ranging from individual sheets for even larger formats (beginning with 9 x 12 cm) up to film rolls that are intended for use in long film or bulk cartridges and are especially productive. In comparison to paper the film as an archiving medium is much more flexible.
Film material offers a whole range of practical advantages:
- information stored on film is usually analog, so there are no technical difficulties in reading and using it.
- it has a great deal of flexibility in both use and distribution
- film is a highly economical storage medium
- film can reproduce almost any form of information
- the technology required to integrate film into a digital workflow is already available
- the space requirements of film material are much smaller than for paper
- through being filmed, digital data have long-term stability
Silver-halide or pigment based film material is much more flexible as an archival medium. First of all there is a wide variety of different types of film material which can be adapted to specific needs. In addition almost anything that needs to be archived can also be preserved on film.
Film material offers a whole range of practical advantages:
- information stored on film is usually analog, so there are no technical difficulties in reading and using it. Using a magnifying glass, even negatives can be viewed directly.
- it has a great deal of flexibility in both use and distribution, because copies and enlargements can be made at any time.
- film is a highly economical storage medium
- film can reproduce almost any form of information: paintings and graphics as well as photographs and even three-dimensional objects, e.g. photographs of buildings, sculptures, important sites etc.
- the technology required to integrate film into a digital workflow is already available. Digital data can also be stored analogically using film material which has the additional positive side-effect of easy accessibility
- the space requirements of film material are much smaller than for paper (cf. the microfilming of documents)
- through being filmed, digital data have long-term stability and no longer suffer from changes in file formats, computer systems or drives.
First, let’s take a look at the long-term stability of film material. This is an area where we have more than 170 years of practical experience. Some of the very first photographs ever made still exist, even though photographic chemistry was still in its infancy and they were never intended to be stored over long periods. In other words, the oldest photographs are already more than three times as old as even the best digital media can ever become.
Generally speaking black & white film (i.e. classical film material based on silver halide) has the longest life-span. But there are also certain color films which have comparable durability. The durability of film material depends to a large extent on its being processed in a way appropriate to archives (development, fixing, adequate film washing) as well as being kept in the right kind of storage environment. Damaging environmental influences have to be prevented, just as for digital media. Although the expenses in this connection are roughly the same for the two types of data storage, one shouldn’t forget that film material will remain usable for several hundred years!
In view of all these advantages, it’s not surprising that microfiche has been in use in archives all over the world for decades and even now it’s potential still hasn’t been exhausted.
Damaging environmental influences such as air pollution or destruction through light etc. can be prevented by subjecting the developed film material to special chemical processes (i.e. selenium tinting or brown tinting) so that the photographic silver (film’s "weak point") can be replaced by a more stable substance. Black & white film treated in this way has an almost unlimited life-span. It’s also more serviceable, because it is no longer susceptible to light.
In addition, there are now low-temperature archives for photographic material where, at a temperature of –20 C, particularly valuable data can be stored. There are any number of these archives such as in Hollywood or at the Corbis high-security archive in Pennsylvania. The life span of material stored in this kind of an environment is limited only by the life of the archive itself. |
The Best of Both Worlds: From Digital to Analog
Despite flexibility, easy distribution and relatively economical storage media, digital data still have a number of important disadvantages.
Long-term storage of digital data is only possible through continuous and costly data transfer to new storage and computer systems.
Analog storage and the digital world can be combined by using a tried and true technology that is also highly efficient.
Despite flexibility, easy distribution and relatively economical storage media, digital data still have a number of important disadvantages: - a maximum life-span of 50 years
- "illegibility" of the file formats and data media every ten years as a result of continuous technological change.
Long-term storage of digital data is only possible through continuous data transfer to new storage and computer systems and the necessary data conversion this involves. This results in costs which are completely beyond the means of most archiving institutions. In contrast, the greatest advantage of archiving with film material is its long-term stability. It could be the keystone to a link between digital data and analog storage.
The best thing about it is that analog storage and the digital world can be combined by using a tried and true technology that is also highly efficient. And what’s more, it’s comparatively economical. All over the world there are a whole series of archives and libraries using archiving systems that are based upon an interaction between film material and digital equipment, i.e. a direct connection between film recorders and computers. This film recording technology is for example in use in Hollywood’s film industry and is widespread among professional photographers and advertising agencies. Now it is also available to smaller institutions.
Film Recorders: Playing It Safe with Data
Digital Workflow with a Film Recorder 
In principle, any document that can be printed out can also be exposed in a film recorder and thus guarantee long-term storage.
By using a film recorder, the costs resulting from continuous data transfer are reduced. If for some reason the data in the digital archive should become unusable, there is always the film archive to fall back on. What’s more, these data can then be scanned and re-digitalized.One can think of a film recorder as a sort of printer which exposes image data or text documents from a computer onto analog film material. Instead of being “printed out” on paper, the data are reproduced on the sort of film material that can be processed with the conventional technology available in almost any archive. This is one reason why the investment required is so modest.
Functional Description
A film recorder is made up primarily of an exposure unit together with either a cathode-ray tube or an LCD display. The image data coming from the computer gets exposed onto film in the exposure unit. A variety of devices with differing capacities can be used to tailor the equipment to the needs of individual users. In addition to the simple film recorder which exposes film in a conventional 35 mm or medium format camera (or also in a sheet-film cartridge) there are also more elaborate film recorders where the film is in large cartridges and the exposures are advanced automatically, image by image.
In principle, any document that can be printed out can also be exposed in a film recorder. This means it fits perfectly into the already existing digital workflow. It is simply an additional data output device with which an archive can make copies intended for long-term storage.
Digital Workflow with film recorder
By using a film recorder, the costs resulting from continuous data transfer are reduced, because documents stored analogically on film are easily archived. If for some reason the data in the digital archive should become unusable, there is always the film archive to fall back on. What’s more, these data can then be scanned and re-digitalized. In this way a film recorder becomes an indispensable and highly economical tool to preserve humanity’s most important documents from the “Digital Dark Age”. Our grandchildren and even their grandchildren will still have access to their own cultural, political or social heritage.
In this way a film recorder becomes an indispensable and highly economical tool to preserve humanity’s most important documents from the "Digital Dark Age".
Download this document as PDF: .
|
