Lessons Learned about Digitization

The recent tragic news from Germany and Italy points out just how fragile our world is. Documents, works of art, architecture, and more from throughout the centuries can be wiped out within seconds by earthquakes, fires, building collapses, wars, and other calamities. The loss to scholars, historians, genealogists, and others is incalculable. Luckily, we now have the technology and the ability to preserve multiple copies of all these priceless objects for future generations.

I would suggest that we digitize everything and make multiple copies of each copy, to be stored in widely dispersed locations.

To be sure, looking at a digital copy is not as satisfying as examining the original. I love the smell and feel of old paper. Looking at a digital image of a census record or some other old document does not match the feelings and emotions of handling the originals. Microfilm or digital imagery are both poor substitutes for the texture and smell of the originals.

In fact, no digital image or even filmed image will ever match the awe of standing in front of Michelangelo's "David" or looking up at the ceiling of the Sistine Chapel. Perhaps the most extreme example would be large buildings and other construction. How can we create digital imagery that reproduces the grandeur and beauty of the Parthenon or the Great Wall of China?

Obviously, we cannot. However, film, microfilm or digital copies do serve as reasonable substitutes. Such copies have served for years to bring information and appreciation to those who were unable to travel to the sites of the originals. After all, worldwide travel is beyond the reach of most of our citizens. Education spreads throughout the world by the use of books, films, and computers.

In a similar manner, I would suggest that every document available to mankind that has even the slightest amount of historical significance should be digitized, and multiple copies should be stored in multiple locations. As technology changes and the storage media of choice becomes obsolete, the images need to be copied to more modern media. History has proven that data maintenance and the copying of images is inexpensive and easily accomplished.

One example is the Social Security Death Index. These records have been digitized for more than forty years. The original records that were recorded in the 1960s were done so at significant expense, and the results were stored on 80-column punch cards. While punch card readers are no longer available, the data is still available and much more easily accessed than ever before. The data that was originally stored on punched cards was later converted to magnetic tape, then converted to magnetic disks, and still later converted to optical disks (CD-ROM, DVD-ROM and Blu-Ray disks).

I am guessing that only single copies of the original punched cards were created. However, as new media appeared, costs dropped. Storage on tape was cheaper than punch cards, storage on disks was cheaper than on tape, and so on and so forth. Each new generation of storage media seems to result in a huge drop in prices. Multiple copies of records became not only practical, but desirable. We now store the same data on CD, DVD, and Blu-Ray disks for a fraction of the cost of the filing cabinets alone that were used to store punched cards! We can expect that pricing will continue to drop in the future.

The lessons learned from the Social Security Death Index and from tens of thousands of other data conversion efforts can be applied to the billions of records that have not yet been digitized.

For an example, let's consider the recent destruction of the city archives of Cologne, Germany. This repository was one of the largest municipal archives in Europe and held records dating back more than a millennium. Some documents are being salvaged from the rubble, but it is known that a high percentage of the holdings will never be seen again by historians, scholars, genealogists, or anyone else.

If those documents had been scanned and digitized in previous years, the loss still would have been significant, but something of value would have been left behind. Those same historians, scholars, and genealogists would still be able to study the documents, even though they no longer existed physically.

To be sure, digitizing a few billion old and fragile documents will not be a trivial undertaking. Such efforts will require decades to complete. The costs are huge. However, the loss of such documents is immeasurable.

Archivists and others always have to handle, maintain, document, catalog, re-organize, and sometimes move old documents. When the items are being handled for other purposes, why not add one more item to the list of tasks to be performed: create a digital image? The time required to do so can be measured in minutes if suitable hardware is already installed and readily available.

A few forward-thinking repositories are already involved in such efforts. FamilySearch, the genealogy organization sponsored by the Church of Jesus Christ of Latter-day Saints (the Mormons) is already converting millions of rolls of microfilm stored in a granite mountain. The National Archives of Great Britain, the National Archives of Scotland, and the U.S. National Archives and Records Administration all are digitizing millions of their most-requested documents. The Library of Congress has made digitized versions of collection materials available online since 1994, concentrating on its most rare collections and those unavailable anywhere else.

These are not "one time efforts." All of the organizations I have mentioned expect their digitization efforts to continue more or less forever. Smaller archives, libraries, and museums are also beginning to see the benefits of digitization. Not only can they make their materials available to more patrons in more locations, but they also can save on building costs. Placing research materials online means a probable cost reduction in making these physical materials available to the general public. The holding organizations do not need to spend as much money on public reading rooms, rest rooms, microfilm equipment, open stacks, and personnel to serve the general public.

The repositories can offer smaller reading rooms and produce the material by reservation only. The new business plan is that any person who wishes to look at materials would first look at them online. If the online images prove to be insufficient, he or she then can schedule an appointment in the new (smaller) reading room, and the materials can be retrieved from the cramped stacks in advance and delivered to the reading room in time for the appointment.

While not as convenient, millions of dollars can be saved in construction costs of new and ever-expanding facilities to be used by the general public. Space in the repository that was used for public reading rooms can now be converted to storage space to house more documents than ever.

Digitizing of records appears to be a win/win situation: it preserves information and also makes that information more widely available to those who cannot travel to the holding repository.

What is your local organization doing about preserving its most precious resource: information?

Comments

[Richard Heaton]

After putting on line surviving family documents (few more to do) - personally I'm gradually scanning my misc. document collection (part of which is already on loan to archives but probably won't be properly catalogued for a while - under fragments of lost lives) and continue to transcribe (when I have chance) my sprawling newspaper collection. If I could afford to digitialise my collection I would do so - but the cost currently is too great. But should A1 size scanners ever fall significantly in price I'll probably scan all the unique newspapers as well (copies not available as far as I'm aware in any other UK collection).

I only wish there was a method for engaging US document collectors (probably a number in Australia, Canada, and the UK) to scan and make available their private collections.

Cheers
Richard H

[Sabrina Stradtner]

At the library I work at we are currently working to digitize the county's marriage licenses. We create a bound book and a CD of the images we take at the court house. We also create a searchable index. We have many county records and pictures in our possession that need to be digitized. Unfortunately, it's easier to come up with projects than actually getting them done. Money and man power are the two biggest issues. None of our projects would get done without our volunteers.

[Peter Hecht]

The destruction of the Cologne archive is certainly a tragedy as it was material saved over centuries by a history conscious country. In many countries, documents and materials are not being saved and thus will never be known to be lost. Canada is one of these countries where fewer and fewer people know about even our recent history.

In the last few years, Ontario, the largest and one of the oldest of provinces decided there was no more space to store land registry records and so preparations were made to dispose of these records. Although some were stored digitally (in questionnable format), it is possible that many boxes were disposed of before anyone knowledgeable noticed.

And our national archives are located in a 1950s or 60s building which suffers regularly from broken pipes in the ceilings. The stacks have been covered by plastic sheets but damage is still done. Many documents are in commercial freezers until they can be saved. Yet at the same time, staff and resources are being cut back. Reading room hours are being severely limited and many documents are made inaccessible.

Although we like to pride ourselves on being superior in many ways to the United States, I only wish that we had an equivalent to your Library of Congress as well as the myriad number of local libraries and historical societies who were able to store documents doomed by the electronic age practices of those who have no faith in the past and who discard paper once only data of immediate interest is entered into a computer system somewhere. At the same time, much data management has been privatised so that citizens must pay a fortune for copies of data collected by public taxpayers expense.

Thank goodness for people in other countries who value our records more than we do. Thanks to you and the hardworkers below the radar for at least keeping records even if they aren't yet available.

[Ron Darrah]

I remember reading that the IRS has thousands of rolls of data files that no one can now access because the machines and coding
has been lost. Is this true and is there any attempt to retrieve
this material?

[Dick Eastman]

NOT true, although I believe that "urban myth" was about the Census Bureau, not the I.R.S. I wrote about that myth six years ago this week. You can read my article at http://www.eogn.com/archives/news0315.htm#1960USCensus

- Dick Eastman

[hslanham]

I am registering much of my research at
www.findagrave.com
I enter a photo of the stone plus general
information plus photos of the person.
I had one distant cousin email me regarding
a stone I photographed about 8 years
ago. He said that my photo is now more
readable than the actual stone.
I also register tombstones of people not
related to me just to get the record out there.

[Margaret Harris]

I would like for every librarian in the land to receive this post! Every archivist, every employee of every repository, everywhere: Push for digitization.

[Mike Brubaker]

By suggesting that digitization replace microfilm you are comparing apples of oranges. Digitization may enhance access, but it does nothing for preservation.
Keep in mind that changing technologies will require updates for digitized projects every few years. You can anticipate throwing more money away as the digital project ages. I also disagree that digital projects are inexpensive. For large projects the costs can be huge.
Microfilm is a one cost project--create the film and use it. Microfilm will last upwards of one hundred years. With current digital technolgy, good luck having the same resources in ten years.
if you have the money and time, do both microfilm and digitize. If your money is limited, microfilm first. You can always digitize later--from the microfilm.

[Dick Eastman]

As you mentioned, "changing technologies will require updates..."

Microfilm is a technology that is going away. In ten or twenty years, there will be no new microfilm. Organizations such as the Mormon Church are already converting their microfilms to digital images because of disappearing technologies. (I doubt if they will ever throw the old microfilms away but they will be less and less useful every year.)

As you wrote, "Microfilm will last upwards of one hundred years." I agree. However, what good will that be when microfilm readers are no longer available? In fact, microfilm CAMERAS are already difficult to purchase and microfilm itself (unexposed reels) is already difficult to find. The number of microfilm readers available for sale has dropped dramatically in the past few years as manufacturers are closing down production. In fact, the "hot item" for the manufacturers today is "microfilm-to-digital conversion devices," namely microfilm scanners.

- Dick Eastman

[Jason Presley]

In the last 15 years, the only changes I've seen in "digitization" technology has been the increase in quality and storage density. The JPGs, TIFFs and PNGs from 10-15 years ago are no less accessible now than they day they were created. There's no reason to think the basic technologies the visual aspects of the WWW have been built around are going to be come obsolete all that quickly. Besides, most of these digitization projects are scanning their materials in at extraordinarily high resolutions, so even conversions to newer formats (if that's even necessary) is unlikely to result is any noticable degradation.

Once something is digitized, duplication of that asset is essentially free, it's only the initial effort that has any real cost associated with it. Copying a reel of microfilm is not free and requires specialized equipment. Duplication of digital materials can be done with any low-end personal computer.

[Jim Lynch]

Although I live in Canada I am originally from the Caribbean (Barbados) and several years ago I put a proposal out there for a CHPI (Caribbean Historical Preservation Institute) whose object would be to digitise every available record throughout the Caribbean.

Attached to the Institute's interests and activities would have been volunteer genealogy tourism, attachments from Universities for archaeologists and paper chemistry researchers, high quality printing, souvenirs, sales over the internet to support ongoing activities, and many other interests.

I envisioned the Insititute starting in one island and developing a workable model; and then exporting model "packages" to the other islands/countries in pre-packed 40-foot containers (vehicle, desks, chairs, digitising equipment, back-up generator, etc.) which on arrival and installation on a pre-poured slab with utilities would serve as hurricane-proof offices and storage. Digitised pages and would be eexported via internet and hard drives to other parts of the world for safe-keeping.

Given the record of hurricanes, earthquakes and ither natural (and man-made) disasters as well as destruction by termites, insects and the open and grossly casual destruction and dumping by island governments (I have heard that in St. Vincent they dump excess records on the damp floors of disused jail cells, and they rot from the ground up), anyone would think this would have been received well and seen as a priority, but so far there have been no takers at all.

I initially tried to gather some people together to register it as a non-profit and approach companies like Xerox for new or used equipment and donations, but have received no responses and no expressions of interest. I approached the Gates Foundation and several other donors and benefactors, but the ones who did reply said that was not in their target field.

As an alternative, I gathered copies of two sets of books on the verge of disappearing due to insect damage and acid-paper deterioration (Vere Langford Oliver's "History of Antigua" and "Caribbeana") and re-published them mnyself under ISBN numbers and with the highest quality materials and binding.

My Proposal remains on the web... and still with no experssions of interest from anyone.
http://www.candoo.com/chpi/index.html

If you are interested in activating the Proposal, you merely need to write to me and ask for access to the Discussion Forum, and we can take it from there.

Jim Lynch

[Joan Griffis]

A recent article in the St. Pete Times" (FL) had information on Brewster Kahle "who would be the Web's librarian" because he is taking on the long time task of creating "the world's largest digital library" (of everything that has ever been published.) His openlibrary.org is intended to provide everything available online. Haven't found it yet, but does anyone know anything more about this project and its validity?