Post on 12-Sep-2021
transcript
Migration and PreservationMigration and Preservationof Mainof Main--frame Tape Dataframe Tape Data
Muller Media Conversions was founded in 1978 to provide documentediting and conversion services to financial printers, banks and law
firms. The company has always developed most of its own software.(We “eat our own cooking” as they say.)
Some of our most interesting projects over the yearshave been for the U.S. National Archives.
That work has made us particularly aware of the requirements andopportunities in handling “Persistent Data”.
Muller Media Conversions www.mullermedia.com
Please excuse our taking just a moment to talk about our background in datapreservation. One day, we saw this article in our local newspaper. It ultimatelylead to our longstanding relationship with the U.S. National Archives…
The NationThe Nation’’s Records Are a Messs Records Are a MessWASHINGTON-- A slice of America's history has become asunreadable as Egyptian hieroglyphics before the discovery ofthe Rosetta stone. Vast untold volumes of historic, scientificand business data are in danger of dissolving into ameaningless jumble of letters, numbers and computersymbols. Much information from the last 30 years is strandedon computer tape from primitive or discarded systems-unintelligible or soon to be so…
...ASSOCIATED PRESS JANUARY 3, 1991
www.mullermedia.com
On an incompatiblemedium ...
In an indecipherable code or format ...
Trapped in an obsolete filing system ...
On tapes in dangerof decay ...
Information Assets in DangerInformation Assets in Danger
Muller Media Conversions www.mullermedia.com
NARA uses tape conversion system topreserve records
The National Archives and Records Administration hasstepped up its preservation of electronic records withthe installation of a $510,000 records maintenancesystem. Responsible for preserving the government’selectronic mail and other data stored in digital formats,NARA’s Center for Electronic Records in 1992contracted Muller Media Conversions Inc. of NewYork, to develop and install…
…GOVERNMENT COMPUTER NEWS, AUGUST, 1994
IAC/IRM honors federal I/T leadersFynette Eaton of the National Archives and RecordsAdministration's Center for Electronic Recordsreceives a GSA Technology Excellence Award fordeveloping the Archival Preservation System to storerecords and capture data about their physical andtechnical attributes.…
…GOVERNMENT COMPUTER NEWS, JUNE, 1996
• Award-winning
• Contract extended continually
• Ongoing enhancements
• MMC owns the software.
MMC and the National ArchivesMMC and the National Archives
www.mullermedia.com
The Archival Preservation System is designed to perform a wide variety of functionswith robust, bullet-proof reliability. In daily operation at NARA and MMC. *DPS is the
designation used for the non-government version of the package.
Copying Functions
• Supports virtually any type of tape drive.
• Copy and compare tape-to-tape, tape-to-disk,disk-to-tape.
• Block-for-block (duplication) or "smart copy”(output can have different label conventions,record formats, block sizes than the input).
• Standard ANSI and IBM tape formats,unlabeled files and several proprietaryformats such as VMS Backup.
• ASCII-to/from-EBCDIC conversion plus acomprehensive scripting language for field &record parsing. (packed, integer, float, etc.)
• ”Tape-Image" capability which enables tapepreservation and duplication or smart-copyingof tapes with only one tape drive.
• Tape-Image files can be preserved on CD,DVD or M/O disks (or even on other higher-capacity tapes). Later, original tape can bereconstructed byte-for-byte.
Data Recovery
• Great flexibility/control in error handling.
• Merge data from two or more tapes to a singleoutput tape or file. (by record or by block)
• Automatic repositioning after interruption for tapecleaning/conditioning.
APS/DPS* CapabilitiesAPS/DPS* Capabilities
Record Keeping/Reporting
Central database for detailed records of:
• Tape processing (e.g.- which system, drives,operators, hard/soft errors, file names, sizes, etc.)
• Media tracking with bar-code reader support.
• Media performance by manufacturer/lot.
• A wide variety of production reports.
Analysis/Review
• Block/record dumps in hex or octal with charinterpretation in EBCDIC, ASCII-7 or ASCII-8.
• Tape map (directory) brief and verbose listings.
…and lots more
Muller Media Conversions www.mullermedia.com
Storm Cloud or Sweet Spot?Storm Cloud or Sweet Spot?
Stormy
• The “vapor trail” includes endangeredmedia containing data of lasting value.
• The ability to handle such media/fileformats is becoming increasingly rare.
• Regulators and compliance officers insistthat it be readable, useful.
Sweet
• A good ratio between capacity of old mediaand inexpensive, durable new media.
• With MMC’s software, experience and skills,this legacy of important data can be verifiedand preserved just as they do it at theUS National Archives.
Time & Computer Media (very rough scale)
Techno-Rocket
Your “Vapor Trail”of Older Information
Time
Muller Media Conversions www.mullermedia.com
Over the past 28 years MMC has developed theOver the past 28 years MMC has developed theskills and tools to transform information for askills and tools to transform information for a
variety of purposes.variety of purposes.
•• Preservation. (Transfer to inexpensive, long-
lived media for statutory or other reasons--
with extensive verification and record-keeping.)
•• Litigation/Investigation. (Rendering electronic
evidence to a format suitable for legal review.)
•• Conversion. (Translation of file content for use
on another platform or software package.)
Some projects involve aspects of two or more of these. Please see
our website for examples of our work in these areas.
Muller Media Conversions www.mullermedia.com
Possibly saved for many years.
(3a) Using DPS™, many types of filescan be processed on PC, no tapeinvolved.
(2) Preserve on hard drives or DVDs.An inexpensive external drive canhold the contents of thousands ofreels or cartridges.
Tapes to be processed
(3b) If needed, any tape can bere-created with bit-for-bitaccuracy. (On same ordifferent medium.)
Basic* OverviewBasic* OverviewTape Conversion/PreservationTape Conversion/Preservation
__________* Other features & variations* Other features & variations
are presented later.are presented later.
(1) DPS system captures datawith 100% accuracy
Muller Media Conversions www.mullermedia.com
Photo of a DPSPhoto of a DPS™™ WorkstationWorkstation
Muller Media Conversions www.mullermedia.com
DPS creates a PC disk file containing byte-for-byte, block-for-block, filemark-for-filemark representation ofthe tape. This is called a Tape Image. The tape ID is automatically picked up and used to uniquely namethe tape image file. (Example: 019246.tap.) Optionally, the tape image is then automatically compared tothe original tape to ensure 100% accuracy—though most tape drives have reliable error checking makingthis step unnecessary. A report file detailing success/failure and counts of blocks, bytes and filemarks isalso captured and named to match the tape image. (Example: 019246.txt.) The “tap” and “txt” files are thencompressed to a single file.
Compressibility of content can vary widely depending on the nature of the material, tape length, block-sizesand so forth. A modern external hard drive can hold thousands of compressed 349x tape images.
The Tape Capture ProcessThe Tape Capture Process
compare
copy
019246.tap
015953.tap
A01293.tap
Muller Media Conversions www.mullermedia.com
… The originalhard drive maybe used, or thetape-imagesmay have beencopied to anynetwork ordirect-attachedstorage youmay choose.
Using an ordinary PC, a SCSI-attached tape drive and MMCsoftware, any captured tape can bere-created with bit-for-bit accuracy.(On same or different medium.)
Restoration/Conversion of Preserved DataRestoration/Conversion of Preserved DataLevel 1Level 1 –– ReRe--Create a TapeCreate a Tape
This approach is used for material that’s wanted back on the mainframe forprocessing, or because of proprietary format can be handled in no other way.See Levels Two and Three for other possibilities.
Muller Media Conversions www.mullermedia.com
Restoration/Conversion of Preserved DataRestoration/Conversion of Preserved DataLevel 2Level 2 –– Extract Individual FilesExtract Individual Files
019246\0001_REND.TFB.G0013V00019246\0002_REND.PCF.G0016V00019246\0003_……
015953\0001_30.ITEMFILE.BACKUP
A01293\0001_LAYOUTA01293\0002_PFINVUPL
*File names in standard IBM labels are limited to 17 characters. The name may have been longer on themainframe’s disk and truncated when written to tape. This can lead to ambiguous names. Knowing the tape “volser”value and the sequential position of the file on the tape enables the mainframe (and us) to identify the file uniquely.
… the DPS program extracts files from any standard ANSI or IBM tape (or tape-image). Files are placed
within folders named according to the tape VOLSER. The complete tape filename is retained, and
optionally pre-pended with a sequence number*. A log of this process captures information about record
types and counts for each file. Files that contain only EBCDIC text can be translated to ASCII in this
step, with optional cr/lf appended to each record. See “Conversion/Transfer” slide for further steps.
Muller Media Conversions www.mullermedia.com
In many instances, extracted files can be transferred toa mainframe via FTP or drive-sharing.
Using our GenScript/RunScript™record-parsing utilities, many typesof files can be translated on PC, nomainframe resources required.
B. For data that must be restored toB. For data that must be restored tothe mainframe.the mainframe.
FTP
A. For files in a translatable format, whereA. For files in a translatable format, wheretarget system is PC or Open System.target system is PC or Open System.019246\0001_REND.TFB.G0013V00
019246\0002_REND.PCF.G0016V00019246\0003_……
015953\0001_30.ITEMFILE.BACKUP
A01293\0001_LAYOUTA01293\0002_PFINVUPL
Mainframe
Restoration/Conversion of Preserved DataRestoration/Conversion of Preserved DataLevel 3Level 3 –– Further Processing via PC or MainframeFurther Processing via PC or Mainframe
Muller Media Conversions www.mullermedia.com
For files with complex layouts containing packed fields, binary integers, floating point, etc., one canuse GenScript to view, and “hack” these files, creating a script to translate each unique file layout.
Subsequently, the RunScript program is used to translate the files to ASCII. Output choices include“flat”, “quotes-and commas”, “tab-delimited”, and “xml-style”. RunScript is quite fast.
NOTE: Given the possibility that the provided record layout may have been incorrect, and the smallincremental cost, it’s often wise to save the “original file” format along with “converted file” format, thuspreserving the possibility of re-conversion.
019246\0001_REND.TFB.G0013V00019246\0002_REND.PCF.G0016V00019246\0003_……
015953\0001_30.ITEMFILE.BACKUP
A01293\0001_LAYOUTA01293\0002_PFINVUPL
GenScriptGenScript
RunScriptRunScript
Scriptfiles
019246\0001_REND.TFB.G0013V00.txt019246\0002_REND.PCF.G0016V00.txt019246\0003_……
015953\0001_30.ITEMFILE.BACKUP.txt
A01293\0001_LAYOUT.txtA01293\0002_PFINVUPL.txt
Record Parsing and ConversionRecord Parsing and Conversion
Muller Media Conversions www.mullermedia.com
WhatWhat’’s Your Preference?s Your Preference?
Chris Muller, presidentChris Muller, presidentMuller Media ConversionsMuller Media Conversions
Preserving the value of your information legacy isPreserving the value of your information legacy isour goal.our goal.
Every situation is somewhat different, and weEvery situation is somewhat different, and weappreciate the opportunity to discuss the needs andappreciate the opportunity to discuss the needs and
technical parameters of your project.technical parameters of your project.
Muller Media Conversions www.mullermedia.com