Salvador Barragan

Curator of Government Records

Nebraska State Historical Society

Public Records in the Digital Age

What is a record?

• A ‘record’ is the complete set of documentation required to provide evidence of a business transaction.

Shifting Media• Before Paper we stored valuable historical data on stone

and papyrus.• In our current period records of historical permanency

were stored on paper and kept in filing cabinets– When the cabinet was full, records were sent to file

room.

• Now records are stored electronically on computers– When the computer is ‘full’ – add more hard drives or

servers.

Basic skills to manage and maintain records has been lost, replaced by infinite storage

Electronic Records Management Goals

1.Bring the record to the forefront of system design activities.

2.Identify electronic records functionality as part of system design.

3.Create electronic records that support legal, fiscal and evidentiary needs.

4.Create long term archival storage for both retention schedule and historical purposes.

Goals con’t

5.Create electronic records that are accessible and usable over time (non-proprietary formats).

6.Integrate diverse document forms and formats into records.

7.Identify need for internal and external primary and secondary access to records.

Three Functional Requirements for Electronic

Records Management & Preservation

1. Records Capture – Records are created or captured and identified to support the business process and meet all records management requirements related to the process.

2. Records Maintenance and Accessibility – Electronic records are maintained so that they are accessible and retain their integrity for as long as they are needed.

3. System Reliability – A system is administrated in accordance with best practices in the information resource management field to ensure the reliability of the records it produces.

What happens when you do not have a

RM system?

Higher Standards

• As electronic records become more integrated into society, producers of those records will be held to higher standards of conduct– HIPPA– Sarbanes Oxley– Federal and State Mandates– Case Law

NE Public Records Laws

002.01 Record. The Records Management Act (Revised Statutes of Nebraska, Chapter

84, Article 12) defines a record as: "any book, document, paper, photograph, microfilm,

sound recording, magnetic storage medium, optical storage medium, or other material

regardless of physical form or characteristics created or received pursuant to law, charter,

or ordinance or in connection with any other activity relating to or having an effect upon

the transaction of public business." A record is information that is inscribed on a tangible

medium or that is stored in an electronic or other medium and is retrievable in

perceivable form.

http://statutes.unicam.state.ne.us/Corpus/statutes/chap84/R8412013.html

Records Retention

The foundation of democracy in America is government accountability to the people and permanency of our culture and heritage.

So the question becomes… who takes care of the

records, and do they have the knowledge

understanding of the new technology?

Caretakers of Information• Historically records sent to file room, staff

maintained access to records and managed lifecycle based on need and legal requirements

• Now records are managed by users and IT staff, based on capacity and cost.

Taking into account the goals of records management and the

function of Records, what are we to do?

Or what is the solution?

•Best Practice Models

•Standards

•Systems

•Digital Archive?

Best Practice Models

•OAIS Model

•Washington State Archives

http://www.digitalarchives.wa.gov/default.aspx

OAIS Model

www.digitalarchives.wa.gov

Standards

When ever possible follow the prevailing best practices and standards.

Standards for E-Records….

•Hardware

•Software

•Formats

•Management

•Authenticity

Hardware• File Room of the 21st century• Capacity and Speed double every 18 months• Many choices

– Tape– Optical– Spinning Disc

First Immutable Law of Digital Archiving

“What hardware you use today will be obsolete within four years”

Internet

DA-WEB1

DA-SE1

DA-BIZ-RS1BizTalk

Receive/SendLocation

Washington State Digital Archives Network Configuration, May 2, 2005

LegendFirewall

Database Server

REVISIONS

1.0

2.0Final network architecture

09/16/2004

05/02/2005Actual configuration

Data Tier

Processing Tier

Services Tier(Search Services)

State/LocalOffice

Web/FTP Server

Web Services

BizTalk Server

Domain Controllers

DA-DC1

HP DL3802 *3GHz HT CPU2GB RAM36GB Mirrored HDWIN 2003 std

HP DL3802 * 3GHZ HT CPU

2GB RAM36GB Mirrored HDMS WIN 2003 std

SAN Storage

Internet Send/Receive

DA-WEB2

DA-DC2

HP DL3802 * 3GHz HT CPU2GB RAM36GB Mirrored HDMS WIN 2003 std2 Coyote HW Loadbalancers

DA-SE2 DA-SE3

DA-BIZ-RS2BizTalk

Receive/SendLocation

BizTalk 2004Database

Cluster

Digital ArchivesAsset Metadata

Cluster

HP DL3802 * 3GHZ HT CPU2GB RAM144GB RAID 5 HDMS WIN 2003 stdMS BizTalk 2004 ent

`

CitizenInternet

User

HP DL580 4 *3GHZ CPU4GB RAM36GB Mirrored HDMS WIN 2003 entMS SQL Server 2000MS Clustering Active/Active

EMC Clariion CX700 SAN 1TB 15K FC

4TB 7200 SATA

Tape Library

HP DL7408 *3GHZ HT CPU

8GB RAM36 GB Mirrored HD

MS WIN 2003 entMS SQL Server 2000

MS Clustering Active/Passive

Shared Disk Array

DA-BIZ-INBOX1 RAW Data “Temp” Storage

Image ConversionXML “Temp” Storage

Administration

ADIC iScalar 2000 10 LTO-2 drives 500 tape slots

Hardware Load Balanced

Hardware Load Balanced

IIS

IIS

IIS IISIIS

http/https

Secure FTP

DA-DMZ-DC2DA-DMZ-DC1

http/https

DA-Media1 & 2 (Images & Streaming

Media)

DA-SE4

DA-SE5

IIS

IIS

80/443

22

80

22

80

80

DMZ

DA-Tectia1 (Secure FTP)

HP DL3802 *3GHz HT CPU2GB RAM36GB Mirrored HDMS WIN 2003 std

HP DL580 4 *3GHZ CPU4GB RAM36GB Mirrored HDMS WIN 2003 ent

Digital Archives Hardware

• Network – Cisco Backbone end to end– LAN and SAN

• EMC – SAN storage– 5 TB now, 20TB by end of Year

• HP – Servers and desktops• ADIC – Tape Library for offsite, disaster

recovery (nightly or weekly back up, remember Katrina and 9-11)

• Microsoft – Software and Development

Archival Software and File Format Standards

• Native• ASCII• TIF• PDF/A (Used by the Federal Courts)• XML http://www.thexmltoolkit.org/guides.asp (metadata

and interoperability)• DoD 5015.2-STD compliant system• Nebraska State Records Guidelines

Whenever possible seek the Open, documented solution!

Remember WordStar and DBase II ???

Metadata & Interoperability

• Cross cultural and contextual boundaries

• Interoperability

• Interoperability & Metadata schema

Interoperability and XML

Content Management

• Essential to maintain control of the information explosion

• Allows hard coded rules and information exchange

• BUT still requires a strong knowledge, understanding and implementation of basic records management

Second Immutable Law of Digital Archiving:

“Data is Data, a Record is a Record, It is the content that drives retention, not the media”

• DoD 5015.2-STD compliant systemhttp://www.dtic.mil/whs/directives/corres/pdf/50152std_061902/p50152s.pdf

Wrap original file in native format

• Wrap XML copy

• Apply metadata & XML for indexing,

searching & retrieval

• Provide chain of custody & authenticity

‘Content Management’

‘Content Management’

• Microsoft Solution

• SQL Server back end

• BizTalk translation utility

• SSH Tectia for secure transport http://www.ssh.com/products/

Washington State Archives Case Study

Authenticity

• Maintain Chain of Custody

• In the care of trusted 3rd party

• Received from trusted, known source

Data Security

• Encrypted SSH FTP transmission

• Issue Digital Certificate

• Verify IP and computer information

• MD5 Hash on all original files

• Copy of FTP on tape prior to ingestion

• DB backups on tape

• Record Level Security for confidential Info

Record Level Security• Restrict records at item, field or series

level

• Restrict to individual, dept, office or global

• Uses authenticated login to reveal fields

• Anonymous users see ‘Restricted’

Deep Storage XML

Deep Storage XML SchemaRecord Common

•Who

•What

•When

•Where

•Original File

•‘web’ file

•Security

•Fixity

Vital Records• Type

Birth• Date of

• Father, Mother

• Hospital

Ingestion Process• MUST be flexible

• Microsoft BizTalk 2004

• Transforms, adds metadata based on business rules

• Creates ‘deep storage’ copy wrapping original file in XML, with Hash

• Creates ‘web’ version of original file

Archive Database

• Designed around latest industry standards

• Open source, non-proprietary file storage

• Applies metadata ‘tags’ to save information

about record

– creator, date, agency, subject, etc.

• Provides chain of custody & authenticity of

record

• Allow search and retrieval of archival records

through a web page

Risks

• Distributed, non-standardized environment

• Limited technology expertise in some agencies

• Unpredictable data growth rate

• Few business models

• Emerging technologies

• Limited internal expertise

Management Issues• Authenticity of record• Metadata• File naming conventions• Corporate Culture• Start small with e-mail, web page• Use existing retention schedules• Educate• Shift AWAY from desktops…• …And move to central servers• Management Software is a must!• Privacy of sensitive data

Third Immutable Law

“Anything that you do today, will need major overhaul in two years or sooner”

Technology and industry changing at unprecedented rates… But, more records are ‘lost’ every day!

– Key is to be flexible and address with systematic forethought

How to handle Records over the Web.

Open Record

Restricted Record

Confidential

E-Commerce

Add to Shopping Cart

• Ecommerce Functionality– Add to Shopping cart

Shopping Cart

Billing Information

View and Submit Order

Why a Digital Archives?• Comply with statutory & regulatory mandates.

– The Law requires preservation of certain public records – it doesn’t specify

whether those records are paper or electronic. All records must be given

the same care.

• Avoid loss of legal & historical records

– As technology changes, the older media (5 ¼” floppy disks, for instance)

become harder to read.

• Centralize Records

– Centralization means uniformity in maintenance

– ‘Trained professionals’ serve as caretakers

• Preserve rare and ‘at-risk’ paper records

• Improved access for citizens

– By centralizing historical electronic records in one location, ‘one-stop

shopping’ will provide the information quicker and easier

The Digital Archives will:

• Preserve electronic records with long-term legal, historical and/or fiscal significance

• Assure platform-neutral retrieval 50, 100, or more years from now

• Provide security back-up of certain permanent electronic legal records (courts, vital records, land records, etc.)

Acknowledgements

• Adam Jansen, Digital Archivist for the Washington State Archives.

• Dr. Ed Papenfuse, State Archivist for the Maryland State Archives.

• Andrea Falling, State Archivist for the Nebraska State Historical Society.

• Cathy Danahy, Assistant Director of the Nebraska Records Management Division.