Processing SIRxs Journals

Presented at SIR-UK ConferenceDublin, Ireland, June 2008

©Tom Shriver, DataVisor 2008

Processing SIRxs Journals

Presented at SIR-UK Conference Dublin, Ireland, June 2008

©Tom Shriver, DataVisor 2008

3

Progress of this Presentation

A Wee Bit of Background Old Journaling vs New Previous Capabilities and SIRxs Options

Mostly About PQL Processing of Journal Will Turn to Tony for some Internals

Especially for Schema Journaling

Discussion / Suggestions There is almost no PQL Processing of

Journalled Schema. What do we need? Want?

4

SIR2002 (& SIRxs)Journal Access: ITEMIZE FILE

Type of header: Journal dataDatabase name: MNYR'From' Update level: 6638 DATE: 09/20/07 TIME: 15:37:31'To' Update level: 6639 Date: 09/20/07 TIME: 15:39:00

Type of header: Journal schemaDatabase name: MNYR'From' Update level: 6639 DATE: 09/20/07 TIME: 15:39:00'To' Update level: 6640 Date: 09/20/07 TIME: 15:39:27

5

SIR2002 (& SIRxs)Journal Access: Journal Upload

Creates a Text File Bare Bones Schema Information Data Formatted Similar to Export format

(example MNYR.JUL & jou_upload.out)

Originally Intended to Download Data from Satellite databases to Central Database

I once wrote a PQL program to process this journal information

6

SIR2002 (& SIRxs) Journal Usage: JOURNAL RESTORE & JOURNAL DOWNLOAD

Apply Data from Journal to Database RESTORE:

Applies Updates synchronized by Update Level

DOWNLOAD: Applies Updates – Update Level Independent

7

What’s in a SIRxs Journal File?

Journal is a Sequential File Tracks Update Levels

What happened at update level When & to what the update level happened Who controlled update (Master, User)

Schema Changes Data Changes

New Records Modified Records Deleted Records

8

Journal File Headers

Two General Types of Headers in Journal Update Level Header Detail (i.e. record specific) Header

within update level

9

Detail Records

Copies of Database Records Nearly Identical Structure Journal has additional header info (new/old/deleted)

New Records: Copy of new record journaled

Modified Records: Copy of “pre-modification” record journaled Copy of “post-modification” record journaled

Deleted Records: Copy of “pre-deletion” record journaled

10

What the Journal Does Not Have

Does Not have a Complete Schema Definition Implications:

The Journal doesn’t “know” simple things like:o What’s the name of Record Type 3?o What’s the value of CURRPOS in this new record?

To interpret what journal data means, we must have a schema attached

11

Practical Implications & Usage

Better Journal Utility Functionality (e.g. Rollback, standard SIR reports)

Better Analytic tools for various things User Defined Recovery &/or Rollback

functionality Audit Trails

12

Journal Processing with PQL

New Tools to Read a Journal File PROCESS JOURNAL

o Steps through Journal Fileo Reads Journal Headers & Returns Information

JOURNAL RECORD ISo Reads detail record from journal and

gets data as we expect from a RETRIEVALo Must have a SCHEMA (i.e. database) availableo Is not indexed

No VIA, NEXT REC, etc.

13

PROCESS JOURNAL SyntaxPROCESS JOURNAL FILENAME = jou_file | REVERSE | FROM = j_slevel | START = j_sdate , j_stime | THRU = j_elevel | END = j_edate , j_etime DATE = up_date ENDDATE = up_edate TIME = up_time ENDTIME = up_etime LEVEL = up_level RECORD = up_rt# TYPE = up_type USER = up_user DATABASE = up_db MASTER = up_mst

Above is example from PQL – to right of “=“ are local PQL variables, not syntax notation.

Options generally indicate content of journal headers.

14

PROCESS JOURNAL Update Types

value labels up_type ( 1)"New record written" ( 2)"Before existing record updated" ( 3)"After existing record updated" ( 4)"Before Record deleted" (-1)"Journal Data Header" (-2)"Unload Schema Header“ | unload! (-3)"Unload Data Header“ | unload! (-4)"Journal Schema Header" (-5)"User Header"

From Example PQL program to process journals.(see jou_summ_dublin.pql)

15

Journal Processing Example: Summary Report

Summarize a daily journal by: Record Type Type of Update User see… jou_summ_dublin.pql

Logic of PQL Create SubProcedures for JOURNAL REC IS PROCESS JOURNAL … . EXECUTE SUBPROCEDURE rec_nn END JOURNAL INCLUDE FILE “created_subprocedures.pql” OUTPUT REPORT

16

Journal Processing Example: Customized Database Recovery

Situation: User was updating database late at night (9-12 PM) Backups Begin at Midnight User (via Master) and BackUps (single user) struggled

for control BackUps Wins Struggle; database is a mess

Solution: Write a PQL program to decipher two journals & fix

database Used modified version of Summary Report;

Was actually quite simple.

17

Journal Processing Usage Potential: Audit Trails (1)

Problems to Overcome What’s an Audit Trail?

All variables all records?Subset of important variables (e.g. adverse reactions)?Need User Explanation for Modifications? Coded? Textual Note Written?

These issues can be standardized and implemented automatically - A place to store “preferences” - A way to automate rule conformance (e.g. screens) - A place to store audited records

18

Journal Processing Usage Potential: Audit Trails (2)

Implementation Strategy Possibilities Keep all Journal Files (or one continuous file) and

Process when Neededo Problem: No Schema in Journalo Maybe: Tony could store schema in Journal

Store Audit Data in Database o Mirror Image Rectypes with additional high level Keys

Problem: What if keys changed? CaseID? Store Audit Data Outside Database

o Audit Databaseo Audit Tabfileo Other File Type (text, CSV, etc)

19

Audit Trail Example: Audit Deleted Recs

At DCC managers and data entry staff send emails to notify of deletions (because deletions show up in standard reports).

Example: jou_audit_del_dublin.pql