ORSYP Dollar Universe Workshop - Aldersoftaldersoft.ca/Dollar-Universe-Workshop.pdfORSYP Dollar...

ORSYP Dollar Universe Workshop

University of Manitoba

Information Services and Technology

Gary F. Alderson

June 11, 2010

Workshop Outline

Day 1 – June 11, 2010

Login to Terminal Server

Basic Dollar Universe Concepts

Object types in Dollar Universe

Workflow of Development

Assignment 1 – Build a simple Session

TASK Objects – Schedule Instances

Assignment 2 – Build a simple Task

Review of Day 1

Workshop Outline

Day 2 – June 18, 2010

Dependancy Scheduling

Assignment 3 – Build an optimized Session

Resources

Assignment 4 – Build a Session using Resources

Non-Similtaneity

Classes

Inter-Session Dependancy Scheduling

Assignment 5 – Build a Multiple Session Schedule

Mulitple Platform Scheduling

Review of Day 2

Workshop Outline

Day 3 – June 25, 2010

Calendars

Command Line Functions

Promotion into Production

Dollar Universe Web Console

Business Views

Crystal Reports – JeffNet Report Runner Batch

Questions and Answers


Day 1 – June 11, 2010

Login to Terminal ServerBasic Dollar Universe Concepts






Review of Day 1


The Microsoft Terminal Server

Windows Server 2003

BeaverPond.cc.UManitoba.CA (ASPterm)

For the exclusive use of Enterprise Systems

Start → Accessories → Communications → Remote Desktop Connection

Enter your Userid and Password

Make sure to go to the Local Resources Tab

Then check Local Devices – Disk Drives to allow you to access your machines disk drives.


• Click the Start Button• Select ”All Programs”• Select ”Accessories”• Select ”Communications”• Click ”Remote Desktop

Connection”


• Click on ”Options” so that see this box.

• Enter ”BeaverPond” in Computer

• Enter your Userid and Password

• Then Click on the ”Local Resources” tab.


• You should see this.• Check ”Disk Drives”

– This will carry across access to your local disk drives.

• Click on ”Connect”• You will now connect –

Well almost ...


• Terminal Server will warn you that your disks are now exposed to the terminal server.

• You have been warned.• Click ”OK”


• Your Terminal Server Desktop should look something like this.

• You should see an ICON for ”Desktop ASDEVL – ASPTERM”.

• Also an ICON for ”Dollar Universe Web Console”

• If not raise your hand.


Day 1 – June 11, 2010


Basic Dollar Universe ConceptsObject types in Dollar Universe





Review of Day 1


Nodes• Physical Nodes are servers

which run a fully contained Dollar Universe environment

• We have 9 nodes

– 3 SUN Solaris– 3 RedHat Linux– 3 Windows Server

• Each platform has one test and two production nodes.


Companies• $U can support multiple

companies on each node.

• Each company is independant and is used to segregate securable entities.

• We currently have two companies: ASDEVL and ASPROD to segregate Development and Production.


Each Company is made up of four Areas - ASDEVL:

• APP – Application

– Source Development– Source Repository

• INT – Integration (Not Used)

• SIM – Simulation

– Developer Testing– Promotion Send Point

• EXP – Production

– Quality Assurance


Each Company is made up of four Areas - ASPROD:

• APP – Application (not used)

• INT – Integration (Not Used)

• SIM – Simulation

– Production Repository– Promotion Receive Point

• EXP – Production

– Active Production


Management Units A logical environment built around logical units

At least one MU per Physical Node. MU Types:

L - Linux U - Solaris W – Windows Server Used as an abstract place to run things. They can represent: Applications / Databases / Entities /

Whatever. Cater for hardware configuration changes Generic jobs (not context-specific) Hierarchical job processes


Hardware Changes MUs can be used to

facilitate hardware consolidation or failure without making any changes to jobs.

Host_1

MUs NodesMU_A Host_1MU_B Host_1

JOB_Yon MU_A

JOB_Xon MU_B

MUs NodesMU_A Host_1MU_B Host_2

Host_1 Host_2

JOB_YOn MU_A

JOB_Xon MU_B

dependent on… dependant on …


Shutdown

Back-up

Updates

Restart

A1A2

A3

Re-use jobs for similar contextsRe-use jobs for similar contextsSession1 = 4 jobs

A30

30 Applications

In most scheduling tools: Jobs are host-specific, application-specific1 stream per application: 5 objects x 30 applications=150 objects to maintain

In most scheduling tools: Jobs are host-specific, application-specific1 stream per application: 5 objects x 30 applications=150 objects to maintain

In $Universe: Jobs are generic / re-usable, Same stream executed against 30 MUs: 5 objects + 30 MUs = 35 objects to maintain

In $Universe: Jobs are generic / re-usable, Same stream executed against 30 MUs: 5 objects + 30 MUs = 35 objects to maintain

Results: Fewer objects (lower cost), ease of administration, etc. Results: Fewer objects (lower cost), ease of administration, etc.

Generic ProcessingGeneric Processing


1. To execute jobs on a hierarchy-based environment2. To adjust the environment to physical changes1. To execute jobs on a hierarchy-based environment2. To adjust the environment to physical changes

CONSOLIDon MU=REGION

REVENUEon MU=TYPE S (all stores, generically)

dependent on…

Region

Store1 Store 2 Store 50

JOBREVENUE

JOBREVENUE

JOBREVENUE

JOB CONSOLID

JOBREVENUE

Store 51

Hierarchical Processing


1 MU name and label

2 Name: 10 alphanumerical characters

3 Management Unit Types

For generic addressing of MUs

MU type = first letter of MU code

4 Node of residenceAn MU resides on one and only one node

One node may host any number of MUs

5 Authorized Universes

1 2

3

4

Make sure all M.U.s of a similar nature start

with the same character

5

Object Types in Dollar Universe

Day 1 – June 11, 2010



Object types in Dollar UniverseWorkflow of Development




Review of Day 1


JobsFunctional LogicScriptsRun Books

Uprocs

Resources

SessionsSchedulesOperations Logic

Calendars

Management Units Execution

Context

Tasks

Schedule InstanceTechnical SpecificationsTime Specifications

Scheduling Rules


Command

Language

Procedure

Conditions

Completion instructions

- A command language procedure + its application’s logic

Definition

1. May contain specific Dollar Universe variables / commands (steps, messages, parameters, etc)

2. Should return an exit code indicating success or failure to Dollar Universe

CL File

May require: 1. Events 2. ResourcesConditions

Completioninstructions

Instructions to purge specific events (opposite of a condition)

Create a job definitionCreate a job definition

Job ID

A Uproc runs : - on a Management Unit - in a Session- under a submission user account - for a Processing Date.

Uprocs


Written in command language (Bourne Shell Script, DOS BAT, etc.)

Internal procedures

The script is built into the Uproc and becomes a Dollar Universe object.

The CL procedure is stored by default in the Area’s Uproc directory.

The script is transferred, distributed or duplicated with the Uproc.

This is the preferred method of coding Uprocs.

External procedures

Only the access path to the script is stored in the Uproc's General Information

Dollar Universe does not manage the script and is common to all Areas.

Never used at University of Manitoba (Will be explained soon).

The CL procedure should be:

Independent of its operating environment (no hard coding of machines, Userids, Passwords, Run Env, etc.)

Consistent in terms of functional, technical and time constraints

Modular


UPROC Editor• Allow you to view and

change of the various attributes of a UPROC.

• The General Tab: Description, UPROC Type, Execution Information, Status Management.

• The Right Pane is for Dependancy information to assess whether to run this job. This is called the “Launch Formula”


UPROC Editor – Classes• Used to define what

Class of job this is.• Used to define what

classes this job cannot run with.

• One method of collision avoidance.


UPROC Editor – Events• Defines: Functional

Period, Memorization, Keep Events, and Completion (cleanup) Instructions.

• Critical to dependancy scheduling.

• More later ...


UPROC Editior – Variables• Variables passed to the

UPROC.• 3 Standard Variables:

DC_ENV, DC_ONCALL, DC_PRODUCTID

• Can be of type: Text, Numeric and Date

• Maximum Length set.• Default Value set.


UPROC Editor – Email

• Can be used to send Email messages to people, pagers, lists.

• Can send on all or certain conditions.

• Can include the LOG or not.

• DC_UPROC_FUNCTIONS and DC_FUNCTIONS will email DC_ONCALL as well.


UPROC Editor – CL Edit

• Used for developing and editing Internal CLs.

• TSE is the default editor.

• Always copy the template UPROCs and then edit.

• Never delete the executable code that is included.

• Comments must be changed.


• An external sequencing model of a group of UPROCs.

• Made up of a Header, Applications and a Trailer.

• UPROCs can be launched in parallel or serially.

• Simple conditions OK and KO are set in the Session.

• Advanced conditions are set in the UPROCs via the Launch Formula.

• They should have a single exit point through the Trailer. (Good Practice. Requires advanced conditions in the Trailer ... the Launch Formula)

Sessions


Day 1 – June 11, 2010








Review of Day 1

In $U objects can only be transferred down Areas in a Company.

EG. APP -> SIM -> EXP

As well, objects must be promoted across Companies at to the same Area or downward.

Thus we have made to following standards to facilitate this:

All object source must reside in Area APP of ASDEVL.

All changes must be done in Area APP of ASDEVL and transferred downward.

All object promotions must occur between Area SIM on ASDEVL to Area SIM on ASPROD.



Request fromAnalysts

Create/ChangeObjects

Launch Syntax and Flow Tests Transfer to SIM

Launch Tests Against TestEnvironment

Evaluate TestsTransfer to EXPInform Analysts

QA Testing Against NEXT Environment

Create ExportsFor Promotion

Create/ChangeDocumentation

Create PromoteRequest to

Prod Support

Review PromoteRequest

Promote ObjectsASDEVL-SIMASPROD-SIM

Transfer to EXP

Request Marked

Resolved

RequestVerified and

Closed


Launching an Object• Select “Launches

and Executions” from Area Application.

• Right Click in the right pane and Click “Create”.


Launch a Uproc• Click Uproc and MU• Set MU to

L_DCLPROD2• Set Uproc to

EE_ASHWENV• Click “GO”


• Select or type in your userid

• Click “OK”• You just launched a

Uproc ....


• Select “Execution” below “Launch”

• You should see all of the Uprocs executing and their status.

• Right click on one of them that is green and lets look around.

Assignment 1 – Simple Session

Day 1 – June 11, 2010





Assignment 1 – Build a simple SessionTASK Objects – Schedule Instances


Review of Day 1

Assignment 1 – Simple Session

• Create a Session called EE_SxxASN1

• For xx use your First and Last Initials.

• Create Header UPROCs EE_HxxASN1 and EE_TxxASN1 from the templates.

• Add child EE_ASHWENV to the Header.

• Add the Trailer to EE_ASHWENV

• Test it under Area=APP using Launch under Area=APP, Launches and Executions.

• Examine your output using Execution. Right click on your UPROCs and Select Job Log.

• If it works as expected, Transfer your objects to Area=SIM and test it again using Launch under Area=SIM, Launches and Executions.

• Examine your output using Execution and right click on your UPROCs and Select Job Log.

• Do you notice the difference ?

TASK Objects

Day 1 – June 11, 2010






TASK Objects – Schedule InstancesAssignment 2 – Build a simple Task

Review of Day 1

Task Objects

Task Editor - General• Basic Task Attributes

– MU, Session and Uproc– Task Type:

• Scheduled – Date/Time Based.• Provoked – Dynamically launched

by another process.• Cyclical – Run multiple times per

Day.• Optional – Run mid-session.

– Scheduling Criteria.• Launch Window.• Calendar Rule to follow.

Task Objects

Task Editor – Technical

• At this point in time, only the User Account needs to be set.

• In future, we will use DQM to define multiple queues for load balancing and failover.

Task Objects

Task Editor – Variables

• Variables that are set in the Header Uproc appear here.

• Launch overides Task

• Task overides Uproc

• This is a good way to make your Uprocs generic.

Task Objects

Task Editor – Launch Windows

• Launch Window Types:– Serial – calculated from the end

of the launch window. (usual).

– Parallel – Use dates/times of each specified window regardless of completion time (avoid).

• Generally just use the General Tab unless you want to fine tune the parameters.

Task Objects

Task Editor – Rules

• Use only if you have multiple scheduling rules during the year.

• EG. Normally run weekly on Monday except during Registration then run daily.

Task Objects

Task Editor – Dates

• Again an exception for special rules.

• EG. Run during Christmas week when normally you use a rule to exclude holidays.

• EG. Patch Sunday.

Assignment 2 – Simple Task

Day 1 – June 11, 2010







Assignment 2 – Build a simple TaskReview of Day 1

Assignment 2 – Simple Task

• Using the Session that you created in Assignment 1, create a Task to run it.

• Use the following criteria:– Don’t run it on weekends nor holidays.– Start the Task after 2:00am.– Allow for a 45 minute start window.– If it cannot start nor execute, email yourself.– If it fails, email yourself and send the log.– Override the environment to run against next.– Run the job under your account.

• Launch the Task in SIM and review its status.• Check it on Monday to see if it worked.

Review of Day 1

Day 1 – June 11, 2010








Review of Day 1

Dependency Scheduling

Day 2 – June 18, 2010



Resources


Non-Similtaneity

Classes




Review of Day 2


Type of Scheduling Conditions

• Launch Formula – Defining dependencies between Uprocs and Sessions.

• Non-Simultaneity – Uprocs that cannot run together.

• Resources – Physical and Logical Semaphores.

• Classes – Types of Uprocs that are incompatible.

Launch Formula

Typical Launch Formula

• EE_HGATEST runs if there is no other EE_SGATEST sessions active.

• EE_ASLP010 runs.

• EE_ASLP015 runs only if EE_ASLP010 is successful.

• EE_ASLP025 runs only if EE_ASLP010 fails.

• EE_TGATEST runs when the session has completed.

• How do we accomplish this ?

Launch Formula

EE_HGATEST runs if there is no other EE_SGATEST sessions active.

• First Edit Uproc EE_HGATEST Events to test for the completion of the event.

Launch Formula

Setup the Completion Instruction

• If EE_TGATEST completed successfully, purge the event if the other criteria match.

• IE. Session, MU and Processing Date.

Launch Formula

Set Session Criteria• Any session won't work as we

want to test for completion of the session.

• Same session will work as we are testing for a previous session of the same name.

• Same session and execution won't work since we want a previous session.

• Specific session will work since but we must select the same session name.

Launch Formula

Select the MU Criteria• Same MU would generally

be correct.

• Specific MU would work but would be hard coded :-(

• MU Type would be preferable if the Session might run on multiple MUs.

• HDP Scary Stuff :-0

Launch Formula

Processing Date Criteria• Any/Any/Any is the safe

choice to avoid taking the date into account.

• There are multiple combinations using Same, Specific Value or Offset (+/-).

• Offset requires that Functional Period be set, so be careful when using it !!!

Launch Formula

• The Resultant Launch Formula using Events … for the Header at least.

• Now we need to set the Event condition to be created in the Trailer.

Launch Formula

Setup the Event in EE_TGATEST

• Check the Radio Button “One memorization per processing date” which is default in the Trailer Template.

• If there will not be any event tests, always select “No memorization” !!!

• Full memorization is seldom required, but can be used when an event needs testing multiple times.

• Remember that it needs to be purged from Job Events !!!

Launch Formula

Rules of Thumb

• Event testing should always be done using only Headers and Trailers.

• Each Session must have exactly one Header and one Trailer.

• Events are tested by the Header.

• Events are set by the Trailer.

• Session Completion is always tested by the Trailer.

• If event testing is ever done in an application uproc, then the uproc cannot be used in another session without the same dependent uprocs. So don’t risk it.

Launch Formula

Dependency Conditions

• Used primarily to test for Session Completion IE. When to run the Trailer.

• As the Session designer you must decide what the acceptable criteria is to signal Session completion.

• There are no right nor wrong answers, only philosophical points of view and repercussions.

Launch Formula

Logical Conditions

• AND is represented by serial (vertical) boxes.

• OR is represented by parallel (horizontal) boxes.

• Each box is a condition.

• Think of water flowing through hoses. Water must flow from top to bottom in order for this Uproc to run.

• Why is this picture overkill ?

Launch Formula

Looking Inside the Box

• Does this look familiar ?

• Like Events, you check the Uproc Status and then qualify it with Session, MU and Processing Date.

• A word of warning from experience … avoid negative conditions like Excluded or Inverted. They tend to bite you.

Launch Formula

Session Criteria• You must use Same

session and execution to avoid bumping into any other one.

Launch Formula

MU Criteria• Again, Same MU is

typical, but MU Type may be advisable if you run on multiple MUs.

Launch Formula

Processing Date Criteria• Unless you have a very very long running schedule, Same/Same/Same will do the trick.

Assignment 3

Day 2 – June 18, 2010


Assignment 3 – Build an optimized SessionResources


Non-Similtaneity

Classes




Review of Day 2

Assignment 3

Create a Session EE_SxxASN3 with a matching Header and Trailer.

Ensure it does not run when other Session instances are outstanding.

It needs to run the following Uprocs:

EE_ASLP010, EE_ASLP015, EE_ASLP020, EE_ASLP025, EE_A_OK and EE_AKABOOM.

All jobs except EE_ASPL025 depend on output generated by EE_ASLP010.

EE_ASLP020 must run before EE_ASLP015.

If EE_ASLP020 should fail, then run EE_AKABOOM.

If EE_ASLP020 succeeds, the run EE_A_OK.

EE_ASLP025 can run independently of the others, but it is critical to session completion.

Hint … you will need to seed a Job Event to prime the process.

Workshop Outline

Day 2 – June 18, 2010



ResourcesAssignment 4 – Build a Session using Resources

Non-Similtanity

Classes




Review of Day 2

Resources

Resources• Types: File, Logical, JMS.

• Can be Exclusive – Yes or No to allow or disallow launches.

• Or Non-Exclusive – Use Quotas to restrict resources.

• Test for Existence or Absence.

• Allocation can create/destroy if Automatic or depend on external factors Manual.

• Can cause Failure or Wait.

• Used by Launch Engine and Supervisor Engine.

Resources

File Resources• Use when in a mixed

Universe/Cron situation.

• File Resources MUST be on the same MU as the Uproc.

• Avoid in favour of Logical if possible.

• Names MUST begin with ss_RF – where ss is a System Identifier. The remaining 15 characters are all yours.

Resources

Logical Resources

• Use when in a pure Dollar Universe situation.

• Logical Resources exist on any MU.

• Use Logical if at all possible.

• Names MUST begin with ss_RL – where ss is a System Identifier. The remaining 15 characters are all yours.

Resources

JMS Resources

• Use when in a Java/Dollar Universe situation.

• JMS Resources can exist on any MU.

• Use when required.

• Specifics are set using the JMS Message editor button.

• Names MUST begin with ss_RJ – where ss is a System Identifier. The remaining 15 characters are all yours.

Resources

Setting Parameters• Deallocation Mode

– Automatic will delete the resource upon completion.

– Manual, let other do it.

• Access Mode– Exclusive: Yes/No

– Non-Exclusive: Use Quotas for limiting resources.

• Scan Frequency is how often the Launch and Supervisor Engine test the resource.

• JMS only has Scan Frequency.

Resources

Using Resources• Resources are used in the

Launch Formula for Uprocs, hence back to the Uproc editor.

• Proposition defines whether they should be there (Expected) or not (Excluded) also whether to Fail if this is not the case (Fatal Condition) or wait.

Resources

Using Resources• Exclusive Resources allow

for checking conditions: Existence, Size, etc.

• Non-Exclsuive Resources defines now many units of quota 1 and 2 are required by this Uproc.

• JMS does not use this.

Resources

• Management Unit– You can specify

the MU to check

• Processing Date– You can define the

Date conditions.

Assignment 4

Day 2 – June 18, 2010



Resources


Non-Similtaneity

Classes




Review of Day 2

Assignment 4

Create a new Session EE_SxxASN4 with its Header and Trailer Uprocs.

Create a Uproc EE_AxxCRF1 which will copy /etc/services to /tmp/EE_SxxASN4.txt. Use a Semaphore to let people know when the file is ready for further processing.

Create a Uproc EE_AxxFTP1 which will wait until File /tmp/EE_SxxASN4.is.ready exists on MU DCLPROD2 then it will FTP /tmp/EE_SxxASN4.txt to DCUPROD1 in /tmp/EE_SxxASN4.txt in binary mode with UMASK 007.

Create Uproc EE_AxxFTP2 which will transfer /tmp/EE_SxxASN4.txt from DCUPROD1 to /tmp/EE_SxxASN4.data on DCLPROD2. Respect the restriction of maximum ftps for DCLPROD2.

To allow Production Support to control the load on DCLPROD2 and to quiesce Production, respect the maximum job limit.

Make sure that this session only runs if there are no other instances of EE_SxxASN4 active.

Non-Similtaneity

Day 2 – June 18, 2010



Resources


Non-SimiltaneityClasses




Review of Day 2

Non-Similtaneity

• Set using the Uproc Conditions editor.

• Select Non-Simultaneity condition.

• Identify the Uproc that this Uproc cannot run with.

• The Launch Formula will do the rest.

Classes

Day 2 – June 18, 2010



Resources


Non-Similtaneity

ClassesInter-Session Dependency Scheduling



Review of Day 2

Classes

Prevent Simultaneous Execution for a Group of Uprocs

• Non-Simultaneity on Steroids.

• Classes are defined in the Uproc Class editor

• Limited to 6 characters. Must start with ss_ where ss is the System Identifier, leaving three characters to be creative with.

• Use with restraint and with discussion with Technical Application Support.

Classes

Prevent Simultaneous Execution for a Group of Uprocs

• Uproc class in incompatibilities are set in the Uproc editor – Classes Tab.

• A Uproc can be a member of only one Class.

• A Uproc can be incompatible with multiple classes.

• A Uproc can be incompatible with its own class.

Inter-Session Dependencies

Day 2 – June 18, 2010



Resources


Non-Similtaneity

Classes




Review of Day 2

Inter-Session Dependencies

• Sessions by definition will often interact with each other.

• Sessions will likely contain the same Uprocs. So putting such checking inside a Application Uproc will lock it to that session. This is a bad thing.

• Headers and Trailers belong to their own Sessions.

• All dependencies must be between the Trailer of one Session setting an event and the Header of another Session checking the event.

• We have already setup Headers and Trailers to prevent collisions with their own Session instances.

Assignment 5

Day 2 – June 18, 2010



Resources


Non-Similtaneity

Classes




Review of Day 2

Assignment 5

The three Sessions that you have developed need to run dependently every work night.

First Session EE_SxxASN1 must run successfully.

Session EE_SxxASN3 must wait for completion of EE_SxxASN1 and must also wait until Resource EE_RFxxONLINE is released. (Note the Semaphore EE_RFxxONLINE should be found in /home/adpc/cservap/tmp/common/locks)

Session EE_SxxASN4 can run anytime provided that Session EE_SxxASN1 is finished.

Create Session EE_SxxASN5 to setup for the next day runs.

All Sessions need to become Tasks.

What needs to be set carefully if EE_SxxASN1 starts at 2:00am.

Also you have discovered that the SLP Uprocs cannot run together in the system. Change things such that this conflict is avoided. Or has it been done already ?

Multi-Platform Scheduling

Day 2 – June 18, 2010



Resources


Non-Similtaneity

Classes



Mulitple Platform SchedulingReview of Day 2


• There will be times when you must run Uprocs on more than one platform, eg. Solaris, Windows and Linux.

• Here is an example and we will discuss issues that need to be addressed.

• This example can be found in nodes DCUPROD1 and DCWPROD1 in Area Production in Session EE_SMD001 courtesy of Matt Dunlop its author.


• All objects, namely Uprocs, Sessions, Tasks, Calendars, Rules, Resources that are going to be used MUST EXIST on all nodes that the Session will be running on.

• Use of File Resources is restricted to the node on which it exists. With NFS, this can be used on Linux and Solaris if you share the same file system but not Windows.

• Parameters can be passed, but only using Dollar Universe “uxset var” dc_uproc_set_session_variables and dc_uproc_get_session_variables is restricted to a single node session.

• Ok lets run this !!!

Review of Day 2

Day 2 – June 18, 2010



Resources


Non-Similtaneity

Classes




Review of Day 2

Date post:	15-Mar-2018
Category:	Documents
Upload:	duongliem
View:	245 times
Download:	8 times

ORSYP Dollar Universe Workshop - Aldersoftaldersoft.ca/Dollar-Universe-Workshop.pdfORSYP Dollar...

Documents