TIBCO EMMA Operation Manual - · PDF filePage 3 of 61 TIBCO EMMA Operational Manual –...

Page 1 of 61

TIBCO EMMA Operational Manual – Supplier

Insert date here

© 2016 Capita Managed IT Solutions. All rights reserved.

02/08/2016 Version 1.2

TIBCO EMMA Operation Manual

Page 2 of 61


Insert date here


Copyright

This document contains material the copyright in which is owned by Capita Managed IT

Solutions Ltd and its associated companies (“Capita”). All rights reserved. No part of this

publication may be reproduced or transmitted in any form or by any means, or stored in any

retrieval system of any nature without prior written permission, except for permitted fair dealing

under the Copyright, Designs and Patents Act 1998, or in accordance with the terms of a

licence issued by the Copyright Licensing Agency in respect of photocopying and/or

reprographic reproduction. Application for permission for other use of copyright material

including permission to reproduce extracts in other published works shall be made to the

publishers. Full acknowledgement of author, publisher and source must be given.

Trade Marks

Capita and the Capita logo are registered trademarks. All other trademarks or product names

mentioned in this document are acknowledged as being owned by their respective owners or

proprietors.

Confidentiality

All of the information in this document is provided in confidence for the sole purpose of your

adjudication and must not be used for any other purposes, or published or disclosed, wholly or

in part, to any other party without Capita’s prior written permission, and further, all of the

information must be held in safe custody.

Contacts

Hugh O’Neill

Service Delivery Manager

Capita Managed IT Solutions

Hillview House

61 Church Road

Newtownabbey

BT36 7LQ

T 028 9085 9085

F 028 9085 9086

E hugh.o’[email protected]

www.capita-mits.co.uk

mailto:[email protected]

Page 3 of 61


Insert date here


Revision History

Version Date Description

1.0 10/06/2016 Issued to Suppliers

1.1 01/07/2016 Process starter check added section 8.2 and IBM HTTPservice names updated section 7.8 & 8.5.

1.2 02/08/2016 Update to section 7 & 8 for starting and stopping the EMMA. Chapter 10 added regarding account passwords

Page 4 of 61


Insert date here


Contents

1 Overview and Purpose ......................................................................... 6

1.1 Objective ..................................................................................... 6 1.2 Scope of document ..................................................................... 6 1.3 Document Structure .................................................................... 7

2 Solution Overview ................................................................................ 9

2.1 Solution Architecture ................................................................... 9 2.2 Folder Structure......................................................................... 10 2.3 EMMA/Hub Message States ..................................................... 13

3 EMMA Health Checks ......................................................................... 15

3.1 Performance .............................................................................. 15 3.2 Disk Space ................................................................................ 16 3.3 Oracle Database ....................................................................... 18 3.4 Deletion of Log Files ................................................................. 25 3.5 Ledger files ................................................................................ 28 3.6 Webforms .................................................................................. 30

4 Backup & Recovery ............................................................................ 32

5 Manual Archiving Batch Script ......................................................... 34

6 Supplier Checklist .............................................................................. 37

7 EMMA Shut-down ............................................................................... 39

7.1 Pre-Shutdown steps .................................................................. 39 7.2 Stop TIBCO services via TIBCO Administrator ......................... 40 7.3 Start TIBCO Administrator ........................................................ 40 7.4 Amend browser security settings .............................................. 40 7.5 Login to TIBCO Administrator ................................................... 42 7.6 Stop TIBCO Services ................................................................ 42 7.7 Stop TIBCO Windows Services ................................................ 44 7.8 Stop WAS server then IBM HTTP server .................................. 46 7.9 Stop Oracle database ............................................................... 47

8 EMMA Start-up .................................................................................... 49

8.1 Start Oracle Database ............................................................... 49 8.2 Start TIBCO windows services.................................................. 49 8.3 Start TIBCO services via TIBCO Administrator ........................ 51 8.4 Start TIBCO Services ................................................................ 52 8.5 Start IBM HTTP server and WAS server .................................. 56

9 Software Update & Patching ............................................................. 58

9.1 Windows Operating System ...................................................... 58 9.2 Applications ............................................................................... 58 9.3 Oracle Database ....................................................................... 59 9.4 Anti-Virus Software ................................................................... 59

10 Application User Passwords ............................................................. 61

Page 5 of 61


Insert date here


Figure 2.1-2: Architecture of Small EMMA .................................................................................... 9 Figure 2.2-2 Inbound Folder Structure ........................................................................................ 12 Figure 2.3-1EMMA to Hub Application State Transitions ............................................................ 14 Figure 3.1-1 Performance CPU .................................................................................................. 15 Figure 3.2-1: Opening Windows Explorer ................................................................................... 16 Figure 3.2-2: Checking Disk Space ............................................................................................. 17 Figure 3.3-1 Checking Statistics of Job Run ............................................................................... 18 Figure 3.3-2: Monitoring Status of Purge EMMA Job.................................................................. 19 Figure 3.3-3 Monitoring PURGE_BC Job ................................................................................... 20 Figure 3.3-4 Missing UNSENT files............................................................................................. 21 Figure 3.3-5 Oracle Reorganisations .......................................................................................... 23 Figure 3.3-6 Oracle Reorganisations .......................................................................................... 24 Figure 3.4-1 Listener Logs .......................................................................................................... 25 Figure 3.4-2 Alert Logs ................................................................................................................ 26 Figure 3.4-3 Trace logs ............................................................................................................... 26 Figure 3.5-1 Ledger files ............................................................................................................. 28 Figure 3.5-2 BC Ledger files ....................................................................................................... 29 Figure 3.6-1 Webforms Log files ................................................................................................. 30 Figure 3.6-1 Manual Archiving .................................................................................................... 34 Figure 3.6-2 Manual Archiving .................................................................................................... 34 Figure 3.6-3 Manual Archiving .................................................................................................... 34 Figure 3.6-4 Manual Archiving .................................................................................................... 35 Figure 3.6-5 Manual Archiving .................................................................................................... 35 Figure 3.6-6 Manual Archiving .................................................................................................... 35 Figure 3.6-7 Manual Archiving .................................................................................................... 35 Figure 7.1-1 Dashboard take EMMA offline ................................................................................ 39 Figure 7.3-1 Start TIBCO Administrator ...................................................................................... 40 Figure 7.4-1 Amend browser security settings ............................................................................ 40 Figure 7.4-3 Amend browser security settings ............................................................................ 41 Figure 7.4-4 Amend browser security settings ............................................................................ 41 Figure 7.4-5 Amend browser security settings ............................................................................ 42 Figure 7.5-1 Login to TIBCO Administrator ................................................................................. 42 Figure 7.6-1 Sequence of steps to stop EMMA - TIBCO Application Components .................... 43 Figure 7.7-1 Stop TIBCO Windows Services .............................................................................. 44 Figure 7.7-2 Stop TIBCO Windows Services .............................................................................. 44 Figure 7.7-4: Stopping a windows service .................................................................................. 45 Figure 7.7-5 Stop TIBCO Windows Services .............................................................................. 45 Figure 7.8-1 Stop WAS server .................................................................................................... 46 Figure 7.8-2 Stop WAS server .................................................................................................... 46 Figure 8.4-1 Sequence of steps to re-start EMMA - TIBCO Application Components ............... 52 Figure 8.4-3 Checking Process Starters ..................................................................................... 53 Figure 8.4-4 Checking Process Starters .................................................................................... 54 Figure 8.4-5 Checking Process Starters Status ......................................................................... 54 Figure 8.4-6 Enabling Process Starters ..................................................................................... 55 Figure 8.4-7 Process Symbol ...................................................................................................... 55 Figure 8.4-8 show a full list of EMMA applications ...................................................................... 55 Figure 8.5-1 Start IBM HTTP server and WAS server ................................................................ 56 Figure 8.5-2 Start IBM HTTP server and WAS server ................................................................ 56

Page 6 of 61


Insert date here


1 Overview and Purpose

Suppliers in both the NI & RoI Retail Electricity Markets require an Electricity Market Message Application (EMMA) to exchange market messages with their respective Market Operators.

Suppliers have a responsibility to

Ensure the computer hardware and software running the EMMA solution is monitored on a regular basis

Any problems discovered are corrected before they impact the performance and stability of their EMMA and the market processes overall

That their EMMA solution is shut down and started in the correct sequence

1.1 Objective

The objective of the document is to provide Suppliers with guidance on the required EMMA checks that they should perform on a regular basis and the correct shutdown and start-up process and sequence.

1.2 Scope of document

The Scope of the document covers the following areas:

Solution Overview

EMMA Health Checks

Backup & Recovery

Manual Archiving Batch Script

Supplier Checklist

EMMA Shut-down

Starting the EMMA

Software Update & Patching

Page 7 of 61


Insert date here


1.3 Document Structure

1.3.1 Solution Overview

Section 2 describes the server build, network setup and messages flow of the

solution.

1.3.2 EMMA Health Checks

Section 3 discusses the regular health checks that a supplier should carry out to

ensure that their EMMA remains available and operational. It covers how to

check CPU, memory, database status and log files.

1.3.3 Backup & Recovery

Section 4 covers the supplier’s responsibility in relation to maintaining backups

of the EMMA systems.

1.3.4 Archiving

Section 5 shows how to run the manual archiving process and discusses the

workings of the automatic archive process.

1.3.5 Supplier Checklist

Section 6 is the checklist for suppliers to use while covering off their health

checks. We would recommend having this to hand for your regular checks.

1.3.6 EMMA Shut-down

Section 7 takes you through the required step-by-step process to take an

EMMA offline and the restart sequence. This would be the starting point for any

planned outages.

1.3.7 Starting the EMMA

Section 8 is the starting point for recovering from any unplanned outages or

restarting the EMMAs applications after a planned outage. It is a step-by-step

guide on the required procedure to start your EMMA system.

1.3.8 Software Update & Patching

Section 9 covers the details regarding the updates or patching of the windows

operating system, TIBCO applications, Oracle database and the impact of

having anti-virus software installed on the EMMAs servers.

1.3.9 Software Update & Patching

Section 10 covers application user passwords and their settings

Page 8 of 61


Insert date here


Section 2

Solution Overview

Page 9 of 61


Insert date here


2 Solution Overview

2.1 Solution Architecture

Your EMMA solution is made up on the following components.

Component Description

Gateway Server Receives the inbound market messages from the HUB

and passes them to the interior server. The Gateway

server sits in the DMZ and adds an extra level of security

to the system by protecting the interior server.

Interior Server Processes the market messages received from the HUB

in readiness for the supplier system and packages market

messages rom the supplier system to send to the HUB.

Database Server The storage of market messages status and application

configuration.

Depending on the size of supplier you are, you may have a large EMMA - where all these components are deployed across different physical (or virtual) servers; or a small EMMA - where all components are deployed on a single physical (or virtual) server, as shown in the diagrams below.

Figure 2.1-1: Architecture of Large EMMA

Figure 2.1-2: Architecture of Small EMMA

Page 10 of 61


Insert date here


2.2 Folder Structure

2.2.1 Outbound Folder Structure

Your EMMA uses a pre-defined folder structure to process, send and store outbound messages that are exchanged between the EMMA and HUB, the diagram below outlines what this outbound folder structure looks likes on your EMMA and gives a brief explanation as to the purpose of each folder.

Figure 2.2-1 Outbound Folder Structure

If a single EMMA instance is managing messages for two or more different Supplier IDs, then one Supplier IDs is declared as primary Supplier, while all other IDs are classed as secondary (Any number of secondary IDs are supported on a single EMMA instance). This nomination influences the naming conventions used for the folders on the EMMA server.

o With reference to the outbound folder structure above’, ‘Sxx’ is the primary Supplier ID and ‘Syy’ and ‘Szz’ are secondary suppliers.

Page 11 of 61


Insert date here


o If the EMMA installation is managing messages for a single Supplier, then the folder setup is as displayed but without the replication of supplier ID-specific folders.

Folder Description

Outbound

The outbound folder is common and always named after the primary supplier ID.

Suppliers will place any new messages to be sent to the HUB in this folder.

TIBCO Business Works component picks the Supplier’s message from the Outbound folder and processes it where it is moved to the Processing folder and validated it against the schema.

Processing

The processing folder is Supplier specific and subdivided into 60 subfolders, one for each second of the minute in which the message was sent.

If valid the message is stored in a Processing sub-folder while it is passed to BusinessConnect (BC) for sending.

Invalid If invalid the message is moved to the Invalid folder, a 601 Rejection message is created in the Inbound folder.

Unsent If no acknowledgement is received the message times out and is moved from the Processing to the Unsent folder.

Failed If negatively acknowledged the message is moved from Processing to the Failed folder. The negative acknowledgement is stored in the AckErr folder.

Archive/Outbound

If positively acknowledged the message is moved from Processing to the Archive/Outbound/YYYY/MM/DD sub-folder for that day. The positive acknowledgement is stored in the Ack folder.

Both the Failed and Unsent folders are covered by the File Recovery Process so messages should not remain in this folder indefinitely. These folders need to be monitored and older messages that are not cleared need to be investigated and if not resolved need to be reported to Capita ([email protected]) for guidance.

Page 12 of 61


Insert date here


2.2.2 Inbound Folder Structure

The general message flow and folder structure for inbound messages is illustrated below.

Figure 2.2-2 Inbound Folder Structure

The figure above shows the main folder setup and naming convention for both single and multiple supplier setup. If the EMMA is managing messages for more than one Supplier ID, then there is an Inbound and Archive/Inbound folder for each Supplier ID being hosted.

If the same Supplier ID is used in both jurisdictions then the messages will appear in the same folder but will be distinguished by the market prefix ‘DSO’ for RoI messages and ‘TDO’ for NI messages.

Folder Description

Inbound The Webforms Message Search function available on the Main Menu allows users to search for and view the status of inbound messages. See Webforms user guide.

Archive/Inbound

The Archive\Inbound folder is subdivided by Year, Month and Day subfolders containing messages received on a particular day.

The messages in the Archive\Inbound folder are held for a certain period until they are archived to new location in zipped folders.

Page 13 of 61


Insert date here


2.3 EMMA/Hub Message States

During the exchange of a message between the EMMA and the HUB, each message will go through various states, which are held within the database, this allows the solution to know the current delivery status of each message and the next stage in the delivery process.

Status Description

Received This is the first status of a market message once it has been

picked up by the EMMA

Processed Processed is the state of a market message while being

handled by the application

Invalid Invalid market messages need to be investigated to be

corrected and resent.

No Supplier If a supplier ID has not been setup on the EMMA then the

market message can’t be received or sent

Pending Pending is waiting on an acknowledgement for the current

processing section of the application

Failed The message has failed to be sent and will be retried

automatically. If it still cannot be delivered then it needs to be

manually investigated.

Complete The message has been fully processed and will be in the

inbound and archive inbound folders ready for the supplier

system to pick up.

Page 14 of 61


Insert date here


Figure 2.3-1EMMA to Hub Application State Transitions

Page 15 of 61


Insert date here


3 EMMA Health Checks

It is the responsibility of the supplier to ensure that their EMMA remains available and operational, and as such there are a number of health checks that must be carried out at the recommended frequency outlined below.

Please note that this is only a recommended frequency and if you feel your EMMA would benefit from more frequent checks then these should be carried out at your discretion.

Health Check Recommended Frequency

Performance (CPU/Memory) Monthly

Disk Space Monthly

Oracle Database Weekly/Daily

Log & Ledger Files Weekly

If during the process of carrying out the above health checks you have any concerns regarding your EMMA, please contact Capita ([email protected]) for guidance.

3.1 Performance

You can view the current performance statistics using the Performance Tab, within the Windows Task Manager, if you are unsure how to access this tab, then please revert to the Microsoft website for guidance, a screenshot has been included below showing this tab.

Figure 3.1-1 Performance CPU

To ensure an accurate view is taken of the performance it is important that you perform this check at periods when the EMMA is actively sending and receiving messages. If you notice that your memory or CPU is higher than expected (sustained periods of processing where CPU is greater than 85/90%) then this should be investigated and if required raised to Capita.

Page 16 of 61


Insert date here


3.2 Disk Space

Your EMMA is used to store your messages both as part of their transmission between the HUB and EMMA (and vice versa) and also as part of archiving. As such the space on your disks may start to fill overtime. How quickly will vary depending on the size of EMMA you have implemented (Large vs Small) and the volume of daily messages received.

Should your solution reach an overall capacity for all (aggregated) disks greater than 80% then the EMMA will place itself into contingency and stop receiving any further messages. However, it is the responsibility of the supplier to ensure that the disk space on the EMMA is correctly monitored and managed, the steps below outline how a supplier can check the storage of each drive within their EMMA solution.

1. Open a Windows Explorer

Figure 3.2-1: Opening Windows Explorer

Page 17 of 61


Insert date here


2. Navigate on the right hand menu to ‘Computer’, where you will be shown the 3 drives used by your EMMA (c:/, d:/, e:/), as show below.

Figure 3.2-2: Checking Disk Space

3. If you notice that space is running low then you should look to remove files that are no longer required, your solution mainly uses the E:/ drive for storing messages, therefore this is the best drive to try and remove any files to reclaim storage – the C: and D: drives are used to store the operating system and TIBCO application and therefore any files should only be removed from these drives by someone who is qualified. If you need assistance to confirm files that can be safely removed please contact Capita at [email protected]


Page 18 of 61


Insert date here


3.3 Oracle Database

Your EMMA solution utilises Oracle database and it is essential that the health of this database is maintained. We have outlined a number of basic health checks that should be performed on your database by a qualified Oracle DBA. Your DBA may wish to include other Oracle bespoke health checks as they see fit to align with your organisations IT processes.

In the checks outlined below, it is assumed that the database version is Oracle 12c as per the EMMA Technical Specifications issued.

3.3.1 Checking Statistics of Job Run (Weekly)

An Oracle job – RUN_STATISTICS is scheduled to run every Sunday at 4 am. This job will gather the statistics of all the Oracle application schemas by calling a stored procedure – SP_RUN_STATS. The user can find the status of each by running the below query (using EMMASCHEMA account)

SELECT * FROM USER_SCHEDULER_JOB_RUN_DETAILS WHERE JOB_NAME =’RUN_STATISTICS’ ORDER BY LOG_DATE DESC;

Figure 3.3-1 Checking Statistics of Job Run

If the STATUS of the top row is ‘NOT SUCCEEDED’, then the supplier will have to report the error to Capita at [email protected]

If the job run failed on a particular Sunday, the same can be run on the same day or the next day after fixing the issue.


Page 19 of 61


Insert date here


3.3.2 Monitoring Status of PURGE_EMMA job (Daily)

The Oracle job PURGE_EMMA is defined under the schema – EMMASCHEMA and runs at 3am every day. The user will need to login to the EMMA database using the EMMASCHEMA account and run the below query to find out the status of the run of this purge job everyday:

SELECT * FROM USER_SCHEDULER_JOB_RUN_DETAILS WHERE JOB_NAME =’PURGE_EMMA’ ORDER BY LOG_DATE DESC;

Figure 3.3-2: Monitoring Status of Purge EMMA Job

If the STATUS of the top row is ‘NOT SUCCEEDED’, then the supplier will have to check the TAB_TR_SP_LOGS table for more details on the error and report the same to the Capita team – [email protected]. If the job run failed on a particular day, the procedure can be run manually on the same day or the next day after fixing the issue.


Page 20 of 61


Insert date here


3.3.3 Monitor PURGE_BC job (Daily)

The Oracle job PURGE_BC is defined under the schema – BCPURGE and runs at 4am every day. The user will need to login to the EMMA database using the BCPURGE account and run the below query to find out the status of the run of this purge job everyday.

Figure 3.3-3 Monitoring PURGE_BC Job

If the STATUS is not SUCCEEDED, then the user will have to check the TAB_TR_SP_LOGS table for more details on the error and report the same to the Capita DBA team – [email protected]. If the job run failed on a particular day, the procedure can be run manually on the same day or the next day after fixing the issue.

3.3.4 Missing UNSENT files

The following query needs to be run on the Emma to see if there are any UNSENT files which has got deleted from the file system (accidently) but the database records still exists. In such cases, an exception record is written to the TAB_TR_EXCEPTION table.

SELECT count(distinct mm_tracking_id) from TAB_TR_EXCEPTION WHERE STACK_TRACE LIKE ‘%\Unsent\% was not found%’;


Page 21 of 61


Insert date here


Figure 3.3-4 Missing UNSENT files

If there are any records found, then each of the messages (MM_TRACKING_ID) needs to be manually checked to see the next course of action. The missing files may be placed to automatically recover these transactions. If the missing files cannot be restored, then the data has to be manually removed from the database using the query below:

DELETE FROM TAB_TR_MM_TRANS_STATUS WHERE MM_TRACKING_ID IN (SELECT MM_TRACKING_ID FROM TAB_TR_EXCEPTION WHERE STACK_TRACE LIKE ‘%\Unsent\% was not found%’);

DELETE FROM TAB_TR_MM_STORE WHERE MM_TRACKING_ID IN (SELECT MM_TRACKING_ID FROM TAB_TR_EXCEPTION WHERE STACK_TRACE LIKE ‘%\Unsent\% was not found%’);

COMMIT;

Page 22 of 61


Insert date here


3.3.5 Oracle Reorganisations

The Business Connect related BCADMIN schema can grow fast due to the amount of traffic. Therefore it is important that this space is reclaimed if the BCADMIN schema size goes beyond certain limit.

If the size of BCADMIN_DATA_01 tablespace goes above 40GB used (Over 50% used), you can perform a one time reorg activity to reclaim the space from BCADMIN schema.

However, BCADMIN schema is full of Large Objects using SECUREFILE and the reorgs cannot be done with the database online. In order to perform the reorg’s, the Business Connect Interior server needs to be shut-down, hence an outage is required.

Once the application is shut-down, you can follow the below steps to reclaim space. As with all EMMA database activities, these steps should only be performed by your Oracle DBA.

Ensure that the database backups are not running on the Emma database as well as the purge jobs have completed. There should not be any database backups scheduled in the next 7 to 8 hours.

Obtain the size of the BCADMIN_DATA_01 tablespace as below

Page 23 of 61


Insert date here


Figure 3.3-5 Oracle Reorganisations

Contact Capita to get the reqired SQL file.

Connect to the Emma database using BCADMIN account and run the provided sql file. This may take about 7 to 8 hours to run (sometimes more depending on the space to be reclaimed and the database server cpu/memory allocations)

Run the below SELECT query to generate indexes which are not valid and run the ALTER INDEX commands generated

o SELECT ‘ALTER INDEX BCADMIN.’||INDEX_NAME||’ REBUILD;’ FROM DBA_INDEXES WHERE OWNER=’BCADMIN’ AND STATUS <> ‘VALID’;

Check the BCADMIN_DATA_01 tablespace usage now to ensure that the space is reclaimed.

Page 24 of 61


Insert date here


Figure 3.3-6 Oracle Reorganisations

As you can see, the space has reduced considerably.

Page 25 of 61


Insert date here


3.4 Deletion of Log Files

Log files should only be deleted when disk space on the actual partition holding the log files is over 80% of capacity.

The process for deleting log files is explained in the following sections.

3.4.1 Listener Logs

Oracle listener log location (Sample is shown for E drive but can be any drive based on your Oracle installation):

E:\app\oracle01\diag\tnslsnr\<server name>\listener\trace

Figure 3.4-1 Listener Logs

How to delete the listener log file without having to stop the database or listener service:

1. Admin privileges required

2. In the command prompt type > LSNRCTL

3. This will open the Listener control utility. Type LSNRCTL> set log_status off

4. Now the user can navigate to the file location and rename the file using Windows explorer.

5. Restart logging LSNRCTL> set log_status on

6. Once the new listener log file is created the user can safely delete the old log file.

Page 26 of 61


Insert date here


3.4.2 Alert logs

Oracle Alert log location (Sample is shown for E drive but can be any drive based on your Oracle installation):

E:\app\oracle01\diag\rdbms\<dbname>\<dbname>\alert

This creates a new log file after every 10 MB and the previous log files can be deleted to free up space.

Figure 3.4-2 Alert Logs

As seen above, all the files other than the current one (log.XML) can be deleted whenever there is any shortage of space. We would recommend this action to be taken when space is at over 80% of capacity.

3.4.3 Trace logs

Oracle Trace log location (Sample is shown for E drive but can be any drive based on your Oracle installation):

E:\app\oracle01\diag\rdbms\<dbname>\<dbname>\trace\alert_<dbname>.txt

Figure 3.4-3 Trace logs

Page 27 of 61


Insert date here


Here, the alert logs can be renamed to .old and then Oracle will start writing to the new log file which will be automatically created. Once this happens it is safe to delete the alert_emma05.old file.

3.4.4 Recycle bin

Please remember that when log files are deleted they are placed in the ‘Recycle Bin’ and disk space is not reclaimed until the log files have been deleted from it.

3.4.5 Users Tablespace

The following table shows the maximum size for each tablespace:

Schema Tablespace Max Size (GB)

EMMASCHEMA EMMASCHEMA_DATA_01 50

EMMASCHEMA EMMASCHEMA_INDEX_01 25

BCADMIN BCADMIN_DATA_01 70

TIBCOADM TIBCOADM_DATA_01 5

FREEDOM FREEDOM_DATA_01 5

IF_CMIS IF_CMIS_DATA_01 2

APP_DATA APP_DATA_DATA_01 2

FEBADMIN FEBADMIN_DATA_01 5

VMM_USER VMM_USER_DATA_01 5

Temporary tablespace TEMPAPP 10

Your DBA should monitor the growth of each tablespace and perform purging of data or the addition of more space as necessary.

Also, the file system usage (the drives on which the tablespace are placed) needs to be monitored regularly so that they are never running out of space.

The BCADMIN schema can grow up to 70 GB or more and the same needs to be extended by the DBA’s after the initial set up.

As Suppliers now have BC and BW data and file Archiving in place and running regularly as a scheduled task it will reduce the risk of the Users tablespace filling up. However it is still recommended to monitor the database size on a regular basis.

Page 28 of 61


Insert date here


3.5 Ledger files

Ledger files are used by the sending component of the TIBCO application. When a message is sent, a copy of the message contents are held in the ledger file until receipt of the message is confirmed by the recipient.

There are two types of ledger files:

BusinessWorks

BusinessConnect

3.5.1 BusinessWorks

When BusinessWorks sends a message to BusinessConnect the BW ledger files are used, under normal circumstances these files remain very small. However, should the acknowledgement not be received the ledger file can become large very quickly.

Location:

D:\app\tibco\bw\5.12

Figure 3.5-1 Ledger files

3.5.2 BusinessConnect

When BusinessConnect sends a message to BusinessWorks the BC ledger files are used. Again under normal circumstances these files remain very small. However, should the acknowledgement not be received the ledger file can, very quickly, become huge.

Location:

D:\ProgramData\<Server Name>\Ledgerfile

Page 29 of 61


Insert date here


Figure 3.5-2 BC Ledger files

Should a ledger file grow excessively, check that all services are running, if they are contact Capita ([email protected]) for guidance. The leger files should NOT be deleted.

Page 30 of 61


Insert date here


3.6 Webforms

3.6.1 Log files

Location: D:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1\

There are three types of log files:

SystemOut.log

SystemErr.log

Trace.log

Websphere application server log files are setup as follows:

When log file size reaches 1MB, archive it and create a new log file

Keeps the current log file plus 4 archive log files

Monitor periodically, the number of files should not increase more than defined in Websphere.

Figure 3.6-1 Webforms Log files

Page 31 of 61


Insert date here


Section 4

Backup & Recovery

Page 32 of 61


Insert date here


4 Backup & Recovery

It is the Supplier’s responsibility to ensure all their EMMA servers and databases are fully backed-up and recoverable.

Page 33 of 61


Insert date here


Section 5

Archiving

Page 34 of 61


Insert date here


5 Manual Archiving Batch Script

This section outlines the procedure for executing the Manual Archiving Batch Script. This needs to be executed only and only if there is a backlog created in E:\HouseKeeping\temp\Inbound or E:\HouseKeeping\temp\Outbound or E:\HouseKeeping\temp\SMM due to any kind of failure in EMMA Archiving Process.

The steps for the running the script are as listed below:-

1. Open command prompt

Figure 3.6-1 Manual Archiving

2. Go to E drive


3. Go to HouseKeeping\Config


Page 35 of 61


Insert date here


4. Execute ManualArchive.bat file


5. Enter appropriate Input I or O or S. It should be based on backlog folder


6. Verify that files have been archived in E:\BW_Archive folder from the temp folder and zip file is created


7. Also verify that temp folder is empty


Page 36 of 61


Insert date here


Section 6

Supplier Checklist

Page 37 of 61


Insert date here


6 Supplier Checklist

EMMA Health Checks – Check List

1 CPU & Memory Performance (Monthly)

2 Disk Space (Monthly)

3 Database Health Checks

1. Run Statistics Check (Weekly) 2. Purge EMMA Check (Daily) 3. Purge BC Check (Daily) 4. Unsent Check (Daily) 5. Oracle Reorganisation (DBA recommendation)

4 Logs (Weekly)

1. Listener Logs 2. Alert Logs 3. Trace Logs 4. Webforms Logs 5. Recycle Bin

5 Users Table space (Monthly)

6 Ledger Files (Weekly)

1. BusinessWorks file size check 2. BusinessConnect file size check

7 Archiving (Weekly)

1. Check E:\BWArchive folder for recent zip file

If, as a result of these checks and follow-up, you have any questions or concerns please contact Capita using the following email address:

[email protected]

Page 38 of 61


Insert date here


Section 7

EMMA Shut-down

Page 39 of 61


Insert date here


7 EMMA Shut-down

7.1 Pre-Shutdown steps

This is the approved process to follow for an EMMA shutdown. The

process has been tested to ensure a smooth EMMA shutdown.

Request EMMA outage

o Email Capita to [email protected]; please give at least 2 days.

Stop any new messages coming from your back-end billing system

Check the outbound, inbound, unsent, failed and invalid messages folders are empty

Take your EMMA off-line using the your EMMA Dashboard

Figure 7.1-1 Dashboard take EMMA offline

Once you have completed these tasks you can start the process of shutting down the EMMA


Page 40 of 61


Insert date here


7.2 Stop TIBCO services via TIBCO Administrator

Before you can stop the TIBCO services you will need to start TIBCO Administrator and login using the username and password provided by Capita.

7.3 Start TIBCO Administrator

On the EMMA server perform the following:

Click Start -> Type TIBCO administrator (as soon as you start typing you’d see a screen as below) -> Select TIBCO Administrator (displayed automatically)

Figure 7.3-1 Start TIBCO Administrator

7.4 Amend browser security settings

Figure 7.4-1 Amend browser security settings

Page 41 of 61


Insert date here


Steps to make it a trusted site:

Go to Internet options in explorer settings


Go to security tab


Go to Sites and click on add button as below, replacing the hostname with the hostname of the EMMA Interior Server


Page 42 of 61


Insert date here


TIBCO Administrator login screen


7.5 Login to TIBCO Administrator

Using the username and password provided by Capita

Figure 7.5-1 Login to TIBCO Administrator

7.6 Stop TIBCO Services

In the left hand panel, navigate to Application Management -> All Service Instances

In order to minimize the number of errors during shutdown, there is a "logical

order" of shutting down components in terms of dependencies from a message

processing perspective.

Page 43 of 61


Insert date here


Seq. Tibco Deployment Components Comments

1 WebFormHandler

2 Gateway Server Note that this component must be stopped via

Windows Service.

3 EMMAMessageDispatcher

4 EMMAMDN

5 Heartbeat

6 EMMAMessageReceiver

7 EMMAMessageEnquiryService

8 EMMADashboardServices

9 EMMAFilesReceiver

10 EMMAArchiving

11 EMMA Interior Server Business Connect Infrastructure - Inbound /

Outbound

Note=Stopping this service from TIBCO

Administrator is equivalent of stopping it via

Windows Service “TIBCO BusinessConnect

(domain name.BusinessConnect.Interior Server)

12 EHC

Figure 7.6-1 Sequence of steps to stop EMMA - TIBCO Application Components

Figure 7.6-2 Stop TIBCO Services

Page 44 of 61


Insert date here


7.7 Stop TIBCO Windows Services

Start services and browse to TIBCO Click on start and type services. On the right hand side of the screen click on services

Figure 7.7-1 Stop TIBCO Windows Services

The service program will load. Scroll down to the TIBCO services.


Ensure services are stopped in the following order

Single Server Setup (Small Supplier)

TIBCO Administrator 5.9(SXX_Environment)

TIBCO Hawk Agent(SXX_Environment)

TIBCO Hawk H2 Database

TIBCO Hawk Event

TIBCO BusinessConnect(SXX_Environment.BusinessConnect.Interior Server)

TIBCO Gateway Server (gs)

TIBCO_rvd_xxxx (xxxx=port value)

Page 45 of 61


Insert date here


Three server setup (Large Supplier)

TIBCO Gateway Server (gs) (On the gateway Server)


TIBCO BusinessConnect(SXX_Environment.BusinessConnect.Interior Server-1)





TIBCO Hawk Event


(SXX_Environment example S01_Production or S01_Test)

To stop a service right click on the name and click stop.


N.B. When Stopping TIBCO Administrator, the following message box may appear. If so, simply click OK and proceed with stopping the remaining services.

Figure 7.7-4: Stopping a windows service

Page 46 of 61


Insert date here


7.8 Stop WAS server then IBM HTTP server

7.8.1 Stop WAS server

Open Services as done in section 7.7

Stop IBM WebSphere Application Server v8.5 Service

To stop a service right click on the name and click stop.

Figure 7.8-1 Stop WAS server

Stop IBM HTTP Server v8.5 Service to stop the server.

Figure 7.8-2 Stop WAS server

Please note that Once the IBM HTTP Server and WAS is stopped, webforms

application stops.

Page 47 of 61


Insert date here


7.9 Stop Oracle database

Open up sql plus and connect to the database as SYSDBA or SYSOPER and issue the “shutdown immediate” command as shown below. Your windows account should be part of ORA_DBA group.

Figure 7.9-1 Stop Oracle database

Page 48 of 61


Insert date here


Section 8

EMMA Start-up

Page 49 of 61


Insert date here


8 EMMA Start-up

This is the approved process to follow for an EMMA start-up. The process

has been tested to ensure a smooth EMMA start-up.

8.1 Start Oracle Database

Open up sql plus and connect to the database as SYSDBA or SYSOPER and issue the “start-up” command as shown below. Your windows account should be part of ORA_DBA group.

Figure 8.1-1 Start Oracle Database

8.2 Start TIBCO windows services

Click on start and type services. On the right hand side of the screen click on services

Figure 8.2-1 Start TIBCO windows services

Page 50 of 61


Insert date here


The service program will load. Scroll down to the TIBCO services.


Ensure services are started in the following order. Each service should be in a running state before starting the next.

Single Server Setup (Small Supplier)


TIBCO Hawk Event




TIBCO Gateway Server (gs)

Three server setup (Large Supplier)





TIBCO Hawk Event




TIBCO Gateway Server (gs) (On the gateway Server)

(SXX_Environment example S01_Production or S01_Test)

Page 51 of 61


Insert date here


To start a service right click on the name and click start.


8.3 Start TIBCO services via TIBCO Administrator

Click Start -> Type TIBCO administrator (as soon as you start typing you’d see a screen as below) -> Select TIBCO Administrator (displayed automatically) Log into the site as previously seen in Section 7.5.2 and 7.5.3

Page 52 of 61


Insert date here


Figure 8.3-1 Start TIBCO services via TIBCO Administrator

8.4 Start TIBCO Services

Open All Service Instances on the left hand side of the screen. There is a "logical order" to restart in terms of dependencies from a message processing perspective.

Seq. Tibco Deployment Components Comments

1 EHC

2 EMMAMessageReceiver

3 Heartbeat

4 EMMAMDN

5 EMMAMessageDispatcher

6 EMMAMessageEnquiryService

7 EMMADashboardServices

8 WebFormHandler

9 EMMAFilesReceiver

10 EMMAArchiving

11 EMMA Interior Server

12 Gateway Server Note that these components must be

started via Windows Service.

Figure 8.4-1 Sequence of steps to re-start EMMA - TIBCO Application Components

Page 53 of 61


Insert date here


Figure 8.4-2 Start TIBCO Services

8.4.1 Checking Process Starters

Click on EHCEMMA process. The next screen shows you a detailed view of the process.

Figure 8.4-3 Checking Process Starters

Click on the ‘BW Processes’ tab and in the drop down seen select ‘Process Starters’

Page 54 of 61


Insert date here


Figure 8.4-4 Checking Process Starters

A list of all the process starters for the process will appear. Each of them should be

enabled.

Check the value on the right hand side of the page. See figure 4 below

Figure 8.4-5 Checking Process Starters Status

If any of the processes are disabled selected the checkbox on the left hand side and click

the ‘Enable’ button above the checkboxes. Figure 5 shows the enable button is available

when a check box is selected.

Page 55 of 61


Insert date here


Figure 8.4-6 Enabling Process Starters

Repeat the steps for each of the applications listed under ‘All Service Instances’ that

have a gold mark beside the cog symbol.

Figure 8.4-7 Process Symbol

Figure 8.4-8 show a full list of EMMA applications

Figure 8.4.1-7 Full list of TIBCO services

Page 56 of 61


Insert date here


8.5 Start IBM HTTP server and WAS server

The IBM HTTP server should be started first, then the WAS server

Open Services as done in section 7.7

Start IBM HTTP Server V8.5.

To start a service right click on the name and click start.

Figure 8.5-1 Start IBM HTTP server and WAS server

Start IBM WebSphere Application Server V8.5 Service

Figure 8.5-2 Start IBM HTTP server and WAS server

Please note that Once the IBM HTTP Server and WAS services are started the

webforms application starts and the website will be available again.

Page 57 of 61


Insert date here


Section 9

Software Update & Patching

Page 58 of 61


Insert date here


9 Software Update & Patching

Market Participants will provide a dedicated server to host the Market Messaging

solution. Only software required to support this solution will be installed on the

server.

Market Participants should inform Capita if they intend to install server-monitoring

tools. Capita will then check with TIBCO to check that there are no known issues

between the TIBCO applications and the monitoring tools. If in Production, the

monitoring tool adversely impacts the Market Messaging solution, it must be

removed.

9.1 Windows Operating System

For the server running the Market Messaging solution, it is not recommended applying

any operating system patches. The solution is built to implement a robust, reliable

Messaging solution and these operating system patches are considered as a potential

risk to the solution.

If the Market Participant’s policy is to apply operating system patches then the final

decision on whether to apply the patch/ patches lies with the individual Market

Participant. If a Market Participant chooses to apply the operating system patch/patches,

they are responsible for applying and testing it in their Test EMMA before applying to the

Production EMMA. If in Production, an operating system patch adversely impacts the

Market Message solution, it must be reversed out.

9.2 Applications

The following applications used by the All-Island Market Messaging solution are as

follows:

TIBCO Business Connect - used for the following:

o Wrap the outbound message in an AS2 envelope and send over

HTTPS to the receiver’s EMMA.

o Unwrap the AS2 envelope from the inbound message and forward

to Business Works

TIBCO Business Works - used for the following:

o Validates the outbound message against the schema and if valid

will forward the message to Business Connect.

o Will receive inbound messages and deliver into the destination

folder

Page 59 of 61


Insert date here


IBM Forms Experience Builder – used to manually create Market Messages

via webforms user interface.

ESB/NIE have a support agreement with TIBCO and IBM and as such will be advised of

any updates to the above applications, the TIBCO and IBM Forms Experience Builder

updates will include the reason for the update. This reason will be reviewed with

respect to the All-Island Market Messaging solution and if applicable ESB/NIE will

request Capita to issue an update to all Market Participants after properly structured

tests. As a guideline the update will only be applied if it:

1. directly affects the All-Island Market Messaging solution, or

2. Is required to maintain product support.

If a release is required, Capita will create it including install instructions; they will also

assist Market Participants with their install.

The release will installed and tested on the Market Participant’s Test EMMA before being

applied to their Production EMMA.

9.3 Oracle Database

The All-Island Market Messaging solution requires Oracle database version 12c.

As per the Operating System updates, It is recommended that database updates or

patches are not applied as they may impact the All-Island Market Messaging solution.

The final decision on which patch/ patches to apply lies with the individual Market

Participant. If a Market Participant chooses to apply the Oracle patch/ update then, they

are responsible for applying and testing it with their Test EMMA before applying to their

Production EMMA. If in Production, the update/ patch adversely impacts the NIE Market

Message solution it must be reversed out.

9.4 Anti-Virus Software

For the server running the All-Island Market Messaging solution, it is not recommended to

install any anti-virus software. This server will only be used to exchange XML market

messages with ESB and NIE Market Systems and any additional software is considered

a potential risk.

The final decision on whether to install anti-virus software lies with the individual Market

Participant. If a Market Participant chooses to install anti-virus software then, they are

responsible for testing it in their Test EMMA before installing it on the Production EMMA.

If in Production, the anti-virus software adversely impacts All-Island Market Messaging

solution, it must be reversed out.

Page 60 of 61


Insert date here


Section 10

Applciation User Passwords

Page 61 of 61


Insert date here


10 Application User Passwords

Please note that the passwords for Application Users (usernames that are used by

Applications) must be set as Non Expiry. Please ensure that the following Application

Users at EMMA in the respective components are set as Non-Expiry.

DATABASE

o EMMASCHEMA

o TIBCOADM

o BCADMIN

o FEBADMIN

o FREEDOM

o IF_CMIS

o APP_DATA

o VMM_USER

o BCPURGE

TIBCO

o tibadmin

o tibadmin_production1

o tibadmin_test

o tibadmin_preprod

FEB

o fileloadadmin

o useradmin

Date post:	06-Mar-2018
Category:	Documents
Upload:	lehanh
View:	216 times
Download:	3 times

TIBCO EMMA Operation Manual - · PDF filePage 3 of 61 TIBCO EMMA Operational Manual –...

Documents