How to setup ITNM Failover with Failover NCIM and ... - IBM · The ncp_virtualdomain entry from the...

IBM Tivoli Network Manager IP

© 2011 IBM Corporation

How to setup ITNM Failover with Failover NCIM

and Shared NCIM

Presenter:Krishna M Kodali - [email protected]


2Presenter: Krishna M Kodali April 28th - 2011 © 2011 IBM Corporation

Agenda● ITNM Failover Configuration

● Architecture of Failover● ITNM Failover with Shared NCIM

● Steps to implement Failover with Shared NCIM● Example Configuration

● ITNM Failover with NCIM Replication● Steps to implement Failover with NCIM replication

● Example Configuration● V3.9 example configuration for both scenarios:● Failover FAQs● Known issues ● Limitations

IBM Presentation Template Full Version



ITNM Failover Configuration

Naming convention across the presentation:– Primary domain - NCOMSP– Backup domain - NCOMSB– Virtual domain - NCOMSV



ITNM failover Architecture – shared NCIM

Model

DISCO

Virtual Domain –NCOMS

NetworkNetwork

Model

Server C

NCIM & NCMonitor+ NCPGUI

Sever A Server B

Domain - NCOMSP Domain - NCOMSB

PollerPoller



V3.8 – Failover with shared NCIM example

Steps on Primary domain:Step # 1:Create ncp_virtualdomain process entry in CtrlServices.NCOMP.cfg

insert into services.inTray( serviceName, servicePath, domainName, argList, dependsOn, retryCount)values( "ncp_virtualdomain", "$PRECISION_HOME/platform/$PLATFORM/bin", "$PRECISION_DOMAIN", [ "-domain" , "$PRECISION_DOMAIN", "-virtualDomain", "NCOMSV", "-backupDomain", "NCOMSB", "-latency" , "1000000", "-debug", "0" ], [ "ncp_model" ], 5);



V3.8 – Failover with shared NCIM exampleStep # 2:Configure a fixed TCP socket connection between the domains:

# On the primary Network Manager server, manually start the ncp_virtualdomain process from the $NCHOME/precision/bin directory:

ncp_virtualdomain -domain PRIMARYDOMAIN_NAME (ex: NCOMSP)

When the ncp_virtualdomain process starts for the first time, it writes a line to the $NCHOME/etc/precision/ServiceData.cfg file, which lists TCP and multicast connection information for Network Manager processes. This line references ncp_virtualdomain, and includes the port on which the Virtual Domain component on the primary server accepts TCP connections from the backup server. For example:

SERVICE: NCP.VIRTUALDOMAIN.QUERY DOMAIN: NCOMSV ADDRESS: 10.10.50.205 PORT: 41838 SERVERNAME: sasun2 DYNAMIC: NO

Tip: The DYNAMIC:NO setting forces the ncp_virtualdomain process to use the same address and port the next time that it starts.# Save the file.# Stop the ncp_virtualdomain process.

Step # 3: Run ncp_mib to populate ncmib.

Step # 4: Start the Primary domain as normal.



V3.8 – Failover with shared NCIM example

Steps on Backup:Step # 5:Create ncp_virtualdomain process entry in CtrlServices.NCOMB.cfg

The CtrlServices.NCOMSB.cfg requires multiple changes unlike Primary:

insert into services.inTray( serviceName, servicePath, domainName, argList, dependsOn, retryCount)values( "ncp_virtualdomain", "$PRECISION_HOME/platform/$PLATFORM/bin", "$PRECISION_DOMAIN", [ "-domain" , "$PRECISION_DOMAIN", "-virtualDomain", "NCOMSV", "-primaryDomain", "NCOMSP", "-latency" , "1000000", "-debug", "0", "-messagelevel", "warn" ], [ "ncp_model" ], 5);



V3.8 – Failover with shared NCIM example Steps on Backup domain: continued.Step # 6

● Add "-backup" parameter for ncp_model and ncp_ncogate processes in CtrlServices.NCOMSB.cfg - an example:

insert into services.inTray( serviceName, servicePath, domainName, argList, dependsOn, retryCount)values( "ncp_model", "$PRECISION_HOME/platform/$PLATFORM/bin", "$PRECISION_DOMAIN", [ "-domain" , "$PRECISION_DOMAIN" , "-latency" , "1000000", "-debug", "0", "-messagelevel", "warn", "-backup"], [ "ncp_config", "ncp_store", "ncp_class" ], 5);



V3.8 – Failover with shared NCIM example Steps on Backup domain: continued.Step # 7

● Poller(ncp_poller) requires new parameter:

insert into services.inTray( serviceName, servicePath, domainName, argList, dependsOn, retryCount)values( "ncp_poller", "$PRECISION_HOME/platform/$PLATFORM/bin", "$PRECISION_DOMAIN", [ "-domain" , "$PRECISION_DOMAIN" , "-latency" , "1000000", "-primaryDomain", "NCOMSP", "-debug", "0", "-messagelevel", "warn" ], [ "nco_p_ncpmonitor", "ncp_ncogate" ], 5);



V3.8 – Failover with shared NCIM example Steps on Backup domain: continued.Step # 8Copy below files from Primary to Backup domain and alter them to be Backup domain specific:

$NCHOME/etc/precision/DbLogins.NCOMSP.cfg$NCHOME/etc/precision/MibDbLogins.NCOMSP.cfg$NCHOME/etc/precision/NcoLogin.NCOMSP.cfg (Only if password is encrypted which isn't default)$NCHOME/etc/security/key/conf.key

Note - Make a Backup of conf.key before replacing with Primary domain specific file.

Step # 9Copy the SERVICE: ncp_virtualdomain DOMAIN: VIRTUAL ... line from the $NCHOME/etc/precision/ServiceData.cfg file on the primary server into the $NCHOME/etc/precision/ServiceData.cfg file on the backup server. Ensure that only a single SERVICE: ncp_virtualdomain DOMAIN: VIRTUAL ... line is present in the file.


Important: The SERVICE: ncp_virtualdomain DOMAIN: VIRTUAL ... line must be identical in the $NCHOME/etc/precision/ServiceData.cfg file in both domains.

Step # 10: Start the Backup domain as normal.



V3.8 – Failover with shared NCIM exampleQ: Can I avoid all configurations files & conf.key exchange (step # 8) from Primary to Backup and have Failover functionality?

A: Yes - if you disable password encryption before configuring SNMP/Telnet Passwords - then there isn't a need to perform above file exchange.

To Deactivating password encryption (CLI Model only - this option is not available via GUI).

You can configure Network Manager to deactivate password encryption. If you do this then passwords entered using the GUIs are written to disk in plain text.

To deactivate password encryption: 1. Edit the ncp_config configuration file, ConfigSchema.cfg. 2. Configure the following insert to the config.settings table:

insert into config.settings( m_DirScanIntvl, m_EncryptPasswords, m_EncryptionKeyFile)values( 60, 0, "$NCHOME/etc/security/keys/conf.key");



ITNM Failover with NCIM Failover Architecture

Model

DISCO Virtual Domain –NCOMS

NetworkNetwork

Model

Server C

NCIM & NCMonitor+GUISever A

Server B

Domain - NCOMSPDomain - NCOMSB

Poller PollerServer D

NCIM & NCMonitor+ GUI



V3.8 – Example for ITNM Failover with NCIM Replication Steps on Primary remains same as Failover with shared NCIM i.e. slides 5 & 6● Steps on Backup:Step # 1Create ncp_virtualdomain process entry in CtrlServices.NCOMB.cfg

The file requires multiple changes unlike Primary:

insert into services.inTray( serviceName, servicePath, domainName, argList, dependsOn, retryCount)values( "ncp_virtualdomain", "$PRECISION_HOME/platform/$PLATFORM/bin", "$PRECISION_DOMAIN", [ "-domain" , "$PRECISION_DOMAIN", "-virtualDomain", "NCOMSV", "-primaryDomain", "NCOMSP", "-latency" , "1000000", "-debug", "0", "-messagelevel", "debug" ], [ "ncp_model" ], 5);



V3.8 – Example for ITNM Failover with NCIM ReplicationSteps on Backup: Continued..Step # 1.a:Add new parameter for ncp_model under CtrlServices.NCOMSB.cfg

insert into services.inTray( serviceName, servicePath, domainName, argList, dependsOn, retryCount)values( "ncp_model", "$PRECISION_HOME/platform/$PLATFORM/bin", "$PRECISION_DOMAIN", [ "-domain" , "$PRECISION_DOMAIN" , "-latency" , "1000000", "-debug", "-primaryDomain", "NCOMSP", "0", "-messagelevel", "warn"], [ "ncp_config", "ncp_store", "ncp_class" ], 5);

Step # 1.b:Add "-backup" parameter for ncp_ncogate:



V3.8 – Example for ITNM Failover with NCIM Replication

Steps on Backup: Continued..Step # 1.c:

● Poller(ncp_poller) requires new parameter:

insert into services.inTray( serviceName, servicePath, domainName, argList, dependsOn, retryCount)values( "ncp_poller", "$PRECISION_HOME/platform/$PLATFORM/bin", "$PRECISION_DOMAIN", [ "-domain" , "$PRECISION_DOMAIN" , "-primaryDomain", "NCOMSP", "-readsnmpconfig", "-latency" , "1000000", "-debug", "0", "-messagelevel", "warn", ], [ "nco_p_ncpmonitor", "ncp_ncogate" ], 5);

Note – if you do not specify “-readsnmpconfig” - Poller can't perform SNMP polling when using NCIM replication.



V3.8 – Example for ITNM Failover with NCIM ReplicationSteps on Backup: Continued..Step # 2:

● As Backup Domain will be talking to its own NCIM db, below two files should be configured accordingly:

$NCHOME/etc/precision/DbLogins.NCOMSB.cfg$NCHOME/etc/precision/MibDbLogins.NCOMSB.cfg

Step # 3:

Copy the SERVICE: ncp_virtualdomain DOMAIN: VIRTUAL ... line from the $NCHOME/etc/precision/ServiceData.cfg file on the primary server into the $NCHOME/etc/precision/ServiceData.cfg file on the backup server. Ensure that only a single SERVICE: ncp_virtualdomain DOMAIN: VIRTUAL ... line is present in the file.


Important: The SERVICE: ncp_virtualdomain DOMAIN: VIRTUAL ... line must be identical in the $NCHOME/etc/precision/ServiceData.cfg file in both domains.

Step # 3: Run ncp_mib to populate ncmib.

Step # 5: Start the Backup domain as normal.



V3.9 – Failover with NCIM Replication or Shared NCIM

● Generic information about v3.9 Failover● Failover is configured primarily via the new ConfigItnm file ($NCHOME/etc/precision/ConfigItnm.cfg)● No changes are required to the default CtrlServices file ● Simplifies set up and reduces risk of configuration errors● The virtual domain process runs by default● No additional command line args are required in either domain● The ncp_virtualdomain entry from the primary domain ServiceData.cfg file must be copied to the backup domain

● The entry is generated automatically in the primary domain when ncp_virtualdomain is run● eg manually run: ncp_virtualdomain –domain <PRIMARY>

● When using shared NCIM – the Encryption key i.e. $NCHOME/etc/security/keys/conf.key, must be manually copied from the primary to the backup domain - refer to slide#10 (Step # 8) for complete information.



V3.9 – Failover with NCIM Replication or Shared NCIM

Contains clear failover configuration in a single place

Object Server name is used even if failover is not enabled

File should have the same contents on the primary and backup domains

Replaces various command line args across processes:– -primaryDomain– -virtualDomain– -backupDomain– -backup– -server



FAQs about Failover:

Q-1: Which data or configuration files are kept in sync via ncp_virtualdomain process?

A: a) All Password configuration files (SNMP & TELNET) i.e.$NCHOME/etc/precision/SnmpStackSecurityInfo.<domain>.cfg &$NCHOME/etc/precision/TelnetStackPasswords.<domain>.cfg, For example configuration - refer slides 35, 36 & 37 @ https://www-304.ibm.com/support/docview.wss?rs=0&uid=swg27018013 b) Polling policies which gets stored in $NCHOME/var/precision/MonitorPolicies.NCOMSP.xml c) Topology Data i.e $NCHOME/var/precision/Store.Cache.kernel.activeModel.NCOMSB. d) Network Views using get_network_views.pl - v3.9 only.

Q-2: Can I setup Primary NCIM to be Oracle and Backup NCIM to be DB2/Mysql/Informix?

A:Yes, that is supported configuration.



FAQs about Failover: continued..Q-3: How often ncp_virtualdomain pushes updates of Topology, Policies (MonitorPolicies.xml), SnmpStackSecurityInfo.cfg & TelnetStackPasswords.cfg to Backup domain? Is that frequency configurable?

A: Yes, the frequency is configurable in ConfigSchema.cfgThe moment changes are made to SnmpStack, TelnetStack or Poll Policies, the timestamp is changed on those files, ncp_config on Primary - picks that change, passes to ncp_virtualdomain, then to Backup ncp_virtualdomain then to Backup ncp_config.

The Backup ncp_config backs up existing SnmpStackSecurityInfo.domain.cfg file i.e. it moves that file to $NCHOME/etc/precision/backup/SnmpStackSecurityInfo.domain.cfg.ttimestamp and creates new file using all entries from Primary. Where as for .xml (policies) we don't maintain backup..

The key is timestamp on the files and the default scan interval is 60 seconds. >ncp_oql -domain NCOMSP -username admin -service Config -password '' -query "select * from config.settings;"

select * from config.settings;{ m_DirScanIntvl=60; m_EncryptPasswords=1; m_EncryptionKeyFile='$NCHOME/etc/security/keys/conf.key';}( 1 record(s) : Transaction complete )During its scan, if it finds a change in timestamp - ncp_config starts to push those changes to Virtualdomain. .> select * from config.files where m_ConfigFile='SnmpStackSecurityInfo';.> go{ m_ConfigFile='SnmpStackSecurityInfo'; m_UpdTime=1299260528; m_SchemaFile='SnmpStackSchema.cfg';}( 1 record(s) : Transaction complete )



FAQs about Failover: continued..

Q-4: Are there any files that requires manual synchronisation between Primary and Backup?

A: Yes & NONo - if you are using out-of-the-box(default) AOC, MIBs, Amos(RCA) rules, Probe rules etc. Yes - if you have added custom aoc or mibs. i.e.

● AOC(Active Object Class) files are stored in $NCHOME/precision/aoc and● MIBs which are stored in $NCHOME/precision/mibs.

These two folders must be kept in sync all the time for accurate polling in a failover environment.

Note: If you do any other customisation on Primary - for ex: Amos rules, Probe rules (ex: nco_p_ncpmonitor) etc - same should be done on Backup too.



Incorrect Configuration issues and the resolution:

● In a Failover environment – the server hosting Backup domains MUST run only the failover domains of Primary.

● If you have additional domains on Backup that doesn’t on Primary – such architecture is unsupported by IBM.

Primary - NCIM

Primary - Sever ABackup - Server B

Domain –XYZ_P

Domain – ABC_P

Domain –XYZ_B

Domain – ABC_B

Domain -WHATEVER

Virtualdomain –ABC_V

Virtualdomain –XYZ_V

Backup - NCIM

Additional domain that doesn’t exist on Primary



Incorrect Configuration issues and the resolution: continued..

Q-5: In a multi-domain environment, when using NCIM replication method, the following error appears in Backup/Failover Model log & trace - 'entityId: xxxxx perhaps it already exists within another domain' - how to fix it?

Then ncp_model.<domain>.trace shows:Warning: A generic non-fatal error has occurred found in file CMdlUpdateThread.cc at line 1038 - Failed to delete entityId: 797861 perhaps it already exists within another domainCMdlUpdateThread::InsertIntoEntityNameCache: 797861 DeviceA.IBM.COM[Fa0/17]Mon Apr 11 15:51:11 2011 Warning: A database 'execute' operation has failed found in file CMdlUpdateThread.cc at line 908 - : CNcpMySQLSth.cc line 364 : Duplicate entry '797861' for key 1( statement: ' INSERT INTO entityNameCache ( ENTITYID, ENTITYNAME, DOMAINMGRID ) VALUES ( ?, ?, ? )' varbinds: 797861,DeviceA.IBM.COM[ Fa0/17 ],24 )CMdlUpdateThread::UpdateEntityNameCache: 0 -> 797861

See previous slide Screen shot for better understanding:

Resolution: Clean up entire Backup NCIM using domain_drop.pl script and start the failover domains one by one to be in sync with Primary.

● In a failover ITNM environment, the ObjectId (Model) which is nothing but entityId in NCIM are kept in sync between Primary and Backup domains.

● This means: ● It is only on Primary server A that new entityIds are ever generated in entityNameCache.● Backup server B will never allocate an new entityId and only get topology data from Primary server A.● In Summary - when using NCIM Replication feature - the Backup NCIM should sync data only from Primary and any

stand-alone domains must NOT be configured to migrate data to Backup NCIM.



Incorrect Configuration issues and the resolution: continued..

Q-6: Failed to decode password on Backup Poller log/trace observed - what does it mean and how to correct it? What's the impact on Polling functionality?

A:Warning: A generic non-fatal error has occurred found in file CCryptPwdMgr.cc at line 503 - Failed to decode password: Unknown error -5013

It means - Backup Poller is not able to decrypt the passwords associated to SNMP enabled devices (ncmonitor.snmpTarget) that are stored in ncmonitor.snmpV1Sec or ncmonitor.snmpV3Sec. To see relation between the table - read more @ https://www-304.ibm.com/support/docview.wss?uid=swg21407317

The affect of the error is : The Backup Poller won't be able to perform any SNMP Polling, only ICMP Polling is possible.

The way to fix it is - follow the steps outlined in slide 10 (Step # 8) i.e. copy necessary files from Primary to Backup(conf.key).



Limitations -ITNM Gui

The ITNM - GUI(Topoviz) does not currently switch to use the active domain. Since TIP has HighAvailability, is that applicable for ITNM-Topoviz Plug-in too?

– Topoviz Plug-in is NOT part of High Availability and TIP will not know to switch to Primary/Backup ITNMCore during failover or failback.

– When using NCIM replication, either:

• Set up 2 TIP servers, one to point to the primary NCIM database, one to the backup, and manually switch upon failover

• Manually reconfigure the NCIM database access parameters in the GUI to switch the single TIP server to point to the backup NCIM database ($TIPHOME/profiles/TIPProfile/etc/tnm/tnm.properties)



Known Issues: 1) Does failover happen if the Primary NCIM is down instead of 'NCP' processes?

● v3.8 - This is a known limitation and there is no workaround or fix available.

● v3.9 - The new ItnmDatabaseConnection events handles such failure, alongside the filter for the NCIM service in the VirtualDomainSchema file.

Excerpt from $NCHOME/etc/precisin/VirtualDomainSchema.cfg:insert into state.filters( m_ServiceName, m_Filter, m_Description)values( "NCIM", "m_CtrlState <> 7 or (m_CtrlState = 7 and m_ChangeTime > eval(time,'$TIME - 300'))", "The NCIM database has been down for more than 300 seconds");



Known Issues:2) https://www-304.ibm.com/support/docview.wss?uid=swg1IZ98663 - Fix will be available in FixPack6 due in Sept'11.

Every customer with large topology - say > 200k elements can hit this bug:

mysql> select * from ncim.domainSummary where domainMgrId=3;+-------------+---------------------+---------------------+-------------+--------------+----------------+| domainMgrId | createTime | changeTime | entityCount | chassisCount | interfaceCount |+-------------+---------------------+---------------------+-------------+--------------+----------------+| 3 | 2011-04-06 23:16:33 | 2011-04-07 00:41:41 | 222381 | 1770 | 113042 |+-------------+---------------------+---------------------+-------------+--------------+----------------+1 row in set (0.00 sec)

ncp_virtualdomain.NCOMSP.log: 03/25/11 13:07:00: Warning: [4132211600t] CRivRvNet.cc(1702) Rendezvous method called from incorrect thread.ncp_virtualdomain.NCOMSP.trace:ing: Rendezvous method called from incorrect thread. found in file CRivRvNet.cc at line 1702ncp_virtualdomain.NCOMSP.trace:11 Warning: Rendezvous method called from incorrect thrROSS::ROSSSendToClient sending packet length 6951 socket 11 at 1301084121

The problem is the socket keep alive messages that are periodically sent across the socket to ensure that it doesn't get marked as inactive and closed by a firewall. The current logic inside ncp_virtualdomain sent one message every 20 seconds and marked the connection as dead if 3 were missed (i.e no keep alive messages for a minute.) The problem occurs when the time taken for the topology download from ncp_model exceeded a minute, because in that case ncp_virtualdomain would mark the socket as dead and try and open a new one.



Known Issues:The fix for IZ98663 is:a) upping the default from 20 seconds to 60 seconds (giving 3 minutes to get the data) and b) making this property configurable in the VirtualDomainSchemal.cfg.

You shouldn't need to modify schemas as the new default of 60 seconds(hard-coded) - but we have listed the schema change in VirtualDomainSchema.cfg for reference incase if one need to increase.

create table config.defaults( m_SocketKeepAlivePeriod int not null, m_HealthCheckPeriod int not null, m_FailoverTime int not null, m_AutoTopologyDownload int not null);

insert into config.defaults( m_SocketKeepAlivePeriod, m_HealthCheckPeriod, m_FailoverTime, m_AutoTopologyDownload)values( 60, 60, 300, 1);



Questions?

Date post:	15-Nov-2019
Category:	Documents
Upload:	others
View:	60 times
Download:	1 times

How to setup ITNM Failover with Failover NCIM and ... - IBM · The ncp_virtualdomain entry from the...

Documents