Date post: | 13-Sep-2014 |
Category: |
Technology |
View: | 954 times |
Download: | 0 times |
Kscope
Case of the Unexplained Support Issue
Charles Beyer
Troubleshooting steps for diagnosing HFM Workspace and FDM
Kscope
About the Speaker
15+ years professional development experience in various languagestechnologies (6 years w Hyperion products)
Significant experience reverse engineering software
Loves to help others solve problems Posts under beyerch2 on OTN Discussion
Boards
Kscope
Before we Beginhellip
To expedite session we will use both screen caps and live demos
lsquoAll-in-onersquo Virtual Machine created for session available upon request
Feel free to ask questions
Kscope
Session Goals
Discover some new tools to be used for troubleshooting (Oracle Third Party)
Acquire some new troubleshooting ideas
Kscope
Challenges Responsible for a large amount of products Foundation Services (OHS Weblogic Shared Services
Workspace EPMA Calc Manager LCM) HFM (HFM Web Client Smartview) FDM Reporting and Analysis (FR IR SQLPR WA) Planning Performance Scorecard Strategic Finance
Probability Cost Disclosure Management Financial Close Management Data Relationship Management Essbase Crystal Ball helliphellip Hyperion Kitchen Sink
Kscope
Challenges (continued) Need to understand underlying technologies OS Servers (WindowsUnix) Application Servers (Weblogic Tomcat) Web Servers
(IIS Apache OHS) Authentication Options (AD Native Custom) Languages (Java NET ASP Classic JSP VBScript) Communication Frameworks (DCOM Web Services) Database (SQL Server Oracle Berkeley DB) Network Principles (Routing Firewall VPN)
Kscope
Challenges (continued) End Users Quantity ndash Intermittent issues due to number of users
stressing system
Internal vs External Less control over environment More points of failure Remote troubleshooting more difficult
User Experience New Users - More hands on but easier problems Power Users ndash Likes to experiment
Kscope
Challenges (continued) Internal Issues aka Politics Separation of Duties
Complicates troubleshooting efforts due to lack of access Delays resolution due to co-ordination efforts
Restrictive Policies Could preventlimit certain types of troubleshooting
NetworkHTTP sniffing No monitor programs on production
Kscope
Challenges (continued)
Cryptic Error Messages
But I didnrsquot click on a folder
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued) Limited OOTB (Out Of The Box) Tools Mostly consists of Log Files System 11 has expanded diagnostic capability SupportOracleCom
Kscope
Troubleshooting Strategy
Kscope
Troubleshooting Strategy
Final Solution
Follow Up
Evaluate
Possible
Solutions
Clearly
Defined
Problem
Kscope
Troubleshooting Strategy (continued) Clearly Define the Problem Ask the right questions Verify Information Where Possible
Research Prioritize and Evaluate potential solutions Use Initial Information and Resources to propose
possible causessolutions Prioritize in order of likelihood Evaluate Potential Solutions Determine Final Solution or Escalate
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Document Get Sign-Off Add Information To Internal Knowledgebase
Kscope
Troubleshooting Strategy (continued) Notes Process Is Easier As You Learn More About The
System Development Test Environment Makes This Easier Donrsquot Fear Leveraging Other Resources Practice Practice Practice
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How Who (is affected)
All Users or Select Group If subset is there commonality (Geographic
HardwareSoftware Data Provider)
What (is happening not happening) What was the attempted procedure What has changed
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How When (does the issue occur)
Consistently Intermittently Pattern
Where (are failures occurring) Client Machine(s) Database Server(s) Web Server(s) App Server(s)
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How How (should the process work)
Do you know how it should work Important to understand expected operation to help
determine point of failure probable cause and solution Research this if you are not sure
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
About the Speaker
15+ years professional development experience in various languagestechnologies (6 years w Hyperion products)
Significant experience reverse engineering software
Loves to help others solve problems Posts under beyerch2 on OTN Discussion
Boards
Kscope
Before we Beginhellip
To expedite session we will use both screen caps and live demos
lsquoAll-in-onersquo Virtual Machine created for session available upon request
Feel free to ask questions
Kscope
Session Goals
Discover some new tools to be used for troubleshooting (Oracle Third Party)
Acquire some new troubleshooting ideas
Kscope
Challenges Responsible for a large amount of products Foundation Services (OHS Weblogic Shared Services
Workspace EPMA Calc Manager LCM) HFM (HFM Web Client Smartview) FDM Reporting and Analysis (FR IR SQLPR WA) Planning Performance Scorecard Strategic Finance
Probability Cost Disclosure Management Financial Close Management Data Relationship Management Essbase Crystal Ball helliphellip Hyperion Kitchen Sink
Kscope
Challenges (continued) Need to understand underlying technologies OS Servers (WindowsUnix) Application Servers (Weblogic Tomcat) Web Servers
(IIS Apache OHS) Authentication Options (AD Native Custom) Languages (Java NET ASP Classic JSP VBScript) Communication Frameworks (DCOM Web Services) Database (SQL Server Oracle Berkeley DB) Network Principles (Routing Firewall VPN)
Kscope
Challenges (continued) End Users Quantity ndash Intermittent issues due to number of users
stressing system
Internal vs External Less control over environment More points of failure Remote troubleshooting more difficult
User Experience New Users - More hands on but easier problems Power Users ndash Likes to experiment
Kscope
Challenges (continued) Internal Issues aka Politics Separation of Duties
Complicates troubleshooting efforts due to lack of access Delays resolution due to co-ordination efforts
Restrictive Policies Could preventlimit certain types of troubleshooting
NetworkHTTP sniffing No monitor programs on production
Kscope
Challenges (continued)
Cryptic Error Messages
But I didnrsquot click on a folder
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued) Limited OOTB (Out Of The Box) Tools Mostly consists of Log Files System 11 has expanded diagnostic capability SupportOracleCom
Kscope
Troubleshooting Strategy
Kscope
Troubleshooting Strategy
Final Solution
Follow Up
Evaluate
Possible
Solutions
Clearly
Defined
Problem
Kscope
Troubleshooting Strategy (continued) Clearly Define the Problem Ask the right questions Verify Information Where Possible
Research Prioritize and Evaluate potential solutions Use Initial Information and Resources to propose
possible causessolutions Prioritize in order of likelihood Evaluate Potential Solutions Determine Final Solution or Escalate
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Document Get Sign-Off Add Information To Internal Knowledgebase
Kscope
Troubleshooting Strategy (continued) Notes Process Is Easier As You Learn More About The
System Development Test Environment Makes This Easier Donrsquot Fear Leveraging Other Resources Practice Practice Practice
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How Who (is affected)
All Users or Select Group If subset is there commonality (Geographic
HardwareSoftware Data Provider)
What (is happening not happening) What was the attempted procedure What has changed
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How When (does the issue occur)
Consistently Intermittently Pattern
Where (are failures occurring) Client Machine(s) Database Server(s) Web Server(s) App Server(s)
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How How (should the process work)
Do you know how it should work Important to understand expected operation to help
determine point of failure probable cause and solution Research this if you are not sure
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Before we Beginhellip
To expedite session we will use both screen caps and live demos
lsquoAll-in-onersquo Virtual Machine created for session available upon request
Feel free to ask questions
Kscope
Session Goals
Discover some new tools to be used for troubleshooting (Oracle Third Party)
Acquire some new troubleshooting ideas
Kscope
Challenges Responsible for a large amount of products Foundation Services (OHS Weblogic Shared Services
Workspace EPMA Calc Manager LCM) HFM (HFM Web Client Smartview) FDM Reporting and Analysis (FR IR SQLPR WA) Planning Performance Scorecard Strategic Finance
Probability Cost Disclosure Management Financial Close Management Data Relationship Management Essbase Crystal Ball helliphellip Hyperion Kitchen Sink
Kscope
Challenges (continued) Need to understand underlying technologies OS Servers (WindowsUnix) Application Servers (Weblogic Tomcat) Web Servers
(IIS Apache OHS) Authentication Options (AD Native Custom) Languages (Java NET ASP Classic JSP VBScript) Communication Frameworks (DCOM Web Services) Database (SQL Server Oracle Berkeley DB) Network Principles (Routing Firewall VPN)
Kscope
Challenges (continued) End Users Quantity ndash Intermittent issues due to number of users
stressing system
Internal vs External Less control over environment More points of failure Remote troubleshooting more difficult
User Experience New Users - More hands on but easier problems Power Users ndash Likes to experiment
Kscope
Challenges (continued) Internal Issues aka Politics Separation of Duties
Complicates troubleshooting efforts due to lack of access Delays resolution due to co-ordination efforts
Restrictive Policies Could preventlimit certain types of troubleshooting
NetworkHTTP sniffing No monitor programs on production
Kscope
Challenges (continued)
Cryptic Error Messages
But I didnrsquot click on a folder
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued) Limited OOTB (Out Of The Box) Tools Mostly consists of Log Files System 11 has expanded diagnostic capability SupportOracleCom
Kscope
Troubleshooting Strategy
Kscope
Troubleshooting Strategy
Final Solution
Follow Up
Evaluate
Possible
Solutions
Clearly
Defined
Problem
Kscope
Troubleshooting Strategy (continued) Clearly Define the Problem Ask the right questions Verify Information Where Possible
Research Prioritize and Evaluate potential solutions Use Initial Information and Resources to propose
possible causessolutions Prioritize in order of likelihood Evaluate Potential Solutions Determine Final Solution or Escalate
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Document Get Sign-Off Add Information To Internal Knowledgebase
Kscope
Troubleshooting Strategy (continued) Notes Process Is Easier As You Learn More About The
System Development Test Environment Makes This Easier Donrsquot Fear Leveraging Other Resources Practice Practice Practice
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How Who (is affected)
All Users or Select Group If subset is there commonality (Geographic
HardwareSoftware Data Provider)
What (is happening not happening) What was the attempted procedure What has changed
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How When (does the issue occur)
Consistently Intermittently Pattern
Where (are failures occurring) Client Machine(s) Database Server(s) Web Server(s) App Server(s)
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How How (should the process work)
Do you know how it should work Important to understand expected operation to help
determine point of failure probable cause and solution Research this if you are not sure
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Session Goals
Discover some new tools to be used for troubleshooting (Oracle Third Party)
Acquire some new troubleshooting ideas
Kscope
Challenges Responsible for a large amount of products Foundation Services (OHS Weblogic Shared Services
Workspace EPMA Calc Manager LCM) HFM (HFM Web Client Smartview) FDM Reporting and Analysis (FR IR SQLPR WA) Planning Performance Scorecard Strategic Finance
Probability Cost Disclosure Management Financial Close Management Data Relationship Management Essbase Crystal Ball helliphellip Hyperion Kitchen Sink
Kscope
Challenges (continued) Need to understand underlying technologies OS Servers (WindowsUnix) Application Servers (Weblogic Tomcat) Web Servers
(IIS Apache OHS) Authentication Options (AD Native Custom) Languages (Java NET ASP Classic JSP VBScript) Communication Frameworks (DCOM Web Services) Database (SQL Server Oracle Berkeley DB) Network Principles (Routing Firewall VPN)
Kscope
Challenges (continued) End Users Quantity ndash Intermittent issues due to number of users
stressing system
Internal vs External Less control over environment More points of failure Remote troubleshooting more difficult
User Experience New Users - More hands on but easier problems Power Users ndash Likes to experiment
Kscope
Challenges (continued) Internal Issues aka Politics Separation of Duties
Complicates troubleshooting efforts due to lack of access Delays resolution due to co-ordination efforts
Restrictive Policies Could preventlimit certain types of troubleshooting
NetworkHTTP sniffing No monitor programs on production
Kscope
Challenges (continued)
Cryptic Error Messages
But I didnrsquot click on a folder
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued) Limited OOTB (Out Of The Box) Tools Mostly consists of Log Files System 11 has expanded diagnostic capability SupportOracleCom
Kscope
Troubleshooting Strategy
Kscope
Troubleshooting Strategy
Final Solution
Follow Up
Evaluate
Possible
Solutions
Clearly
Defined
Problem
Kscope
Troubleshooting Strategy (continued) Clearly Define the Problem Ask the right questions Verify Information Where Possible
Research Prioritize and Evaluate potential solutions Use Initial Information and Resources to propose
possible causessolutions Prioritize in order of likelihood Evaluate Potential Solutions Determine Final Solution or Escalate
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Document Get Sign-Off Add Information To Internal Knowledgebase
Kscope
Troubleshooting Strategy (continued) Notes Process Is Easier As You Learn More About The
System Development Test Environment Makes This Easier Donrsquot Fear Leveraging Other Resources Practice Practice Practice
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How Who (is affected)
All Users or Select Group If subset is there commonality (Geographic
HardwareSoftware Data Provider)
What (is happening not happening) What was the attempted procedure What has changed
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How When (does the issue occur)
Consistently Intermittently Pattern
Where (are failures occurring) Client Machine(s) Database Server(s) Web Server(s) App Server(s)
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How How (should the process work)
Do you know how it should work Important to understand expected operation to help
determine point of failure probable cause and solution Research this if you are not sure
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Challenges Responsible for a large amount of products Foundation Services (OHS Weblogic Shared Services
Workspace EPMA Calc Manager LCM) HFM (HFM Web Client Smartview) FDM Reporting and Analysis (FR IR SQLPR WA) Planning Performance Scorecard Strategic Finance
Probability Cost Disclosure Management Financial Close Management Data Relationship Management Essbase Crystal Ball helliphellip Hyperion Kitchen Sink
Kscope
Challenges (continued) Need to understand underlying technologies OS Servers (WindowsUnix) Application Servers (Weblogic Tomcat) Web Servers
(IIS Apache OHS) Authentication Options (AD Native Custom) Languages (Java NET ASP Classic JSP VBScript) Communication Frameworks (DCOM Web Services) Database (SQL Server Oracle Berkeley DB) Network Principles (Routing Firewall VPN)
Kscope
Challenges (continued) End Users Quantity ndash Intermittent issues due to number of users
stressing system
Internal vs External Less control over environment More points of failure Remote troubleshooting more difficult
User Experience New Users - More hands on but easier problems Power Users ndash Likes to experiment
Kscope
Challenges (continued) Internal Issues aka Politics Separation of Duties
Complicates troubleshooting efforts due to lack of access Delays resolution due to co-ordination efforts
Restrictive Policies Could preventlimit certain types of troubleshooting
NetworkHTTP sniffing No monitor programs on production
Kscope
Challenges (continued)
Cryptic Error Messages
But I didnrsquot click on a folder
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued) Limited OOTB (Out Of The Box) Tools Mostly consists of Log Files System 11 has expanded diagnostic capability SupportOracleCom
Kscope
Troubleshooting Strategy
Kscope
Troubleshooting Strategy
Final Solution
Follow Up
Evaluate
Possible
Solutions
Clearly
Defined
Problem
Kscope
Troubleshooting Strategy (continued) Clearly Define the Problem Ask the right questions Verify Information Where Possible
Research Prioritize and Evaluate potential solutions Use Initial Information and Resources to propose
possible causessolutions Prioritize in order of likelihood Evaluate Potential Solutions Determine Final Solution or Escalate
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Document Get Sign-Off Add Information To Internal Knowledgebase
Kscope
Troubleshooting Strategy (continued) Notes Process Is Easier As You Learn More About The
System Development Test Environment Makes This Easier Donrsquot Fear Leveraging Other Resources Practice Practice Practice
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How Who (is affected)
All Users or Select Group If subset is there commonality (Geographic
HardwareSoftware Data Provider)
What (is happening not happening) What was the attempted procedure What has changed
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How When (does the issue occur)
Consistently Intermittently Pattern
Where (are failures occurring) Client Machine(s) Database Server(s) Web Server(s) App Server(s)
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How How (should the process work)
Do you know how it should work Important to understand expected operation to help
determine point of failure probable cause and solution Research this if you are not sure
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Challenges (continued) Need to understand underlying technologies OS Servers (WindowsUnix) Application Servers (Weblogic Tomcat) Web Servers
(IIS Apache OHS) Authentication Options (AD Native Custom) Languages (Java NET ASP Classic JSP VBScript) Communication Frameworks (DCOM Web Services) Database (SQL Server Oracle Berkeley DB) Network Principles (Routing Firewall VPN)
Kscope
Challenges (continued) End Users Quantity ndash Intermittent issues due to number of users
stressing system
Internal vs External Less control over environment More points of failure Remote troubleshooting more difficult
User Experience New Users - More hands on but easier problems Power Users ndash Likes to experiment
Kscope
Challenges (continued) Internal Issues aka Politics Separation of Duties
Complicates troubleshooting efforts due to lack of access Delays resolution due to co-ordination efforts
Restrictive Policies Could preventlimit certain types of troubleshooting
NetworkHTTP sniffing No monitor programs on production
Kscope
Challenges (continued)
Cryptic Error Messages
But I didnrsquot click on a folder
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued) Limited OOTB (Out Of The Box) Tools Mostly consists of Log Files System 11 has expanded diagnostic capability SupportOracleCom
Kscope
Troubleshooting Strategy
Kscope
Troubleshooting Strategy
Final Solution
Follow Up
Evaluate
Possible
Solutions
Clearly
Defined
Problem
Kscope
Troubleshooting Strategy (continued) Clearly Define the Problem Ask the right questions Verify Information Where Possible
Research Prioritize and Evaluate potential solutions Use Initial Information and Resources to propose
possible causessolutions Prioritize in order of likelihood Evaluate Potential Solutions Determine Final Solution or Escalate
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Document Get Sign-Off Add Information To Internal Knowledgebase
Kscope
Troubleshooting Strategy (continued) Notes Process Is Easier As You Learn More About The
System Development Test Environment Makes This Easier Donrsquot Fear Leveraging Other Resources Practice Practice Practice
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How Who (is affected)
All Users or Select Group If subset is there commonality (Geographic
HardwareSoftware Data Provider)
What (is happening not happening) What was the attempted procedure What has changed
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How When (does the issue occur)
Consistently Intermittently Pattern
Where (are failures occurring) Client Machine(s) Database Server(s) Web Server(s) App Server(s)
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How How (should the process work)
Do you know how it should work Important to understand expected operation to help
determine point of failure probable cause and solution Research this if you are not sure
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Challenges (continued) End Users Quantity ndash Intermittent issues due to number of users
stressing system
Internal vs External Less control over environment More points of failure Remote troubleshooting more difficult
User Experience New Users - More hands on but easier problems Power Users ndash Likes to experiment
Kscope
Challenges (continued) Internal Issues aka Politics Separation of Duties
Complicates troubleshooting efforts due to lack of access Delays resolution due to co-ordination efforts
Restrictive Policies Could preventlimit certain types of troubleshooting
NetworkHTTP sniffing No monitor programs on production
Kscope
Challenges (continued)
Cryptic Error Messages
But I didnrsquot click on a folder
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued) Limited OOTB (Out Of The Box) Tools Mostly consists of Log Files System 11 has expanded diagnostic capability SupportOracleCom
Kscope
Troubleshooting Strategy
Kscope
Troubleshooting Strategy
Final Solution
Follow Up
Evaluate
Possible
Solutions
Clearly
Defined
Problem
Kscope
Troubleshooting Strategy (continued) Clearly Define the Problem Ask the right questions Verify Information Where Possible
Research Prioritize and Evaluate potential solutions Use Initial Information and Resources to propose
possible causessolutions Prioritize in order of likelihood Evaluate Potential Solutions Determine Final Solution or Escalate
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Document Get Sign-Off Add Information To Internal Knowledgebase
Kscope
Troubleshooting Strategy (continued) Notes Process Is Easier As You Learn More About The
System Development Test Environment Makes This Easier Donrsquot Fear Leveraging Other Resources Practice Practice Practice
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How Who (is affected)
All Users or Select Group If subset is there commonality (Geographic
HardwareSoftware Data Provider)
What (is happening not happening) What was the attempted procedure What has changed
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How When (does the issue occur)
Consistently Intermittently Pattern
Where (are failures occurring) Client Machine(s) Database Server(s) Web Server(s) App Server(s)
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How How (should the process work)
Do you know how it should work Important to understand expected operation to help
determine point of failure probable cause and solution Research this if you are not sure
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Challenges (continued) Internal Issues aka Politics Separation of Duties
Complicates troubleshooting efforts due to lack of access Delays resolution due to co-ordination efforts
Restrictive Policies Could preventlimit certain types of troubleshooting
NetworkHTTP sniffing No monitor programs on production
Kscope
Challenges (continued)
Cryptic Error Messages
But I didnrsquot click on a folder
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued) Limited OOTB (Out Of The Box) Tools Mostly consists of Log Files System 11 has expanded diagnostic capability SupportOracleCom
Kscope
Troubleshooting Strategy
Kscope
Troubleshooting Strategy
Final Solution
Follow Up
Evaluate
Possible
Solutions
Clearly
Defined
Problem
Kscope
Troubleshooting Strategy (continued) Clearly Define the Problem Ask the right questions Verify Information Where Possible
Research Prioritize and Evaluate potential solutions Use Initial Information and Resources to propose
possible causessolutions Prioritize in order of likelihood Evaluate Potential Solutions Determine Final Solution or Escalate
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Document Get Sign-Off Add Information To Internal Knowledgebase
Kscope
Troubleshooting Strategy (continued) Notes Process Is Easier As You Learn More About The
System Development Test Environment Makes This Easier Donrsquot Fear Leveraging Other Resources Practice Practice Practice
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How Who (is affected)
All Users or Select Group If subset is there commonality (Geographic
HardwareSoftware Data Provider)
What (is happening not happening) What was the attempted procedure What has changed
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How When (does the issue occur)
Consistently Intermittently Pattern
Where (are failures occurring) Client Machine(s) Database Server(s) Web Server(s) App Server(s)
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How How (should the process work)
Do you know how it should work Important to understand expected operation to help
determine point of failure probable cause and solution Research this if you are not sure
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Challenges (continued)
Cryptic Error Messages
But I didnrsquot click on a folder
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued) Limited OOTB (Out Of The Box) Tools Mostly consists of Log Files System 11 has expanded diagnostic capability SupportOracleCom
Kscope
Troubleshooting Strategy
Kscope
Troubleshooting Strategy
Final Solution
Follow Up
Evaluate
Possible
Solutions
Clearly
Defined
Problem
Kscope
Troubleshooting Strategy (continued) Clearly Define the Problem Ask the right questions Verify Information Where Possible
Research Prioritize and Evaluate potential solutions Use Initial Information and Resources to propose
possible causessolutions Prioritize in order of likelihood Evaluate Potential Solutions Determine Final Solution or Escalate
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Document Get Sign-Off Add Information To Internal Knowledgebase
Kscope
Troubleshooting Strategy (continued) Notes Process Is Easier As You Learn More About The
System Development Test Environment Makes This Easier Donrsquot Fear Leveraging Other Resources Practice Practice Practice
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How Who (is affected)
All Users or Select Group If subset is there commonality (Geographic
HardwareSoftware Data Provider)
What (is happening not happening) What was the attempted procedure What has changed
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How When (does the issue occur)
Consistently Intermittently Pattern
Where (are failures occurring) Client Machine(s) Database Server(s) Web Server(s) App Server(s)
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How How (should the process work)
Do you know how it should work Important to understand expected operation to help
determine point of failure probable cause and solution Research this if you are not sure
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued) Limited OOTB (Out Of The Box) Tools Mostly consists of Log Files System 11 has expanded diagnostic capability SupportOracleCom
Kscope
Troubleshooting Strategy
Kscope
Troubleshooting Strategy
Final Solution
Follow Up
Evaluate
Possible
Solutions
Clearly
Defined
Problem
Kscope
Troubleshooting Strategy (continued) Clearly Define the Problem Ask the right questions Verify Information Where Possible
Research Prioritize and Evaluate potential solutions Use Initial Information and Resources to propose
possible causessolutions Prioritize in order of likelihood Evaluate Potential Solutions Determine Final Solution or Escalate
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Document Get Sign-Off Add Information To Internal Knowledgebase
Kscope
Troubleshooting Strategy (continued) Notes Process Is Easier As You Learn More About The
System Development Test Environment Makes This Easier Donrsquot Fear Leveraging Other Resources Practice Practice Practice
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How Who (is affected)
All Users or Select Group If subset is there commonality (Geographic
HardwareSoftware Data Provider)
What (is happening not happening) What was the attempted procedure What has changed
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How When (does the issue occur)
Consistently Intermittently Pattern
Where (are failures occurring) Client Machine(s) Database Server(s) Web Server(s) App Server(s)
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How How (should the process work)
Do you know how it should work Important to understand expected operation to help
determine point of failure probable cause and solution Research this if you are not sure
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued) Limited OOTB (Out Of The Box) Tools Mostly consists of Log Files System 11 has expanded diagnostic capability SupportOracleCom
Kscope
Troubleshooting Strategy
Kscope
Troubleshooting Strategy
Final Solution
Follow Up
Evaluate
Possible
Solutions
Clearly
Defined
Problem
Kscope
Troubleshooting Strategy (continued) Clearly Define the Problem Ask the right questions Verify Information Where Possible
Research Prioritize and Evaluate potential solutions Use Initial Information and Resources to propose
possible causessolutions Prioritize in order of likelihood Evaluate Potential Solutions Determine Final Solution or Escalate
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Document Get Sign-Off Add Information To Internal Knowledgebase
Kscope
Troubleshooting Strategy (continued) Notes Process Is Easier As You Learn More About The
System Development Test Environment Makes This Easier Donrsquot Fear Leveraging Other Resources Practice Practice Practice
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How Who (is affected)
All Users or Select Group If subset is there commonality (Geographic
HardwareSoftware Data Provider)
What (is happening not happening) What was the attempted procedure What has changed
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How When (does the issue occur)
Consistently Intermittently Pattern
Where (are failures occurring) Client Machine(s) Database Server(s) Web Server(s) App Server(s)
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How How (should the process work)
Do you know how it should work Important to understand expected operation to help
determine point of failure probable cause and solution Research this if you are not sure
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued) Limited OOTB (Out Of The Box) Tools Mostly consists of Log Files System 11 has expanded diagnostic capability SupportOracleCom
Kscope
Troubleshooting Strategy
Kscope
Troubleshooting Strategy
Final Solution
Follow Up
Evaluate
Possible
Solutions
Clearly
Defined
Problem
Kscope
Troubleshooting Strategy (continued) Clearly Define the Problem Ask the right questions Verify Information Where Possible
Research Prioritize and Evaluate potential solutions Use Initial Information and Resources to propose
possible causessolutions Prioritize in order of likelihood Evaluate Potential Solutions Determine Final Solution or Escalate
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Document Get Sign-Off Add Information To Internal Knowledgebase
Kscope
Troubleshooting Strategy (continued) Notes Process Is Easier As You Learn More About The
System Development Test Environment Makes This Easier Donrsquot Fear Leveraging Other Resources Practice Practice Practice
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How Who (is affected)
All Users or Select Group If subset is there commonality (Geographic
HardwareSoftware Data Provider)
What (is happening not happening) What was the attempted procedure What has changed
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How When (does the issue occur)
Consistently Intermittently Pattern
Where (are failures occurring) Client Machine(s) Database Server(s) Web Server(s) App Server(s)
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How How (should the process work)
Do you know how it should work Important to understand expected operation to help
determine point of failure probable cause and solution Research this if you are not sure
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued) Limited OOTB (Out Of The Box) Tools Mostly consists of Log Files System 11 has expanded diagnostic capability SupportOracleCom
Kscope
Troubleshooting Strategy
Kscope
Troubleshooting Strategy
Final Solution
Follow Up
Evaluate
Possible
Solutions
Clearly
Defined
Problem
Kscope
Troubleshooting Strategy (continued) Clearly Define the Problem Ask the right questions Verify Information Where Possible
Research Prioritize and Evaluate potential solutions Use Initial Information and Resources to propose
possible causessolutions Prioritize in order of likelihood Evaluate Potential Solutions Determine Final Solution or Escalate
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Document Get Sign-Off Add Information To Internal Knowledgebase
Kscope
Troubleshooting Strategy (continued) Notes Process Is Easier As You Learn More About The
System Development Test Environment Makes This Easier Donrsquot Fear Leveraging Other Resources Practice Practice Practice
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How Who (is affected)
All Users or Select Group If subset is there commonality (Geographic
HardwareSoftware Data Provider)
What (is happening not happening) What was the attempted procedure What has changed
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How When (does the issue occur)
Consistently Intermittently Pattern
Where (are failures occurring) Client Machine(s) Database Server(s) Web Server(s) App Server(s)
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How How (should the process work)
Do you know how it should work Important to understand expected operation to help
determine point of failure probable cause and solution Research this if you are not sure
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Challenges (continued)
Cryptic Error Messages
Kscope
Challenges (continued) Limited OOTB (Out Of The Box) Tools Mostly consists of Log Files System 11 has expanded diagnostic capability SupportOracleCom
Kscope
Troubleshooting Strategy
Kscope
Troubleshooting Strategy
Final Solution
Follow Up
Evaluate
Possible
Solutions
Clearly
Defined
Problem
Kscope
Troubleshooting Strategy (continued) Clearly Define the Problem Ask the right questions Verify Information Where Possible
Research Prioritize and Evaluate potential solutions Use Initial Information and Resources to propose
possible causessolutions Prioritize in order of likelihood Evaluate Potential Solutions Determine Final Solution or Escalate
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Document Get Sign-Off Add Information To Internal Knowledgebase
Kscope
Troubleshooting Strategy (continued) Notes Process Is Easier As You Learn More About The
System Development Test Environment Makes This Easier Donrsquot Fear Leveraging Other Resources Practice Practice Practice
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How Who (is affected)
All Users or Select Group If subset is there commonality (Geographic
HardwareSoftware Data Provider)
What (is happening not happening) What was the attempted procedure What has changed
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How When (does the issue occur)
Consistently Intermittently Pattern
Where (are failures occurring) Client Machine(s) Database Server(s) Web Server(s) App Server(s)
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How How (should the process work)
Do you know how it should work Important to understand expected operation to help
determine point of failure probable cause and solution Research this if you are not sure
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Challenges (continued) Limited OOTB (Out Of The Box) Tools Mostly consists of Log Files System 11 has expanded diagnostic capability SupportOracleCom
Kscope
Troubleshooting Strategy
Kscope
Troubleshooting Strategy
Final Solution
Follow Up
Evaluate
Possible
Solutions
Clearly
Defined
Problem
Kscope
Troubleshooting Strategy (continued) Clearly Define the Problem Ask the right questions Verify Information Where Possible
Research Prioritize and Evaluate potential solutions Use Initial Information and Resources to propose
possible causessolutions Prioritize in order of likelihood Evaluate Potential Solutions Determine Final Solution or Escalate
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Document Get Sign-Off Add Information To Internal Knowledgebase
Kscope
Troubleshooting Strategy (continued) Notes Process Is Easier As You Learn More About The
System Development Test Environment Makes This Easier Donrsquot Fear Leveraging Other Resources Practice Practice Practice
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How Who (is affected)
All Users or Select Group If subset is there commonality (Geographic
HardwareSoftware Data Provider)
What (is happening not happening) What was the attempted procedure What has changed
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How When (does the issue occur)
Consistently Intermittently Pattern
Where (are failures occurring) Client Machine(s) Database Server(s) Web Server(s) App Server(s)
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How How (should the process work)
Do you know how it should work Important to understand expected operation to help
determine point of failure probable cause and solution Research this if you are not sure
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Strategy
Kscope
Troubleshooting Strategy
Final Solution
Follow Up
Evaluate
Possible
Solutions
Clearly
Defined
Problem
Kscope
Troubleshooting Strategy (continued) Clearly Define the Problem Ask the right questions Verify Information Where Possible
Research Prioritize and Evaluate potential solutions Use Initial Information and Resources to propose
possible causessolutions Prioritize in order of likelihood Evaluate Potential Solutions Determine Final Solution or Escalate
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Document Get Sign-Off Add Information To Internal Knowledgebase
Kscope
Troubleshooting Strategy (continued) Notes Process Is Easier As You Learn More About The
System Development Test Environment Makes This Easier Donrsquot Fear Leveraging Other Resources Practice Practice Practice
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How Who (is affected)
All Users or Select Group If subset is there commonality (Geographic
HardwareSoftware Data Provider)
What (is happening not happening) What was the attempted procedure What has changed
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How When (does the issue occur)
Consistently Intermittently Pattern
Where (are failures occurring) Client Machine(s) Database Server(s) Web Server(s) App Server(s)
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How How (should the process work)
Do you know how it should work Important to understand expected operation to help
determine point of failure probable cause and solution Research this if you are not sure
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Strategy
Final Solution
Follow Up
Evaluate
Possible
Solutions
Clearly
Defined
Problem
Kscope
Troubleshooting Strategy (continued) Clearly Define the Problem Ask the right questions Verify Information Where Possible
Research Prioritize and Evaluate potential solutions Use Initial Information and Resources to propose
possible causessolutions Prioritize in order of likelihood Evaluate Potential Solutions Determine Final Solution or Escalate
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Document Get Sign-Off Add Information To Internal Knowledgebase
Kscope
Troubleshooting Strategy (continued) Notes Process Is Easier As You Learn More About The
System Development Test Environment Makes This Easier Donrsquot Fear Leveraging Other Resources Practice Practice Practice
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How Who (is affected)
All Users or Select Group If subset is there commonality (Geographic
HardwareSoftware Data Provider)
What (is happening not happening) What was the attempted procedure What has changed
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How When (does the issue occur)
Consistently Intermittently Pattern
Where (are failures occurring) Client Machine(s) Database Server(s) Web Server(s) App Server(s)
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How How (should the process work)
Do you know how it should work Important to understand expected operation to help
determine point of failure probable cause and solution Research this if you are not sure
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Strategy (continued) Clearly Define the Problem Ask the right questions Verify Information Where Possible
Research Prioritize and Evaluate potential solutions Use Initial Information and Resources to propose
possible causessolutions Prioritize in order of likelihood Evaluate Potential Solutions Determine Final Solution or Escalate
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Document Get Sign-Off Add Information To Internal Knowledgebase
Kscope
Troubleshooting Strategy (continued) Notes Process Is Easier As You Learn More About The
System Development Test Environment Makes This Easier Donrsquot Fear Leveraging Other Resources Practice Practice Practice
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How Who (is affected)
All Users or Select Group If subset is there commonality (Geographic
HardwareSoftware Data Provider)
What (is happening not happening) What was the attempted procedure What has changed
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How When (does the issue occur)
Consistently Intermittently Pattern
Where (are failures occurring) Client Machine(s) Database Server(s) Web Server(s) App Server(s)
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How How (should the process work)
Do you know how it should work Important to understand expected operation to help
determine point of failure probable cause and solution Research this if you are not sure
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Document Get Sign-Off Add Information To Internal Knowledgebase
Kscope
Troubleshooting Strategy (continued) Notes Process Is Easier As You Learn More About The
System Development Test Environment Makes This Easier Donrsquot Fear Leveraging Other Resources Practice Practice Practice
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How Who (is affected)
All Users or Select Group If subset is there commonality (Geographic
HardwareSoftware Data Provider)
What (is happening not happening) What was the attempted procedure What has changed
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How When (does the issue occur)
Consistently Intermittently Pattern
Where (are failures occurring) Client Machine(s) Database Server(s) Web Server(s) App Server(s)
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How How (should the process work)
Do you know how it should work Important to understand expected operation to help
determine point of failure probable cause and solution Research this if you are not sure
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Strategy (continued) Notes Process Is Easier As You Learn More About The
System Development Test Environment Makes This Easier Donrsquot Fear Leveraging Other Resources Practice Practice Practice
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How Who (is affected)
All Users or Select Group If subset is there commonality (Geographic
HardwareSoftware Data Provider)
What (is happening not happening) What was the attempted procedure What has changed
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How When (does the issue occur)
Consistently Intermittently Pattern
Where (are failures occurring) Client Machine(s) Database Server(s) Web Server(s) App Server(s)
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How How (should the process work)
Do you know how it should work Important to understand expected operation to help
determine point of failure probable cause and solution Research this if you are not sure
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How Who (is affected)
All Users or Select Group If subset is there commonality (Geographic
HardwareSoftware Data Provider)
What (is happening not happening) What was the attempted procedure What has changed
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How When (does the issue occur)
Consistently Intermittently Pattern
Where (are failures occurring) Client Machine(s) Database Server(s) Web Server(s) App Server(s)
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How How (should the process work)
Do you know how it should work Important to understand expected operation to help
determine point of failure probable cause and solution Research this if you are not sure
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How When (does the issue occur)
Consistently Intermittently Pattern
Where (are failures occurring) Client Machine(s) Database Server(s) Web Server(s) App Server(s)
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How How (should the process work)
Do you know how it should work Important to understand expected operation to help
determine point of failure probable cause and solution Research this if you are not sure
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Strategy (continued) Clearly Define The Problem ndash Answer the Who What When Where How How (should the process work)
Do you know how it should work Important to understand expected operation to help
determine point of failure probable cause and solution Research this if you are not sure
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Iterative Process Defined Problem + Internal Knowledge + External
Knowledge + Leveraged Resources + Tools = Potential Solutions
Prioritize Potential Solutions KISS Occamrsquos Razor ndash Start with most likely obvious first
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Strategy (continued) Research Prioritize and Evaluate potential solutions Evaluate Potential Solutions
In TestDev if possible Promote to production after verifying your fix doesnrsquot cause
additional problems If you exhaust all possible causes escalate Donrsquot spin
wheels Do not discount workarounds
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Strategy (continued) Close Out The Problem Get Positive Confirmation That Issue Is Resolved Add Issue to Internal Knowledgebase For Future
Review SharePoint Track-It Shared Outlook Folder
Share Information When Necessary
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Internal Tools
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools - Internal
Knowledge Tools httpsupportoraclecom ndash Knowledgebase Patches amp
Update and ldquoCommunitiesrdquo
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools OTN Discussion Forum (httpforumsoraclecom)
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont)
Knowledge Tools Product Documentation
System 11121 - httpdocsoraclecomcdE24674_01indexhtm System 93x - httpdocsoraclecomcdE10530_01welcomehtml
NOTE These move periodically Google lsquoHyperion documentation libraryrsquo if you cannot find what you are looking for
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools - Internal
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont) Knowledge Tools Tips and Tricks from EPM System Infrastructure
Development Issues 1 to 72 -
httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-1-72-427329pdf
Issues 73 and up - httpwwworaclecomtechnetworkmiddlewarebi-foundationepm-tips-issues-73-up-399995pdf
NOTE Constantly updating so there will be more etc
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Log Files (see Table)
HFMErrorLookupexe ndash Resolves Error Codes to Text Description
System 11- ltEPM ORACLE HOMEgtEPMSystem11R1productsFinancialManagementConsultant Utilities
System 9 - ltHYPERION HOMEgtFinancialManagementConsultant Utilities
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering HFMErrorLogViewerexe ndash Utility to view HFM error logs
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering FRConfigcmd ndash FR utility that shows app Mem CPU
utilization System 11 - ltEPM_ORACLE_HOMEgtproductsfinancialreportingbin
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering ManageUserPOVcmd ndash ViewEdit POV for users
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Remote Diagnostic Agent (RDA) ndash Gathers system
information about environment [See KB ID 13048851]
oPatch [System 11 Only] ndash Install and list installed patches Location - ltEPM ORACLE HOMEgtOpatch List Installed products = Opatch lsinventory
NOTE If you get an error running about OUI Home make sure environment variable pointing to EPM ORACLE HOME which is most like OracleMiddlewareEPMSystem11R1
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Compsxml [System 11 Only] ndash List of all installed components
on machine ltEPM ORACLE HOMEgtinventoryContentsXML
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Epmsys_registrybat [System 11 Only] ndash Generates a detailed
report of everything registered in Shared Services Registry Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Reports - ltMIDDLEWARE HOMEgtuser_projectsdiagnosticsreportsregistryhtml
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering lsquoAbout Usrsquo pages ndash Shows currently installed programs and
specific version info (Applies to Workspace FDM Shared Service web pages etc)
Click on the lsquoShow Detailsrsquo button to see version info per product NOTE This version info MAY NOT change even if you have
installed patches
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httphostnameportworkspacebrowseconfigInfo - Returns detailed information about products installed and links to other configuration files
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httpltshared_services_servergt28080interopframeowrkgetCSSConfigFile ndash Returns configuration information for Shared Services
System 9 ndash Port Number is 58080 System 11 ndash You may need to administratively enable this
page for it to be visible
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont)
Data Gathering Informational Web Pages
httplthfm_web_servergt80hfm ndash Returns lsquohfmrsquo if HFM is running properly and integrated with workspace (else opens HFM Web)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs Validatebat [System 11 Only] - Kicks off process that will test
installed components on the machine for proper operation Generates detailed report
Can be run on product by product basis or all at once Program - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1bin Report - ltMIDDLEWARE HOMEgtuser_projectsepmsystem1diagnostics
reports NOTE If you are building an environment you should be
running this after your complete installconfig
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont)
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash Internal (cont)
Utility Programs DB_Verify DB_Recover [Pre-System 11 Only] - Checks and
corrects Shared Services OpenLDAP database issues
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
3rd Party Tools
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash 3rd Party Data Gathering WebEx GoTo Meeting Log Me In VNC Lync ndash
Remote Access Software A must for remote support
Windows Event Log
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Ping Tracert ndash Simple connectivity test tools
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash 3rd Party
Basic Connectivity Tools Telnet ndash Connectivity tool for any port
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Advanced Connectivity Tools Fiddler ndash Web proxy that allows you to monitor (and
simulate) HTTPHTTPS traffic httpfiddler2comfiddler2
Great Tool for HTTPHTTPS traffic
Warning You can sniff decode HTTPS traffic Warning Some programs may send logins
passwords in plain text
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash 3rd Party (cont) Advanced Connectivity Tools Wireshark ndash Network traffic analyzer ndash
httpwwwwiresharkorg Network packet analyzer Monitors ALL network traffic not just HTTPHTTPS Warning Can intercept all types of traffic such as
Streaming Audio Video VoiP Types of Encrypted Data Plain Text User Name Password
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools Service Manager (servicesmsc) ndash View services to see
which are running
Process Monitor ndash Allows you to monitor processesrsquo file use registry access memory utilization etc httptechnetmicrosoftcomen-ussysinternalsbb896645
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Server Tools SQL Profiler ndash Allows you to monitor database activity in
SQL Server ndash httpmsdnmicrosoftcomen-uslibraryms181091aspx
[Oracle Enterprise Manager has similar functionality for Oracle databases]
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Tools ndash 3rd Party (cont)
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Troubleshooting Problems
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Migrate VM Server Error
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Migrate VM Server Error To aid with session built VMVirtual PC System
11 ldquoAll In Onerdquo Environment
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Migrate VM Server Error (cont)
After copying VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Migrate VM Server Error (cont)
But not everything was deadhellip VM to another machinehellip
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Migrate VM Server Error (cont)
IDEAS What changed
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Migrate VM Server Error (cont)
Lets get some machine info
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Migrate VM Server Error (cont)
Lets get some config (epmsys_registrybat)
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Migrate VM Server Error (cont)
IDEAS Anyone see anything suspect so far I did and have a suspicion but need
more information to confirm
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(Wireshark)
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Migrate VM Server Error (cont)
IDEAS Why would there be calls to an IP
address that does not belong to the machine
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Migrate VM Server Error (cont)
Lets do some sniffinghellip(WinGrep)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Migrate VM Server Error (cont)
Lets do some more sniffinghellip(SQLLocator)
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Migrate VM Server Error (cont)
IDEAS Looks pretty reasonable that hardcode
IP addresses are breaking things since the machine no longer has the IP address buried in config database
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Migrate VM Server Error (cont)
How To Fix Reinstall Reconfigure Manually Search and Replace
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Manual Repair
(Because we can)
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Migrate VM Server Error (cont)
Lets fix the SQL Damage First
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Migrate VM Server Error (cont)
Lets fix config file damage ContentXML ConfigXML GlobalContextXML httpdconf mod_wl_ohsconf Serverxml
Warning In some places you DO NOT want to replace the hardcoded IP address as it is part of a Unique Identifier That could be bad
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Migrate VM Server Error (cont)
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Migrate VM Server Error (cont)
Reboot
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Migrate VM Server Error (cont)
Mission Accomplished
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom
Kscope
Questions If you have any questions with the sample code or presentation feel free to askcontact me charlescharlescbeyercom