dn0251007x5x0xen

5.0

Maintenance Manual

Guidelines for MPC Problem Reporting

Document number/Issue DN0251007 Issue 5

Copyright Nokia Networks Oy

The information in this document is subject to change without notice and describes only the product defined in the introduction of this documentation. This document is intended for the use of Nokia Networks' customers only for the purposes of the agreement under which the document is submitted, and no part of it may be reproduced or transmitted in any form or means without the prior written permission of Nokia Networks. The document has been prepared to be used by professional and properly trained personnel, and the customer assumes full responsibility when using it. Nokia Networks welcomes customer comments as part of the process of continuous development and improvement of the documentation.

The information or statements given in this document concerning the suitability, capacity, or performance of the mentioned hardware or software products cannot be considered binding but shall be defined in the agreement made between Nokia Networks and the customer. However, Nokia Networks has made all reasonable efforts to ensure that the instructions contained in the document are adequate and free of material errors and omissions. Nokia Networks will, if necessary, explain issues which may not be covered by the document.

Nokia Networks' liability for any errors in the document is limited to the documentary correction of errors. Nokia Networks WILL NOT BE RESPONSIBLE IN ANY EVENT FOR ERRORS IN THIS DOCUMENT OR FOR ANY DAMAGES, INCIDENTAL OR CONSEQUENTIAL (INCLUDING MONETARY LOSSES), that might arise from the use of this document or the information in it.

This document and the product it describes are considered protected by copyright according to the applicable laws.

NOKIA logo is a registered trademark of Nokia Corporation.

Other product names mentioned in this document may be trademarks of their respective companies, and they are mentioned for identification purposes only.

Copyright Nokia Networks Oy 2004. All rights reserved.

No. of pages 29/KaS

Edited by/Translator 29 Dec 2003 V Saukkonen

Author 29 Dec 2003 V Saukkonen

Approved by 14 Jan 2004 P Pajari

Previous issue(x) approved

15 May 2003


Copyright Nokia Networks Oy Page

2

Contents

Contents

Changes made between issues 4 and 3 5 Changes made between issues 5 and 4 5

1. About this document 6

2. Scope of a problem report 7 2.1. Hardware problem 7 2.2. System defect or problem 7 2.3. Documentation problem 7 2.4. Product proposal 7

3. Problem report fields 8 3.1. Title field in Problem Report 8 3.2. Description field in Problem Report 8 3.3. Delivery field in Problem Report 9 3.4. Release field in Problem Report 9 3.5. Severity 9

3.5.1. Critical 10 3.5.2. Major 10 3.5.3. Minor 11 3.5.4. Information Request (IR) 11

3.6. Enhancement or improvement proposal 12

4. Information needed in problem reports 13 4.1. 2G SGSN related problems 13

4.1.1. System restart 13 4.1.2. Unit restart 14 4.1.3. Process restart 14 4.1.4. Backup 15 4.1.5. Mobility management 15 4.1.6. Session management 16 4.1.7. Charging 16 4.1.8. Statistics 17 4.1.9. Storing and/or transferring statistical data 18 4.1.10. Alarms 18 4.1.11. OSI 19 4.1.12. HW configuration 19 4.1.13. I/O devices 20 4.1.14. MML 20 4.1.15. CCS7 20

4.2. GGSN related problems 21 4.3. LIG related problems 21

4.3.1. LIG release 1 21 4.3.2. LIG release 2 23

4.4. 3G SGSN related problems 23 4.4.1. 3G SGSN install log file 23



3

Contents

4.4.2. Configuration files 24 4.4.3. Using grep to sort out messages 25 4.4.4. All core dump files 25

4.5. TA related problems 25 4.5.1. General 25 4.5.2. TA subsystem traces 25 4.5.2.1. TA Kernel Module Trace 26 4.5.2.2. ita_chargingd (RADIUS between TA and OSC)

tracing 27 4.5.2.3. Other TA subsystems (ita_httpd, ita_wapd, ita_rtspd,

ita_smtpd, ita_ftpd) tracing 27 4.5.3. Ita commands 28 4.5.4. Miscellaneous tips and hints 28

4.6. IPSO information 28 4.6.1. Run ipsctl command 28 4.6.2. Complete configuration file 28 4.6.3. All core dump files 29

4.7. Documentation 29



4

Contents

Summary of changes Fifth issue:

This is the customers version of MPC Problem Reporting Guide.

Changes made between issues 4 and 3

Section 2.5 Other has been removed. Section 3.5. Severity has been changed to reflect Help Desk wizard severity classification in NOLS. Section 3.5.4. Information Request (IR) has been added to the severity classes. Section describing enhancements or improvement proposals has been separated from severities to its own section 3.6. Enhancement or improvement proposal. Section 4.4.4. All core dump files has been added to this document. Changes made between issues 5 and 4

Section 4.5. TA related problems and 4.6. IPSO information has been added to this document. Numbering of Section 4.7. Documentation has changed respectively.



5

About this document

1. About this document Fast and reliable problem solving is beneficial to all parties involved. Time wasted in Further Information Requests can easily be saved by careful filling and proper attached data of Problem Reports. This document instructs how to fill in a Problem Report and what steps to take when collecting data for a reported problem. Following these instructions will improve fault location and speed up the correction process. The guidelines given here should be regarded as the minimum information required in a Problem Report. Any other information is welcome, but for example message monitoring without the knowledge of the exact process causing the problem might not be worth the effort. You should also mention whether any special actions have taken place prior to the problem (for example, Change Delivery installation, network configurations and extensions). As it might not be possible to execute all the commands given as examples in this document, they need to be considered case by case. Avoid using local abbreviations and terminology, or at least explain them carefully. This document does not handle hardware Problem Reports that are attached to faulty units being sent to the repair centre.



6

Scope of a problem report

2. Scope of a problem report Scope of a Problem Report is to report defects and problems that occur in software, hardware and documents of Mobile Packet Core.

2.1. Hardware problem When you suspect that some hardware of the mobile packet core causes the problem but you cannot determine the faulty unit, you can report it with a PR. Check that the TYPE of the PR is set to the value HARDWARE. If you find the faulty unit, fill in the hardware problem report, attach it to the faulty unit, and send them over to your local Nokia Hardware service (HWS).

2.2. System defect or problem The most common usage of a PR is reporting system defects and problems. They relate to the software or data configuration of the MPC product. Check that the TYPE of the PR is set to the value SOFTWARE.

2.3. Documentation problem Whenever you discover a deficiency in MPC documentation, make a PR. Remember not only to report the actual problem but also to specify:

The subject where you have discovered a deficiency Identification and issue information concerning the text (for example

DN9912345, Issue 1-2) The release it is in

Check that the TYPE of the PR is set to the value DOCUMENTATION. If you are using the NED, click the circled green 'i' on the top right corner of the text window. You will now be able to see the identification and version information.

2.4. Product proposal Whenever there is a need for a product proposal, please contact your local DMS.



7

Problem report fields

3. Problem report fields The details of Problem Reporting (for example Problem Report format and delivery to Nokia) should be agreed with the local Nokia DMS. Nokia provides a tool called Advanced Services Management System (ASMS) for customers to store and deliver Problem Reports. For more information, contact your local Nokia DMS. The following sections describe the most important fields in the Fault Report (PR) where you should pay special attention when filling them in.

3.1. Title field in Problem Report Make sure that the Title field clearly describes the main content of the problem.

3.2. Description field in Problem Report When filling in the Description field make sure that you include the following important information:

The actual problem The impact of the problem to the operator, mobile subscriber, or a

third party Description of the procedures and the environment before the

problem occurred, for example parameters of the MML commands Description of the problem situation, how it appeared and effected

the system, and what is the likehood that the problem appears again How the situation was recovered The possible cause for the problem What external packages (e.g. log files) were collected and where

they were stored, and How to reproduce the problem.

Often this information is not clearly stated and, as a result, it is very difficult or even impossible to investigate the problem. This results in the correction delivery to be delayed. Fill in the description field properly. For example, stating See enclosed files is not a valid description, even if the enclosures have good explanations.



8


Note Each problem needs to be reported separately, that is, only one

fault per Problem Report. This is important because different problems are normally handled in different software groups, and the management of multi-PRs is not possible.

Avoid using terms or abbreviations not commonly known in GSM/GPRS vocabulary (for switch names, special services, etc.), unless you explain them.

3.3. Delivery field in Problem Report It is important to fill in the right delivery code of the customer to the Delivery field. This is important the software status can be checked with the help of the delivery code. The delivery code is a unique identification given by MPC care.

3.4. Release field in Problem Report Select the right release code for the software from release field. Then choose MPC and the software. Selecting the right software release expedites the handling of the Problem Report and prevents the problem report going to wrong product line.

3.5. Severity In addition to the visual severity described in the Description field, use also the Severity field to indicate the scale of the severity that the problem has in relation to all the other Problem Reports. The scale to be used is from critical to info request, so that only very urgent cases should be indicated as Critical. It does not speed up the correction process if all Problem Reports are given severity Critical, because thus really important cases cannot be separated from the lower cases. However, the correct use of this field will help detecting the most important problems and thus speed up the investigation. The following sections describe the different severity values with the help of examples. The examples should help you to classify the fault to the correct severity value.



9


3.5.1. Critical

Conditions that severely affect service, capacity/traffic, billing and maintenence capabilities and require immediate corrective action, regardless of time of day or day of the week as viewed by a customer on discussion with the supplier. For example: a loss of service that is comparable to the total loss of effective functional capability of an entire switching or transport system. A reduction in capacity or traffic handling capability such that expected loads cannot be handled. Any loss of safety or emergency capability (e.g., 911 calls). Example situations are:

Total traffic interruption System restart Continual traffic disturbance PDP context drop frequently PDP context connects incorrectly Charging not working Continual disruption in charging Billing centre interface does not work Signalling connections down Authorities requirements don't work Business critical operating commands don't work Critical application area doesn't work Upgrade fails Impossible to take network element in use (configuration MMLs do

not work) 3.5.2. Major

Conditions that seriously affect system operation, maintenance and administration, etc. and require immediate attention as viewed by the customer on discussion with sthe supplier. The urgency is less than in critical situations because of a lesser immediate or impending effect on system performance, customers and the customer's operation and revenue. For example: reduction in any capacity/traffic measurement function. Any loss of functional visibility and/or diagnostic capability. Short outages equivalent to system or subsystem outages, with accumulated duration of greater than 2 minutes in any 24-hour period, or that continue to repeat during longer periods. Repeated degradation of DS1 or higher rater spans or connections. Prevention of access for routine administrative activity. Degradation of access for maintenance or recovery operations. Degradation of the system's ability to provide any required critical or major trouble notification. Any significant increase in product related customer trouble reports. Billing error rates that exceed specifications. Corruption of system or billing databases.



10


Example situations are:

Single restart of computer units Single performance measurement is not working Problems with back-up Configuration changes (HW and SW) are not working Alarm management of objects (SGSN, functional units) are not

working Subscriber related MPC functionality is not working completely Capacity/quality related functionality is not working.

3.5.3. Minor

Conditions that do not significantly impair the functioning of the system and do not significantly affect service to customers. These problems are not traffic affecting. Note: Engineering complaints are classified as minor unless otherwise negotiated between the customer and supplier. Example situations are:

All other problems not listed in the other impact level examples Errors in MML syntax Cosmetic errors in MML/Statistic output.

Example cases for documentation faults are:

Alarm or description is missing from a document (Alarm Reference Manuals, Command Reference Manuals, Operating Manuals)

Vital documents are missing from the MPC library Pictures or tables are missing from a document.

Faults concerning documentation may have as a solution a Documentation Technical Note or a work-around solution. 3.5.4. Information Request (IR)

Includes requests for design changes, or enhancements (feature requests). An IR is a service request for which a customer with technical expertise and acquaintance with the product could have determined or resolved on their own. An IR may have one or more of the following characteristics: It documents a problem that the customer could have resolved independently of the supplier. The customer asks a question on procedures that are covered in the documentation, which is shipped with the product. The customer asks for information on the supplier's product that will be used to help interface the supplier's product with a competitor's product. The customer asks for help on a 'problem' that turns out to be a problem, bug, or failure, but rather is due to a lack of understanding of the product.



11


3.6. Enhancement or improvement proposal You can suggest enhancements or improvement proposals for existing features or suggest totally new ones. Targets for improvement or change proposals are both the MPC software and documentation. Please refer to section 2.4. Product proposal when you have a suggestion for an improvement or feature.



12

Information needed in problem reports

4. Information needed in problem reports This chapter instructs what kind of additional information you should give in the Problem Report and how you can obtain it.

Note: As it might not always be possible to execute all the commands given as examples, they need to be considered case by case.

The chapter is divided into sections according to different problem types. You can thus use this document as a handbook for a quick review at the time the problem occurs. For most cases it is important to collect information on the problem right away, as later it may no longer be possible and the real reason for the problem may thus remain unsolved. Whatever the problem may be, always report detailed description of what was done or what happened prior to the problem and what actions were taken during it. It is important to explain all the operational activities, even if you think they are not related to the problem itself. Also, describe possible special conditions, for example in call profiles.

Note: It is very important to always attach the active CD-level of the software to every problem report. Also mention possible beta change notes which have been installed.

Remember also to fill in the Title, Description, and Impact fields as instructed in chapter Problem Report Fields. As a default, the following information should be required for the Problem Reports we receive:

Regularity of the fault How the end user experiences the fault How operator can see the fault

4.1. 2G SGSN related problems

Following sections give instructions for 2G SGSN related problem reporting. 4.1.1. System restart

In case of problems in system restart, collect information on the following:



13


The blackboxes of working (WO) and spare (SP) computer units The alarm history and blocked alarms of the network element

starting four hours before the system restart The MML log files of those users which were active prior to reset

for possible further investigations. Use the following commands: ZDDE::"ZLE:1,BOXANAGX","Z1U"; ZAHP:::< ime>; date,tZIGO::USERID=:; ZWNH:NAME=; In case of a unit restart, which usually occurs because of process exception then also give command: ZDDE::"ZGLP"; 4.1.2. Unit restart

When encountering problems with unit restarts, collect information on the following:

The blackbox of the unit The alarm history of the unit starting four hours before the restart The MML log files of those users which were active prior to reset

for possible further investigations. Use the following commands: ZDDE::"ZLE:1,BOXANAGX","Z1U"; ZAHP:::; ZIGO::USERID=:; ZWNH:NAME=; ZDDE::"ZGLP"; 4.1.3. Process restart

When encountering problems with process restarts, collect the following information:

The alarm history of the unit starting four hours before the process restart

The logs, operating system error counters, and used buffers of the unit The version of the restarted process.



14


Use the following commands: ZAHP:::; ZDDE::"ZSLP","ZSLE","ZSB","ZSCFE"; ZWNH:NAME=; 4.1.4. Backup

When encountering problems with backup, collect the following information:

Alarms those that are on, alarm history and blocked alarms The MML log of user (note: Command Calendar is also user =

COMCAL) Used backup media Used procedure. Computer log of the unit in question

Use the following commands: ZAHO; ZAHP:::; ZABO; ZIGO::USERID=:; ZDDE::"ZGSC"; ZWNH:NAME=; 4.1.5. Mobility management

When encountering problems with mobility management, collect the following information from the time when the problem has occured:

Message monitoring of the processes GSP, GMG, GIP and GGB If the problem is not related to flow control the message number

9906 of GGB is not needed Gb interface trace

Use the following commands: ZDDS; ZOEK; ZOEBR:20:FFFFF,0 (replace 20 with the respective PAPUs) ZOEC:20:SR:(OFAM=409,40A,403)AND(NOT(NUM(9906)) ZOEM:20 ZE ZDDS:PAPU,0; ZRS:20,30BE ZOQ:33FF CTRL+C ZE



15


ZOES:20 ZOEG:20 ZWNH:NAME=; 4.1.6. Session management

When encountering problems with session management, collect the following information from the time when the problem has occured:

Message monitoring from processes GTP, GSP, GMG, GIP and

GGB If the problem is not related to flow control the message number

9906 of GGB is not needed Gb interface trace Gn interface trace

Use the following commands: ZDDS; ZOEK; ZOEBR:20:FFFFF,0 (replace 20 with the respective PAPUs) ZOEC:20:SR:(OFAM=408,409,40A,403)AND(NOT(NUM(9906)) ZOEM:20 ZE ZDDS:PAPU,0; ZRS:22,30BE ZOQ:33FF CTRL+C ZE ZOES:20 ZOEG:20 ZWNH:NAME=; 4.1.7. Charging

When encountering charging principle or charging data collection problems, collect to following information from the time when the problem has occured.

GTP and GMG monitorings from the PAPU's related to the subscriber movement.

GCH monitorings from MCHU. Computer logs from MCHU. Alarm history. CDR status interrogation (GHH - Interrogate CDR Status) Sample CDRs from CG. Charging files from MCHU's disk if exist.



16


Collect computer logs from MCHU using the following commands: ZDDE:MCHU:"ZGSC"; ZWNH:NAME=; If problem is related to missing CDRs, inconsistence of datavolumes or time stamps, then several hours of message monitorings should be taken from GMG (from all PAPUs). When encountering charging protocol problems, collect to following information from the time when the problem has occured.

GCH, GIG and GTP monitorings from MCHU. IP traffic monitoring between SGSN and CG. A list of CG IP addresses and statuses (GHI - Interrogate CG

addresses and statuses) Charging release level (EJH - Output SGSN Parameters) Computer logs from MCHU. Alarm history.

Sample CDRs from CG are also needed. 4.1.8. Statistics

When encountering problems with statistics, collect the following information:

GST message monitoring from MCHU Erroneous measurement file, which can be found from MCHU's disk

from SGSTAT directory. Measurement files are called MExxxx.DAT. Measurement file with

the biggest number is the newest one. Copy the measurement file to a disk. Don't get it from NMS.

Alarm history Computer logs from MCHU Active software packet name

Use the following commands for GST message monitoring: ZDDS; ZOEK; ZOEBR:10:FFFFF,0 ZOEC:10:SR:(OFAM=40E) (replace 10 with the working MCHU) ZOEM:10 ZOES:10 ZOEG:10



17


Collect alarm history and computer logs from MCHU using the following commands: ZAHO; ZAHP:::; ZDDE:MCHU:"ZGSC"; ZWNH:NAME=; 4.1.9. Storing and/or transferring statistical data

When encountering problems in storing or transferring statistical data, collect the following information:

Computer logs of the OMU, PAPU, and MCHU The connections of the logical files in OMU Output of VDS status Statuses of I/O devices of OMU

Use the following commands: ZDDE::"ZGSC"; ZIID:,OMU; ZIFO:OMU,SGSTAT:1&&; ZISI:,; ZWNH:NAME=; 4.1.10. Alarms

When encountering problems with alarms, collect the following information:

Alarms active alarms, the alarm history, and blocked alarms Computer log of the unit in question

Use the following commands: ZAHO; ZAHP:::; ZABO; ZDDE:< >:"ZGSC"; unitZWNH:NAME=; If alarm 3040 SGSN Charging Failure exists with 3th field value greater than A (a hexadecimal value), then several hours message monitorings should be taken from GMG (from all PAPUs).



18


4.1.11. OSI

When encountering problems related to OSI, collect the following information:

Alarms active alarms, alarm history, and blocked alarms OSI configuration Computer log of the unit in question

Use the following commands: ZAHO; ZAHP:::; ZABO; ZQTI; ZQSI: ZQLP; ZQLI; ZQGI; ZQEI; ZQDI; ZQCI; ZQBI; ZDDE::"ZGSC"; ZWNH:NAME=; 4.1.12. HW configuration

When encountering problems related to the HW configuration, collect the following information:

Alarms active alarms, alarm history, and blocked alarms Output of HW configuration of the unit where problem occurs Output of the response of the command currently used

Use the following commands: ZAHO; ZAHP:::; ZABO; ZWTI:P:; ZWNH:NAME=; Also check the serial number and interchangeability of the plug-in unit.



19


4.1.13. I/O devices

When encountering problems related to the I/O devices, collect the following information:

Alarms active alarms, alarm history, and blocked alarms Statuses of the I/O devices of the unit concerned Computer log of the unit concerned The connections of the logical files in the unit concerned

Use the following commands: ZAHO; ZAHP:::; ZABO; ZISI:,:"ZGSC"; unitZWNH:NAME=; 4.1.15. CCS7

When encountering problems related to CCS7, collect the following information:

Alarms active alarms, alarm history, and blocked alarms CCS7 network configuration Computer log of the unit in question



20


Use the following commands: ZAHO; ZAHP:::; ZABO; ZNET; ZNRI; ZNSI; ZNHI; ZNGI; ZDDE::"ZGSC"; ZWNH:NAME=; Your local DMS will support you with additional information, if needed.

4.2. GGSN related problems When you have problems considering the GGSN software the following has to be taken into account:

For PDP context related faults ggsntunnel.log is needed For charging related faults ggsntunnel.log and ggsncs.log are needed

For more information, please contact your local DMS.

4.3. LIG related problems

4.3.1. LIG release 1

In case of malfunction in the LIG system is observed, the administrator has to perform immediately the actions specified below. Timing is crucial in order to collect all necessary troubleshooting information because the log files have a predefined lifetime (by default 12 hours) and, therefore, not all information might necessarily be recovered later. For each LIC network element, do the following: a) From the LIC console, execute the following commands:

/opt/licontroller/bin/licTroubleShoot.sh mv /var/admin/lic.tgz /var/admin/lic.tgz

b) Transfer the /var/admin/lic.tgz file to a local FTP server



21


For each LIB network element, do the following: a) From the LIB console, execute the following commands:

/opt/librowser/bin/libTroubleShoot.sh cd /var/browse tar cvzf /var/browse browsedata.tgz mv /var/admin/lib.tgz /var/admin/lib.tgz

b) Transfer the /var/admin/lib.tgz file to a local FTP server. c) Transfer the /var/browse/browsedata.tgz file to a local FTP server For each GGSN in the network perform the following actions: a) tar -cvz -f lie.tgz var/log/* /config /var/tmp b) Transfer the /var/admin/lie.tgz file to a local FTP server For each alarm destination address and directory perform the following actions (if possible):

a) Execute the following command in the alarm destination directory (if using other than Unix computer, use the available archiving and compression tools)

tar -cvz -f li.tgz alarm_* cancel_*

b) Transfer the generated file li.tgz to a local FTP server Attach all the:

lic.tgz lib.tgz lie.tgz li.tgz

files to the troubleshooting note. The contents of /var/log/ directory should also be sent to us. If there is anything in directory /var/tmp/, please gzip it and send it to us. Command to be executed: 'gzip '



22


4.3.2. LIG release 2

All databases are encrypted so you can collect the following information and send to PL. LIC

Log into LIC via telnet by admin user. Go to /opt/licontroller/bin -directory Execute licTroubleshoot script by typing ./licTroubleshoot.sh Go to /var/admin -directory Transfer lic.tgz to FTP server

LIB

Log into LIB via telnet by admin user. Go to /opt/librowser/bin -directory Execute libTroubleshoot script by typing ./libTroubleshoot.sh Go to /var/admin -directory Transfer lib.tgz to FTP server

For more information, please contact your local DMS.

4.4. 3G SGSN related problems At the moment trouble shooting of 3G SGSN is monitoring of different interfaces and checking configuration of SGSN so it needs quite good knowledge of different protocols, procedures and SGSN parameters. Internal debugging methods of SGSN will be implemented later and those will be added to this document later. When you send in problem reports, please take into account the following requirements:

Attach good explanation of the problem case Synchronize time of all related equipment. This way comparing of

different logs is easier.

4.4.1. 3G SGSN install log file

/var/log/sgsn-install log file includes information of package and files which are loaded to different slots. This file includes good information if you have to solve start-up problems.



23


4.4.2. Configuration files

Log into CRP via telnet as user admin. Run the run the following script: /opt/sgsn/bin/sgsn_pr_info This script executes the following tasks: 1. generates a list of all currently running processes and puts it in a file 2. checks the free disk space and writes it to a file 3. dumps the ipsctl database to a file 4. dumps the dmesg output to a file 5. generates a tarball that includes all the files from steps 1-4, /var/log/messages, and the configuration database for the crp and all gplc slots. Attach the file: /var/admin/sgn1_pr_info_*.tgz to the troubleshooting note (a new one is generated every time /opt/sgsn/bin/sgsn_pr_info is run so you might want to remove some old ones to avoid filling the disk if this is done several times). Always run the sgsn_pr_info as soon as possible after the problem occurred and attach the produced file to the problem report. Also if the sgn1_pr_info-xxx.tgz gets too big and you have a problem you can reproduce, the logs can be rotated by running /etc/daily After that, reproduce the problem, run sgsn_pr_info and the resulting sgn1_pr_info-xxx.tgz file will be a lot smaller. Also, the logs are rotated every night at around 2AM so if you have a problem, for instance 17:15 and run sgsn_pr_info at 8:00 the next morning, the script will not catch the logs when the problems have occurred. If this is the case, the instructions should be to do a ls -l /var/log/messages* Have a look at the timestamps and attach the appropriate messages.x.gz (x denotes a number) too. For more information, please contact your local DMS.



24


4.4.3. Using grep to sort out messages

To use grep to sort out log messages from crp's message file. For example, something has happened e.g. at 08:34:48 that interests us. We don't need whole messages file, instead we would look into events happened just before and after the problem situation > grep "Oct 22 08:3" /var/log/messages > info.log One should use common sense on figuring out how to filter out the information that is relevant for problem analysis.

4.4.4. All core dump files

Core dump files are stored in /var/admin and /var/tmp directories. Attach the files: /var/admin/kernel.x.tar.gz /var/admin/wmcore.x.tar.gz /var/tmp/ipm.core-- /var/tmp/ipm -- to the troubleshooting note.

4.5. TA related problems

4.5.1. General

The following info needs to be attached to a TA problem report:

1. itainfo output. 2. If applicable: tcpdump capture file from all interfaces. 3. If applicable: TA subsystem trace (see the next chapters on details). 4. If applicable: ita command output (see the next chapters on details). 5. In case of incorrect behaviour: reference to the document against

which TA conflicts with, including page number. 1-4 are needed from all cluster members. 4.5.2. TA subsystem traces

Each TA subsystem can be set to a debug mode, where the subsystem generates extensive debug information to /var/log/messages.



25


For a suspected fault reports:

1. Enable DEBUG information collection to syslog: #dbset syslog:action:file:/var/log/messages:selector:all.debug t

2. Activate the subsystem trace (next chapters). 3. Repeat the test case. 4. Deactivate the subsystem trace. 5. Disable DEBUG information collection to the syslog:

#dbset syslog:action:file:/var/log/messages:selector:all.debug Attach /var/log/messages to the problem report.

4.5.2.1. TA Kernel Module Trace

One must be VERY careful when activating TA kernel module traces, as the amount of trace can be huge. A short extended trace (including trace from all TA kernel module components) snapshot can be made with the following command: #ita L; sleep 15; ita l Here a 15 second trace snapshot was made. The L option enabled traces and the l option disabled them. It is also possible to do as follows: #ita L ##rem here we just wait #ita l But as said, this may be dangerous. If the component in TA kernel module can be guessed, tracing from that one component can be activated. This is done as below (15 secs sleep is used. If it is not enough, use a longer sleep or use the L and l options as described above)

core component; all userspace/kernel module activities are traced: #ita L 1FFFF; sleep 15; ita l

IP component; all IP packets are traced, a lot of data is typically

generated: #ita L 2FFFF; sleep 15; ita l

SubsDB; traces some subscriber database activities. Not very much

data is generated: #ita L 4FFFF; sleep 60; ita l



26


Charging component; traces all charging related activities, a lot of

data may be generated: #ita L 8FFFF; sleep 60; ita l

Cluster component; traces all cluster related activities, a lot of data

may be generated: #ita L 10FFFF; sleep 60; ita l

RADIUS component; traces all RADIUS packets from the GGSN.

Not very much data is generated unless there is a lot of RADIUS messages in the air: #ita L 20FFFF; sleep 60; ita l

You can combine the components, for example to trace both cluster and

RADIUS: #ita L 30FFFF; sleep 60; ita l

4.5.2.2. ita_chargingd (RADIUS between TA and OSC) tracing

Tracing is enabled from Voyager TA RADIUS (prepaid) settings. Set Error logging to Debug and activate all Tracing checkboxes.

NOTE: Do not forget to set the error logging mode back to what it previously was.

4.5.2.3. Other TA subsystems (ita_httpd, ita_wapd, ita_rtspd, ita_smtpd, ita_ftpd) tracing

These subsystems do not have Voyager page to enabled tracing. The tracing is enabled by sending USR2 signal to the process. These subsystems react to the USR2 signal by flipflopping the debug level. Initially the debug mode is disabled and USR2 signal enables it. The next USR2 signal disables it and so on. The signal must be sent to the parent process whose PID can be found from /var/run directory.

A convenient way is to use the following command: #kill USR2 `cat /var/run/ita_xxxx.pid` Here xxxx is the subsystem in question.



27


For example to enable ita_httpd traces, issue the following command:

#kill USR2 `cat /var/run/ita_httpd.pid`

And to disable ita_httpd traces, issue then the following command: #kill USR2 `cat /var/run/ita_httpd.pid`

4.5.3. Ita commands

Commands ita s and ita S print TA statistics (capital letter S gives more).

To list all subscribers use command ita d To get subscriber data use command ita -G To force intermediate changing updates (CDR generation and RADIUS

interims to OSC) use command ita -F 0 Use command ita x (small letter x) in the master to force

remastering. 4.5.4. Miscellaneous tips and hints

Top utility is installed with TA package (1.3.7 or later) When you use command vmstat m note that TA kernel module

memory allocations are reported to 'Firewall-1'.

4.6. IPSO information The following information is needed when there is a suspected problem in the IPSO platform of 3G SGSN, GGSN, LIG and TA.

4.6.1. Run ipsctl command

Run ipsctl -a command and send the result with the problem report. 4.6.2. Complete configuration file

Run command "ipsoinfo" Log file can be found from /var/admin directory and file name is

"ipsoinfo-routername-date.tar.gz" Attach the file to the problem report



28




29

4.6.3. All core dump files

See Section 4.4.4. All core dump files.

4.7. Documentation When you are reporting a problem in a MPC document, collect the following information from the document:

Document name Document number (for example CAG 12345 or DN994321) Document issue Which MPC release is in question What is wrong in the document or what is missing Did you find the fault in the paper library or in the Nokia Electronic

Documentation (NED)

Fifth issue:Changes made between issues 4 and 3Changes made between issues 5 and 4About this documentScope of a problem reportHardware problemSystem defect or problemDocumentation problemProduct proposal

Problem report fieldsTitle field in Problem ReportDescription field in Problem ReportDelivery field in Problem ReportRelease field in Problem ReportSeverityCriticalMajorMinorInformation Request (IR)

Enhancement or improvement proposal

Information needed in problem reports2G SGSN related problemsSystem restartUnit restartProcess restartBackupMobility managementSession managementChargingStatisticsStoring and/or transferring statistical dataAlarmsOSIHW configurationI/O devicesMMLCCS7

GGSN related problemsLIG related problemsLIG release 1LIG release 2

3G SGSN related problems3G SGSN install log fileConfiguration filesUsing grep to sort out messagesAll core dump files

TA related problemsGeneralTA subsystem tracesTA Kernel Module Traceita_chargingd (RADIUS between TA and OSC) tracingOther TA subsystems (ita_httpd, ita_wapd, ita_rtspd, ita_smtpd, ita_ftpd) tracing

Ita commandsMiscellaneous tips and hints

IPSO informationRun ipsctl commandComplete configuration fileAll core dump files

Documentation

Date post:	08-Nov-2015
Category:	Documents
Upload:	nizar-touaiti
View:	2 times
Download:	1 times

dn0251007x5x0xen

Documents