Date post: | 03-Apr-2015 |
Category: |
Documents |
Upload: | gustavo-villafana |
View: | 219 times |
Download: | 2 times |
Performance Monitoring on AIX
byPeter Lee
IT Specialist
Performance Tuning Process
Indentify Workloads
Define and prioritize goals
Identify the required resoureces
Minimize resource requirements
Control the allocation of resource
Apply additional resources as indicated
Performance Analysis Tools
CPU Memory Subsystem I/O Subsystem Network
Subsystemvmstat, iostat vmstat iostat lsattr
ps lsps vmstat netstatsar svmon lsps nfsstat
gprof/prof/tprof filemon lsattr entstattime/timex bf, bfrpt lsdev netpmonnetpmon lspv/lsls/lsvg ifconfig
stem fileplace iptrace/ipreportsyscalls filemon tcpdumplockstatemstat
performance toolbox
performance toolbox
performance toolbox
performance toolbox
trace, trcrpt, utld trace, trcrpt trace, trcrpt trace, trcrpt
Performance Tuning Tools
CPUs Memory Subsystem I/O Subsystem Network
Subsystem
nice/renice vmtune vmtune no
schedtune chps/mkps chdev nfso
bindprocessor fdpr migratepv chdev
chdev chdev chlv ifconfig
setpri rmss reorgvg
Performance Tuning Flowchart
Actions
Is System CPU-Bound?
Actions
Is System Memory-Bound?
Is System Disk-Bound?
Is System Network-Bound?
Actions
Actions Additonal
Actions
NO
YES
NOYES YES
NO
NOYES
vmstat Commandvmstat Command Reports:
Summary of paging and system events since initialization (-s)Number of forks since initilization (-f)Device interrupts since initilization or per interval (-i)Periodic reports of processes, paging, real memory, and CPUUsage : vmstat [ -ifs ] [drives] [interval] [count]
vmstat 1 10kthr memory page faults cpu----- ----------- ------------------------ ------------ -----------r b avm fre re pi po fr sr cy in sy cs us sy id wa0 0 2514 24166 0 0 0 0 0 0 121 1 18 0 0 99 00 0 2514 24165 0 0 0 0 0 0 126 178 19 1 0 99 00 0 2514 24165 0 0 0 0 0 0 118 60 24 0 0 99 00 0 2514 24165 0 0 0 0 0 0 114 54 20 0 0 99 00 0 2514 24165 0 0 0 0 0 0 121 117 21 0 0 99 00 0 2514 24165 0 0 0 0 0 0 119 54 20 0 1 99 00 0 2514 24165 0 0 0 0 0 0 126 54 22 1 0 99 00 0 2514 24165 0 0 0 0 0 0 127 54 19 0 1 99 00 0 2514 24165 0 0 0 0 0 0 125 54 21 0 0 99 00 0 2514 24165 0 0 0 0 0 0 127 54 18 0 0 99 0
root@bmserver[/tmp]>vmstat -i
priority level type count module(handler)0 0 hardware 0 i_misc_pwr(b0e5c)0 24 hardware 0 i_epow(9d820)0 24 hardware 0 /etc/drivers/hscsidd(18bc578)0 48 hardware 0 i_scu(b0e50)1 2 hardware 66 /etc/drivers/rsdd(196de4c)3 4 hardware 3 /etc/drivers/middd_loadpin(19ce63c)3 5 hardware 1736 /etc/drivers/hscsidd(18bc56c)3 7 hardware 21636015 /etc/drivers/tokdd(19e78fc)3 14 hardware 54189 /etc/drivers/hscsidd(18bc56c)5 62 hardware 117295403 clock(9d670)10 63 hardware 7647 i_softoff(9d628)
vmstat Command
root@bmserver[/tmp]>vmstat -s
210376 total address trans. faults5724 page ins55944 page outs
0 paging space page ins0 paging space page outs0 total reclaims
78419 zero filled pages faults1551 executable filled pages faults
0 pages examined by clock0 revolutions of the clock hand0 pages freed by the clock
8790 backtracks0 lock misses0 free frame waits0 extend XPT waits
3760 pending I/O waits56326 start I/Os56326 iodones
21347782 cpu context switches139338639 device interrupts
0 software interrupts0 traps
1873766 syscalls
vmstat Command
fre : the number of pages of real memory on the free list at the time of the interval sample
re : the number of page reclaims per second observed in the sample interval. If a page fault occurs and this page is currently on the freelist and has not yet been reused, this is considered a reclaim since no new I/O request has to be initiated. This is not to be confused with the term repage which refers to a page which has already incurred a page-fault (the page could be currently on the freelist, filesystem, or in paging space).
pi or po : the number of page ins or page outs per second from/to paging space during the sample interval
fr/sr : the number of pages per second that were freed/examined by the page replacement algorithm during the interval
cy : the number of cycles per second of the clock algorithm
in/sy/cs : the number of device interrupts/system calls/thread context switches per second observed in the interval
CPU : CPU utilization observed in the interval, divided into us-user, sy-system, id-idle, wa-wait
disk xfer the number of disk requests per second observed to the PV
vmstat Command
topas Command
iostat Command Reports:CPU utilization is %user + %system approaching 100%?Terminal I/ODisk I/O Balanced?CD-ROM I/O
Hardware Dependent
Local disk information only
No special authority required
iostat Command
iostat Command
usage: iostat [-t] [-d] [drives] [interval [count]]-t specifies tty/cpu report only-d specifies drive report only-t and -d cannot both be specified
root@bmserver[/tmp]>iostat
tty: tin tout avg-cpu: % user % sys % idle% iowait
0.0 0.5 0.0 0.1 99.80.1
Disks: % tm_act Kbps tps Kb_read Kb_wrtnhdisk0 0.1 0.2 0.0 40264 220277hdisk1 0.0 0.0 0.0 0 0cd0 0.0 0.0 0.0 500 0
sar Command
sar Command
Data TypesAll data types collected; only specified data types reported on
Timing Intervals: the extraction interval can differ from the collection interval
Permissions: need to be in system group
sadc: back end for sar, responsible for the data collection
timex -s: displays sar statistics for the duration of the command executed by timex
sar Command
There are two forms of the sar command: the first form is used to sample, display, and/or save systemstatistics; the second form is used to process and display previously-captured data.
e.gCollection: sar -o filename 60 300 > /dev/null &Extraction: sar -A -f filenameExtraction: sar -u -f filename -s 10:20 -e 14:50Monitoring: sar -c 60 300
sar Command
Data Types:-a file access system routines-b buffer activity-c system calls-k kernel process statistics-m message and semaphore activity-q statistics on run queue and wait queue-r paging statistics-u CPU utilization-v kernel tables: i-node, process and file-w process switching activity-y reports on tty statistics-A Show all of the above-P [processor_id[,...] | ALL] Show individual processor utilization on SMP systems
# sar -u 5 319:28:02 %usr %sys %wio %idle19:28:07 0 6 0 9419:28:12 1 4 0 9519:28:07 7 4 0 90Average 3 4 0 93
# sar -c 1 319:28:25 scall/s sread/s swrit/s fork/s exec/s rchar/s wchar/s19:28:26 134 36 1 0.00 0.00 2691306 151719:28:27 46 34 1 0.00 0.00 2716922 153119:28:28 46 34 1 0.00 0.00 2716922 1531Average 75 35 1 0.00 0.00 2708329 1527
# sar -q 5 319:31:42 runq-sz %runocc swpq-sz %swpocc19:31:47 1.0 100 1.0 10019:31:52 2.0 100 1.0 10019:31:57 1.0 100 1.0 100Average 1.3 95 1.0 95
sar Command
ps Command
ps CommandThe ps command writes the current status of active processes and (if the -m flag is given) associated kernel threads to standard output.A new option to ps in AIX Version 4 shows if a process is bound to a particular CPU. This option is -o THREAD. If you see the -m option as well (-mo THREAD), then you can see the thread or threads that belongs to each process.Usage : ps [-Aaedfklm] [-n namelist] [-F Format] [-o specifier[=header],...] [-p proclist][-G|-g grouplist] [-t termlist] [-U|-u userlist]Usage: ps [aceglnsuvwxU] [t tty] [processnumber]
ps Command
root@bmserver[/tmp]>ps -elF S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
200003 A 0 1 0 0 60 20 2805 252 - 0:00 init240001 A 0 2548 1 0 60 20 8831 68 5a6cf58 - 2:32 syncd40401 A 0 2814 1 0 60 20 1923 276 95378 - 0:00 errdemon240001 A 0 2902 1 0 60 20 921 244 - 0:00 srcmstr240001 A 0 3528 1 0 60 20 d21a 168 54aeccc - 0:13 cron240001 A 0 3694 2902 0 60 20 59eb 608 - 0:00 sendmail240001 A 0 3968 2902 0 60 20 c9f9 984 - 0:42 snmpd240001 A 0 4218 2902 0 60 20 a1f4 172 - 0:00 inetd240001 A 0 4468 2902 0 60 20 79ef 292 - 0:00 portmap240001 A 0 4712 2902 0 60 20 9e1 128 - 0:02 syslogd240001 A 0 4998 2902 0 60 20 2204 468 - 0:00 dpid2240001 A 0 5866 8424 0 60 20 5acb 348 pts/0 0:00 ksh240001 A 0 6054 2902 0 60 20 8a11 68 - 0:00 biod240001 A 0 6326 2902 0 60 20 4208 144 - 0:00 rpc.statd240001 A 0 6598 2902 0 60 20 f21e 68 * - 0:03 rpc.lockd240001 A 0 6858 1 0 60 20 1a23 32 9ec0170 - 0:00 install_as200001 A 0 6932 5866 11 65 20 7acf 188 pts/0 0:00 ps200001 A 0 8424 4218 2 61 20 82d0 268 - 0:00 telnetd
ps Commandroot@bmserver[/tmp]>ps -emo THREAD
USER PID PPID TID ST CP PRI SC WCHAN F TT BND COMMANDroot 1 0 - A 0 60 1 - 200003 - 0 /etc/init
- - - 463 S 0 60 1 - 400 - 0 -root 2548 1 - A 0 60 1 5a6cf58 240001 - 0 /usr/sbin/syncd 60
- - - 3325 S 0 60 1 5a6cf58 400 - 0 -root 2814 1 - A 0 60 1 95378 40401 - 0 /usr/lib/errdemon
- - - 3335 S 0 60 1 95378 400 - 0 -root 2902 1 - A 0 60 1 - 240001 - 0 /usr/sbin/srcmstr
- - - 2399 S 0 60 1 - 8400 - 0 -root 3528 1 - A 0 60 1 54aeccc 240001 - 0 /usr/sbin/cron
- - - 3795 S 0 60 1 54aeccc 400 - 0 -root 3694 2902 - A 0 60 1 - 240001 - 0 sendmail: accepting
connections- - - 4727 S 0 60 1 - 8400 - 0 -
root 3968 2902 - A 0 60 1 - 240001 - 0 /usr/sbin/snmpd- - - 4233 S 0 60 1 - 8400 - 0 -
root 4218 2902 - A 0 60 1 - 240001 - 0 /usr/sbin/inetd- - - 4483 S 0 60 1 - 8400 - 0 -
root 4468 2902 - A 0 60 1 - 240001 - 0 /usr/sbin/portmap- - - 4989 S 0 60 1 - 8400 - 0 -
root 4712 2902 - A 0 60 1 - 240001 - 0 /usr/sbin/syslogd- - - 5233 S 0 60 1 - 8400 - 0 -
root 4998 2902 - A 0 60 1 - 240001 - 0 /usr/sbin/dpid2- - - 5519 S 0 60 1 - 8400 - 0 -
root 5866 8424 - A 0 60 1 - 240001 pts/0 0 -ksh- - - 8185 S 0 60 1 - 400 - 0 -
root 6054 2902 - A 0 60 1 - 240001 - 0 /usr/sbin/biod 6- - - 6833 S 0 60 1 - 400 - 0 -
root 6326 2902 - A 0 60 1 - 240001 - 0 /usr/sbin/rpc.statd- - - 7105 S 0 60 1 - 8400 - 0 -
root 6598 2902 - A 0 60 3 * 240001 - 0 /usr/sbin/rpc.lockd- - - 7377 S 0 60 1 9f64c98 400 - 0 -- - - 7639 S 0 60 1 54aa988 1400 - 0 -- - - 7897 S 0 60 1 2735dc 1400 - 0 -
root 7008 5866 - A 12 66 1 - 200001 pts/0 0 ps -emo THREAD- - - 9071 R 12 66 1 - 0 - 0 -
ps Commandroot@bmserver[/tmp]>ps auxUSER PID %CPU %MEM SZ RSS TTY STAT STIME TIME COMMANDroot 5522 0.0 0.0 8 16 - A Aug 20 0:00 kprocroot 5866 0.0 1.0 348 408 pts/0 A 06:01:57 0:00 -kshroot 6054 0.0 0.0 68 96 - A Aug 20 0:00 /usr/sbin/biod 6root 5264 0.0 0.0 8 12 - A Aug 20 0:01 kprocroot 4468 0.0 1.0 292 360 - A Aug 20 0:00 /usr/sbin/portmaproot 4712 0.0 0.0 128 172 - A Aug 20 0:02 /usr/sbin/syslogdroot 4998 0.0 2.0 468 548 - A Aug 20 0:00 /usr/sbin/dpid2root 7308 0.0 0.0 8 12 - A Aug 30 0:00 kprocroot 6598 0.0 0.0 68 108 - A Aug 20 0:03 /usr/sbin/rpc.lockroot 8424 0.0 1.0 268 324 - A 06:01:57 0:01 telnetdroot 7534 0.0 1.0 188 232 pts/0 A 07:27:18 0:00 ps auxroot 6326 0.0 1.0 144 196 - A Aug 20 0:00 /usr/sbin/rpc.statroot 6858 0.0 0.0 32 44 - A Aug 20 0:00 /usr/sbin/install_root 7024 0.0 1.0 208 252 pts/0 A 07:27:18 0:00 moreroot 1032 0.0 0.0 40 48 - A Aug 20 2:25 kprocroot 1896 0.0 0.0 8 12 - A Aug 20 0:00 kprocroot 2228 0.0 0.0 8 12 - A Aug 20 0:00 kprocroot 774 0.0 0.0 8 12 - A Aug 20 0:25 kprocroot 0 0.0 0.0 4 8 - A Aug 20 2:51 swapperroot 1 0.0 1.0 252 304 - A Aug 20 0:00 /etc/initroot 516 99.9 0.0 0 4 - A Aug 20 19067:57 kproc
emstat Command
Instruction Emulation can be determined
Different processors have different instruction sets
PowerPC architecture deletes about 35 POWER instruc-tions
To maintain binary compatibility, deleted instructions are emulated
Instruction emulation increases the pathlength for the instruction
Emulation can be avoided by compiling applications using Common Mode option
emstat Command
# /usr/samples/kernel/emstat 1 7emstat total count emstat interval count
21951937 2195193721967488 1555121974877 738921994354 1947722007329 1297522018171 1084222018171 0
New tool to detect instruction emulation - shipped with bos.adt.samples
Usage : /usr/samples/kernel/emstat [interval] [count]
Scheduling Policies
SCHED_FIFONon-preemptive schedulingFixed piority
SCHED_RRRound-Robin schedulingFixed piorityRun until interrupted or end of time slice
SCHED_OTHERDefault scheduling policySame as SCHED_RR but applies to threads with non-fixed pioritiesPiority degrades with CPU Usage
Process and Thread Piority
Child Process /Threads
Real-Time Process Piorities
Parent Process
User Process Piorities
fork()
Inherited Process Piority
Initial PRI = NICE + 40pe -el (to view values)ps -emo THREAD (to view threads)
Process Piorities
setp
ri( )
nice
reni
ce
High = 0
Low = 127
40
nice/renice Commands
Initial priority of non-fixed priority thread is: 40 + NICE where NICE has a default value of 20. NICE can range from 0-40.
Nice value can be set at thread creation time by using the nice command. Syntax of nice:
usage: nice [-n increment] commandusage: nice [-increment] command
Nice value can be changed after thread has already been created using the renice command. Syntax of renice:
usage: renice [-n increment] [ -g| -p| -u] ID...Nice values are reduced by using negative increment values and increased by using positive increment values. Reducing nice value makespriority more favored. Increasing nice value makes priority less favored.
schedtune Command
Modifies/Disables Memory Load Control
Modifies CPU scheduling algorithms
Modifies CPU time slices
Restores defaults
Must repeat at every boot
schedtune Command
Change the Scheduling policy from default to Round-Robin using the following command:
/usr/samples/kernel/schedtune -r 0 -d 0
Each timeslice is in units of clock ticks (10ms). Default timeslice is 1 clock tick and is tunable using
/usr/samples/kernel/schedtune -t <# of ticks>
# /usr/samples/kernel/schedtune
THRASH SUSP FORK SCHED-h -p -m -w -e -f -d -r -t -sSYS PROC MULTI WAIT GRACE TICKS SCHED_D SCHED_R TIMESLICE MAXSPIN6 4 2 1 2 10 16 16 1 1
bindprocessor Process [ ProcessorNum ] | -q | -u Process
Purpose
Binds or unbinds the kernel threads of a process to a processor.
bindprocessor Command
svmon Command
svmon Command is an AIX Specific analysis tool for virtual memory
Purpose:Captures a snapshot of the current state of memory.The display of information can be analyzed using four different reports:global [ -G] process [-P]segment [-S]detailed segment [-D]
In AIX Version 4, svmon is packaged as part of perfagent.tools fileset.
svmon Command
usage: svmon[ -G ][ -P[n|s|a] [ pid1 ... pidN ] ][ -P[n|s|a][u|p|g|r] [ count ] ][ -S sid1 ... sidN ][ -S{[n|s|a][u|p|g|r]} [ count ] ][ -D sid1 ... sidN ][ -i interval [ numintervals ] ][ -r ]
# svmon -G -i 5 3m e m o r y i n u s e p i n p g s p a c e
size inuse free pin work pers clnt work pers clnt size inuse49152 48134 1018 2613 10674 37460 0 2613 0 0 98304 810449152 48139 1013 2613 10678 37461 0 2613 0 0 98304 810549152 48139 1013 2613 10678 37461 0 2613 0 0 98304 8105
svmon Command1. To print out global statistics:
$ svmon -G
2. To print out the memory usage statistics for all processes (limit statistics to non-system segments):
$ svmon -Pn
3. To print out the memory usage statistics for processes 6746 and 10078 (includestatistics on all segments):
$ svmon -Pa 6746 10078
4. To sort system segments by the number of pages in real memory, and print outthe top 10 segments of the resulting list:
$ svmon -Ssu 10
5. To print detailed memory usage statistics on segment 340d:$ svmon -D 340d
6. To print out the global statistics, repeated 10 times at 30 second intervals:$ svmon -G -i 30 10
rmss Command
Reduced Memory System Simulator
Simulate RISC System/6000 with various size memories
Run an application over a range of memory sizes
rmss is packaged as part of perfagent.tools
usage: rmss [-s startmemsize] [-f finalmemsize] [-d deltamemsize][-n numiterations] [-o outputfile] command
rmss -c memsizermss -rrmss -p
rmss Command
To change memory size# rmss -c 32Simulated memory size changed to 32 Mb.
To display the current memory size:# rmss -pSimulated memory size is 32 Mb.
To reset the memory size to the real size:rmss -rSimulated memory size changed to 192 Mb.
To measure the performance of the command cc -O foo.c on memory size 32, 24, 16, 8MB, run and measure the command once at each memory size and write the report to the file cc.rmss.out
rmss -s 32 -f 8 -d 8 -n 1 -o cc.rmss.out cc -O foo.c
filemon Command
The filemon tool collects and presents trace data on the various layers of file system utilization including the logical file system, virtual memory segments, LVM, and physical disk layers. Data can be collected on all the layers, or layers can be specified with the -O layer option. The default is to collect data on the VM, LVM, and physical layers.
filemon will only collect data for those files opened after filemon was started unless you specify the -u flag.
Normally, only the top 20 logical files and segments are reported unless the -v (verbose) flag is used.
filemon Command
Usage: filemon [-i file] [-o file] [-d] [-v] [-u] [-O opt]-i file: open input file (default is real-time trace)-o file: open output file (default is stdout)-d: deferred trace (until ‘ trcon’ )-T num: set trace kernel buf sz (default 32000 bytes)-P: pin monitor process in memory-v: verbose mode (print extra details)-u: print unnamed file activity via pid-O opt: other monitor-specific options
valid -O options: lf,vm,lv,pv,alllf: monitor logical file I/Ovm: monitor virtual memory I/Olv: monitor logical volume I/Opv: monitor physical volume I/Oall: short for lf,vm,lv,pv
(default is: vm,lv,pv)
filemon Command
# filemon -O lv,pv -o fmon.out# dd if=/dev/rhdisk1 of=/dev/null bs=512 count=1024# cp /unix /tstc/junk# trcstop# more fmon.outFri Aug 2 15:11:21 1996System: AIX magic Node: 4 Machine: 000001455C005.450 secs in measured intervalCpu utilization: 72.6%Most Active Logical Volumes------------------------------------------------------------------------util #rblk #wblk KB/s volume description
------------------------------------------------------------------------0.36 0 1588 145.7 /dev/lv04 /tstc Compressed: Frag_Sz.=10240.00 0 16 1.5 /dev/loglv00 jfslog
Most Active Physical Volumes------------------------------------------------------------------------util #rblk #wblk KB/s volume description
------------------------------------------------------------------------0.75 1024 1604 241.1 /dev/hdisk1
netstat Command
netstat CommandShows state of interfacesShows interface statisticsShows routing statisticsShows per-protocol statisticsShows network device driver statisticsShows network memory usage statistics
netstat Commandroot@bmserver[/tmp]>netstat -iName Mtu Network Address Ipkts Ierrs Opkts Oerrs Colllo0 16896 <Link> 473 0 473 0 0lo0 16896 127 loopback 473 0 473 0 0tr0 1492 <Link>10.0.5a.c9.62.a4 10832411 0 22191 0 0tr0 1492 9.184.64 bmserver 10832411 0 22191 0 0
root@bmserver[/tmp]>netstat -I tr0 1input (tr0) output input (Total) output
packets errs packets errs colls packets errs packets errs colls10832898 0 22267 0 0 10833371 0 22740 0 0
3 0 1 0 0 3 0 1 0 07 0 1 0 0 7 0 1 0 04 0 1 0 0 4 0 1 0 05 0 1 0 0 5 0 1 0 010 0 1 0 0 10 0 1 0 07 0 1 0 0 7 0 1 0 06 0 1 0 0 6 0 1 0 011 0 1 0 0 11 0 1 0 010 0 1 0 0 10 0 1 0 09 0 1 0 0 9 0 1 0 04 0 1 0 0 4 0 1 0 09 0 1 0 0 9 0 1 0 07 0 1 0 0 7 0 1 0 07 0 1 0 0 7 0 1 0 0
netstat Commandroot@bmserver[/tmp]>netstat -m
34 mbufs in use:16 mbuf cluster pages in use72 Kbytes allocated to mbufs0 requests for mbufs denied0 calls to protocol drain routines
Kernel malloc statistics:
******* CPU 0 *******By size inuse calls failed free hiwat freed32 108 115 0 20 640 064 122 765 0 70 320 0128 111 1666 0 17 160 0256 134 21771388 0 42 384 0512 37 2470 0 11 40 01024 1 481 0 3 20 02048 0 773 0 6 10 04096 21 1510233 0 6 120 08192 0 66 0 2 10 016384 1 6 0 19 24 7
By type inuse calls failed memuse memmax mapbmbuf 34 21747482 0 8704 17920 0mcluster 16 1512910 0 65536 95744 0socket 111 698 0 18048 19008 0pcb 78 427 0 12032 12864 0routetbl 8 13 0 1312 2080 0ifaddr 5 5 0 576 576 0mblk 30 1198 0 6272 6528 0mblkdata 3 1070 0 12288 22528 0strhead 13 57 0 3776 4800 0strqueue 21 132 0 10752 13312 0strmodsw 21 21 0 1344 1344 0strosr 0 23261 0 0 768 0strsyncq 29 388 0 2944 3648 0streams 148 275 0 16000 17280 0kernel table 14 15 0 45920 46432 0temp 5 12 0 5312 9920 0
Streams mblk statistic failures:0 high priority mblk failures0 medium priority mblk failures0 low priority mblk failures
netstat Commandroot@bmserver[/tmp]>netstat -D
Source Ipkts Opkts Idrops Odrops-------------------------------------------------------------------------------ent_dev0 0 0 0 0tok_dev0 21755956 24738 58384 0
---------------------------------------------------------------Devices Total 21755956 24738 58384 0-------------------------------------------------------------------------------ent_dd0 0 0 0 0tok_dd0 21697572 24738 0 0
---------------------------------------------------------------Drivers Total 21697572 24738 0 0-------------------------------------------------------------------------------ent_dmx0 0 N/A 0 N/Atok_dmx0 10836721 N/A 10860851 N/A
---------------------------------------------------------------Demuxer Total 10836721 N/A 10860851 N/A-------------------------------------------------------------------------------IP 5171634 2654588 18061 18061TCP 26292 21120 0 0UDP 2628010 473 2 2370
---------------------------------------------------------------Protocols Total 7825936 2676181 18063 20431-------------------------------------------------------------------------------lo_if0 473 473 0 0tr_if0 10834304 22321 0 7
---------------------------------------------------------------Net IF Total 10834777 22794 0 7-------------------------------------------------------------------------------
---------------------------------------------------------------NFS/RPC Total N/A 12 0 0-------------------------------------------------------------------------------(Note: N/A -> Not Applicable)
root@bmserver[/tmp]>netstat -sip:
1079372 total packets received0 bad header checksums0 with size smaller than minimum0 with data size < data length0 with header length < data size0 with data length < header length0 with bad options0 with incorrect version number0 fragments received0 fragments dropped (dup or out of space)0 fragments dropped after timeout0 packets reassembled ok266328 packets for this host13 packets for unknown/unsupported protocol0 packets forwarded14007 packets not forwardable0 redirects sent14202 packets sent from this host0 packets sent with fabricated ip header0 output packets dropped due to no bufs, etc.0 output packets discarded due to no route0 output datagrams fragmented0 fragments created0 datagrams that can't be fragmented799024 IP Multicast packets dropped due to no receiver37 successful path MTU discovery cycles8 path MTU rediscovery cycles attempted0 path MTU discovery no-response estimates1 path MTU discovery response timeout0 path MTU discovery decreases detected39 path MTU discovery packets sent0 path MTU discovery memory allocation failures0 ipintrq overflows0 with illegal source0 packets processed by threads0 packets dropped by threads0 packets dropped due to the full socket receive buffer0 dead gateway detection packets sent0 dead gateway detection packet allocation failures
0 dead gateway detection gateway allocation failures
icmp:4559 calls to icmp_error0 errors not generated because old message was icmpOutput histogram:
echo reply: 6destination unreachable: 25
0 messages with bad code fields0 messages < minimum length0 bad checksums0 messages with bad lengthInput histogram:
echo reply: 37destination unreachable: 13echo: 6
6 message responses generatedigmp:
1 message received0 messages received with too few bytes0 messages received with bad checksum0 membership queries received0 membership queries received with invalid field(s)1 membership report received1 membership report received with invalid field(s)0 membership reports received for groups to which we belong4 membership reports sent
netstat Command
tcp:13593 packets sent
11369 data packets (4102411 bytes)89 data packets (108779 bytes) retransmitted1842 ack-only packets (1062 delayed)0 URG only packets0 window probe packets1 window update packet292 control packets
15930 packets received11390 acks (for 4104523 bytes)217 duplicate acks0 acks for unsent data9910 packets (154901 bytes) received in-sequence195 completely duplicate packets (592 bytes)0 old duplicate packets5 packets with some dup. data (1572 bytes duped)307 out-of-order packets (4588 bytes)0 packets (0 bytes) of data after window0 window probes262 window update packets0 packets received after close0 packets with bad hardware assisted checksum2 discarded for bad checksums0 discarded for bad header offset fields0 discarded because packet too short0 discarded by listeners579 ack packet headers correctly predicted3412 data packet headers correctly predicted
10 connection requests311 connection accepts316 connections established (including accepts)318 connections closed (including 5 drops)0 connections with ECN capability0 times responded to ECN5 embryonic connections dropped11180 segments updated rtt (of 11262 attempts)0 segments with congestion window reduced bit set0 segments with congestion experienced bit set0 resends due to path MTU discovery
netstat Command0 path MTU discovery terminations due to retransmits86 retransmit timeouts
0 connections dropped by rexmit timeout2 fast retransmits
0 when congestion window less than 4 segments0 newreno retransmits0 times avoided false fast retransmits6 persist timeouts
0 connections dropped due to persist timeout149 keepalive timeouts
149 keepalive probes sent0 connections dropped by keepalive
0 times SACK blocks array is extended0 times SACK holes array is extended0 packets dropped due to memory allocation failure0 connections in timewait reused0 delayed ACKs for SYN0 delayed ACKs for FIN0 send_and_disconnects0 spliced connections0 spliced connections closed0 spliced connections reset0 spliced connections timeout0 spliced connections persist timeout0 spliced connections keepalive timeout
udp:250354 datagrams received0 incomplete headers0 bad data length fields0 bad checksums4559 dropped due to no socket245436 broadcast/multicast datagrams dropped due to no
socket0 socket buffer overflows359 delivered381 datagrams output
entstat Command
The entstat command displays the statistics gathered by the specified Ethernet device driver.
This command is also invoked when the netstat command is run with the -v flag. The netstat command does not issue any entstat command flags.
usage: entstat [-drt] <device name>-d Displays all the statistics, including the device-specific statistics. -r Resets all the statistics back to their initial values. This flag can only be issued by privileged users. -t Toggles debug trace in some device drivers.
root@bmserver[/tmp]>entstat -d ent0-------------------------------------------------------------ETHERNET STATISTICS (ent0) :Device Type: IBM 10/100 Mbps Ethernet PCI Adapter (23100020)Hardware Address: 00:04:ac:17:74:a0Elapsed Time: 4 days 18 hours 8 minutes 43 seconds
Transmit Statistics: ReceiveStatistics:---------------------------------------Packets: 0 Packets: 0Bytes: 0 Bytes: 0Interrupts: 0 Interrupts: 0Transmit Errors: 6 Receive Errors: 0Packets Dropped: 1 Packets Dropped:0
Bad Packets: 0Max Packets on S/W Transmit Queue: 2S/W Transmit Queue Overflow: 0Current S/W+H/W Transmit Queue Length: 0
Broadcast Packets: 0 BroadcastPackets: 0Multicast Packets: 0 MulticastPackets: 0No Carrier Sense: 6 CRC Errors: 0DMA Underrun: 0 DMA Overrun: 0Lost CTS Errors: 0 Alignment Errors:0Max Collision Errors: 0 No ResourceErrors: 0Late Collision Errors: 0 Receive CollisionErrors: 0Deferred: 0 Packet Too ShortErrors: 0SQE Test: 0 Packet Too LongErrors: 0Timeout Errors: 0 Packets Discardedby Adapter: 0Single Collision Count: 0 Receiver StartCount: 0Multiple Collision Count: 0Current HW Transmit Queue Length: 0
entstat CommandGeneral Statistics:-------------------No mbuf Errors: 0Adapter Reset Count: 0Adapter Data Rate: 200Driver Flags: Up Broadcast Running
Simplex AlternateAddress 64BitSupportPrivateSegment DataRateSet
IBM 10/100 Mbps Ethernet PCI Adapter Specific Statistics:------------------------------------------------Chip Version: 26RJ45 Port Link Status : downMedia Speed Selected: Auto negotiationMedia Speed Running: UnknownReceive Pool Buffer Size: 384Free Receive Pool Buffers: 128No Receive Pool Buffer Errors: 0Inter Packet Gap: 96Adapter Restarts due to IOCTL commands: 1Packets with Transmit collisions:1 collisions: 0 6 collisions: 0 11collisions: 02 collisions: 0 7 collisions: 0 12collisions: 03 collisions: 0 8 collisions: 0 13collisions: 04 collisions: 0 9 collisions: 0 14collisions: 05 collisions: 0 10 collisions: 0 15collisions: 0Excessive deferral errors: 0x0
NFS Tuning ConceptsNFS Version 3 or Version 2
(Version 3 is recommended)
UDP or TCP protocol (NOTE: TCP is recommended)
Increase/decrease number of biods/nfsds
nfs socket size and sb_max
adapter transmit queues
Filesystem read/write buffer sizes
RAM for caching file pages (vmtune -P)
I/O tuning (multiple disks, separate log disk)
Tuning with nfso
nfsstat Command
Displays statistics on:Network File System (NFS)Remote Procedure Calls (RPC)
Syntax - /usr/sbin/nfsstat:-c client information-s server information-n NFS information only-r RPC information only-z reset statistics (root only)-m mount statisticsdefault: -csnr
nfsstat Command
root@bmserver[/tmp]>nfsstat -rc
Client rpc:Connection oriented:calls badcalls badxids timeouts newcreds badverfs timers1 0 0 0 0 0 0nomem cantconn interrupts0 0 0Connectionless:calls badcalls retrans badxids timeouts newcreds badverfs1448 0 12 0 12 0 0timers nomem cantsend22 0 0
nfsstat Command
# nfsstat -m/nfs/retain/pmrs from /nfs/retain/pmrs:ciaFlags:vers=2,proto=udp,auth=unix,soft,intr,dynamic,rsize=8192,wsize=8192,retrans=5Lookups: srtt=2 (5ms), dev=2 (10ms), cur=1 (20ms)All: srtt=2 (5ms), dev=2400 (12000ms), cur=600 (12000ms)
/nfs/retain/bin from /nfs/retain/bin:ciaFlags:vers=2,proto=udp,auth=unix,hard,intr,dynamic,rsize=8192,wsize=8192,retrans=5Lookups: srtt=14 (35ms), dev=6 (30ms), cur=4 (80ms)Reads: srtt=18 (45ms), dev=5 (25ms), cur=4 (80ms)All: srtt=15 (37ms), dev=7 (35ms), cur=5 (100ms)
/nfs/cust from /nfs/cust:zenFlags: vers=3,proto=tcp,auth=unix,soft,intr,link,sym-link,rsize=32768,wsize=32768,retrans=5All: srtt=0 (0ms), dev=2400 (12000ms), cur=600 (12000ms)
nfsstat Command# netstat -p udp
udp:27136572 datagrams received0 incomplete headers0 bad data length fields0 bad checksums20 dropped due to no socket8 broadcast/multicast datagrams dropped due to no socket39476 socket buffer overflows27097068 delivered177636 datagrams output
socket buffer overflows = Too few nfsd daemons and/or too small nfs_socketsize
Increase the number of nfsd daemons (NFS V2)
Increase value of nfs_socketsize and sb_max
nfso Command
nfso Command# nfso -aportcheck= 0udpchecksum= 1nfs_socketsize= 60000nfs_tcp_socketsize= 60000nfs_setattr_error= 0nfs_gather_threshold= 4096nfs_repeat_messages= 0nfs_udp_duplicate_cache_size= 0nfs_tcp_duplicate_cache_size= 0nfs_server_base_priority= 0nfs_dynamic_retrans= 1nfs_iopace_pages= 0nfs_max_connections= 0nfs_max_threads= 8nfs_use_reserved_ports= 0nfs_device_specific_bufs= 1nfs_server_clread= 1nfs_rfc1323= 0nfs_max_write_size= 0nfs_max_read_size= 0nfs_allow_all_signals= 0