[Linux] Linux Performance Tuning

CPU Tools
1 – top
2 – vmstat
3 – ps aux
4 – mpstat -P all
5 – sar -u
6 – iostat
7 – oprofile
8 – gnome-system-monitor
9 – KDE-monitor
10 – /proc

 

Memory Tools
1 – top
2 – vmstat -s
3 – ps aur
4 – ipcs
5 – sar -r -B -W
6 – free
7 – oprofile
8 – gnome-system-monitor
9 – KDE-monitor
10 – /proc

 

Process Tools
1 – top
2 – ps -o pmem
3 – gprof
4 – strace,ltrace
5 – sar

 

Disk Tools
1 – iostat -x
2 – vmstat -D
3 – sar -DEV #
4 – nfsstat
5 – NEED MORE!

 

 

# numastat

 

 

/proc
● meminfo
● slabinfo
● cpuinfo
● pid<#>/maps
● vmstat(RHEL4 & RHEL5)
● zoneinfo(RHEL5)

● sysrqtrigger

 

 

 

cat /proc/vmstat
nr_anon_pages 98893
nr_mapped 20715
nr_file_pages 120855
nr_slab 23060
nr_page_table_pages 5971
nr_dirty 21
nr_writeback 0
nr_unstable 0
nr_bounce 0
numa_hit 996729666
numa_miss 0
numa_foreign 0
numa_interleave 87657
numa_local 996729666
numa_other 0
pgpgin 2577307
pgpgout 106131928
pswpin 0
pswpout 34
pgalloc_dma 198908
pgalloc_dma32 997707549
pgalloc_normal 0
pgalloc_high 0
pgfree 997909734
pgactivate 1313196
pgdeactivate 470908
pgfault 2971972147
pgmajfault 8047.
CONTINUED...
pgrefill_dma 18338
pgrefill_dma32 1353451
pgrefill_normal 0
pgrefill_high 0
pgsteal_dma 0
pgsteal_dma32 0
pgsteal_normal 0
pgsteal_high 0
pgscan_kswapd_dma 7235
pgscan_kswapd_dma32 417984
pgscan_kswapd_normal 0
pgscan_kswapd_high 0
pgscan_direct_dma 12
pgscan_direct_dma32 1984
pgscan_direct_normal 0
pgscan_direct_high 0
pginodesteal 166
slabs_scanned 1072512
kswapd_steal 410973
kswapd_inodesteal 61305
pageoutrun 7752
allocstall 29
pgrotated 73

 

 

Alt Sysrq T
bash R current 0 1609 1606
(NOTLB)
Call Trace: [<c02a1897>] snprintf [kernel] 0x27 (0xdb3c5e90)
[<c01294b3>] call_console_drivers [kernel] 0x63 (0xdb3c5eb4)
[<c01297e3>] printk [kernel] 0x153 (0xdb3c5eec)
[<c01297e3>] printk [kernel] 0x153 (0xdb3c5f00)
[<c010c289>] show_trace [kernel] 0xd9 (0xdb3c5f0c)
[<c010c289>] show_trace [kernel] 0xd9 (0xdb3c5f14)
[<c0125992>] show_state [kernel] 0x62 (0xdb3c5f24)
[<c01cfb1a>] __handle_sysrq_nolock [kernel] 0x7a (0xdb3c5f38)
[<c01cfa7d>] handle_sysrq [kernel] 0x5d (0xdb3c5f58)
[<c0198f43>] write_sysrq_trigger [kernel] 0x53 (0xdb3c5f7c)
[<c01645b7>] sys_write [kernel] 0x97 (0xdb3c5f94)
* logged in /var/log/messages

 

 

Alt Sysrq W and P

SysRq : Show CPUs
CPU0:
ffffffff8047ef48 0000000000000000 ffffffff80437f10 ffffffff8019378b
0000000000000000 0000000000000000 0000000000000000 ffffffff801937ba
ffffffff8019378b ffffffff80022b27 ffffffff800551bf 0000000000090000
Call Trace:
[<ffffffff80069572>] show_trace+0x34/0x47
[<ffffffff80069675>] _show_stack+0xd9/0xe8
[<ffffffff801937ba>] showacpu+0x2f/0x3b
[<ffffffff80022b27>] smp_call_function_interrupt+0x57/0x75
[<ffffffff8005bf16>] call_function_interrupt+0x66/0x6c
[<ffffffff8002fcc2>] unix_poll+0x0/0x96
[<ffffffff800551f5>] mwait_idle+0x36/0x4a
[<ffffffff80047205>] cpu_idle+0x95/0xb8
[<ffffffff8044181f>] start_kernel+0x225/0x22a
[<ffffffff8044125b>] _sinittext+0x25b/0x262

 

 

oprofile – builtin to RHEL4 & 5 (smp)
 opcontrol – on/off data
● start    start collection
● stop    stop collection
● dump   output to disk
● event=:name:count

Example:
# opcontrol –start
# /bin/time test1 &
# sleep 60
# opcontrol –stop
# opcontrol dump

opreport – analyze profile
● r    reverse order sort
● t    [percentage] theshold to view
● f    /path/filename
● d    details


opannotate
● s    /path/source
● a    /path/assembly

 

 

Performance Tuning – (RHEL4 and RHEL5)

/proc/sys/vm/swappiness
/proc/sys/vm/min_free_kbytes
/proc/sys/vm/dirty_ratio
/proc/sys/vm/dirty_background_ratio
/proc/sys/vm/pagecache

 

 

Controls how aggressively the system reclaims “mapped” memory:
● Anonymous memory swapping
● Mapped file pages – writing if dirty and freeing
● System V shared memory swapping
 Decreasing: more aggressive reclaiming of unmapped pagecache memory
 Increasing: more aggressive swapping of mapped memory

RHEL4 /proc/sys/vm/swappiness

Sybase server with /proc/sys/vm/swappiness set to 60(default)
procs memoryswapiosystemcpur
b swpd free buff cache si so bi bo in cs us sy id wa
5 1 643644 26788 3544 32341788 880 120 4044 7496 1302 20846 25 34 25 16

Sybase server with /proc/sys/vm/swappiness set to 10
procs memoryswapiosystemcpur
b swpd free buff cache si so bi bo in cs us sy id wa
8 3 0 24228 6724 32280696 0 0 23888 63776 1286 20020 24 38 13 26

 

 

 

Directly controls the page reclaim watermarks in KB
RHEL4&5 /proc/sys/vm/min_free_kbytes

# echo 1024 > /proc/sys/vm/min_free_kbytes
Node
0 DMA free:4420kB min:8kB low:8kB high:12kB
Node 0 DMA32 free:14456kB min:1012kB low:1264kB high:1516kB
echo
2048 > /proc/sys/vm/min_free_kbytes
Node
0 DMA free:4420kB min:20kB low:24kB high:28kB
Node 0 DMA32 free:14456kB min:2024kB low:2528kB high:3036kB

 

Memory reclaim Watermarks min_free_kbytes

Pages High – kswapd sleeps above High
                    kswapd reclaims memory

Pages Low – kswapd wakesup at Low
                   kswapd reclaims memory

Pages Min – all memory allocators reclaim at Min
                  user processes/kswapd reclaim memory

 

 

RHEL4&5 /proc/sys/vm/dirty_ratio

Absolute limit to percentage of dirty pagecache memory
● Default is 40%
● Lower means less dirty pagecache and smaller IO streams
● Higher means more dirty pagecache and larger IO streams

 

 

 

RHEL4&5 /proc/sys/vm/dirty_background_ratio

Controls when dirty pagecache memory starts getting written.
● Default is 10%
● Lower
● pdflush starts earlier
● less dirty pagecache and smaller IO streams
● Higher
● pdflush starts later
● more dirty pagecache and larger IO streams

 

- 100% of pagecache RAM dirty
                                   pdflushd and write()'ng processes write dirty buffers
- dirty_ratio(40% of RAM dirty) – processes start synchronous writes
                                  pdflushd writes dirty buffers in background
- dirty_background_ratio(10% of RAM dirty) – wakeup pdflushd
                                  do_nothing
- 0% of pagecache RAM dirty

 

 

RHEL4&5 /proc/sys/vm/pagecache

Controls when pagecache memory is deactivated.
Default is 100%
Lower
       ● Prevents swapping out anonymous memory
Higher
      ● Favors pagecache pages
      ● Disabled at 100%

 

 

 NUMAstat to display system NUMA characteristics on a numasystem






NUMActl to control process and memory”

RHEL4&5 NUMAstat and NUMActl
[root@perf5 ~]# numastat
node3 node2 node1 node0
numa_hit 72684 82215 157244 325444
numa_miss 0 0 0 0
numa_foreign 0 0 0 0
interleave_hit 2668 2431 2763 2699
local_node 67306 77456 152115 324733
other_node 5378 4759 5129 711
numactl [ interleave nodes ] [ preferred node ] [ membind nodes ] [ cpubind nodes ] [ localalloc] command {arguments ...}

 

● App < memory single NUMA zone
Numactl use –cpubind cpus within same socket
● App > memory of a single NUMA zone
Numactl –interleave XY and –cpubind XY

 

 

EXAMPLES
numactl --interleave=all

bigdatabase arguments Run big database with its memory interleaved on all CPUs.

numactl --cpubind=0 --membind=0,1

process Run process on node 0 with memory allocated on node 0 and 1.

numactl --preferred=1 --numactl show
Set preferred node 1 and show the resulting state.

numactl --interleave=all --shmkeyfile /tmp/shmkey

Interleave all of the sysv shared memory regiion specified by /tmp/shmkey over all nodes.

numactl --offset=1G --length=1G --membind=1 --file /dev/shm/A --touch
Bind the second gigabyte in the tmpfs file /dev/shm/A to node 1.

numactl --localalloc /dev/shm/file

Reset the policy for the shared memory file file to the default localalloc policy.

 

 

RHEL5.2 CPUspeed and performance:
 Enabled = governor set to “ondemand”
 Looks at cpu usage to regulate power
     - Within 35% of performance for cpu loads
     - IO loads can keep cpu stepped down 15-30%

Supported in RHEL5.2 virtualization
To turn off – else may leave cpu’s in reduced step
     - If its not using performance, then:
        # echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
       Then check to see if it stuck:
       # cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
      Check /proc/cpuinfo to make sure your seeing the expected CPU freq.

Proceed to “normal” service disable
    - Service cpuspeed stop
    - Chkconfig cpuspeed off

 

 

 

The Translation Lookaside Buffer (TLB) is a small CPU cache of recently used virtual to physical address mappings
TLB misses are extremely expensive on today's very fast, pipelined CPUs
Large memory applications can incur high TLB miss rates
HugeTLBs permit memory to be managed in very large segments
● E.G. Itanium®:
    Standard page: 16KB  
    Default huge page: 256MB
   16000:1 difference

● File system mapping interface

● Ideal for databases
    E.G. TLB can fully map a 32GB Oracle SGA

 

 

JVM Tuning
 Eliminate swapping
      ● Lower swappiness to 10%(or lower if necessary).
 Promote pagecache reclaiming
      ● Lower dirty_background_ratio to 10%
      ● Lower dirty_ratio if necessary
 Promote inode cache reclaiming
      ● Lower vfs_cache_pressure

 

 

General Performance Tuning Guidelines
 Use hugepages whenever possible.
 Minimize swapping.
 Maximize pagecache reclaiming
 Place swap partition(s) on quite device(s).
 Direct IO if possible.
 Beware of turning NUMA off.

 

 

Benchmark Tuning
 Use Hugepages.
 Dont overcommit memory
 If memory must be over committed
      ● Eliminate all swapping.
      ● Maximize pagecache reclaiming
      ● Place swap partition(s) on separate device(s).
 Use Direct IO
 Dont turn NUMA off.