[AIX] AIX 10Gbe Recomended Performance Set-up

Daniel Martin-Corben | June 26 2013 | Comment (1) | Visits (10117) 

 

Out of the box 10G Ethernet is not tuned at all, which can result in total system throughput only slightly better than 1G. To max out 10G's capabilities, you need to make some adjustments.  I hope that this blog will give you a number of tips to tuning your system for the best 10Gbe performance.

Bring down the adapter -

      # ifconfig en1 down detach

Set jumbo frames -

      # chdev -l ent1 -a jumbo_frames=yes

 

NOTE - The transmission speed of Virtual Ethernet adapters is in the range of 1-3 Gigabits per second, depending on the transmission (MTU) size. The Virtual Ethernet Adapter supports, as Gigabit (Gb) Ethernet, Standard MTU-Sizes of 1500 Byte and Jumbo frames with 9000 Byte. Additionally for Gb Ethernet, the MTU-Size of 65280 Bytes is also supported in Virtual Ethernet. So, the MTU of 65280 Bytes can be only used inside a Virtual Ethernet ONLY.

Set the MTU -

      # chdev -l en1 -a mtu=9000

 

The final tuning step is to enable Ethernet flow control throughout your network. Flow control prevents retransmissions, and the resulting network congestion, by throttling transmissions at their source when any buffer on the path between two endpoints approaches overflow. For AIX, use the following command, though your also need to speak to your networking team to see if its been enable on the related switches:

 

# chdev -l ent0 -a flow_ctrl=yes

Bring up the adapter -

      # chdev -l en0 -a state=up

 

Testing the network bandwidth with ftp between two AIX (or two Unix) LPARs, first the 10Gbe adapter:

# ftp <AIX box="box">

provide login credentials

 

230 User root logged in.

ftp> put "|dd if=/dev/zero bs=1m count=100" /dev/null

200 PORT command successful.

150 Opening data connection for /dev/null.

100+0 records in.

100+0 records out.

226 Transfer complete.

104857600 bytes sent in 0.3988 seconds (2.567e+05 Kbytes/s)

local: |dd if=/dev/zero bs=1m count=100 remote: /dev/null

ftp> by

221 Goodbye.

 

Then on the traditional 1Gb adapter:

# ftp <AIX box="box">

provide login credentials

 

230 User root logged in.

ftp> put "|dd if=/dev/zero bs=1m count=100" /dev/null

200 PORT command successful.

150 Opening data connection for /dev/null.

100+0 records in.

100+0 records out.

226 Transfer complete.

104857600 bytes sent in 3.956 seconds (2.588e+04 Kbytes/s)

local: |dd if=/dev/zero bs=1m count=100 remote: /dev/null

 

The following table shows suggested sizes for the tunable values to obtain optimal performance, based on the type of adapter and the MTU size:

 

Device Speed   MTU size        tcp_sendspace           tcp_recvspace            sb_max1 rfc1323

Gigabit           1500            131072                   65536                     131072          0

Gigabit           9000            131072                   65535                     262144          0

Gigabit           9000            262144                   1310722                  524288          1

10 Gigabit      1500             131072                   65536                     131072          0

10 Gigabit      9000             262144                   131072                    262144          1

 

There are five ISNO parameters for each supported interface: rfc1323, tcp_nodelay, tcp_sendspace, tcp_recvspace, and tcp_mssdflt. When set, the values for these parameters override the system-wide parameters of the same names that had been set with the no command.

The network option use_isno, set with the no command, must have a value of 1 for the ISNOs to take effect. The default value for use_isno is 1.

Verify general system and interface support using the no and lsattr commands.

Ensure the use_isno option is enabled using a command similar to the following:

# no -a | grep isno

use_isno=1

 

Enable isno –

# no -p -o use_isno=1

     Modification to restricted tunable use_isno, confirmation required

yes/no: yes

Warning: a restricted tunable has been modified

Setting use_isno to 1

Setting use_isno to 1 in nextboot file

 

NOTE – This seems to be only true on older OS levels, use_isno seems to be enabled as default in AIX7.

 

Ensure the interface supports the five new ISNOs using the lsattr -El command, as shown in the following:

 

# lsattr -E -l en0 -H

attribute         value  description

rfc1323                      N/A

tcp_nodelay                  N/A

tcp_sendspace                N/A

tcp_recvspace                N/A

tcp_mssdflt                  N/A

 

Set the interface specific values, using either the ifconfig or chdev command. The ifconfig command sets values temporarily, which is recommended for testing. The chdev command alters the ODM, so customized values remain valid after reboot.

Set the tcp_recvspace and tcp_sendspace to 64K and enable tcp_nodelay by using one of the following:

 

# ifconfig en0 tcp_recvspace 65536 tcp_sendspace 65536 tcp_nodelay 1

# chdev -l en0 -a tcp_recvspace=65536 -a tcp_sendspace=65536 -a tcp_nodelay=1

 

Enable the rfc1323 Option, this enables TCP enhancements as specified by RFC 1323, TCP Extensions for hight perforrmance. To do this AIX system wide  - 

 # no -p -o rfc1324=1

 

Alternatively, assuming the no command reports an rfc1323=1 global value, the root user can turn rfc1323 off for all connections over en0 with the following commands:

 

# ifconfig en0 rfc1323 1

# chdev -l en0 -a rfc1323=1

 

Verify the settings using the ifconfig or lsattr command, as shown in the following example:

 

# ifconfig en0 <UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT>

en0: flags=e080863 

inet 9.19.19.19 netmask 0xffffff00 broadcast 9.19.19.255

tcp_sendspace 65536 tcp_recvspace 65536 tcp_nodelay 1 rfc1323 1

# lsattr -El en0

rfc1323         0           N/A               True

tcp_nodelay     1           N/A               True

tcp_sendspace   65536       N/A               True

tcp_recvspace   65536       N/A               True

tcp_mssdflt                 N/A               True

 

Tags:  aix 10gbe performance network networking power