Stale TCP connections with tg3 on Red Hat Enterprise Linux 6

00시 28분 2014년 4월 2일 업데이트

문제

  • Stale TCP connections
  • Connection are being reset unexpectedly
  • SSH disconnect just after login
  • Page allocation failure messages seen in /var/log/messages:
Jan  9 06:36:35 xxxxxxx kernel: swapper: page allocation failure. order:4, mode:0x20
Jan  9 06:36:35 xxxxxxx kernel: Pid: 0, comm: swapper Not tainted 2.6.32-358.14.1.el6.x86_64 #1

환경

  • Red Hat Enterprise Linux 6
    • Broadcom tg3 NIC driver <= 3.132 [3.132 tg3 driver is included in RHEL 6 update 5]
    • GSO/TSO on

해결

  • This issue is being tracked via private Bugzilla #1029192, it does not currently have a resolution. If this issue is encountered please contact Red Hat Support.

Workaround

  • Disable TSO/GSO on all interfaces using the tg3 NIC driver: 
# ethtool -K <interface> gso off tso off

근본 원인

  • The tg3 driver has a workaround for a hardware issue on some models that may cause some packets to be reallocated during TX. The problem is when this happens with an GSO/TSO offloaded packet (>= MTU size) and when system memory is heavily fragmented. When all this aligns up, as this reallocation happens under softirq context and the allocation has to be atomic, the allocation may fail and the driver will not be able to pass down this big packet to the network card itself.
  • Therefore, if this allocation fails, the packet will be silently dropped, just like a drop at network. But when TCP tries to retransmit, it will likely hit the same allocation failure, leading to a connection stall.
  • Disabling GSO/TSO will avoid such large packets at tg3 TX path, and should alleviate the situation.

진단 단계

  • Check dmesg output for traces containing:
Nov  8 07:59:25 SERVER kernel: swapper: page allocation failure. order:3, mode:0x20
Nov  8 07:59:25 SERVER kernel: Pid: 0, comm: swapper Not tainted 2.6.32-358.18.1.el6.x86_64 #1
Nov  8 07:59:25 SERVER kernel: Call Trace:
Nov  8 07:59:25 SERVER kernel: <IRQ>  [<ffffffff8112c257>] ? __alloc_pages_nodemask+0x757/0x8d0
Nov  8 07:59:25 SERVER kernel: [<ffffffff81166d92>] ? kmem_getpages+0x62/0x170
Nov  8 07:59:25 SERVER kernel: [<ffffffff811679aa>] ? fallback_alloc+0x1ba/0x270
Nov  8 07:59:25 SERVER kernel: [<ffffffff811673ff>] ? cache_grow+0x2cf/0x320
Nov  8 07:59:25 SERVER kernel: [<ffffffff81167729>] ? ____cache_alloc_node+0x99/0x160
Nov  8 07:59:25 SERVER kernel: [<ffffffff811688f0>] ? kmem_cache_alloc_node_trace+0x90/0x200
Nov  8 07:59:25 SERVER kernel: [<ffffffff81168b0d>] ? __kmalloc_node+0x4d/0x60
Nov  8 07:59:25 SERVER kernel: [<ffffffff8143ddad>] ? __alloc_skb+0x6d/0x190
Nov  8 07:59:25 SERVER kernel: [<ffffffff8143eed0>] ? skb_copy+0x40/0xb0                            <---
Nov  8 07:59:25 SERVER kernel: [<ffffffffa012627c>] ? tg3_start_xmit+0xa8c/0xd50 [tg3]              <---
Nov  8 07:59:25 SERVER kernel: [<ffffffff814493b8>] ? dev_hard_start_xmit+0x308/0x530
Nov  8 07:59:25 SERVER kernel: [<ffffffffa01d2d5b>] ? bond_start_xmit+0x53b/0x5d0 [bonding]
Nov  8 07:59:25 SERVER kernel: [<ffffffff8146773a>] ? sch_direct_xmit+0x15a/0x1c0
Nov  8 07:59:25 SERVER kernel: [<ffffffff814493b8>] ? dev_hard_start_xmit+0x308/0x530
Nov  8 07:59:25 SERVER kernel: [<ffffffff8144d0c0>] ? dev_queue_xmit+0x3b0/0x550
Nov  8 07:59:25 SERVER kernel: [<ffffffffa01d2687>] ? bond_dev_queue_xmit+0x67/0x200 [bonding]
Nov  8 07:59:25 SERVER kernel: [<ffffffffa01d2d5b>] ? bond_start_xmit+0x53b/0x5d0 [bonding]
Nov  8 07:59:25 SERVER kernel: [<ffffffff814493b8>] ? dev_hard_start_xmit+0x308/0x530
Nov  8 07:59:25 SERVER kernel: [<ffffffff8143ddc1>] ? __alloc_skb+0x81/0x190
Nov  8 07:59:25 SERVER kernel: [<ffffffff8144cf15>] ? dev_queue_xmit+0x205/0x550
Nov  8 07:59:25 SERVER kernel: [<ffffffff814857f8>] ? ip_finish_output+0x148/0x310
Nov  8 07:59:25 SERVER kernel: [<ffffffff81485a78>] ? ip_output+0xb8/0xc0
Nov  8 07:59:25 SERVER kernel: [<ffffffff81055f96>] ? enqueue_task+0x66/0x80
Nov  8 07:59:25 SERVER kernel: [<ffffffff81484d75>] ? ip_local_out+0x25/0x30
Nov  8 07:59:25 SERVER kernel: [<ffffffff81485250>] ? ip_queue_xmit+0x190/0x420
Nov  8 07:59:25 SERVER kernel: [<ffffffff8149a04e>] ? tcp_transmit_skb+0x40e/0x7b0
Nov  8 07:59:25 SERVER kernel: [<ffffffff8149c45b>] ? tcp_write_xmit+0x1fb/0xa20
Nov  8 07:59:25 SERVER kernel: [<ffffffff8149ce10>] ? __tcp_push_pending_frames+0x30/0xe0
Nov  8 07:59:25 SERVER kernel: [<ffffffff814948a3>] ? tcp_data_snd_check+0x33/0x100
Nov  8 07:59:25 SERVER kernel: [<ffffffff81498471>] ? tcp_rcv_established+0x371/0x800
Nov  8 07:59:25 SERVER kernel: [<ffffffffa004e609>] ? do_hpsa_intr_msi+0x149/0x290 [hpsa]
Nov  8 07:59:25 SERVER kernel: [<ffffffff814a04e3>] ? tcp_v4_do_rcv+0x2e3/0x430
Nov  8 07:59:25 SERVER kernel: [<ffffffff810e1760>] ? handle_IRQ_event+0x60/0x170
Nov  8 07:59:25 SERVER kernel: [<ffffffff814a1d6e>] ? tcp_v4_rcv+0x4fe/0x8d0
Nov  8 07:59:25 SERVER kernel: [<ffffffff8143d6d7>] ? __kfree_skb+0x47/0xa0
Nov  8 07:59:25 SERVER kernel: [<ffffffff8147f9fd>] ? ip_local_deliver_finish+0xdd/0x2d0
Nov  8 07:59:25 SERVER kernel: [<ffffffff8147fc88>] ? ip_local_deliver+0x98/0xa0
Nov  8 07:59:25 SERVER kernel: [<ffffffff8147f14d>] ? ip_rcv_finish+0x12d/0x440
Nov  8 07:59:25 SERVER kernel: [<ffffffff8147f6d5>] ? ip_rcv+0x275/0x350
Nov  8 07:59:25 SERVER kernel: [<ffffffff814488ab>] ? __netif_receive_skb+0x4ab/0x750
Nov  8 07:59:25 SERVER kernel: [<ffffffff8149f08a>] ? tcp4_gro_receive+0x5a/0xd0
Nov  8 07:59:25 SERVER kernel: [<ffffffff8144ac88>] ? netif_receive_skb+0x58/0x60
Nov  8 07:59:25 SERVER kernel: [<ffffffff8144ad90>] ? napi_skb_finish+0x50/0x70
Nov  8 07:59:25 SERVER kernel: [<ffffffff8144d339>] ? napi_gro_receive+0x39/0x50
Nov  8 07:59:25 SERVER kernel: [<ffffffffa0122c14>] ? tg3_poll_work+0x784/0xe50 [tg3]
Nov  8 07:59:25 SERVER kernel: [<ffffffffa012332c>] ? tg3_poll_msix+0x4c/0x150 [tg3]
Nov  8 07:59:25 SERVER kernel: [<ffffffff8144d453>] ? net_rx_action+0x103/0x2f0
Nov  8 07:59:25 SERVER kernel: [<ffffffff810770b1>] ? __do_softirq+0xc1/0x1e0
Nov  8 07:59:25 SERVER kernel: [<ffffffff810e1760>] ? handle_IRQ_event+0x60/0x170
Nov  8 07:59:25 SERVER kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
Nov  8 07:59:25 SERVER kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
Nov  8 07:59:25 SERVER kernel: [<ffffffff81076e95>] ? irq_exit+0x85/0x90
Nov  8 07:59:25 SERVER kernel: [<ffffffff815176e5>] ? do_IRQ+0x75/0xf0
Nov  8 07:59:25 SERVER kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
Nov  8 07:59:25 SERVER kernel: <EOI>  [<ffffffff812d3cae>] ? intel_idle+0xde/0x170
Nov  8 07:59:25 SERVER kernel: [<ffffffff812d3c91>] ? intel_idle+0xc1/0x170
Nov  8 07:59:25 SERVER kernel: [<ffffffff814155f7>] ? cpuidle_idle_call+0xa7/0x140
Nov  8 07:59:25 SERVER kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
Nov  8 07:59:25 SERVER kernel: [<ffffffff8150756c>] ? start_secondary+0x2ac/0x2ef
Nov  8 08:04:37 SERVER kernel: swapper: page allocation failure. order:3, mode:0x20
Nov  8 08:04:37 SERVER kernel: Pid: 0, comm: swapper Not tainted 2.6.32-358.18.1.el6.x86_64 #1

 

 

저작자표시

'OS > RedHat Bug Report' 카테고리의 다른 글

[BUG] Why does system reboot fail with "init: unable to open console: Input/output error" error message?  (0) 2014.04.24
[BUG] Unable to turn TSO on on virtual interfaces on RHEL 6.5  (0) 2014.04.24
[BUG] Kernel crash at "cpufreq_governor_dbs+0x397" or "__cpufreq_governor+0x2b"  (0) 2014.04.24
[BUG] NFSv4 clients write to the same NFS file on a NFSv4 server with write delegations  (0) 2014.04.24
[BUG] Failed GFP_ATOMIC allocations (dropped network packets) result in kernel warnings and backtrace  (0) 2014.04.24
[BUG] EROFS ("Read-only file system") error is incorretly returned when trying to open already existing file with 'O_CREAT' on a read-only filesystem  (0) 2014.04.24
[BUG] certain versions of Red Hat Enterprise Linux 6 kernels become unresponsive/hung or incur a kernel panic  (0) 2014.04.24
[BUG] Filesystem corruption: "ext3_new_block: Allocating block in system zone"  (0) 2014.04.24
[BUG] RHEV guests hanging and crashing  (0) 2014.04.24
[BUG] Problem with Hot-adding memory in VMWare RHEL guest  (0) 2014.04.24

+ Recent posts

티스토리툴바