[RHEL] Enabling hugepages for use with Oracle Database

문제

• Frequent unresponsiveness of server and application during heavy load
• Near constant io wait on the server above 30%
• How do I determine how many hugepages to preallocate for my Oracle instance?

환경

• Red Hat Enterprise Linux 5
• Red Hat Enterprise Linux 6
• Oracle Database

해결

¶1. Calculate the amount of hugepages that will need to be allocated for your Oracle instance.

NUMBER_OF_HUGEPAGES = ( SGA + PGA + (20KB * # of Oracle_processes)) / 2MB

¶2. SGA and PGA are values that are obtained from the Oracle instance. The correct number here is crucial. If you do not allocate enough hugepages for use with Oracle, zero pages will be used, causing "lost" memory from your available pool.

¶3. Enable the oracle user to be able utilize huge pages in the /etc/sysctl.conf file.

vm.hugetlb_shm_group=`id -g oracle`

¶4. Set the amount of huge pages to allocate in the /etc/sysctl.conf file.

vm.nr_hugepages=NUMBER_OF_HUGEPAGES

¶Note: NUMBER_OF_HUGEPAGES should be replaced with the number calculated in step 1.

¶5. Calculate the amount of pages to put into the 'memlock' parameter.

AMOUNT_OF_MEMORY = NUMBER_OF_HUGEPAGES * 1024 * 2

¶6. Set the 'memlock' parameter in the /etc/security/limits.conf file.

oracle    -    memlock    AMOUNT_OF_MEMORY

¶Note: AMOUNT_OF_MEMORY should be replaced with the value calculated from step 5.

¶7. Restart the system.

• ¶These settings can be changed on the fly in a live environment; however, memory defragmentation will begin to occur which will have a negative impact on the system and could result in the system being unresponsive for an extended period of time while a large block of free memory is allocated. It is recommended to restart the machine completely to avoid this from happening.

• ¶If you're having trouble calculating your correct amount of huge pages, you may be able to use the following script to automatically determine the correct value

#!/bin/bash
 KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'`

 # Find out the HugePage size
  HPG_SZ=`grep Hugepagesize /proc/meminfo | awk '{print $2}'`

 # Start from 1 pages to be on the safe side and guarantee 1 free HugePage
  NUM_PG=1

 # Cumulative number of pages required to handle the running shared memory segments
  for SEG_BYTES in `ipcs -m | awk '{print $5}' | grep "[0-9][0-9]*"`
   do
    MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`
    if [ $MIN_PG -gt 0 ]; then
      NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`
    fi
  done

 # Finish with results
  case $KERN in
    '2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;
           echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;
    '2.6') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    '3.10')echo "Recommended setting: vm.nr_hugepages = $NUM_PG";;
        *) echo "Unrecognized kernel version $KERN. Exiting." ;;
  esac

• ¶More information regarding tuning of your RHEL machine for Oracle database can be found here:
  http://red.ht/U1LXGQ.

• ¶For specific memory tuning recommendations, take a look at the following Red Hat Knowledgebase article:
  Tuning Red Hat Enterprise Linux for Oracle and Oracle RAC performance

근본 원인

• ¶By default, the kernel will attempt to make the most efficient use of your memory by swapping pages out from memory to disk, or by caching disk within the memory. This works well for almost all environments, however, when utilizing a database application, you may see problems occurring such as high iowait or an unresponsive server due to everything waiting on the storage.  This is caused by a high visit-to-disk count when trying to swap in and swap out pages of memory while your application is trying to access data.

• ¶Huge pages allows for a large pre-allocated amount of space be set aside for use by your application. This memory does not swap in/out from disk, and should alleviate some of the pressure that was previously seen in the environment.  It also provides other advantages, such as lower overhead for memory management.