[AIX] AIX 6.1 what cause high memory utilization

 

I have LPAR aix 6100-07-02 with dedicated memory mode
Svmon shows ~13,7 real memory used as below from with 13.3 is working segment (nmon shows ~96% as computional memory).

bash-4.0# svmon -G -O unit=GB
Unit: GB

---

size inuse free pin virtual available mmode
memory 13.8 13.7 0.09 2.97 14.0 0.06 Ded
pg space 13.8 0.78

work pers clnt other
pin 2.40 0 0 0.57
in use 13.3 0 0.37

Question is what cause this high memory utilization??
Svmon -U shows few users with some memory in use, but sum is under 3GB.
I`ve tried to sum memory utilization by process and it gives ~14GB but of course it`s wrong way as processes use shared segments. Any ideas how to determine high memory utilization cause?

bash-4.0# svmon -U -O unit=GB
Unit: GB
===============================================================================
User Inuse Pin Pgsp Virtual
wasana01 0.97 0.04 0.05 0.98
wasana02 0.61 0.04 0.42 0.98
root 0.33 0.05 0.15 0.43
catnd 0.15 0.04 0.11 0.26
Rroot_rn 0.12 0.03 0 0.12
ibm 0.12 0.03 0 0.12
nobody 0.12 0.03 0 0.12
zabbix 0.12 0.03 0 0.12
daemon 0.12 0.03 0 0.12
bin 0 0 0 0
sys 0 0 0 0
adm 0 0 0 0
uucp 0 0 0 0
(...)

 

 

• #ps gv | head -n 1; ps gv | egrep -v "RSS" | sort +6b -7 -n -r |head -11
  
  RSS—The amount of RAM used for the text and data segments per process. PID 15256 is using 2888k.
  %MEM—The actual amount of the RSS / Total RAM. Watch for processes that consume 40-70 percent of %MEM.
  TRS—The amount of RAM used for the text segment of a process in kilobytes.
  SIZE—The actual amount of paging space allocated for this process (text and data)
  
  #Using svmon to display memory statistics for a given process
  #svmon -P | grep -p <PID>
  

  ---

  Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd LPage
  
  
   

  • Re: AIX 6.1 what cause high memory utilization

    Mar 19, 2012 in response to SystemAdmin

    #Memory usage for each active process
    #ps gv | head -n 1; ps gv | egrep -v "RSS" | sort +6b -7 -n -r
    ID TTY STAT TIME PGIN SIZE RSS LIM TSIZ TRS %CPU %MEM COMMAND
    15256 - A 64:15 755 2572 2888 xx 2356 316 0.9 0.0 /usr/lpp/
    22752 - A 0:08 261 1960 1980 32768 465 20 0.0 0.0 dtwm
    14654 - A 0:00 324 1932 1932 xx 198 0 0.0 0.0 /usr/sbin
    20700 - A 0:07 271 1868 1896 32768 95 28 0.0 0.0 /usr/dt/b
    20444 - A 0:03 203 1736 1824 32768 551 88 0.0 0.0 dtfile
    17602 - A 0:00 274 948 1644 32768 817 696 0.0 0.0 sendmail:
    13218 - A 0:00 74 1620 1620 xx 116 0 0.0 0.0 /usr/sbin
    
    RSS—The amount of RAM used for the text and data segments per process. PID 15256 is using 2888k.
    %MEM—The actual amount of the RSS / Total RAM. Watch for processes that consume 40-70 percent of %MEM.
    TRS—The amount of RAM used for the text segment of a process in kilobytes.
    SIZE—The actual amount of paging space allocated for this process (text and data)
    
    #Using svmon to display memory statistics for a given process
    #svmon -P | grep -p 15256
    

    ---

    Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd LPage
    15256 X 12102 3221 0 12022 N N N
    #List of top memory usage of segments
    # svmon -Sut 10
    Vsid Esid Type Description PSize Inuse Pin Pgsp Virtual
    70c4e - pers large file /dev/lv01:26 s 84625 0 - -
    22ec4 - work s 29576 0 0 29586
    8b091 - pers /dev/hd3:123 s 24403 0 - -
    7800f - work kernel heap s 22050 3199 19690 22903
    a2db4 - pers /dev/hd3:105 s 15833 0 - -
    80010 - work page frame table s 15120 15120 0 15120
    7000e - work misc kernel tables s 13991 0 2388 14104
    dc09b - pers /dev/hd1:28703 s 9496 0 - -
    730ee - pers /dev/hd3:111 s 8568 0 - -
    f001e - work s 4857 0 36 6823

    
     

    • Re: AIX 6.1 what cause high memory utilization

      Mar 19, 2012 in response to SystemAdmin

      MrBhatt thanks for answer.
      As I wrote earlier I did sum of processes displayed by svmon -P (output in attachment in .rar file) and it gives me 13.74GB. The problem is that processes share memory so calculating sum this way make no sense. I did it few times and often total was x time more than actual physical memory. I could sum exclusive segments but it still problem with adding shared segment. More accurate way was in my opinion to check global utilization by user, but as you see total utilization is ~x6 less than it should be (less than 3GB with almost 14GB RAM).
      
      Top processes showed by ps or svmon, or even nmon (btw. on ibm sites you can find nice article about correletion between numbers show by ps and svmon - it`s not so simply

      
      http:
      //publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=%2Fcom.ibm.aix.prftungd%2Fdoc%2Fprftungd%2Fcorr_svmon_ps_outputs.htm
      

      ) could give me answer what cause major memory utilization but still not what causes almost 100% utilization.
      In attachment is jpg from nmon analyser showing memory utilization by command in MBytes. It`s clear that java is the top one (2x processes to be strict) but still it`s only <1300MB and than nothing.
      
      From the other side even if we could forget about shared segments sum of all processes will be more less correct but svmon -U (by user) will show only <3GB utilization - why???
      
      Short summary:
      Although I know that java is top process that "eat" memory, I don`t know what use majority of ram.
      I`m afraid that I`m making some rookie mistake, but where? :D

      	
      					
      					
      						
          MrBhatt thanks for answer.
      
           As I wrote earlier I did sum of processes displayed by svmon -P (output in attachment in .rar file) and it gives me 13.74GB. The problem is that processes share memory so calculating sum this way make no sense. I did it few times and often total was x time more than actual physical memory. I could sum exclusive segments but it still problem with adding shared segment. More accurate way was in my opinion to check global utilization by user, but as you see total utilization is ~x6 less than it should be (less than 3GB with almost 14GB RAM).
      
           
      
           Top processes showed by ps or svmon, or even nmon (btw. on ibm sites you can find nice article about correletion between numbers show by ps and svmon - it`s not so simply 
      <pre class="jive-pre">
      http:
      //publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=%2Fcom.ibm.aix.prftungd%2Fdoc%2Fprftungd%2Fcorr_svmon_ps_outputs.htm
      </pre>
          ) could give me answer what cause major memory utilization but still not what causes almost 100% utilization. 
      
           In attachment is jpg from nmon analyser showing memory utilization by command in MBytes. It`s clear that java is the top one (2x processes to be strict) but still it`s only <1300MB and than nothing.
      
           
      
           From the other side even if we could forget about shared segments sum of all processes will be more less correct but svmon -U (by user) will show only <3GB utilization - why??? 
      
           
      
           Short summary:
      
           Although I know that java is top process that "eat" memory, I don`t know what use majority of ram.
      
           I`m afraid that I`m making some rookie mistake, but where? :D
        
      					
      				

      Re: AIX 6.1 what cause high memory utilization

      • Mar 29, 2012 in response to pacynka

        Please review the svmon -O options. Something like this will filter the computational memory:
        svmon -P -Ofiltercat=exclusive -Ofiltertype=working -Osegment=on
        
        Then use: cat /proc/sys/fs/jfs2/memory_usage
        
        To figure out what the inode cache size is, which isn't reported by any other tool.
        
        

         

        • Re: AIX 6.1 what cause high memory utilization

          Feb 5 in response to Steve_ATS

          Hi Steve,

          Thanks for pointing out memory usage - /proc/sys/fs/jfs2/memory_usage. What these number are telling us:

          /proc/sys/fs/jfs2>>cat memory_usage

          metadata cache: 145985536 Are these in Kb or bytes?

          inode cache: 844103680

          total: 990089216

          
           

• Re: AIX 6.1 what cause high memory utilization

  Feb 5 in response to pacynka

  As described before it's probably shared memory allocated by one or more processes.

  If it's IPC shared memory (shmat), you should get more information with the ipcs command.

  If it's a mmap shared memory, you won't see this memory with ps or nmon.

  In this case get the perfpmr package and run the memdetails.sh! It basically uses svmon and beautifies the output :-)

  Check http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.genprogc/doc/genprogc/understanding_mem_mapping.htm for details about the ways to create shared memory.

  • Log in to reply.