Techniques or tools for Memory or Performance Analysis in Linux Machine?

As we are listening some of the BUZZZZ words from management,

•  Increase the performance of the system
•  Reduce the memory usage by the application etc.. etc..

In this post, i want to give you few techniques or tools help to start analysis of your application/system to increase the performance.

Step 1: Check whether any process consuming more CPU time using "top" command

command #  top - display Linux processes

root@m:~# top
top - 13:48:55 up 3 days,  6:25,  3 users,  load average: 1.58, 1.22, 1.09
Tasks: 146 total,   2 running, 144 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.8 us, 10.3 sy,  0.0 ni, 85.9 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem:   4120948 total,   771684 used,  3349264 free,   224024 buffers
KiB Swap:   511996 total,        0 used,   511996 free.   389264 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
   13 root      20   0       0      0      0 S   0.7  0.0  47:35.07 ksoftirqd/1
   18 root      20   0       0      0      0 S   0.7  0.0  34:44.90 ksoftirqd/2
15607 root      20   0    5532   1428   1008 R   0.7  0.0   0:00.14 top
    7 root      20   0       0      0      0 S   0.3  0.0  11:16.14 rcu_sched
   17 root      rt   0       0      0      0 S   0.3  0.0   3:37.20 migration/2
   23 root      20   0       0      0      0 S   0.3  0.0  35:48.12 ksoftirqd/3
18099 root      20   0    2904    576    492 R   0.3  0.0   0:00.01 ps
    1 root      20   0    4552   2512   1424 S   0.0  0.1   1:08.98 init
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0  44:12.70 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H

* Verify for the CPU and MEM percentage whether process is taking more or not

 #
Check the CPU time and memory for a particular process using process ID.

top -p <process ID>

root@50:~# ps -eaf | grep -i rsyslogd
syslog   11009     1  0 07:21 ?        00:00:00 rsyslogd
root     30034  3215  0 08:47 pts/3    00:00:00 grep --color=auto -i rsyslogd

root@50:~# top -p 11009


                  ****  Different ways to fetch the process information *****
 
# ps -aux

# pstree 

# pstree -clans


# ps -eaf | grep -i  < process> 
root@50:~# ps -eaf | grep -i rsyslogd
syslog   11009     1  0 07:21 ?        00:00:00 rsyslogd
root     30034  3215  0 08:47 pts/3    00:00:00 grep --color=auto -i rsyslogd

# ps -p <pid> -o %cpu,%mem,cmd
root@50:~# ps -p 11009 -o %cpu,%mem,cmd
%CPU %MEM CMD
 0.0  0.0 rsyslogd
 
# ps axo pid,etime,%cpu,%mem,cmd | grep 'processname' | grep -v grep

 
Step 2: Verify the availability of RAM memory by "free" command
free  - displays  the total amount of free and used physical and swap memory in the system, as well as the buffers used by the kernel.  In order words display RAM details.

          -b, --bytes
         Display the amount of memory in bytes.

       -k, --kilo
              Display the amount of memory in kilobytes.  This is the default.

       -m, --mega
              Display the amount of memory in megabytes.

       -g, --giga
              Display the amount of memory in gigabytes.

       --tera Display the amount of memory in terabytes.

       -h, --human
              Show all output fields automatically scaled to shortest three digit unit and display the units of print out.  

-> Kilo Bytes
root@m:~# free
             total       used       free     shared    buffers     cached
Mem:       4120948     778564    3342384        892     224048     396376
-/+ buffers/cache:     158140    3962808
Swap:       511996          0     511996

-> Mega Bytes
root@S341640-50:~# free -m
             total       used       free     shared    buffers     cached
Mem:          4024        760       3264          0        218        387
-/+ buffers/cache:        154       3870
Swap:          499          0        499

-> GIGA Bytes
root@S341640-50:~# free -g
             total       used       free     shared    buffers     cached
Mem:             3          0          3          0          0          0
-/+ buffers/cache:          0          3
Swap:            0          0          0

-> Human Readable Format
root@S341640-50:~# free -h
             total       used       free     shared    buffers     cached
Mem:          3.9G       759M       3.2G       892K       218M       387M
-/+ buffers/cache:       153M       3.8G
Swap:         499M         0B       499M

Let's understand the free -h  output:

Line 1:  Mem: 3.9G       760M       3.2G       892K       218M       387M

3.9G : It represents memory physically available in Machine.
760M : It represents memory/RAM used by system. And this include even buffers and cached data size as well.
3.2G : It represents Total RAM free and available for new process to run.
892K : Indicates shared memory (Don't worry about this)
218M : Indicates total RAM buffered by different applications in Linux
387M : Indicates total RAM used for Caching of data

Line 2: -/+ buffers/cache:       154M       3.8G
Buffers:A buffer is a temporary location to store data for particular application and this data is not used by any other application.
Cache: is a memory location to store frequently used data for faster access.
The main difference between buffer and cache is that cache can be used multiple times where as buffer is used single time. And both are used for temporary storage of data.

(154M :  Actual size of RAM used -(buffers + cache)
Used RAM = 760M
Line 1 -> Used Buffers = 218M
Line 1 -> Used Cache   = 387M
760M-(218M + 387M) = ~154M

3.8G = (Total RAM - Buffers (line 2))
Total RAM = 3.9G
Buffer Size = 154M
Total actual available RAM = 3.9G  - 154M = ~3.8G

** 3.8G is actual RAM available. So the Linux machine has available is 3.8 G.***

Line 3: It indicates the SWAP memory. Swap is a virtual memory created on Hard disk to increase RAM size virtually. 

Step 3:  Verify the RAM  usage by "watch" command
"free" command can be made even more useful by using watch's -d (i.e., difference) option, which highlights changes in output, and its -n option used to get memory updates every n seconds.

Try watch -n 5 free to monitor memory usage with updates every five seconds.
watch -n 5 free
watch -n 5 -d free -m

Step 4:  Check the full details of RAM usage by  "cat /proc/meminfo" command
root@S341640:~# free
             total       used       free     shared    buffers     cached
Mem:       2027408     893072    1134336      49704      89364     353536
-/+ buffers/cache:     450172    1577236
Swap:      1949692          0    1949692

root@muni:~# cat /proc/meminfo
MemTotal:        2027408 kB
MemFree:         1134432 kB
Buffers:           89364 kB
Cached:           353536 kB
SwapCached:            0 kB
Active:           532680 kB
Inactive:         296572 kB
Active(anon):     395196 kB
Inactive(anon):    40856 kB
Active(file):     137484 kB
Inactive(file):   255716 kB
Unevictable:           0 kB
Mlocked:               0 kB
HighTotal:       1146956 kB
HighFree:         681860 kB
LowTotal:         880452 kB

To know more details about this command : http://www.cloudibee.com/know-about-procmeminfo/

Step 5:  There is one more command which is useful to get the virtual memory statistics  by  "vmstat" command

vmstat - Report virtual memory statistics

root@S341640:~# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 1129516  90336 358992    0    0    84    34  287  669  3  1 96  1  0

Use "-s" option for better statistics.

root@muni:~# vmstat -s      2027408 K total memory
       897728 K used memory
       536484 K active memory
       296960 K inactive memory
      1129680 K free memory
        90440 K buffer memory
       359072 K swap cache
      1949692 K total swap
            0 K used swap
      1949692 K free swap
        14913 non-nice user cpu ticks
            0 nice user cpu ticks
         2755 system cpu ticks
       464452 idle cpu ticks
         3648 IO-wait cpu ticks
            0 IRQ cpu ticks
          132 softirq cpu ticks
            0 stolen cpu ticks
       397525 pages paged in
       162776 pages paged out
            0 pages swapped in
            0 pages swapped out
      1385938 interrupts
      3223970 CPU context switches
   1487662504 boot time
        14022 forks

HTOP tool:

You can checkout the CPU, memory by user friendly tool like htop.
htop is an interactive system-monitor process-viewer. It is designed as an alternative to the Unix program top. It shows a frequently updated list of the processes running on a computer, normally ordered by the amount of CPU usage. Unlike top, htop provides a full list of processes running, instead of the top resource-consuming processes. Htop uses color and gives visual information about processor, swap and memory status.

Install using: sudo apt-get install htop 



Step 6:  Check the disk space as well.
df displays the amount of disk space available on the file system containing each file name argument


root@m:~# df -h

Filesystem      Size  Used Avail Use% Mounted on
udev            980M  4.0K  980M   1% /dev
tmpfs           198M  3.6M  195M   2% /run
/dev/dm-1        20G  9.7G  9.3G  51% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M   12K  5.0M   1% /run/lock
none            990M     0  990M   0% /run/shm
none            100M  4.0K  100M   1% /run/user

Step 7:  Tuning Virtual Memory or Swappiness value.

What is Swappiness?

------------------------

Most of Linux users that have installed a distribution before, must have noticed the existence of the “swap space” during the partitioning phase (it is usually found as /sda5). This is a dedicated space in your hard drive that is usually set to at least twice the capacity of your RAM, and along with it constitutes the total virtual memory of your system. From time to time, the Linux kernel utilizes this swap space by copying chunks from your RAM to the swap, allowing active processes that require more memory than it is physically available to run.

Swappiness is the kernel parameter that defines how much (and how often) your Linux kernel will copy RAM contents to swap. This parameter's default value is “60” and it can take anything from “0” to “100”. The higher the value of the swappiness parameter, the more aggressively your kernel will swap. 

Moreover, the swappiness of a system is a primary factor that determines the overall functionality and speed performance of an OS.  

There are some maths involved in the swappiness that should be considered when changing your settings. The parameter value set to “60” means that your kernel will swap when RAM reaches 40% capacity. Setting it to “100” means that your kernel will try to swap everything. Setting it to 10 (like I did on this tutorial) means that swap will be used when RAM is 90% full, so if you have enough RAM memory, this could be a safe option that would easily improve the performance of your system.

 [ Source :https://www.howtoforge.com/tutorial/linux-swappiness/]

To temporarily set the swappiness in Linux, write the desired value (e.g. 10) to /proc/sys/vm/swappiness using the following command, running as root user:

# Set the swappiness value as root
echo 10 > /proc/sys/vm/swappiness

# Alternatively, run this 
sysctl -w vm.swappiness=10

# Verify the change
cat /proc/sys/vm/swappiness
10

# Alternatively, verify the change
sysctl vm.swappiness
vm.swappiness = 10

To make the changes permanent , then /etc/sysctl.conf via the following configuration line (inserted)

vm.swappiness = 10

Step 7:  Identify the unwanted processes or Zombie process and  remove them to save memory.

The above command shows the list of processes running in the system,, if there are unwanted process running in the system, remove it from process table.

 A "defunct" processes is also known as a "zombie" processes. A Zombie process is referred as dead process. This Dead process will be in process table and consuming RAM (memory). If there are more defunct processes then it will consume more memory causes slow down of the system.

So it is always better to kill the defunct process  to free RAM and make system stable.

Find out zombie process?
# ps aux | awk '{ print $8 " " $2 }' | grep -w Z 

STAT of "ps aux" command shows as 'Z' means Zombie process.

How to kill Zombie process:
Zombie process is already dead process, we need to restart the service of parent or kill the parent.
But you can try the possible ways:

Option #1
ps -ef | grep defunct
kill defunct-pid

Option #2
kill -9 defunct-pid

Option #3
kill parent-id-of-defunct-pid (or) kill -9 parent-id-of-defunct-pid

Remove unwanted processes to Save memory or CPU time: 

•  ps aux 
•  service --status-all

 Example:

•  kill -9 httpd
•  service httpd status
•  chkconfig httpd of
•  service httpd status

Find out unwanted kernel Modules using lsmod and remove them using rmmod, if it is not required.

Step 8:  Write a cron Job  that runs daily to get the memory statistics of the machine and analyze which process is consuming more memory.

Step 9:  Below you can find python script to get memory statistics.
http://www.pixelbeat.org/scripts/ps_mem.py


Step 10:   
If any of your process is taking more CPU time and memory. Debug using valgrind tools like kcachegrind, massify, callgrind etc.

Don't limit to above steps, there are many ways to tweak linux machine to get more performance.... All the best....