Interpreting /proc/meminfo and free output for Red Hat Enterprise Linux 5, 6 and 7

Solution Verified - UpdatedOctober 2 2018 at 6:35 AM -

Environment

Red Hat Enterprise Linux (RHEL) 5
Red Hat Enterprise Linux (RHEL) 6
Red Hat Enterprise Linux (RHEL) 7

Issue

I need an interpretation of /proc/meminfo output.
I want to compare the output of free -k to cat /proc/meminfo.

Resolution

For definition of /proc/meminfo fields in Red Hat Enterprise Linux (RHEL) releases prior to RHEL 5, please look at What is indicated by each value in /proc/meminfo?
Each field of cat /proc/meminfo will be discussed in the Diagnostics Steps.
The RHEL 5 output differs in some settings. This is also marked in the Diagnostic Steps.
RHEL 5 also has some fields no longer present in RHEL 6. For explanation on this issue have a look at Why are LowTotal, LowFree, HighTotal, and HighFree missing from /proc/meminfo on x86_64 RHEL 6?
For more information on the output of the free command see How do I view system memory utilization in Red Hat Enterprise Linux?
RHEL 7 has an additional field called MemAvailable in /proc/meminfo
RHEL 7 has a slightly changed output of the free command

Comparing the output

free -k output (RHEL 5 and RHEL 6):

             total       used       free     shared    buffers     cached
Mem:       7778104    2971960    4806144          0     211756    1071092
-/+ buffers/cache:    1689112    6088992
Swap:      4194296          0    4194296

free -k output (RHEL 7):

              total        used        free      shared  buff/cache   available
Mem:        1012952      252740      158732       11108      601480      543584
Swap:       1048572        5380     1043192

Relevant fields from /proc/meminfo to match them against the output of free -k:

MemTotal:        7778104 kB
MemFree:         4806144 kB
Buffers:          211756 kB
Cached:          1071092 kB
SwapTotal:       4194296 kB
SwapFree:        4194296 kB

For RHEL 7 there is an additional field available, which is used instead of the calculation for -/+ buffers/cache line:

MemAvailable:     543584 kB

Matching output of `free -k` to `/proc/meminfo`

The following table shows how to get the free output matched to the /proc/meminfo fields in Red Hat Enterprise Linux 5, 6 and 7.0.

`free output`	coresponding `/proc/meminfo` fields
`Mem: total`	`MemTotal`
`Mem: used`	`MemTotal - MemFree`
`Mem: free`	`MemFree`
`Mem: shared` (can be ignored nowadays. It has no meaning.)	N/A
`Mem: buffers`	`Buffers`
`Mem: cached`	`Cached`
`-/+ buffers/cache: used`	`MemTotal - (MemFree + Buffers + Cached)`
`-/+ buffers/cache: free`	`MemFree + Buffers + Cached`
`Swap: total`	`SwapTotal`
`Swap: used`	`SwapTotal - SwapFree`
`Swap: free`	`SwapFree`

The following table shows how to get the free output matched to the /proc/meminfo fields in Red Hat Enterprise Linux 7.1 or later (procps-ng 3.3.10).

`free output`	coresponding `/proc/meminfo` fields
`Mem: total`	`MemTotal`
`Mem: used`	`MemTotal - MemFree - Buffers - Cached - Slab`
`Mem: free`	`MemFree`
`Mem: shared`	`Shmem`
`Mem: buff/cache`	`Buffers + Cached + Slab`
`Mem:available`	`MemAvailable`
`Swap: total`	`SwapTotal`
`Swap: used`	`SwapTotal - SwapFree`
`Swap: free`	`SwapFree`

Root Cause

Analyzing memory consumption

Diagnostic Steps

Most stuff is taken from the kernel documentation (Documentation/filesystems/proc.txt and Documentation/vm/hugetlbpage.txt)

High Level statistics

RHEL 5, RHEL 6 and RHEL 7

MemTotal: Total usable memory
MemFree: The amount of physical memory not used by the system
Buffers: Memory in buffer cache, so relatively temporary storage for raw disk blocks. This shouldn't get very large.
Cached: Memory in the pagecache (Diskcache and Shared Memory)
SwapCached: Memory that is present within main memory, but also in the swapfile. (If memory is needed this area does not need to be swapped out AGAIN because it is already in the swapfile. This saves I/O and increases performance if machine runs short on memory.)

RHEL 7 only

MemAvailable: An estimate of how much memory is available for starting new applications, without swapping.

Detailed Level statistics

RHEL 5, RHEL 6 and RHEL 7

Active: Memory that has been used more recently and usually not swapped out or reclaimed
Inactive: Memory that has not been used recently and can be swapped out or reclaimed

RHEL 6 and RHEL 7 only

Active(anon): Anonymous memory that has been used more recently and usually not swapped out
Inactive(anon): Anonymous memory that has not been used recently and can be swapped out
Active(file): Pagecache memory that has been used more recently and usually not reclaimed until needed
Inactive(file): Pagecache memory that can be reclaimed without huge performance impact
Unevictable: Unevictable pages can't be swapped out for a variety of reasons
Mlocked: Pages locked to memory using the mlock() system call. Mlocked pages are also Unevictable.

Memory statistics

RHEL 5, RHEL 6 and RHEL 7

SwapTotal: Total swap space available
SwapFree: The remaining swap space available
Dirty: Memory waiting to be written back to disk
Writeback: Memory which is actively being written back to disk
AnonPages: Non-file backed pages mapped into userspace page tables
Mapped: Files which have been mmaped, such as libraries
Slab: In-kernel data structures cache
PageTables: Amount of memory dedicated to the lowest level of page tables. This can increase to a high value if a lot of processes are attached to the same shared memory segment.
NFS_Unstable: NFS pages sent to the server, but not yet commited to the storage
Bounce: Memory used for block device bounce buffers
CommitLimit: Based on the overcommit ratio (vm.overcommit_ratio), this is the total amount of memory currently available to be allocated on the system. This limit is only adhered to if strict overcommit accounting is enabled (mode 2 in vm.overcommit_memory).
Committed_AS: The amount of memory presently allocated on the system. The committed memory is a sum of all of the memory which has been allocated by processes, even if it has not been "used" by them as of yet.
VmallocTotal: total size of vmalloc memory area
VmallocUsed: amount of vmalloc area which is used
VmallocChunk: largest contiguous block of vmalloc area which is free
HugePages_Total: Number of hugepages being allocated by the kernel (Defined with vm.nr_hugepages)
HugePages_Free: The number of hugepages not being allocated by a process
HugePages_Rsvd: The number of hugepages for which a commitment to allocate from the pool has been made, but no allocation has yet been made.
Hugepagesize: The size of a hugepage (usually 2MB on an Intel based system)

RHEL 6 and RHEL 7 only

Shmem: Total used shared memory (shared between several processes, thus including RAM disks, SYS-V-IPC and BSD like SHMEM)
SReclaimable: The part of the Slab that might be reclaimed (such as caches)
SUnreclaim: The part of the Slab that can't be reclaimed under memory pressure
KernelStack: The memory the kernel stack uses. This is not reclaimable.
WritebackTmp: Memory used by FUSE for temporary writeback buffers
HardwareCorrupted: The amount of RAM the kernel identified as corrupted / not working
AnonHugePages: Non-file backed huge pages mapped into userspace page tables
HugePages_Surp: The number of hugepages in the pool above the value in vm.nr_hugepages. The maximum number of surplus hugepages is controlled by vm.nr_overcommit_hugepages.
DirectMap4k: The amount of memory being mapped to standard 4k pages
DirectMap2M: The amount of memory being mapped to hugepages (usually 2MB in size)

A quick rundown on /proc’s files:

/proc/cmdline – Kernel command line information.
/proc/console – Information about current consoles including tty.
/proc/devices – Device drivers currently configured for the running kernel.
/proc/dma – Info about current DMA channels.
/proc/fb – Framebuffer devices.
/proc/filesystems – Current filesystems supported by the kernel.
/proc/iomem – Current system memory map for devices.
/proc/ioports – Registered port regions for input output communication with device.
/proc/loadavg – System load average.
/proc/locks – Files currently locked by kernel.
/proc/meminfo – Info about system memory (see above example).
/proc/misc – Miscellaneous drivers registered for miscellaneous major device.
/proc/modules – Currently loaded kernel modules.
/proc/mounts – List of all mounts in use by system.
/proc/partitions – Detailed info about partitions available to the system.
/proc/pci – Information about every PCI device.
/proc/stat – Record or various statistics kept from last reboot.
/proc/swap – Information about swap space.
/proc/uptime – Uptime information (in seconds).
/proc/version – Kernel version, gcc version, and Linux distribution installed.

Within /proc’s numbered directories you will find a few files and links. Remember that these directories’ numbers correlate to the PID of the command being run within them. Let’s use an example. On my system, there is a folder name /proc/12:

# cd /proc/12
# ls

Sample Output

attr        coredump_filter  io         mounts      oom_score_adj  smaps    wchan
autogroup   cpuset           latency    mountstats  pagemap        stack
auxv        cwd              limits     net         personality    stat
cgroup      environ          loginuid   ns          root           statm
clear_refs  exe              maps       numa_maps   sched          status
cmdline     fd               mem        oom_adj     schedstat      syscall
comm        fdinfo           mountinfo  oom_score   sessionid      task

If I run:

# cat /proc/12/status

I get the following:

Name:	xenwatch
State:	S (sleeping)
Tgid:	12
Pid:	12
PPid:	2
TracerPid:	0
Uid:	0	0	0	0
Gid:	0	0	0	0
FDSize:	64
Groups:
Threads:	1
SigQ:	1/4592
SigPnd:	0000000000000000
ShdPnd:	0000000000000000
SigBlk:	0000000000000000
SigIgn:	ffffffffffffffff
SigCgt:	0000000000000000
CapInh:	0000000000000000
CapPrm:	ffffffffffffffff
CapEff:	ffffffffffffffff
CapBnd:	ffffffffffffffff
Cpus_allowed:	1
Cpus_allowed_list:	0
Mems_allowed:	00000000,00000001
Mems_allowed_list:	0
voluntary_ctxt_switches:	84
nonvoluntary_ctxt_switches:	0

So, what does this mean? Well, the important part is at the top. We can see from the status file that this process belongs to xenwatch. Its current state is sleeping, and its process ID is 12, obviously. We also can see who is running this, as UID and GID are 0, indicating that this process belongs to the root user.

In any numbered directory, you will have a similar file structure. The most important ones, and their descriptions, are as follows:

cmdline – command line of the process
environ – environmental variables
fd – file descriptors
limits – contains information about the limits of the process
mounts – related information

You will also notice a number of links in the numbered directory:

cwd – a link to the current working directory of the process
exe – link to the executable of the process
root – link to the work directory of the process

This should get you started with familiarizing yourself with the /proc directory. It should also provide insight to how a number of commands obtain their info, such as uptime, lsof, mount, and ps, just to name a few.