Lesson 15: Paging

Let’s talk about paging space in general. Paging space is the extension of memory under VMM control. Both, the real memory and the virtual memory are divided into 4kb units called frames and pages accordingly. At any given time, there might be more virtual than real memory. Any excess virtual memory will be stored on disk (inside logical volumes type=paging). If the excess memory continues to grow, its consumption of paging space grows with it and if unchecked eventually it will bring the system to its knees. This fact often is announced with the message similar to this one fork failed: not enough memory ….. I do not remember the exact wording, but this is close enough. If you see a message like this one than it is usually already too late to do anything except rebooting the system. You may try to issue a command and to wait for a few minutes to see if it will be executed or not….Who knows maybe you will be the lucky one today.

With the introduction of lru_file_repage=0 (kernel parameter), we do not often deal with issues related to high paging consumption. But yesterday, one of Annowy’s machines generated a paging alert and he resolved the issue executing the steps described bellow.

Let’s say that the rootvg has this topology (for clarity I left only the swap information):

MarcoPolo:/root> lsvg -l rootvg
rootvg:
LV NAME   TYPE  LPs PPs PVs LV STATE MOUNT POINT
paging00 paging 122 122 2 open/syncd    N/A
hd6      paging 112 112 2 open/syncd    N/A
MarcoPolo:/root>

To check swap consumption, you execute:

MarcoPolo:/root>lsps -s
Total Paging Space Percent Used
29952MB 89%

This is a high utilization, something has to be done about it or the machine may soon lock itself and you out. You need to give yourself enough time to diagnose and fix the real issue – the cause, the reason why so much of paging space was consumed. The following method really works.

First, create and activate a new paging volume. What disk to use? It does not really matter. Try, if you can to make it at least as big as the existing ones.Pierw, stwórz nowy tom logiczny rodzaju paging. Który dysk należy użyć do tego celu? Dowolny dysk, to nie ma żadnego znaczenia w tym momencie. Jeżeli możesz daj temu nowemu tomowi taki sam lub zbliżony rozmiar do już istniejących tomów stronicowych.

MarcoPolo:/root>mkps -n -s 100 rootvg

The -n switch activates the new paging volume as soon as it is made.

MarcoPolo:/root>lsps -a
Page Space Physical Volume Volume Group  Size %Used Active Auto Type
paging01       hdisk0        rootvg     12800MB 45   yes   yes   lv
paging00       hdisk1        rootvg     15616MB 46   yes   yes   lv
hd6            hdisk1        rootvg     14336MB  1   yes   yes   lv
MarcoPolo:/root>

Now, the system will start distributing the paging space to the new (paging01) swap volume, and as the result the paging space utilization will begin to decrease. This process make take few or more minutes, wait for it to stabilize. As you waiting, you may look for the answer to the today’s question ”who’s done it?” What, who or both is responsible for your high blood pressure and your rapid hear beat?

To identify the highest/biggest ”users/consumers” of swap you can use svmon -Pgt5 | grep -p Pid | grep ‘^.*[0-9]’. As you could expect its output will produce the five (5) heaviest swap consuming processes.

After you notice that all paging volumes are equally or almost equally %Used turn off and later remove the newest paging volume (in this case paging01).

MarcoPolo:/root> swapoff /dev/paging01; rmps paging01

In your case the additional paging volume most likely will not be named paging01, this is system dependent.