Skip to content


HDS (USP_V) hdisk/controller relationship

For AIX host with HDS (dlnkmgr) delivered storage, especially when there are multiple HDS controllers in use, it may be important to be able to identify what controllers deliver exactly what hdisks.

The following command delivers the answers:

# dlnkmgr view -lu -item -c
Product S/N     LUs iLU    SLPR HDevName VG    Paths  OnlinePaths
USP_V   0029386  14 00000A    0 hdisk14  -          8           8
                    00000D    0 hdisk15  -          8           8
                    000013    0 hdisk16  -          8           8
                    000019    0 hdisk17  -          8           8
                    00001D    0 hdisk18  -          8           8
                    00001E    0 hdisk19  -          8           8
                    00002B    0 hdisk20  -          8           8
                    00002E    0 hdisk21  -          8           8
                    00002F    0 hdisk22  -          8           8
                    000030    0 hdisk23  -          8           8
                    00004C    0 hdisk24  hb1_vg     8           8
                    000052    0 hdisk26  -          8           8
                    000065    0 hdisk29  -          8           8
                    000066    0 hdisk30  -          8           8
USP_V   0048835  16 00000A    0 hdisk2   epcdbm_vg  8           8
                    00000E    0 hdisk3   epcdbm_vg  8           8
                    00000F    0 hdisk4   epcdbm_vg  8           8
                    000010    0 hdisk5   epcdbm_vg  8           8
                    000011    0 hdisk6   epcdbm_vg  8           8
                    000012    0 hdisk7   epcdbm_vg  8           8
                    000013    0 hdisk8   epcdbmjrnl_vg 8        8
                    000018    0 hdisk31  -          8           8
                    00002D    0 hdisk9   epcdbmjrnl_vg 8        8
                    00002F    0 hdisk10  epcdbm_vg  8           8
                    000037    0 hdisk11  epcdbm_vg  8           8
                    000038    0 hdisk12  hb2_vg     8           8
                    000039    0 hdisk13  mksysbvg   8           8
                    000052    0 hdisk25  epcdbm_vg  8           8
                    000058    0 hdisk27  -          8           8
                    000059    0 hdisk28  epcdbm_vg  8           8
KAPL01001-I The HDLM command completed normally. Operation name = view, completed on time = 2012/03/01 13:12:01

This command is very useful why trying to establish mirroring with disks from appropriate controllers. It is easily identifies disks, volume groups and HDS controllers managing the disks.

Posted in Real life AIX.

Tagged with , , , .


local print queue removal in AIX

I have a ticket in my work queue to remove a considerable amount of print queues. Following the “usual” path executing smitty rmque I get an unexpected message:

There are currently no additional SMIT screen entries available for this item.  This item may require installation of additional software before it can be accessed.

Well, I do not have time to identify and to install this “additional” software, this has to wait for tomorrow. Today, the selected print queues have to be deleted. AIX deletes print queues in two steps. First, it remove the definition of the devices that does the actual printing and then it proceeds deleting the queue associated with the device removed in step one.

The command called lsque followed with the name of a print queue produces all the details we may need, for example:

lsque -c -qspdB_land
#name:device:discipline:up:acctfile:host:s_statfilter:l_statfilter:rq
spdB_land:@spdB_land:fcfs:TRUE:FALSE:spdB_land:/usr/lib/lpd/aixshort:/usr/lib/lpd/aixlong:spdB_land

The -c flag in the last command is responsible for the output being “colon” delimited. The first token repeats the queue name while the second delivers the name of the associated device.

Now the script:

#!/usr/bin/ksh

queue=$1

if [[ $# -ne 1 ]]
then
            print "\tMissing queue name, aborting."
            exit 1
fi

lsque -c -q$queue | grep $queue | awk -F ':' '{print $2}' | while read dev
do
            rmquedev -d $dev -q $queue 2>/dev/null
            rmque -q $queue 2>/dev/null
done

This script takes one argument – the name of the print queue to be removed.

Posted in AIX, Real life AIX.

Tagged with , , , , , .


installing Shadow Image on AIX host

An active node in a cluster has mirrored volume groups which data we want to backup with a minimal interruption (downtime). Each mirror consists of SAN disks (LUNs) from one of two SAN fabrics. One node in this cluster has ShadowImage environment configured to use disks from one of these fabrics (PVOLs and SVOLs belong to the same fabric). Due to site/host migration/re-configuration ShadowImage has to be “moved” (re-installed) on the other node of the same cluster using disks from the other SAN fabric. ShadowImage has to be installed and configured accordingly. This post documents how this is done.

Posted in Real life AIX.

Tagged with , , , , , , .


removing orphaned home directories

To remove a user account, we execute the rmuser command which by design does not remove the user’s home directory which from this moment on will show removed used id in place of his/hers login name. Eventually, with years of doing nothing a host home may end up like this one:

rwxr-xr-x    2 21564    lawson          256 Mar 04 2004  teagued
drwxr-xr-x    2 20337    lawson          256 Mar 04 2004  taylore
drwxr-xr-x    2 22391    lawson          256 Mar 04 2004  tantillo
drwxr-xr-x    2 258      lawson          256 Mar 04 2004  studnt9
drwxr-xr-x    2 256      lawson          256 Mar 04 2004  studnt7
drwxr-xr-x    2 255      lawson          256 Mar 04 2004  studnt6
drwxr-xr-x    2 305      lawson          256 Mar 04 2004  studnt50
drwxr-xr-x    2 254      lawson          256 Mar 04 2004  studnt5
drwxr-xr-x    2 294      lawson          256 Mar 04 2004  studnt40
drwxr-xr-x    2 253      lawson          256 Mar 04 2004  studnt4

It is time to clean. Today, this host was migrated from the local to the LDAP authentication and as the part of this process we decided to clean /home. After its execution only the locally authenticated accounts (files) will have their homes. The LDAP defined users home directories will be automatically created at the first login, thanks to the following entry in the /etc/security/login.cfg

mkhomeatlogin = true

And now the script which did the cleaning:

#!/usr/bin/ksh

for nbr in `ls -l /home | awk '{print $3}' | grep ^[0-9]`
do
        for usr in `ls -l /home | grep $nbr | awk '{print $9}'`
        do
                lsuser -R files $usr

                if [[ $? -ne 0 ]]
                then
                        rm -rf /home/$usr
                fi
        done
done

Posted in ldap, Real life AIX, scripts.

Tagged with , , , .


using NIM to install mksysb image – building spot from mksysb

Today, is the day to start migration of an Oracle cluster which ends in a pair of new lpars with all virtual resources (including two disks for rootvg) which are loaded with a mksysb image of the “current” Oracle servers (AIX 5.3.8.6). Our NIM master is running AIX 5300-08-10-1015. Yes, there is a “slight” difference between the nim client and its nim master and this process may and it probably will fail, but we are willing to take my chances. Well, we took our chances and we failed or to say it better the installation failed….. 🙂

So let’s start from a clean slate. We will remove the NIM client definition from the NIM server as shown next.

# nim -Fo reset MarcoPoloNew
# nim -o deallocate -a subclass=all MarcoPoloNew

0042-124 c_ch_nfsexp: NFS option public is NOT supported

# nim -o remove MarcoPoloNew
warning: 0042-140 m_rmmac: unable to remove the /etc/niminfo file on
"MarcoPoloNew"

# lsnim -l MarcoPoloNew
0042-053 lsnim: there is no NIM object named "reset MarcoPoloNew"

where the MarcoPoloNew represents the hostname of the new host we want to install.

Posted in AIX, Real life AIX.

Tagged with , , , , .


mounting ISO images in AIX6.1

Recently, I had to access contents of few ISO images. Onces for a developer and next because I had to start learning RedHat….. Yes, the “acceptance” of the DarkSide in already in progress.
If your host operating system is AIX 6.1 or above this task is easily done utilizing the “new” command called – loopmount – what a meaningfull name indeed!

loopmount -i IBM_Install_Toolkit.iso -o "-o ro -V cdrfs" -m /mnt

Above, the ISO image will be mounted and accessed via the /mnt directory.
By the way, lately I dream about penguins … Why?

Posted in Real life AIX.

Tagged with , , .


in-place editing of files

There are times, when mind is not exactly communicating with your hands – especially with their extremities called fingers….
There are times, when you wish you had some tool that could allow you to edit the mistakes of brain/fingers miscommunication on a number of hosts via some script….
AIX native version of sed is missing this ability but the “LINUX” version that you can install does it. So if your host has the /opt/freeware/bin/sed than the following will work for you:

sed -i -e 's/hello/jello/' hello.txt

Which replaces the first located hello with jello.

If you want to replace every hello with jello than do this instead:

 sed -i -e 's/hello/jello/g' hello.txt

But not all of us have this version of sed installed. What to do? You have perl. Using it to solve the previous leads to:

perl -pi -e 's/hello/jello/g' hello.txt

Do you know about any other way? If so, please let us now.

Today, I found a gem of simplicity in a shape of a comment. Marcin Wiśnos from unixwear.blogspot.com showed me how to use the tool (vi) that is synonymous with UNIX to do what I described above. Apparently, there is no need for GNU sed or perl… 🙂

# vi +%s/ferris:/ferrisj:/g +wq user
"user" 13120 lines, 163733 characters "user" 13120 lines, 163734 characters
# grep ferrisj user
ferrisj:

# grep -p ferrisj user
ferrisj:
SYSTEM = LDAP
registry = LDAP
loginretries = 0

It works like a charm, thanks Marcin!

Posted in Real life AIX, scripts.

Tagged with , , , , .


FC Adapters and the associated with them devices (disks and tapes)

I have been asked this question already a few times and I expect the same question to be repeated over the next few days – we are switching from the physical to virtual tape libraries. So what FC disks and tape drives are associated with what FC adapters on this machine? Here is the answer in form of a script:

#!/usr/bin/ksh93
### W.M.Duszyk
### map disks/tapes to their FC adapters

set -A FC `lsdev -Cc adapter | grep FC | awk '{print $1" "$3}'`
print ""
cntr=0

for line in `echo ${FC[@]}`
do
        if [[ $cntr = 1 ]]
        then
                print -n "$FC: "
                devices=`lsdev -Cc disk | grep $line | \
                             awk '{print $1}'`

                if [ -z $devices ]          # if there are no disks
                then                        # look for tapes
                        devices=`lsdev -Cc tape | grep $line | \
                                     awk '{print $1}'`
                        if [ -n $devices ]  # if there are tapes
                        then                # print tapes
                                print $devices 
                        fi
                else                        # print disks
                        print $devices
                fi
                print ""
                cntr=0
        else
                cntr=$(($cntr+1))
        fi
    FC=$line
done

Executed, its output looks like this:

# ./mapDisksTapes.ksh

fcs1: hdisk6 hdisk7 hdisk8 hdisk9 hdisk46 hdisk47 hdisk48 hdisk49 hdisk50 hdisk51 hdisk52 hdisk53 hdisk54 hdisk55 hdisk92 hdisk93

fcs2: hdisk37 hdisk42 hdisk43 hdisk44 hdisk45 hdisk56 hdisk57 hdisk58 hdisk59 hdisk64 hdisk65 hdisk66 hdisk67 hdisk68 hdisk69 hdisk70 hdisk71 hdisk72 hdisk73 hdisk74 hdisk75 hdisk76 hdisk77 hdisk78 hdisk79 hdisk80 hdisk81 hdisk82 hdisk83 hdisk84 hdisk85 hdisk86 hdisk87 hdisk88 hdisk89 hdisk90 hdisk91

fcs3: hdisk10 hdisk11 hdisk12 hdisk13 hdisk14 hdisk15 hdisk16 hdisk17 hdisk18 hdisk19 hdisk20 hdisk21 hdisk22 hdisk23 hdisk24 hdisk25 hdisk26 hdisk27 hdisk28 hdisk29 hdisk30 hdisk31 hdisk32 hdisk33 hdisk34 hdisk35 hdisk36

fcs4: rmt0048f203 rmt0048f204 rmt0048f206 rmt0048f208 rmt0048f20a rmt0048f20c rmt0f491802 rmt0f491804 rmt0f491806 rmt0f491808 rmt0f49180a rmt0f49180c smc00f49180c

fcs5: hdisk38 hdisk39 hdisk40 hdisk41

fcs6: rmt0048f201 rmt0048f202 rmt0048f205 rmt0048f207 rmt0048f209 rmt0048f20b rmt0f491801 rmt0f491803 rmt0f491805 rmt0f491807 rmt0f491809 rmt0f49180b smc00048f201 smc00f491801

I do scripts occasionally, do not look for finesse here. Let me know if you have a better way – I am always willing to learn 🙂

UPDATE:
A few hours after I published this post, one reader left here a comment showing a very different approach. Later at night, I could not sleep so I started to look at this “issues” again. Looking at Radu code, I came with my equivalent which generates the same output but using fewer lines of code. See for yourself:

#!/usr/bin/ksh

lsdev -F name | grep fscsi | while read fc_adapter;do
           echo $fc_adapter: `lsdev -p $fc_adapter -F name | \
                                      egrep -e "hdisk|rmt" `
           echo "\n"
done

Today, looking at these two samples, it seems to me like the first one is a programmer’s work and the second adminstrator’s….. 🙂

Posted in Real life AIX, scripts.

Tagged with , , , .


Methods for Identifying Memory Leaks in AIX Systems

The previous post is the result of Adi’s question about a particular Oracle process memory (virtual) consumption. Next, “ironman” suggested using “workloads” to answer such question. This resulted in my brain generating the following thought – “is it possible to use workload(s) to restrict consumption of resources” to prevent the “not enough memory to fork a process” message from showing up again? Well, you know what the implication of this message, right? For you who has never seen it the message indicates that it could already be too late to login and all that you can do is reboot the box ……….
Well, we gona try it with Adi sometime next week – I will let you know how did it work. By the way, today on a test LAWSON server, I noticed paging space utilized at 63% (suddenly, and growing) with JAVA being the main offender so this “host/application component” could become a second candidate for using a workload to arrest unlimited resources consumption by an application?

While searching for workload wisdom, I found another little gem which is responsible for the title of this post. I found a document authored by two IBMers Barry J. Saad and Harold R. Lee titled “Methods for Identifying Memory Leaks in AIX Systems“. This document not only sheds light on the heap and memory allocation techniques but it provides tools that every AIX administrator can use to identify the presence of a “leaky” application. The simplicity of this document in comparison to the built-in difficulty and complexity of its scope is simply mind boggling. It takes two subject matter masters to write a technical document that even a preschooler is able to follow and to understand – no kidding.

By the way, the authors use ps -gv pvid command to obtains the current value (under the SIZE colum) of the virtual memory (RAM + PAGING) used by a process. This value is expressed in units of KB not pages!

Posted in AIX, WLM.

Tagged with , , , , , .


aix process memory consumption – how much does it use?

It is Friday night, and I am back to the americanbulldogrescue.org. We used to have two boxers, one was white and many mistook Kajtek for an american bulldog…. I am looking at the dogs waiting for an adoption trying to come with an excuse I could use to convince my wife. I already tried “this is the best intruder detection “system” which does not a require a backup power supplies ……” – I failed short. She just gave me the look number FIVE!

While I am looking at the available pooches, Adi “reaches and touches” me via the MS Communicator; “Mark, are you there?”. He has some issues with a database server and ask me few questions about memory consumption. “Do you know if DSMC agent really should be using “so and so” memory? Where the “so and so” refers to a specific number expressed in MBs. Well, to be honest I have no idea. So we chat for a few more minutes, I kill some processes and then to make the long story short I offer to reboot this machine which I can do at will since it is not a production one – low level testing server which currently see no usage. Adi agrees and I reboot the host, text him good night and I get out and into my bed.

The question remains, how to establish memory usage of a single process? The most difficult aspect of this question is to remember that the value found is expressed in pages! So if you now the process ID (the number in the second column generated executing ps -ef | grep particular_process_name) you can find its memory consumption like that:

svmon -P 17367068 | more
----------------------------------------------------------------
Pid Command   Inuse      Pin     Pgsp  Virtual 64-bit Mthrd  16MB
17367068 java   440648     9786        0   435928      Y     Y     N

PageSize         Inuse        Pin       Pgsp    Virtual
s    4 KB        31080       1450          0      26360
m   64 KB        25598        521          0      25598

The same could be rephrased or its subject extended to for example – “what are this system the five largest memory consumers”? There are at least two ways to answer this question.

>svmon -Pt5 | perl -e 'while(<>){print if($.==2||$&&&!$s++);$.=0 if(/^-+$/)}'
----------------------------------------------------------------
Pid Command   Inuse      Pin     Pgsp  Virtual 64-bit Mthrd  16MB
17367068 java  442000  9786      0   435928      Y     Y     N
13893728 java  423720  9663      0   417884      Y     Y     N
17957084 java  383216  9220      0   379021      Y     Y     N
10158108 java  219702  8445      0   218189      Y     Y     N
15990974 kulagent 215901  8380   0    44549      Y     Y     N

Yes, I do not use the previous command as I am old and my memory is not what is used to be….. I use this instead.

svmon -Pt5 | grep -p Pid
-----------------------------------------------------------------
Pid Command   Inuse      Pin     Pgsp  Virtual 64-bit Mthrd  16MB
17367068 java 441251  9786        0   435927      Y     Y     N

-----------------------------------------------------------------
Pid Command   Inuse      Pin     Pgsp  Virtual 64-bit Mthrd  16MB
13893728 java 422329  9663        0   417884      Y     Y     N

-----------------------------------------------------------------
Pid Command   Inuse      Pin     Pgsp  Virtual 64-bit Mthrd  16MB
17957084 java 384788  9220        0   379021      Y     Y     N

-----------------------------------------------------------------
Pid Command   Inuse      Pin  Pgsp  Virtual 64-bit Mthrd  16MB
10158108 java 219710 8445        0   218189      Y     Y     N

------------------------------------------------------------------
Pid Command   Inuse      Pin  Pgsp  Virtual 64-bit Mthrd  16MB
15990974 kulagent  215932 8380        0    44549      Y     Y     N

The output is not as compact but at least the command is easy to remember. If you know about the other ways and you feel like doing so, please left a comment.
By the way, if you are interested in finding the five top most users of your host paging space, execute svmon -Pgt 5.

UPDATE:

In one of the comments, ironman suggested using the memdetails.sh script that is part of the perfpmr package (free downloads from ftp.software.ibm.com) and/or the workload manager to accurately defined the amount of memory used. I downloaded and executed the script and and the amount of information its generates left me speechless and scratching my head – I do not think it can be used to list usage of a specific process. On the other hand, I am about to believe that ironman’s second suggestion is more of what I am looking for and need. You may start learning about workloads (if you are like me and are new to workloads) reading Nigel Griffith’s presentation titled “Setting up AIX Workload Manager in 30 minutes“. There is also a RedBook wholly dedicated to this subject titled “AIX 5L Workload Manager (WLM)”.

Posted in Real life AIX.




Copyright © 2016 Waldemar Mark Duszyk. All Rights Reserved. Created by Blog Copyright.