Skip to content

10Gbit Ethernet, bad assumption and Best Practice

Lately, we have some issues with network performance among our virtualized hosts. We found two posts by the same author Gareth M. Coates which are priceless.
The first one can be read here Good info indeed.

The second post by the same author, you can read here:

The next page is a printout of Gareth’s first post.

Posted in Real life AIX.

(re-)installing OSP’s VMware “tools” on a RedHat guest

VMware tools are not running on one of our guests and Igor, the creator of our VMware template is not around …. This post documents the process that I followed (after a few mistakes, u-turns, etc) to have these tools re-installed from the external repository instead of the outdated local one.

Posted in LINUX.

Tagged with , .

VIO Network Performance Tip

What follows is an excerpt from the February IBM News Letter USA East which you can read following this link

By Doug Herman –
When using the Virtual I/O server (VIO) for virtualizing physical networks to client logical partitions, a Shared Ethernet Adapter (SEA) is configured by using the physical Ethernet adapter and the virtual Ethernet adapter to create a layer 2 bridge. One way to improve network performance is to use the largesend option on the VIO SEA and the client logical partitions. The largesend feature allows sending large data packets over virtual Ethernet adapters without breaking up the packets into smaller MTU size packets. Starting with AIX 6.1 TL7-SP1 and AIX 7.1 TL0-SP1, the operating system supports the mtu_bypass attribute for the shared Ethernet adapter to provide a persistent way to enable the largesend feature:

Using largesend (mtu_bypass) on the AIX interfaces boosts throughput between logical partitions within the hypervisor of the Power server, without using additional processor utilization. Set largesend on the VIO SEA, and mtu_bypass (largesend) on the AIX LPAR interfaces. This lowers both the sending AIX LPAR and the sending VIO processor usage when transferring to an outside machine. All MTU sizes remain at 1500. There is no requirement for Jumbo Frames.

Some examples of largesend attributes for performance:

# ifconfig en0 largesend 

(LPAR to LPAR, virtual to virtual, in same machine single stream, binary FTP dd test) 1Gb per second without largesend 3.8Gb per second with largesend – Higher throughput
Processor utilization slightly higher on sender and slightly lower on receiver

largesend=1 on VIO SEA and largesend on client interfaces

– Much lower processor utilization on sender and on sending VIO

VIO SEA physical adapters should have both large_send and large_receive set to yes

$ lsdev -dev ent0 -attr |grep lar

large_receive yes Enable receive TCP segment aggregation True
large_send yes Enable hardware Transmit TCP segmentation

To make change settings permanent:

VIO Server:

$ chdev -dev ent# -attr largesend=1 large_receive=yes


# chdev –l en# -a mtu_bypass=on

Posted in Real life AIX.

Tagged with , .

fixing missing paths to SAN disks – part 2

In the past I posted information showing how to “activate” paths that for some reason were no longer on-line. I used the combination of lspath command to weed our the Failed paths and the chpath command to change their state to the Enabled one.
Recently we cleaned the connectors of our FC fabric and as it was expected some of the effected paths switched state to Failed. As the number of paths to a disk (LUN) decreases the I/O to this disk suffers too.

We noticed that this time the described earlier procedure did not work … To re-activate the missing paths we removed the associated with them FC adapter, immediately followed with execution of the “configuration mangler” as we affectionately call here the cfgmgr command.

As a sort of a remainder: to check the state of paths in the XIV environment you can use the command xiv_devlist. In the listing bellow, hdisk7 and hdisk15 show missing paths.

# xiv_devlist
XIV Devices
Device        Size (GB)  Paths  Vol Name        Vol Id   XIV Id   XIV Host
/dev/hdisk2   10.7       12/12  BIN_MS  243      7801370  BINORTPU001
/dev/hdisk3   10.7       12/12  BIN_E6  364      7802518  BINORTPU001
/dev/hdisk7   1754.5     6/12   BIN_E1  367      7802518  BINORTPU001
/dev/hdisk15  532.6      6/12   BIN_D1  371      7802518  BINORTPU001
/dev/hdisk16  532.6      12/12  BIN_D1  263      7801370  BINORTPU001

The previous post showed this method of located the FC adapters associated with the missing paths. You could also use the lspath command.

# lspath -l hdisk7 -H -F"name parent path_id connection status"
name   parent path_id connection                     status
hdisk7 fscsi0 0       5001738009d60142,5000000000000 Enabled
hdisk7 fscsi0 1       5001738009d60182,5000000000000 Enabled
hdisk7 fscsi0 2       5001738009d60162,5000000000000 Enabled
hdisk7 fscsi1 3       5001738009d60142,5000000000000 Enabled
hdisk7 fscsi1 4       5001738009d60182,5000000000000 Enabled
hdisk7 fscsi1 5       5001738009d60162,5000000000000 Enabled
hdisk7 fscsi2 6       5001738009d60150,5000000000000 Enabled
hdisk7 fscsi2 7       5001738009d60190,5000000000000 Enabled
hdisk7 fscsi2 8       5001738009d60170,5000000000000 Enabled
hdisk7 fscsi3 9       5001738009d60150,5000000000000 Failed
hdisk7 fscsi3 10      5001738009d60190,5000000000000 Failed
hdisk7 fscsi3 11      5001738009d60170,5000000000000 Failed

Now, to selectively enable the paths execute:

# chpath -s enabled -l hdisk7 -p fscsi3
# lspath -l hdisk7 -H -F"name parent path_id connection status"
name   parent path_id connection                     status
hdisk7 fscsi0 0       5001738009d60142,5000000000000 Enabled
hdisk7 fscsi0 1       5001738009d60182,5000000000000 Enabled
hdisk7 fscsi0 2       5001738009d60162,5000000000000 Enabled
hdisk7 fscsi1 3       5001738009d60142,5000000000000 Enabled
hdisk7 fscsi1 4       5001738009d60182,5000000000000 Enabled
hdisk7 fscsi1 5       5001738009d60162,5000000000000 Enabled
hdisk7 fscsi2 6       5001738009d60150,5000000000000 Enabled
hdisk7 fscsi2 7       5001738009d60190,5000000000000 Enabled
hdisk7 fscsi2 8       5001738009d60170,5000000000000 Enabled
hdisk7 fscsi3 9       5001738009d60150,5000000000000 Enabled
hdisk7 fscsi3 10      5001738009d60190,5000000000000 Enabled
hdisk7 fscsi3 11      5001738009d60170,5000000000000 Enabled

As I mention above, this time the procedure could not “fix” all our issues with the Failed paths. So, we resolved to something more drastic, which in this case was

# rmdev -dl fsc3 -R; cfgmgr

These two steps were executed against each adapter with missing paths, one adapter at a time. This procedure worked for all of our cases – XIV, SVC and HDS.

Posted in Real life AIX.

Tagged with , , .

read iso images without a cd-drive

After gaining the root access to the host (see the last post) it is the time to finally upgrade its disk drivers … In this case, they landed on my desktop in form of an iso image. This is all nice and dandy but the host does not have a cd/dvd drive … – this post shows how to unpack an iso image without it.

Use, whatever is appropriate to move the iso image to the host that needs it. Next, create a file system (I call it /iso) of the appropriate size and with appropriate attributes (-pro, -tno).

# crfs -v jfs2 -g vg_name -a size=5G -m /iso -Ano -pro -tno

Since, I allowed LVM to create the logical volume creating the file system above, now it is time to learn its name:

# lsfs /iso | grep fslv

In the next line the if= defines the location of image, and the of defines just identified logical volume. To load the image into the file system we used the command as old as UNIX:

# dd if=/home/duszyk/dlmglm_074001.iso of=/dev/fslv00/ bs=100M

Now, mount the file system (mount /iso) and explore its contents.

It came quick and swift – I am talking about the two comments to this post 🙂 . Thank you Marcus and Ku!!!! For the reader – if you are on AIX 6.1.4 and above, please user the loopmount command instead of creating a file system and populating it with the image with dd command!

By the way, see one of the previous post for loopmount command –
It looks like I need to add a few GB’s to my own memory ….. 🙂

Posted in AIX, Real life AIX.

Tagged with .

recovering forgotten root password with NIM

A decision was made that the fastest way to answer the need for a new AIX host will be to re-purpose an existing dormant partition. After CPU and RAM quantities were set in its profile the partition was powered-on. The next came the discovery that the currently user root password does not work here …… With a working NIM server nothing seems to be a problem :-).

What follows documents the process which with help of a NIM server allow you to reset root password in AIX – it does not matter if this is a stand alone or a partition, the procedure remains the same.

Posted in Real life AIX.

Tagged with , , .

password grammar in LINUX – PAM style

In AIX to control password grammar (composition rules), expiration and all other not mentioned here “attributes”, both on a local or global scale is very easy. All is collected and ready to be manipulated to satisfy your fancy in a single file called user located in directory called /etc/security.

This file as many other files in AIX is based on stanzas. There is the default stanza where all attributes are set to their “default” values. This stanza is followed by any number of other stanzas, each representing a particular user whose password/login attributes differ from the ones set in the default stanza. It is nice and easy. Does it answer all needs in this area? Maybe not, most likely not. Is it dated solution? It could be. In my opinion it is clean and efficient and if anything else is called for in addition and above what this file provides, one has to remember that AIX has PAM,yes AIX supports PAM. Having said that, I admit that I have never used PAM with AIX and have no idea do what extend PAM is supported by this operating system. What PAM stands for? PAM is short for the Pluggable Authentication Modules.

Here comes LINUX. A few days ago, I started to think about RedHat – How to set a different set of password controls on a RedHat host with an configured and operational Active Directory authentication? What to do to apply a different set of password attributes for the local users? This post is a way to share with you what I have so far discovered. As always, let me know if you find an error, omission or a better solution to what I have shown here.

In RedHat to make sure that the local users (the ones present in the file /etc/passwd) are authenticating by the local means and not via AD or LDAP or something else you have to check the files inside directory called /etc/pam.d for the presence of the following entry:

account       sufficient   

On my host, these entry is present in a few files like fingerprint-auth, password-auth, password-auth-ac, smartcard-auth, system-auth – not all of them are actually needed as for example this host does not employ any finger print recognition device to authorize the login-in user. I think, that for most cases it is enough to have this entry in both files which names start with password.

Next comes the grammar. To make sure that the local users passwords are sufficiently strong and they contain the correct ratio of numerals/upper/lower/special characters and so forth we look into the same directory again, but now seeking a different PAM module. This time we look for a PAM module called, which we find in the file called password-auth. For example the following entry:

password	required \
                    dcredit=-1 ucredit=-1 lcredit=-1 minlen=8

requires a password to contain at the minimum one (1) digit (dcredit), one (1) upper case (ucredit) character, one (1) lower case (lcredit) character and be not shorter than (minlen 8 characters. Not to mention that this module will automatically check the selected password against a dictionary an if it is found to be a word …. guess what? It will be refused and the user will have to specify to select a different one.

Now, what about the previous passwords? How to prevent their repetitive usage? First, verify that this file is present:

# ls -l /etc/security/opasswd
-rw-------. 1 root root 0 Apr  5  2012 /etc/security/opasswd

This file is the “keeper” of users previous passwords. The number of previous password per user stored inside this file is controlled by the value of the token called remember – thanks the developer for using meaningful terms!!!!!!!!!! The next line shows this setting in action (an excerpt form the file /etc/pam.d/password-auth).

password    sufficient \
                 sha512 shadow nullok try_first_pass use_authtok \

On this host, user is not allowed to repeat any of his four last passwords. By the way, the maximum for remember is 400 (past passwords).

For now this is all I have, stay warm,


Posted in Linux.

Tagged with , , .

iptrace as a service

There are many tools for observing the activity, both normal and pathological,on the network. Some run under AIX, others run on dedicated hardware.
One tool that can be used to obtain a detailed, packet-by-packet description of the LAN activity generated by a workload is the combination of the iptrace daemon and the ipreport command. The iptrace daemon can only be started by root. By default, iptrace traces all packets.

An option (-a) allows exclusion of address resolution protocol (ARP) packets. Other options can narrow the scope of tracing to a particular source host (-s), destination host (-d), or protocol (-p). For more information, see AIX Commands Reference.

Because iptrace can consume significant amounts of processor time, you should be as specific as possible in describing the packets you want traced. Since iptrace is a daemon, it should be started with a startsrc command rather that directly from the command line. This makes it easier to control and shut down cleanly. A typical invocation would be:

 # startsrc -s iptrace -a "-a -i en0 /user/iptrace/log1" 

For more information on all the available options, read the iptrace manual page in the AIX documentation. This command starts the iptrace daemon with directions to trace all activity on the interface, en0, and place the trace data in /user/iptrace/log1.

To stop the daemon, use:

 # stopsrc -s iptrace 

If you hadn’t started it with startsrc, you would have to find its process ID with ps and kill it.

The ipreport command is a formatter for the log file. Its output is written to stdout. Options allow recognition and formatting of RPC packets (-r), identifying each packet with a number (-n), and prefixing each line with a 3-character string that identifies the protocol (-s). A typical ipreport command to format the log1 file just created (which is owned by root) would be:

 # ipreport -rns log1 > log1_formatted

This would result in a sequence of packet reports similar to the following example. The fields of most interest are: the source (SRC) and destination (DST) host address, both in dotted decimal and in ASCII; the IP packet length (ip_len); and the indication of the higher-level protocol in use (ip_p).

the ‘stuff” above is a quote from an old Bull (France) AIX manual.

To see how to start iptrace as a process follow to this post “iptrace on AIX host

Posted in Real life AIX.

Tagged with , , , .

changing the value of GID – its consequences

Often during migration from a local to a global authentication mechanism system administrator discovers that the same user group has a different numerical GID on one or more hosts. To change it system administrator either edits the file /etc/group or he/she could executes the command called chgroup.
The results of this action are quickly apparent as immediately after such change every file object which belongs to the group will show instead of the group name its previous GID value.

Now, system administrator has to comb through the file systems looking for each file/directory showing the old GID and replace it with group name.

Another AIX administrator, before the change of the group GID may collect the /path/file_name of every file owned by the group in a text file (find / -group Group_Name -ls >> FileList.txt) and use its contents after the GID change as follows:

for myFile in `cat FileList.txt | awk '{print $11}'`
        chgrp GROUP_NAME $myFile

One may quickly find out that if there is a substantial amount of files to process and the length of the command arguments list (ncargs) is not adequate the AIX shell fill refuse to execute these directives responding with the messages of “Not enough memory to execute“.

Before you start looking into one of my earlier posts suggesting how to deal with such outcome, you could try to replace the for loop with its while equivalent.

cat FileList.txt | while read line
        myFile=`echo $line | awk '{print $11}'`
        chgrp GROUP_NAME $myFile

You might be surprised but this time it may work like a charm.

Before we call it a day, we have to remember that any user defined in /etc/passwd whose primary group GID was modified still shows the “old” GID! So any users whose primary group was the one you modified will not be allowed to login!
We could use vi to open the /etc/passwd file and execute (for example) %s/333/335/g. If we did just that, what would happened to the group with GID od 2333 that is also present in this file? Its GID equals now 2335 which is not what it should be, right? So the “proper” way to change GUID is to execute %s/:333:/:335:/g instead.
Finally, I was really surprised when I noticed that GID change “sometimes” remove the (primary group) from among the user attributes. If you do the lsuser -R files .... you may not find pgrp= in the output. Fortunately, the chuser -R files pgrp=......... resolves this one and everything is back to normal.

Posted in Real life AIX.

Tagged with , , .

login with no password LINUX style

Back in the September of the previous year, I put out a post with the procedure outlining the process of setting a password-less logins to other AIX hosts. This very procedure can also be applied to whatever OS it is as long as it supports ssh/ssl.

Today, I had to harvest CPU/RAM information from a large number of LINUX machines and I decided to use the same procedure – I will set one LINUX host so from it I will be able log-in with no password or execute a command also without being asked to authenticate. As I was happily getting started with my tasks, Mike the Red Fedora wearer (really he comes to the office wearing a red fedora) Swierczynski (“ski” for short) entered my cube and after a glance at my screen asked me if I am aware that LINUX has a better tool for the task at hand. What is it Mike?

“LINUX has a command called ssh-copy-id that works for LINIX to LINUX to AIX to SOLARIS, and which will/can set the whole environment so even if a target hosts does not have the .ssh setup in the user’s home directory this command will make sure this entities will be created and populated so the user will not be asked to authenticated at login or at the instant of a command executed remotely on his behalf and carried by SSH.”

In my case, to enable a passwordless root login from hostA to hostB, I executed the following command

# ssh-copy-id -i ~root/.ssh/ rootB@host_name

Actually this command became a body of a for loop which was fed with host names from a file (some twenty or more) called hostsList.

hostA#  for host in `cat hostsList`; do ssh-copy-id -i .ssh/ root@$host; done

I almost forgot, Mike also showed me the dmidcode which spits out page after page of info which usefulness I have yet to learn.

By the way, the file was created with the same command as in September:

# ssh-keygen -t rsa

a good day everybody

Posted in Linux, LINUX, Real life AIX.

Copyright © 2016 - 2017 Waldemar Mark Duszyk. All Rights Reserved. Created by Blog Copyright.