Skip to content

Building SystemMirror 7.1 cluster

Many months ago, I attempted to migrate from 6.1 5o 7.1 and all my attempts failed one by one. The cluster would not sync and there seem to be no way to get it up and running. Eventually, I gave up, put it on a “back burner”. A week ago I had time and the nodes in the “cluster” had time for another shot.

I started by removing cluster definition (using smitty hacmp) followed by removal of all “hacmp cluster” related file sets. Next I installed them again (SystemMirror followed with upgrade_all using the code. At this time, I and re-created /etc/hosts and /etc/cluster/rhosts, I made sure they were identical on each node.

Each node had the same hostname/uname as the label in /etc/hosts associated with it boot IP address. Next, since our bootable addresses are not routable, each node received an IP alias on the same network as the “service” address followed with setting the gateway address on the bootable interfaces. Yes, both nets use the same netmask!

Reboot both nodes start clcomd and configure the cluster. It took a few sync failures before the Sun start shining in my neck of the woods. Few first sync's failed with no apparent reason (asking to contact IBM…) but I noticed that there was no heartbeat volume group and the associated with it file system aka caavg_private. What helped is shown bellow. You guessed it – it was executed on each node.

# rmcluster -f -r hdisk9
# rmcluster -f -r hdisk10
# rmdev -dl hdisk10
# rmdev -dl hdisk9
# cldare -rtV normal

# shutdown -Fr

Originally (and the plan still has not been changed), I set hdisk10 as the heartbeat disk and hdisk9 as its backup so I went back to the cluster configuration menu and set them up again.

The next sync almost succeeded but it failed as an entry was found missing in the /etc/snmpv3.conf file. Why in this file? I had no idea – we use snmp ver.2. But I followed and added the missing entry. Here it is:

smux clsmuxpd_password

Another sync, which took really long and the long awaited OK prompt showed up! Next week, I have another cluster to build.
But this time I will try to set it up via a command line alone – I have never done it, it should be fun!

Posted in HACMP, Real life AIX.

Tagged with , , .

to reboot RedHat host in a future

it could be done with the at command or with crontab -e executed as root or using the plain old reboot provided with the appropriate time, like for example

#  nohup shutdown -r 13:00 &

The system is going down for reboot in 90 minutes!

Broadcast message from duszyk@sysmgttl1
        (unknown) at 12:00 ...

The system is going down for reboot in 60 minutes!

Broadcast message from duszyk@sysmgttl1
        (unknown) at 12:15 ...

The system is going down for reboot in 45 minutes!

Broadcast message from duszyk@sysmgttl1
        (unknown) at 12:30 ...

The system is going down for reboot in 30 minutes!

Broadcast message from duszyk@sysmgttl1
        (unknown) at 12:45 ...

The system is going down for reboot in 15 minutes!

Broadcast message from duszyk@sysmgttl1
        (unknown) at 12:51 ...

The system is going down for reboot in 9 minutes!

Broadcast message from duszyk@sysmgttl1
        (unknown) at 12:52 ...

The system is going down for reboot in 8 minutes!

Broadcast message from duszyk@sysmgttl1
        (unknown) at 12:53 ...

The system is going down for reboot in 7 minutes!

Broadcast message from duszyk@sysmgttl1
        (unknown) at 12:54 ...

The system is going down for reboot in 6 minutes!

The messages might be a bit annoying if not redirected but on the other hand they might server as a reminder….?

Posted in LINUX.

Tagged with , , , .

RedHat security patching for AIX administrator

Hi, this is a “reprint” from RedHat Knowledge base” that today I had found very useful.

• Red Hat Enterprise Linux 6.x
• Red Hat Enterprise Linux 5.1 and later
• Red Hat Network Hosted
• Red Hat Satellite


• Install the yum-security plugin. It is now possible to limit yum to install only security updates (as opposed to bug fixes or enhancements) using Red Hat Enterprise Linux 5 and 6. To do so, simply install the yum-security plugin:

For Red Hat Enterprise Linux 6

 # yum install yum-plugin-security

For Red Hat Enterprise Linux 5

 # yum install yum-security

Alternatively, download the yum-security package from the Red Hat Network (RHN) and manually install it on the system.

For Red Hat Enterprise Linux 6 using yum-security plugin:

• To list all available erratas without installing them, run:

# yum updateinfo list available

• To list all available security updates without installing them, run:

 # yum updateinfo list security all
 # yum updateinfo list sec

• To get a list of the currently installed security updates this command can be used:

 # yum updateinfo list security installed

For Red Hat Enterprise Linux 5 using yum-security plugin

• To list all available erratas without installing them, run:

# yum list-sec

• To list all available security updates without installing them, run:

 # yum list-security --security

For both Red Hat Enterprise Linux 6 and Red Hat Enterprise Linux 5:

• To list all available security updates with verbose descriptions of the issues they apply to:

 # yum info-sec

• Run the following command to download and apply all available security updates from Red Hat Network hosted or Red Hat Network Satellite:

 # yum -y update --security

NOTE: It will install the last version available of any package with at least one security errata thus can install non-security erratas if they provide a more updated version of the package.
• To only install the packages that have a security errata use

 # yum update-minimal --security -y

• yum-security also allows installing security updates based on the CVE reference of the issue. To install a security update using a CVE reference run:

 # yum update --cve <CVE>


 # yum update --cve CVE-2008-0947

Viewing available advisories by severities:

 # yum updateinfo list
This system is receiving updates from RHN Classic or RHN Satellite.
RHSA-2014:0159 Important/Sec. kernel-headers-2.6.32-431.5.1.el6.x86_64
RHSA-2014:0164 Moderate/Sec.  mysql-5.1.73-3.el6_5.x86_64
RHSA-2014:0164 Moderate/Sec.  mysql-devel-5.1.73-3.el6_5.x86_64
RHSA-2014:0164 Moderate/Sec.  mysql-libs-5.1.73-3.el6_5.x86_64
RHSA-2014:0164 Moderate/Sec.  mysql-server-5.1.73-3.el6_5.x86_64
RHBA-2014:0158 bugfix         nss-sysinit-3.15.3-6.el6_5.x86_64
RHBA-2014:0158 bugfix         nss-tools-3.15.3-6.el6_5.x86_64

If you want to apply only one specific advisory:

 # yum update --advisory=RHSA-2014:0159

However, if you would like to know more information about this advisory before to apply it:

 # yum updateinfo RHSA-2014:0159

For more commands consult the manual pages of yum-security with

 # man yum-security

If you face any missing dependency issue while applying security patches on system then refer to yum update --security fails with missing dependency errors.

Posted in Real life AIX.

reboot after patching?

The procedure described bellow applied to Linux (RedHat).
Even withing the same environment, patching done due to security concerns or something else has different meaning for different hosts. Some must be rebooted immediately to activate the “fixes”, while some may wait for a more appropriate occasion.

How to decide if a reboot can wait? Well, it depends (among others) on the location of the hosts. Is it in DMZ on not? Still, even if it is in DMZ a reboot may be delayed depending on the libraries effected by the “fix” (errata) and services using them.

For example, let’s say that we need to upgrade the glibc rpms due to just published errata CVE-2015-0235. To identify what services are using its functionality you could execute the following command:

$ lsof +c 15 | grep libc- | awk '{print $1}' | sort -u

From the resulting list, identify the public-facing services and restart them. Remember that while this process may work as a temporary workaround, it is not supported by Red Hat and, should a problem arise, you will be requested to reboot the system before any troubleshooting begins.

Posted in Linux.

How to monitor telnet traffic in AIX

1. Create /etc/security/authlog file containing the following lines:

/usr/bin/logger -t tsm -p "`/usr/bin/tty` login from $@ " 

2. Make it executable

# chmod +x /etc/security/authlog 

3. Modify the "/etc/security/login.cfg" file adding the following two lines just under the default: stanza.

program = /etc/security/authlog 

4. Modify the field "auth2" in the "/etc/security/user" file:

auth2 = authlog 

The above can be done for all the users by the default: or for a specific user modifying only the correspondent user stanza.

5. Configure syslogd to log the information:

# vi /etc/syslog.conf 
*.info /var/adm/authinfo.log

6. Create the logfile

# touch /var/adm/authinfo.log

7. Restart syslogd

# stopsrc -s syslogd 

# startsrc -s syslogd

8. Log in and check the authinfo.log:

# cat authinfo.log 

You should see the successful logins.

Posted in AIX.

Tagged with , , .

executing commands remotely from Satellite server

to be able to execute commands on clients of RedHat Satellite server you have to equip them with the following rpm

# yum -y install rhncfg-actions

Next, execute the following command on the host:

# rhn-actions-control -enable-all

Finally, check if this directory structure exists /etc/sysconfig/rhn/allowed-actions/script and that it contains an empty file called run

# ls -l /etc/sysconfig/rhn/allowed-actions/script
total 0
-rw-r--r-- 1 root root 0 Jan 29 08:27 run 

Posted in LINUX.

Satellite server – syncing and cloning

Satellite server is something like a NIM server plus much more… Satellite server has “Base” channels and associated with them “Clone” channels. The latter ones are the sources of operating system (RedHat Linux) images (rpms) that a system administrator uses to patch his/hers hosts. Usually, clone channels are not updated automatically – the Base channels, on the other had are usually synchronized with the RedHaNetwork automatically (cron).
There are many possible ways to synchrinise Clone with its Base channel. One way which result is a new Clonned channel syncronized up to a specific date (that you provieded) is shown bellow.

First make sure you have the latest packages in the already existing on yout Satellute server Red Hat channel (in this case called rhel-86_64-server-6), to sync this channel with the latest packages execute the next command.

# satellite-sync -c rhel-86_64-server-6

Once the packages are synced, run the spacewalk-clone-by-date utility to create its clone as of today.

# spacewalk-clone-by-date -u satadmin \
                   -l rhel-x86_64-server-6 
                   prod-rhel6u6-clone1224 \
                   -d 2014-12-24

The last command creates the new channel with name prod-rhel6u6-clone1224 containing the latest errata as of today.

Now, you have to associate your host with this particular clone and patch it with yum -y update .

In another post, I will show you how to synchronize an existing clone channel.

Posted in LINUX.

Tagged with , , , .

scp and hidden files

Today, I recognized that scp does not copy hidden files……. or that I do not know how to use this command to copy hidden files (directories).
Today, I also recognized that rsync is a very handy tool I have to learn how to use.

For example, to copy contents of one file system form one host to another file system on another machine, do:

# rsync -av /source_fs/ loginName@targetHostName:/target_fs/

Posted in AIX, Linux.

Tagged with , , .

uploading files to RedHat

Today, I had to attach files to my case (PMR) and doing it with my web browser kept failing so I had to use command lines to deliver the files.

# redhat-support-tool addattachment -c 01285064 ./spacewalk-debug.tar.bz2

were -c defines the case (PMR) number and the last argument specifies path/file to be uploaded to RH support.
One more thing, your host may not have this tool installed. In this case, execute

# yum install redhat-support-tool

Posted in LINUX.

Editing crontab from a script

Several ORACLE LINUX “unbreakable” kernel (patching requires no reboot) need their root crontab file modified to allow scheduled and “unattended” monthly patching.
The file containing the effected hosts (one host per line) is called eSOA.list.
The script, called Crons.bsh does the job. Here it is:

#! /bin/bash
CronFile='/tmp/RootCron.txt'             # define storage file

chkconfig crond on                       # make sure cron starts at boot
crontab -u root -l > $CronFile           # extract root crontab into the file
crontab -u root -l > RootCrontabSaved    # saved copy, just in case

echo "MAILTO=UnixTeam" >>$CronFile       # add this line
echo "0 5 1 * * yum -y update" >>$CronFile # add this line too

crontab -u root /tmp/RootCron.txt         # replace root crontab 
/etc/init.d/crond restart                 # restart crond 
rm $CronFile                              # remove storage file

# create a new alias to receive "patching" emails
echo "UnixTeam:," >> /etc/aliases
newaliases                                # activate new aliases

With the above in Crons.bsh, the process of implementing this change on the selected hosts is trivial.

# for h in `cat eSOA.list`
scp Crons.bsh $h:/root/
ssh $h '/root/Crons.bsh'
ssh $h 'rm /root/Crons.bsh'

The last step, was of course executed from the host that has the “password-less” priviledge to all other UNIX hosts in my data centers.

Posted in Real life AIX.

Tagged with , , .

Copyright © 2016 Waldemar Mark Duszyk. All Rights Reserved. Created by Blog Copyright.