Skip to content


Building SystemMirror 7.1 cluster

Many months ago, I attempted to migrate from 6.1 5o 7.1 and all my attempts failed one by one. The cluster would not sync and there seem to be no way to get it up and running. Eventually, I gave up, put it on a “back burner”. A week ago I had time and the nodes in the “cluster” had time for another shot.

I started by removing cluster definition (using smitty hacmp) followed by removal of all “hacmp cluster” related file sets. Next I installed them again (SystemMirror 7.1.3.0) followed with upgrade_all using the 7.1.3.2 code. At this time, I and re-created /etc/hosts and /etc/cluster/rhosts, I made sure they were identical on each node.

Each node had the same hostname/uname as the label in /etc/hosts associated with it boot IP address. Next, since our bootable addresses are not routable, each node received an IP alias on the same network as the “service” address followed with setting the gateway address on the bootable interfaces. Yes, both nets use the same netmask!

Reboot both nodes start clcomd and configure the cluster. It took a few sync failures before the Sun start shining in my neck of the woods. Few first sync's failed with no apparent reason (asking to contact IBM…) but I noticed that there was no heartbeat volume group and the associated with it file system aka caavg_private. What helped is shown bellow. You guessed it – it was executed on each node.

# export CAA_FORCE_ENABLED=1
# rmcluster -f -r hdisk9
# rmcluster -f -r hdisk10
# rmdev -dl hdisk10
# rmdev -dl hdisk9
# cldare -rtV normal

# shutdown -Fr

Originally (and the plan still has not been changed), I set hdisk10 as the heartbeat disk and hdisk9 as its backup so I went back to the cluster configuration menu and set them up again.

The next sync almost succeeded but it failed as an entry was found missing in the /etc/snmpv3.conf file. Why in this file? I had no idea – we use snmp ver.2. But I followed and added the missing entry. Here it is:

smux 1.3.6.1.4.1.2.3.1.2.1.5 clsmuxpd_password

Another sync, which took really long and the long awaited OK prompt showed up! Next week, I have another 7.1.3.2 cluster to build.
But this time I will try to set it up via a command line alone – I have never done it, it should be fun!
🙂

Posted in HACMP, Real life AIX.

Tagged with , , .


3 Responses

Stay in touch with the conversation, subscribe to the RSS feed for comments on this post.

  1. Shawn Bodily says

    Hey check out my demos on upgrading on youtube.

    https://www.youtube.com/user/PowerHAguy

    Also another good tip, especially when creating the cluster fails on creating the CAA cluster, is change the CAA entry in syslogd.conf from info to debug.

  2. MarkD:-) says

    Shawn,

    thanks for your comment and link!
    I will watch it.

    Mark

  3. Israel Garcia says

    Hi,
    The same happened to me..
    After a nightmare migrating our cluster powerha-glvm from 6.1SP12 to 7.1.3SP3 because of the mkcluster fails.. the only solution was to install this apar on 3 nodes ( IV61060m3b.150128.epkg.Z) and finally the migration completed. 😉
    Instead of install this apar, we upgrade to latest AIX6.1TL9SP4 from fixcentral.

    cheers.



Some HTML is OK

or, reply to this post via trackback.

WordPress Anti Spam by WP-SpamShield



Copyright © 2016 - 2017 Waldemar Mark Duszyk. All Rights Reserved. Created by Blog Copyright.