Skip to content


Kerberos, Active Directory and ftp

It is not surprising that more and more users looks for Active Directory as the way to unify and to simplify the way to authenticate/authorize users and to save some money too.
After all Active Directory and Windows are almost in every office on this planet. Not to mention that if you work in a heavily audited environment the ability to have a single store of users definitions is really a blessing!

For about the last two months now, I am trying to get ftp working for AIX users authenticated with Kerberos services provided by Active Directory (2012).
I am not doing it alone, I have an open PMR and after numerous iptraces, snaps, and so forth my KERBEROS authenticated users still cannot use ftp…. and this sucks!
The locally (admin accounts) defined users can but the “flash & bone” users, the ones “living” in Active Directory cannot.

Is there someone out there who got this working and who is willing to share his/her knowledge, please?

Thanks,
MarkD:-)

Posted in Real life AIX.

Tagged with .


nim client removal

I have not done any patching for a while and today, when I had to remove a nim client definition I could not remember the second command to use. Now, I do so here it is for the record the process:

First, reset the client

# nim -F -o reset NIM_CLIENT_NAME

Now, remove all associated with the client resources.

# nim -o deallocate -a subclass=all  NIM_CLIENT_NAME

At this stage the client can be removed.

# nim -o remove -F  NIM_CLIENT_NAME

the NIM_CLIENT_NAME is the hostname of the client to be removed.

Posted in Real life AIX.

Tagged with , .


issues with a file system............

Last Friday evening, SAN administrator migrated disks of some hosts from one contraption to another. The effected AIX machines immediately lost their sanity and to get the file system to a usable state they all had to be rebooted. A day later, an application administrator sent out an email informing us that he has two hosts which are missing the same file system. These file systems are not “shared”, each of these two machines has its own disks (SAN delivered), the identical file system name is the only thing common to them both.

Since we follow the practice of making logical volumes and the associated with them file system names as close as they can be it was easy to figure out what is going in. Look at the output bellow:

# lsvg -l epcshreu001_vg
epcshreu001_vg:
LV NAME        TYPE   LPs  PPs     PVs  LV STATE      MOUNT POINT
epicbin_lv     jfs2   4       4    1    closed/syncd  #
engaudit_lv    jfs2   4       4    1    open/syncd    /epic/engaudit
epicprd_lv     jfs2   80      80   1    open/syncd    /epic/prd
epicjournal_lv jfs2   120     120  1    open/syncd    /epic/jrnlshde1

I decided to mount the file system using its appropriate name which was correctly represented as a stanza in /etc/filesystems. It worked like a charm 🙂

# grep -p /epic/bin: filesystems
/epic/bin:
        dev             = /dev/epicbin_lv
        vfs             = jfs2
        log             = INLINE
        mount           = true
        check           = false
        options         = rw
        account         = false
# mount /dev/epicbin_lv /epic/bin 

Inspecting this file system reveled its contents were intact!. I created a new filesystem and filled it with copy of the /epic/bin, just in case. Next, the /epic/bin and the other file systems in this volume group were unmounted, volume group varied off and exported. Nect,the volume group was imported and varied on and it still showed the strange looking # instead of the file system name (executing lsvg -l epcshreu001_vg).

The syncvg -v epcshreu001_vg and the syncvg -l epicbin_lv commands were executed and nothing changed…..

Here we go again, the file systems were all unmounted, the volume group varied off and exported. Next, make copy of the /etc/filesystems and inspection of this file. We are looking for the stanza /epic/bin:. Yes, it is there! But it should not be there – every time a volume group is exported this file is updated removing the file systems stanzas associated with the exported volume group! I think, this is how it works, right?

The stanza was removed, and the volume group imported and varied on. The next execution of the lsvg -l epcshreu001_vg shows that the offending # character has been replaced with the /epic/bin as it should be! The next command (mount all) mounts all the file systems and opens them for user access.

Now, what happened here? Is this the result of the recent SAN migration. No, no, no! I do not think so. I bet my dollar, that someday in the past manual edit of the file left behind some hidden “special” character behind, which prevented AIX from removing this file system stanza…… the SAN migration and the following it reboot just happened to expose this fact and then when the second host was built (copy of the first) one the /etc/filesystems was copied too…..

Posted in Real life AIX.

Tagged with .


Power7, SMT, CPU utilization, etc

There is a lot of room for misunderstanding CPU utilization with active SMT (either 2 or 4 threads). Lately, I am in situation where not only I have to know what is going on with CPU utilization but also I have to be able to show and explain it my clients and my bosses as well.
For all of you who need to learn more about SMT and CPU utilization – check at least these two post by Mr. Nigel Griffiths, IBM.

nmon – I can’t see all the CPUs on-screen. Please Help!

nmon – new online Physical CPU Graphs arrive for latest AIX 6.1

Another reading material after comment from Rob: Power7 CPU and Virtual Processors . You may need to download this document to be able to read it (PowerPoint presentation).

Posted in Real life AIX.


Method error (/usr/lib/methods/cfallvpath -2): 0514-068 Cause not known

To migrate to PowerHA System Mirror 7.1 my cluster needs a pair of disks. But something is wrong, the cfgmgr command fails with the following messages:

# cfgmgr
Method error (/usr/lib/methods/cfallvpath -2):
        0514-068 Cause not known.
sh: /usr/lib/methods/cfallvpath:  not found.

What adds to the mystery is the fact that the /usr/lib/methods/cfallvpath is absent from this host and from all the other nodes in this cluster and as I can see later – it is absent from all of my AIX boxes (6.1.8 and 7.1.3). Something here does not add up…. Do I really need to reboot these nodes in order to get the disks in? Maybe not if only I could delete the offending “method” from ODM.

Let’s start and backup the current configuration rules, just in case…

# cd /etc/objrepos; cp Config_Rules Config_Rules.BACKUP

Now, let see (this step should be really the first one) these rules/methods for the last time.

# odmget -q "rule='/usr/lib/methods/cfallvpath -2' " Config_Rules

Config_Rules:
        phase = 2
        seq = 50
        boot_mask = 0
        rule = "/usr/lib/methods/cfallvpath -2"

Config_Rules:
        phase = 3
        seq = 50
        boot_mask = 0
        rule = "/usr/lib/methods/cfallvpath -2"

Look and check that the backup you made is really there when you left it. Ready to go, let’s remove the rules.

# odmdelete -q "rule='/usr/lib/methods/cfallvpath -2' " -o Config_Rules
0518-307 odmdelete: 2 objects deleted.

Is this the truth and they are gone?

# odmget -q "rule='/usr/lib/methods/cfallvpath -2' " Config_Rules

No output means that the rules have been removed. Now, it is the time for some housekeeping.

# savebase -v
saving to '/dev/hd5'
81 CuDv objects to be saved
366 CuAt objects to be saved
27 CuDep objects to be saved
22 CuVPD objects to be saved
405 CuDvDr objects to be saved
110 CuPath objects to be saved
216 CuPathAt objects to be saved
0 CuData objects to be saved
0 CuAtDef objects to be saved
Number of bytes of data to save = 51308
Compressing data
Compressed data size is = 16280
        bi_start     = 0x3600
        bi_size      = 0x1b20000
        bd_size      = 0x1b00000
        ram FS start = 0x917e30
        ram FS size  = 0x10ec71a
        sba_start    = 0x1b03600
        sba_size     = 0x20000
        sbd_size     = 0x3f9c
Checking boot image size:
        new save base byte cnt = 0x3f9c
Wrote 16284 bytes
Successful completion

Now, ready to run the ConfigurationMangler as Mr.Mike F. affectionately calls it ….

# cfgmgr 

It returns with no errors and the lspv shows the two new disks as expected. Have a good weekend Sys Admins!

Posted in Real life AIX.


NIM, KRB5/AD rsh, ftp …….

Lately, I am busy trying to get ftp and rsh to work with KRB5/AD as the authentication engine. Apparently, there are still applications that need both ftp and rsh… NIM is one such example, it still needs rsh. Well, this is exactly what I have thought till this morning when I discovered Chris Gibson article http://www.ibmsystemsmag.com/aix/administrator/systemsmanagement/nimsh_nimadm/ showing what to do in order to change this requirement! For me this is a “WIN” situation as now I can put back our NIM servers into the KRB5 and they will still work! This is really ironic as two days ago during a meeting with IBM reps I expressed my surprise that NIM still needs rsh. As I see it now, my idea was at least several months old.

Looking for more NIM info, I found a really nice blog I recommend all to take a look at – “NIM Less known features : HANIM, nimsh over ssl, DSM”
http://chmod666.org/index.php/nim-less-known-features-hanim-nimsh-over-ssl-dsm/.

This rocks! Thanks Gents!

Posted in Real life AIX.


AIX LDAP client + KRB5A with Active Directory 2012

If you select this method to authenticate/authorise you may notice that a user group membership is missing – a user with multiple groups will be shown to belong to only one group!

# lsuser -a pgrp groups mannt
mannt pgrp=lawson groups=lawson

The mannt user belongs to more than one group, really. So why when we ask (using AIX LDAP client) Active Directory to deliver this information it does come to us truncated?

It could be that the Active Directory administrator did not follow this procedure:

Active Directory object management
As is the case with any other authentication mechanism, we need to configure the user objects for the users that are to use the system. However, if you are implementing this solution, more than likely your users already have Windows accounts. In that case, all we need to do is to modify the objects to be POSIX compliant.
1.	Open the Active Directory Users and Groups management tool.
   a.) Modify a group object to function as a POSIX group. 
   b.) Right-click on the user group for assignment of a GID. 
   c.) Click on the Unix Attributes tab. 
   d.) Populate the NIS Domain dropdown and the GID number as appropriate.
2.	Modify a user object to function as a POSIX user.
   a.) Locate and activate the tab that says Unix Settings. 
   b.) Under Unix Settings, set the UID and GID for the user, as well         as the home directory location (on the Linux filesystem /home/). 
   Note: You will need to ensure that the directory exists with the appropriate user object having access to the directory. 
   c.) Reset the user's password. This causes the AD password and the Unix password attributes to synchronize.
3.	Add the user as a Unix member of the group.
   a.) After you have added the user as a Unix user, you will also need to come back to the group properties and add the user as a member on the Unix Attributes tab. Otherwise, the user will not be populated in the msSFU30PosixMember attribute.

Next, you have to modify the /etc/security/ldap/sfur2group.map file, which default is presented bellow:

groupname  SEC_CHAR   cn                 s       na      yes
id         SEC_INT    gidNumber          s       na      yes
users      SEC_LIST   cn                 m       na      yes

Depending how your user group membership is declared in Active directory, you have to replace the last line of this file so it looks either like this:

users      SEC_LIST    msSFU30PosixMember m      na      yes

or like this

users      SEC_LIST   member              m      na      yes

Follow this modifications with execution of the restart-secldapclntd command and list the user again. Now, his full group membership is really shown.

# lsuser -a pgrp groups mannt
mannt pgrp=lawson groups=lawson,shell,payroll,operator,printq

I will be able to provide you with the Active Directory “side” of this procedure as soon as my college Igor Zilberman (the greatest AD/CITRIX administrator I have been lucky to work with! 🙂 ) documents this process – Igor thanks in advance!
These two different attributes (member and msSFU30PosixMember) you use in sfur2group.map do really have an effect on how you assign UNIX attributes to AD users….

ATTENTION:
Tu Vo (IBM) just told me that the default *.map files may be overwritten the next time AIX is patched!!! With this knowledge at hand, I copied the original sfur2group.map into sfur2AD2012group.map (am I creative or not?), edited it as described above and next, I modified the appropriate entry in the /etc/security/ldap/ldap.cfg so know it looks like that:

groupattrmappath:/etc/security/ldap/sfur2AD2012group.map

After a few days, a user tried to use the sftp command and failed. Fixing his issues, I noticed a “strange” behavior (AIX 7.1 host) – for a casual user the id command did not work:

# id
uid=934960 gid=4141 groups=216(operator)

The host/user lost the ability to translate (to show) the user login name and his/her groups names – it just showed their numerals. When the same user tried to ssh to another host he would receive this pleasant message:

# ssh markd@hostB
You don't exist, go away!

Well, tell it to the user that he does not exist! How dare you? Tu Vo (IBM) delivered the resolution to this issue letting me know that KRB5A is “depreciated” (on its way out….). Tu Vo advise was to replace in /etc/methods.cfg file every KRB5A with just KRB like that

KRB5:
        program = /usr/lib/security/KRB5
        program_64 = /usr/lib/security/KRB5_64
        options = authonly,is_kadmind_compat=no,tgt_verify=no

LDAP:
        program = /usr/lib/security/LDAP
        program_64 =/usr/lib/security/LDAP64

KRB5LDAP:
        options = auth=KRB5,db=LDAP

next, you must do the same in the /etc/security/user file – make sure that registry and SYSTEM also show KRB5LDAP instead of KRB5ALDAP.

After the change, you either has to restart the secldapclntd or to flush its cache (flush-secldapclntd). Now, login as the ordinary user and execute the id command, does it work? YES!!!! Now it is time to do ssh and sftp do they work? YES!!!

Thanks Tu Vo!
🙂

What about the earlier ssh issue? Well, it was not justKRB5 it was also the key in his ~/.ssh/known_hosts …….

Posted in Real life AIX.

Tagged with , , , , , , , , , , .


PCI DSS – how to show CVE report is wrong?

The Payment Card Industry Data Security Standard (PCI DSS) is a set of specific security standards designed to ensure that all companies that process, store or transmit credit card information maintain a secure environment during and after a financial transaction………

For a company to maintain a “good PCI standing” its internet facing infrastructure has to be “scanned” to verify that all identified security vulnerabilities has been addressed (implemented). If your company follows the PCI DSS you may receive an email notification about PCI CVE vulnerabilities and exposures (or something like that). Often, these emails will really be bogus. If the internet facing hosts are systematically patched they will be bogus and you will have to prove it.

So how do you do that? If you http server is running Red Hat, you can use the method shown next.
Let’s say that the PCI compliance email identifies the following – “Apache HTTPD: error responses can expose cookies (CVE-2012-0053)”. The “CVE-2012-0053” is known as the “Common Vulnerability and Exposures” identified in 2012 under the ID of 0053.

It is a well know fact, that the ones tasked with creating “scanning” tools (regardless of their intended targets AIX or LINUX) as a rule are no able to follow UNIX patching “logic” and always flag as missing something that already has been fully addressed. To verify that the particular CVE andthe associated with it issues have already been addressed, login to the appropriate host and identify all rpm's that could be exposed (the web base stuff – Apache and HTTPD):

# rpm -qa | egrep -i "apache|httpd"
httpd-tools-2.2.15-29.el6_4.x86_64
httpd-2.2.15-29.el6_4.x86_64

Now, you have to see if the packages listed by the last command contain any information about the CVE-2012-0053. The next command looks “inside” all installed httpd rpm’s.

# rpm -q --changelog httpd | grep -C1 'CVE-2012-0053'
* Mon Feb 06 2012 Joe Orton <jorton@redhat.com> - 2.2.15-16
- add security fixes for CVE-2011-4317, CVE-2012-0053, CVE-2012-0031,
  CVE-2011-3607 (#787599)

Analyzing the output above, you clearly see that on 02/06/12 one Joe Orton added four fixes to the httpd rpms. One of them is the one that the PCI scan identified as missing ……

The credit goes to Mike “Ski” Swierczynski who showed me this procedure, thanks Mike.

Posted in Linux.

Tagged with , , .


issuess with alt_disk_install and AIX 6.1.8.3 and above…..

when “migrating” to this version of AIX using the alt_disk_install, you may be presented with this unwelcomed message

0301-150 bosboot: Invalid or no boot device specified!
usage:  bosboot {-a | -v} [-d device] [-p proto] [-k kernel] [-l lvdev]
                [-b file] [-M primary|standby|both] [-D|-I] [-LTq]
        Where:
        -a              Create boot image and write to device or 
                        file.
        -v              Verify, but do not build boot image.
        -d device       Device for which to create the boot image.
        -p proto        Use given proto file for RAM disk file 
                        system.
        -k kernel       Use given kernel file for boot image.
        -l lvdev        Target boot logical volume for boot image.
        -b file         Use given file name for boot image name.
        -D              Load kernel debugger.
        -I              Load and Invoke kernel debugger.
        -M primary|standby|both Boot mode - primary or standby.
        -T platform     Specifies the hardware platform type.
        -q              Query disk space required to create boot 
                        image.
        -L              Enable MP locks instrumentation.
0505-120 alt_disk_install: Error running bosboot in the cloned
root volume group.
Cleaning up.

Executing lspv or lsvg will show you that there is no altinst_rootvg. If you check the state of the current bootlist you most likely will find it in a questionable state….

# bootlist -m normal -o
hdisk0 blv=hd5

Why “questionable”? Because the output of the last command should really look like this

# bootlist -m normal -o
hdisk1 blv=hd5 pathid=0

Searching the net for 0505-120 alt_disk does not returned nothing for me. One short call to IBM SUPPORT and the issues was identified and resolved. Apparently there is a bug and for the alt_disk_install to work you have to install the appropriate fileset – in my case the one included in the AIX 6.1.8.3 collection. A quick nfs mount of the file system on our NIM server containing the AIX 6.1.8.3 filesets followed with smitty install_all and selection of the bos.alt_disk_install.rte from among the available file sets

# lslpp -l | grep alt_disk
bos.alt_disk_install.boot_images
bos.alt_disk_install.rte 6.1.8.16 COMMITTED Alternate Disk Install
bos.msg.en_US.alt_disk_install.rte
bos.alt_disk_install.rte 6.1.8.16 COMMITTED Alternate Disk Install

resolved the issues and the follow up execution of

# alt_disk_install -C -F update_all -I acNgX \
              -l /ptfs/tl8_sp3_update hdisk1

resulted (after a few minutes) in this very welcomed display:

...........................................................
install_all_updates: Log file is /var/adm/ras/install_all_updates.log
install_all_updates: Result = SUCCESS
Modifying ODM on cloned disk.
Building boot image on cloned disk.
forced unmount of /alt_inst/var/nmon
forced unmount of /alt_inst/var/nmon
forced unmount of /alt_inst
...........................................................
forced unmount of /alt_inst
Changing logical volume names in volume group descriptor area.
Fixing LV control blocks...
Fixing file system superblocks...
Bootlist is set to the boot disk: hdisk1 blv=hd5
You have mail in /usr/spool/mail/root

# bootlist -m normal -o
hdisk1 blv=hd5 pathid=0

After the reboot this host shows the expected new version of AIX. By the way, before I migrate the next machine to 6.1.8.3, I wll first upgrade its bos.alt_disk_install.rte ….. 🙂

Today, I witnessed my upgrade from 7100-01-06 to 7100-03-01 fail just the same way. So I installed the bos.alt from 7100-01-06, restarted the process and watched it work = install_all_updates: Result = SUCCESS

Posted in Real life AIX.

Tagged with .


mksysb backups and their integrity

It is a good practise to use this type of backup just in case “something” happens to the rootvg. In my opinion, mksysb backup is the easiest and the fastest way to “recover/recreate” a host. Occasionally, it may also used for a quick restore of a file or a directory that was removed but it should not be – as long as it resided in the rootvg file system structure as mksysb works only with the rootvg.

Many of us automate mksysb backups with cron using the value returned by this command as a trigger for a possible alert in the case failure – in a perfect world this would work every time…….

Over the weekend, I have discovered that backups of some of the servers did not contain the files are needed to restore. It could be that the emails triggered by a faulty mksysb were not delivered or they were delivered but I have missed them among dozens of other emails delivered daily to my mailbox. Regardless of the reason or excuse, the result was the same ….. What I was looking for I could not find because it was not there!

A few hours later, after the issues were finally resolved and my mind was free to think about something else, I started to think about the steps needed to prevent this what has happened today – what to do to make sure the mksysb really contains everything it should and administrator is notified and it is aware of the fact that the backup failed?

The email sent from the backup script is really mandatory. Is it a good idea to re-send the “failure” email a few hours later? Maybe it is a perfect idea to keep re-sending email until sys admin logs into the host sending these emails?

I modified the backup script. Now, mksysb failure is marked with creation of the “mksysb.failed” file, which presence is the reason of sending the “mksysb failed on $HOST” emails (once a day). To disable this mechanism, system administrator needs to login and delete the “mksysb.failed” file. Simple, right?

Nothing, no procedure can substitute a “admin’s eye”. How to make sure that the mksysb backup is indeed good and it contains the data you may need? You have to do it yourself! So no matter how “sophisticated” is the script, occasionally you have to login and interrogate its backup.
In this case there are at least two options – one uses the restore and the other the lsmksysb command. For example:

# lsmksysb -lf /path/to/mksysb_backup
# restore -Tqvf /path/to/mksysb_backup

List its contents, look for a specific file or files. Can you “get” them?

Ok, so this is one side of the store. Now, let’s talk about the second one. Where is this backup and the previous ones being stored? In my case they are backed up and stored by TSM. To verify that they really are there execute the next command.

# dsmc restore -pick /path/to/mksysb_backup -subdir=yes -inactive
...................................................
TSM Scrollable PICK Window - Restore

  #    Backup Date/Time     File Size A/I  File
        -------------------------------------------------------------
  1. | 01/22/14   07:36:50  14.69 GB  A   /mksysbimg/mksysbimg.tds1
  2. | 01/21/14   11:45:12  13.77 GB  I   /mksysbimg/mksysbimg.tds1
  3. | 01/21/14   02:00:14 308.25 MB  I   /mksysbimg/mksysbimg.tds1
  4. | 01/20/14   02:00:15 306.10 MB  I   /mksysbimg/mksysbimg.tds1
  5. | 01/19/14   02:00:16 304.63 MB  I   /mksysbimg/mksysbimg.tds1
  6. | 01/18/14   02:00:30   7.22 GB  I   /mksysbimg/mksysbimg.tds1
  7. | 01/17/14   08:41:29   7.23 GB  I   /mksysbimg/mksysbimg.tds1

The last output shows that something is not right. Why their size is so drastically different? Something needs to be checked here for sure.

Also, the backup command has been modified to include the -p option – the backup seems to work better with it.

# /usr/bin/mksysb -i -e -X -p /mksysbimg/mksysbimg.${HOSTNAME}

Posted in Real life AIX.

Tagged with .




Copyright © 2015 - 2016 Waldemar Mark Duszyk. - best viewed with your eyes.. Created by Blog Copyright.