Skip to content


adding swap space – RH7.x

a. stop swap activity on the existing swap partition or partitions

# swapoff -v /dev/mapper/rootvg-swap

create new swap volume (this host is not using fdisk partitions but logical volume manager), give it 2GB.

# lvcreate rootvg -n swap-vol2 -L 2G

turn it into swap

# mkswap /dev/mapper/swap-vol2

activate it

# swapon -v /dev/rootvg/swap-vol2

edit /etc/fstab to so it comes back every time the hosts is booted.

# echo "/dev/mapper/rootvg-swap-vol2 swap                swap    defaults        0 0" >> /etc/fstab

Posted in LINUX.

Tagged with , , , , , .


deploying ssh keys to remote hosts

I have discovered Ansible and as the result, I have to deploy ssh keys to a few hundred UNIX/LINUX boxes….. Yes, there is the ssh-copy-id which is fine for one machine but this is not going to work for me. The idea of repetitively entering root password is making me sick….. Is there anything else….?
Yes, there is actually more than one way to finish this task. There is the sshpass that you download from EPEL which works with ssh-copy-id like that:

# sshpass -f pass.txt ssh-copy-id -i ~root/.ssh/id_rsa.pub target_host_name

where pass.txt file contains the root password, and the target_host_name is the destination host name.

But there is even a better way! Thanks to Travis Bear who created ssh-deploy-key. You can learn more about it following this link https://bitbucket.org/travis_bear/ssh-deploy-key

This is an excerpt from Travis doc’s:

Here is Travis comparison of ssh-deploy-key with some other common ways to deploy a key.

“Deploying ssh keys by hand is a time-honored technique that in general works pretty well. However, in almost all cases, using ssh-deploy-key is a better option. It’s faster, easier, more reliable, and more repeatable. When deploying to more than one host at a time, these advantages only multiply with ssh-deploy-key’s bulk deployment abilities. There is one use case where deploying by hand is a better bet: when the remote host is on a different network, behind a jump box. ssh-deploy-key does not handle that scenario.

ssh-copy-id is a great tool, but it’s not the ideal solution for every scenario.
ssh-copy-id is not installed by default on all systems, notably on Mac OS.
ssh-copy-id has no concept of ‘smart append’. It will append a key to a remote host’s authorized keys file regardless of whether that key is already present.
• Scripting the use of ssh-copy-id for deploying to multiple remote hosts can be challenging:

The password is entered interactively for each host. In the case where there are numerous remote hosts that have not seen before, you’d need to interactively allow each host to be added to your known_hosts file.

Configuration management tools (like Puppet, Ansible, etc.) can do a terrific job deploying ssh key(s). But if you are not already set up to use them for key distribution, these general-purpose solutions can be overkill, especially when compared with a dedicated tool like ssh-deploy-key that only does one thing.”

To install this utility requires two steps:

# yum -y install python-pip python-devel
# pip install ssh-deploy-key

The spacecmd system_list command (Satellite/Spacewalk) generated all hosts names which where collected inside the HOSTS file. The actual processing the list of hosts was done extremely easy – "ssh-deploy-key -d < HOSTS"

The -d flag could be a very important one to remember. Without it the target host /root/.ssh/authorized_hosts file will be overwritten - every host already defined there will be gone!!!

Posted in LINUX.


switching groups and mixed case logins with sssd

Several users, each with its own group set needs to collaborate on as data located in a folder which is owned by a yet different group. The solution is to add the data folder group to each users group set and next executing the gpasswd command set a password on the “data” group.
To be able to “enter” the data folder a user sets its group as his/her primary group executing the newgrp - group_name command, enters the group password and move to the data folder.

To force LINUX with SSSD to always display login names in lowercase regardless of the format used in Active Directory you have to have this entry in your /etc/sssd/sssd.conf file

[domain/xxxxxxx]
case_sensitive = False 

Edit this file, restart the sssd service and clean its cache with sss_cache -E command and you are as good as new.

Posted in LINUX.


migrating from “THICK” to “THIN” vmware disks with pvmove

I have a linux guest that one disk is set as THICK (by a mistake or an act of God). It is claimed by a volume group with one almost full logical_volume/file_system. I need to change the storage type (thin) and simultaneously provide more capacity to accommodate the constant growth of data.
There are numerous ways to resolve this situation. For example, to change the disk type one could use the VMotion (assuming one has a license) or vmkfstools. Next, the converted disk could be “grown” to the required capacity with vmtools.
Or you could create an new thinly provisioned disk of the required capacity and add it to the volume group and move the data either via a mirror or relocation of physical partitions. Finally, remove the “thick” disk from its volume group and from the guest.
In this case the THICK disk is /dev/sdb and the new THIN one is /dev/sdc
The logical volume is defined as zoom_vg-zoom_lv

# vgextend zoom_vg /dev/sdc
# pvmove -n zoom_vg-zoom_lv /dev/sdb /dev/sdc
  Detected pvmove in progress for /dev/sdb
  Ignoring remaining command line arguments
  /dev/sdb: Moved: 11.1%
  /dev/sdb: Moved: 12.5%
  /dev/sdb: Moved: 13.9%
  /dev/sdb: Moved: 15.4%
...........................
  /dev/sdb: Moved: 98.4%
  /dev/sdb: Moved: 99.9%
  /dev/sdb: no pvmove in progress - already finished or aborted.

During the migration, one can use the lvs command to gauge its progress.

# lvs -a
  LV        VG      Attr       LSize     ......... Log Cpy%Sync
  lv_home   vg_sys  -wi-ao----   1.95g
  lv_root   vg_sys  -wi-ao----  10.84g
  lv_swap   vg_sys  -wi-ao----   3.91g
  lv_temp   vg_sys  -wi-ao----   3.91g
  lv_usr    vg_sys  -wi-ao----   7.91g
  lv_var    vg_sys  -wi-ao----   5.91g
  [pvmove0] zoom_vg p-C-aom--- 199.00g     /dev/sdb     48.20
  zoom_lv   zoom_vg -wI-ao---- 199.00g

# lvs -a | grep pvmove
  [pvmove0] zoom_vg p-C-aom--- 199.00g     /dev/sdb     49.34

# lvs -a | grep pvmove
  [pvmove0] zoom_vg p-C-aom--- 199.00g     /dev/sdb     49.62

When pvmoce finishes, we drop /dev/sdb from its volume group.

# vgreduce zoom_vg /dev/sdb
  Removed "/dev/sdb" from volume group "zoom_vg"
# pvs
  PV         VG      Fmt  Attr PSize   PFree
  /dev/sda2  vg_sys  lvm2 a--   34.61g 196.00m
  /dev/sdb           lvm2 ---  200.00g 200.00g
  /dev/sdc   zoom_vg lvm2 a--  300.00g 101.00g

To finish, we need to remove it form the guest definition in VMWare and the “thick” is finally gone.

Posted in LINUX.


vmare, snapshots, etc

Before Oracle Linux hosts were unregistered from ULN and registered with an internal SpaceWalk system a snapshot was made. Now, a month later there are a large number of snapshots that have to be removed.
It is very easy to find these hosts (VMWare guests) – their names start with “EIE”.

PowerCLI in action…..

To list selected guests and their snapshots execute the following command:

get-vm | where {$_.name -match "EIE"} | Get-Snapshot | format-list vm,name
VM   : EIExxxx2
Name : ULN_Registered
VM   : EIEyyyy1
Name : ULN_Registered
VM   : EIEqqqq2
Name : ULN_Registered
VM   : EIEwwwww3
Name : ULN_Register

To delete a snapshot with conformation:

get-vm | where {$_.name -match "EIE"} | Get-Snapshot | Remove-Snapshot 

You will be asked to verify that you really mean it and only after you say “Yes” the selected snapshot will be removed.

To remove all snapshots without being asked to confirm, execute:

get-vm | where {$_.name -match "EIE"} | Get-Snapshot | Remove-Snapshot -Confirm:$false

Posted in LINUX.

Tagged with , , .


client not talking to spacewalk

from the set of the migrated Oracle Linux hosts two stopped communicating with Spacewalk server….. The tasks scheduled to execute would not run….. Executing /usr/sbin/osad -N -v -v -v -v allows to bring this service to the “front” and after a while of looking at the screen this message showed up:

2016-06-07 09:06:49 jabber_lib.process: 180
Error caught:
Traceback (most recent call last):
  File "/usr/share/rhn/osad/jabber_lib.py", line 121, in main
    self.process_forever(c)
  File "/usr/share/rhn/osad/jabber_lib.py", line 179, in process_forever
    self.process_once(client)
  File "/usr/share/rhn/osad/osad.py", line 250, in process_once
    client.process(timeout=180)
  File "/usr/share/rhn/osad/jabber_lib.py", line 1055, in process
    data = self._read(self.BLOCK_SIZE)
SSLError: ('OpenSSL error; will retry', "(-1, 'Unexpected EOF')")

2016-06-07 09:06:49 jabber_lib.main: Sleeping 108 seconds

A few seconds later Google delivers this gem of advice:

Check /etc/rhn/rhn.conf and make sure the below 2 variables match the hostname of your Spacewalk server

server.jabber_server = yourspacewalk.server.com
osa-dispatcher.jabber_server = yourspacewalk.server.com

Also, verify that the hostname listed in your SSL certificate matches the hostname of your Spacewalk server..

/var/www/html/pub/RHN-ORG-TRUSTED-SSL-CERT

My hosts have this in place in the desired format. The next step in the article provides this advice for osad client:

# service osad stop
# rm /etc/sysconfig/rhn/osad-auth.conf
# service osad start

I do it on each of the two and they start talking with Spacewalk!!!!

By the way, this article has this advice for the Spacewalk (I did not do this step) server:

# service jabberd stop
# rm -Rf /var/lib/jabberd/db/*
# service jabberd start

Keep in mind that the advice presented here applies to both environments – Spacewalk and Satellite.

Posted in LINUX.

Tagged with , , , .


migrating /boot to a larger partition

I cannot install Oracle’s Unbreakable kernel on two hosts (vmware guests) because their /boot is just too small!

# df /boot
/dev/sda1 240M 224M 0 100% /boot

they were build in a peculiar way. The first disk (/dev/sda) contains the /boot (/dev/sda1) and the second primary partition (/dev/sda2) is taken by LVM to create logical volumes used for Linux binaries, etc. Other disks have been added later to accommodate application needs. Currently our “disks” are

# ls /dev/sd*
/dev/sda /dev/sda1 /dev/sda2 /dev/sdb /dev/sdc

My plan is to get an additional ~600MB disk and using the dd command, copy the contents of /dev/sda1 to this bigger disk. Next, I will have to modify the /boot/grub/grub.conf and the box should be ready for the “unbreakable” kernel.

Since this is VMWare, then new disk attachment is simple one step operation the new disk is called /dev/sdd. The new disk is partitioned using the fdisk command.

# fdisk /dev/sdd
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xeb108be0.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').

Command (m for help): p

Disk /dev/sdd: 629 MB, 629145600 bytes
64 heads, 32 sectors/track, 600 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xeb108be0

Device Boot Start End Blocks Id System

Command (m for help): n
Command action
e extended
p primary partition (1-4)

Partition number (1-4): 1
First cylinder (1-600, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-600, default 600):
Using default value 600

Command (m for help): a
Partition number (1-4): 1

Command (m for help): p

Disk /dev/sdd: 629 MB, 629145600 bytes
64 heads, 32 sectors/track, 600 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xeb108be0

Device Boot Start End Blocks Id System
/dev/sdd1 * 1 600 614384 83 Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

A new primary partition is created first and next it is marked as the “bootable” one (with the a toggle). To keep the changes, they must be written to the disk (the w does it)!

There is no longer need for /dev/sda1 to be marked as the boot-able one.

# fdisk /dev/sda

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').

Command (m for help): p

Disk /dev/sda: 75.2 GB, 75161927680 bytes
255 heads, 63 sectors/track, 9137 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00084012

Device Boot Start End Blocks Id System
/dev/sda1 * 1 33 262144 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 33 9138 73137152 8e Linux LVM

Command (m for help): a
Partition number (1-4): 1

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

Time, to edit the /boot/grub/grub.conf

In this file activate the boot line containing reference to /dev/sda and replace it with /dev/sdd:

#boot=/dev/sda

into

boot=/dev/sdd

Modify existing kernel with the new boot disk info.

# mkinitrd /boot/initramfs-`uname -r`.img `uname -r` --force

Copy the contents of old boot partition to the new one.

# dd if=/dev/sda1 of=/dev/sdd1 conv=sync
524288+0 records in
524288+0 records out
268435456 bytes (268 MB) copied, 74.2698 s, 3.6 MB/s

The dd command copies “verbatim” – the new disk gets the same UUID as its “source”, and the file system still has as much capacity as /dev/sda1.
To make /dev/sda1 UUID again unique:

# tune2fs /dev/sda1 -U `uuidgen`

Now, let’s resize the /boot file system on /dev/sdd1 to get ready for the new kernel.

# resize2fs /dev/sdd1 500M

Let’s reboot the host and validate that /boot is now mounted on /dev/sdd1.

Now, I can proceed with loading the “unbreakable” kernel.

# yum -y install uptrack-updates-`uname -r`

and whatever comes next.

Posted in LINUX.

Tagged with , , , .


registering ULN 5 with SpaceWalk2.4

For a Oracle Unbreakable Linux 5.x host which previously has been registered with ULN and now you want to register it with you local SpaceWalk server you have to remove these packages:

# rpm -e --nodeps  pirut, up2date up2date-gnome

While still registered with ULN, download (they could be missing) these rpms

# yum install python-dmidecode python-ethtool

Next, create a SpaceWalk tools repo, like for example

[ol5_spacewalk22_client]
name=Spacewalk Client 2.2 for Oracle Linux 5 ($basearch)
baseurl=http://public-yum.oracle.com/repo/OracleLinux/OL5/spacewalk22/client/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=1

With this repo in place, install the following new packages which are the ones that make SpaceWalk registration possible:

# yum install rhn-client-tools \
rhn-check rhn-setup rhnsd m2crypto yum-rhn-plugin

Finally, use the rhn_register to register with your SpaceWalk

Posted in LINUX.

Tagged with , , , , .


Migrate from RedHat to Oracle ULN

I have a bunch of RedHat 6.7 hosts registered with Satellite 5.7 which I am moving to Oracle ULN 6.7 and simultaneously registering with Oracle’s SpaceWalk2.2
Scheduling reboots for patching is just too much pain in a neck.

This is the procedure I follow.

a. disable any repositories found in /etc/yum.repos.d of the host about to be converted.
b. make copies of up2date and systemid files located in the /usr/share/rhn folder just in case, why am I doing it – paranoid?
c. install the keys required by the SpaceWalk and ULN registration process

# wget -q -O /usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT  \
http://sysspwpl1.chop.edu/pub/RHN-ORG-TRUSTED-SSL-CERT

# wget http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6 -O \
/etc/pki/rpm-gpg/RPM-GPG-KEY-oracle

# cd /etc/pki/rpm-gpg; rpm --import RPM-GPG-KEY-oracle

d. register with SpaceWalk executing the following snippet

# rhnreg_ks \
--sslCACert=/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT \
--serverUrl=https://SpaceWalkHostName.xxx.zzz/XMLRPC \
--activationkey=1-15506df7d95db,1-bf89408ea391c867 \
--force

e. clean yum cache and show new repos

# yum clean all; yum repolist

f. remove old version of vmware tools, install the latest vmware-tools repository

# for pkg in `rpm -qa | grep vmware`
do 
  rpm -e -vv --allmatches --nodeps --noscripts --notriggers $pkg
done

# cd /usr/lib
# rm -rf ./vmware-tools
# cd /etc
# rm -rf ./vmware-tools

# rpm -ivh https://packages.vmware.com/tools/esx/6.0u1/repos/vmware-tools-repo-RHEL6-9.10.5-1.el6.x86_64.rpm

# rpm --import http://packages.vmware.com/tools/keys/VMWARE-PACKAGING-GPG-DSA-KEY.pub

g. install Oracle logos

# yumdownloader oraclelinux-release.x86_64 \
        oraclelinux-release-notes.x86_64

# rpm -Uhv --force oraclelinux-release*

# yum install oracle-logos.noarch

h. install ULN support packages, install vmware, etc.

# yum -y shell
remove libreport-plugin-rhtsupport php-common-5.3.3-46.el6_7.1.x86_64 
remove kernel-firmware.noarch
update libreport abrt
install php55 php55-ldap 
install kernel-uek kernel-uek-firmware uptrack-offline
install vmware-tools-esx-kmods vmware-tools-esx
run
quit

# yum -y install uptrack-updates-`uname -r`;yum -y update

The running kernel is still the “old” one

# # uptrack-uname -r
2.6.32-573.22.1.el6.x86_64

Check that the kernel in force following the next reboot is the Unbreakable one.

# cat /boot/grub/grub.conf
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
password --encrypted $1$NaCl$BQ1C3cAnHEG4PprqLREJa1
title Oracle Linux Server Unbreakable Enterprise Kernel (3.8.13-118.4.2.el6uek.x86_64)
        root (hd0,0)
        kernel /vmlinuz-3.8.13-118.4.2.el6uek.x86_64 .....

Posted in LINUX, NonAix, Satellite.

Tagged with , , , , , , , .


the latest take on installing/patching VMWare tools

I have a set of rel6.x RedHat and Oracle Linux guests with outdated and some with no “tools” installed at all. Bellow, is my latest take on on “patching/re-installing”.

Just in the case the this has previously been installed.

# vmware-uninstall-tools.pl  

Just in the case, there are vmware packages laying around.

# for pkg in `rpm -qa | grep vmware`; do rpm -e -vv --allmatches --nodeps --noscripts --notriggers $pkg; done

Remove any previous “leftover” directories.

# cd /usr/lib
# rm -rf ./vmware-tools
# cd /etc
# rm -rf ./vmware-tools

Install VMWare repository of the appropriate version.

# rpm -ivh https://packages.vmware.com/tools/esx/6.0u1/repos/vmware-tools-repo-RHEL6-9.10.5-1.el6.x86_64.rpm 

Just in the case the keys have never been installed or they are gone.

# rpm --import http://packages.vmware.com/tools/keys/VMWARE-PACKAGING-GPG-DSA-KEY.pub

Finally, we are ready to roll and do the install.

# yum -y install vmware-tools-esx-kmods vmware-tools-esx

As the last step, login to vCenter GUI and see the tools running under the guest management.

Posted in LINUX.

Tagged with , , , .




Copyright © 2015 - 2016 Waldemar Mark Duszyk. - best viewed with your eyes.. Created by Blog Copyright.