Skip to content


a SEA in LIMBO ……

Yesterday was a memorable day because PowerBall (Lottery) prize went up to some 430 million US dollars high. Yesterday, I also upgraded one VIO server to 2.2.2.2. there are two VIO servers bounded with a SEA adapter. The SEA of the VIO server that got “patched” was the PRIMARY adapter, so just after I accepted the license and shortly before the reboot, its SEA (10gbps) was “failed over” to the other VIO server, which in turn became the PRIMARY one. The following shows hot to “flip” the role of the SEA adapter.

$ lsdev | grep –i shared
ent12      Shared Ethernet Adapter

$ entstat -all ent12 | grep State
    State: PRIMARY
LAN State: Operational
LAN State: Operational
$ chdel -dev ent12 -a attr ha_mode=standby

or

# chdev -l ent12 -a ha_mode=standby 

Today is another memorable day but for entirely different reasons. First, I work from home (which I do not really like too much). Second, “Power Ball” did not bring me even a single dollar, which really sucks. Third, has to do with the SEA adapter of the first VIO server (the one I patched yesterday).
After the second VIO server was patched, just after I accepted the license agreement and just before the reboot, I changed the state of its SEA so the first VIO server becomes the one, which SEA is the PRIMARY adapter.

# chdev –l ent12 –a ha_mode=auto

A little voice inside my head told me to verify it.

$ entstat -all ent12 | grep State
    State: LIMBO
LAN State: Operational
LAN State: Operational 

A quick search of the net reviled that a SEA adapter can be transitioned into the LIMBO state due to any of these conditions:

“when there is a physical link down, unknown state adapter, or if netaddr is not pingable, SEA goes to LIMBO state. If netaddr is removed, SEA is left in LIMBO state and it does not recover until a reachable netaddr is configured or SEA is reconfigured. . I wonder, if these are all possible reasons?

What they mean by “the SEA being reconfigured” is accomplished with the rmdev/mkdev commands. With the second VIO server reboot on hold, let’s proceed and reconfigure the first VIO server SEA adapter. Let’s start collecting all the details of this device.

$ lsdev -dev ent12 -attr
attribute     value                 description                                user_settable

accounting    disabled Enable per-client accounting of network statistics 
ctl_chan      ent10    Control Channel adapter for SEA failover          
gvrp          no       Enable GARP VLAN Registration Protocol (GVRP)  
ha_mode       auto     High Availability Mode                                    
hash_algo     0        Hash algorithm used to select a SEA thread      
jumbo_frames  no       Enable Gigabit Ethernet Jumbo Frames         
large_receive no       Enable receive TCP segment aggregation               
largesend     1        Enable Hardware Transmit TCP Resegmentation      
lldpsvc       no       Enable IEEE 802.1qbg services                               
netaddr       0        Address to ping                                                   
nthreads      7        Number of SEA threads in Thread mode                
pvid          1        PVID to use for the SEA device                               
pvid_adapter  ent9     Default virtual  for non-VLAN-tagged packets    
qos_mode      disabled N/A                                                             
queue_size    8192     Queue size for a SEA thread                           
real_adapter  ent15    Physical adapter associated with the 
thread        1        Thread mode enabled (1) or disabled (0)              
virt_adapters ent9     List of virtual adapters associated with the SEA

From the listing above, we learn that the SEA adapter aka ent12 is associated with the physical adapter ent15 and that its control channel (the adapter used to communicate with the SEA in the other VIO server) adapter is ent10. Its virtual adapter used to receive the “un-tagged” packets is ent9 and finally its PVID equals 1.

As mentioned earlier, to reconfigure SEA means to destroy it and to create it anew. First, to destroy it, I will get out of the restricted shell associated with the padmin login – oem_setup_env command serves this purpose. Before we kill the SEA adapter, it is always a good idea to check if this adapter has an associated IP information aka if this adapter is also used to login to the VIO server. If this is the case, after the SEA is re-created the previously assigned to it IP information (IP address, netmask, router, etc) has to be re-created as well.

$ oem_setup_env
# ifconfig -a
en11: flags=1e080863,580<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MUFLOAD(ACTIVE),CHAIN>
        inet 10.19.80.1 netmask 0xffffff00 broadcast 10.19.80.255
         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1
lo0: flags=e08084b,1c0<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTIC >
        inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255
        inet6 ::1%1/0
         tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1

# lsdev -Cc adapter | grep ent
ent0      Available          Logical Host Ethernet Port (lp-hea)
ent1      Available 0A-00    4-Port 10/100/1000 Base-TX PCI-Express
ent2      Available 0A-01    4-Port 10/100/1000 Base-TX PCI-Express
ent3      Available 0B-00    4-Port 10/100/1000 Base-TX PCI-Express
ent4      Available 0B-01    4-Port 10/100/1000 Base-TX PCI-Express
ent5      Available 06-00    4-Port 10/100/1000 Base-TX PCI-Express
ent6      Available 06-01    4-Port 10/100/1000 Base-TX PCI-Express
ent7      Available 07-00    4-Port 10/100/1000 Base-TX PCI-Express
ent8      Available 07-01    4-Port 10/100/1000 Base-TX PCI-Express
ent9      Available          Virtual I/O Ethernet Adapter (l-lan)
ent10     Available          Virtual I/O Ethernet Adapter (l-lan)
ent11     Available          Virtual I/O Ethernet Adapter (l-lan)
ent12     Available          Shared Ethernet Adapter
ent15     Available 0C-00-00 10 Gigabit Ethernet Adapter (ct3)

# lsattr -El ent12
accounting    disabled Enable per-client accounting of network stat
ctl_chan      ent10    Control Channel adapter for SEA failover
gvrp          no       Enable GARP VLAN Registration Protocol (GVRP
ha_mode       disabled      High Availability Mode
hash_algo     0        Hash algorithm used to select a SEA thread
jumbo_frames  no       Enable Gigabit Ethernet Jumbo Frames
large_receive no       Enable receive TCP segment aggregation
largesend     1        Enable Hardware Transmit TCP Resegmentation
lldpsvc       no       Enable IEEE 802.1qbg services
netaddr       0        Address to ping
nthreads      7        Number of SEA threads in Thread mode
pvid          1        PVID to use for the SEA device
pvid_adapter  ent9     Default virtual adapter to use for non-VLAN-
qos_mode      disabled N/A
queue_size    8192     Queue size for a SEA thread
real_adapter  ent15    Physical adapter associated with the SEA
thread        1        Thread mode enabled (1) or disabled (0)
virt_adapters ent9     List of virtual adapters associated with the SEA

The last commands, document that the SEA adapter is just a pass through the LAN with no IP address. The SEA adapter is removed using exactly the same procedure as any “normal” network adapter.

# ifconfig en12 down detach
# rmdev –dl ent12 –R
# exit

After returning back to padmin restricted shell, SEA adapter is recreated with the previous parameters.

$ mkvdev -sea ent15 \
                   -vadapter ent9 \
                   -default ent9 \
                   -defaultid 1 \
                   -attr ctl_chan=ent10 \
                    ha_mode=auto \
                    largesend=1
ent12
ent12

In what state is the new adapter?

 $ entstat -all ent12 | grep State
    State: LIMBO
LAN State: Operational
LAN State: Operational 

Apparently, the re-creation did not help. Just for a peace of mind, I recreated the SEA a few more times with zero luck. It was time to check what is going on with the switch port this adapter is associated with – the network team went to work. Meanwhile, having suddenly nothing to do and nothing else to lose, I decided to set this adapter back as the PRIMARY adapter with a slim hope that this somehow may change the SEA adapter state.

$ chdev –l ent12 –a ha_mode=auto

These efforts were nothing but futile – did not accomplished what I hoped for. It took only a few minutes for the network team to establish that the port to which the first VIO server SEA was using was DOWN in the linkFlapErrDisabled state. Network engineer tried to re-set this port but was not successful so the GBIC was replaced, which immediately changed the state of the LIBMOed SEA adapter! It was back on-line and operational as the PRIMARY SEA adapter. The Sun is shining bright!

$ entstat -all ent12 | grep State
    State: PRIMARY
LAN State: Operational
LAN State: Operational

With this adapter back as the PRIMARY (providing LAN services to all LPARs of this managed system), it was time to reboot the second VIO server. What I saw next, I really did not expected to see. Now, this server’s SEA was in the LIMBO state!
Talking to other UNIX administrators, I learnt that around here this is a known problem. Could it be the combination of hardware/software? Who knows, maybe this is the case. For us this is one more reason to keep the GBICs warranties handy.
If you wonder what switch I worked with it was CISCO Nexus 5540 with 3.5.0 BIOS.

A few days later:
as soon as the counters were reset and the port was de-activated/activated the SEA adapter returned on-line. It seems to me that port switch counters should be set to higher values while using 10gbps adapters as SEA devices.

Posted in Real life AIX, VIO.

Tagged with , , , , , , , .


One Response

Stay in touch with the conversation, subscribe to the RSS feed for comments on this post.

  1. Alex says

    awesome – i had the same issue – dzięki



Some HTML is OK

or, reply to this post via trackback.

WordPress Anti Spam by WP-SpamShield



Copyright © 2016 - 2017 Waldemar Mark Duszyk. All Rights Reserved. Created by Blog Copyright.