AIX VIO 2.2.1.3 Etherchannel 802.3ad on 10 Gb Ethernet SR PCI Express Dual Port Adapter (771000801410b003)
id : nmxkft3z4t
category : computer
blog : unixlinux
created : 04/04/12 - 14:06:31

  • A bug in adapter microcode was found by IBM. If network switch reboots, ethernetchannel adapter in LACP (802.3ad) mode seems to fail.
  • The only things to do to correct the problem was to destroy SEA adapter, and etherchannel adapter shutdown VIO server, and recreate all configuration from scratch.
padmin# errlog | grep -i etherchannel
CE9566DF   0403124312 P H ent6           TOTAL ETHERCHANNEL FAILURE
CE9566DF   0402234812 P H ent6           TOTAL ETHERCHANNEL FAILURE
CE9566DF   0330154112 P H ent6           TOTAL ETHERCHANNEL FAILURE

  • IBM supports gave us a solution, update adapter microcode level, and install an efix (IV16920s02) on VIO Server.
Microcode update
  • as root, get current adapter microcode level :
# lsmcode -d ent0
The current microcode level for ent0
is 010508.

  • as root, install new microcode
# rpm -ivh 771001801410af03.010522.aix.rpm
[..]
#  ls /usr/lib/microcode | grep 010522
771001801410af03.010522

  • as padmin, remove all ip configuration, and sea if exists :
padmin# rmtcpip -all
[..]
padmin# rmdev -dev <sea_adapter>
padmin# rmdev -dev <etherchannel_adapter>

  • as root, download new microcode on ethernet adapter :
# diag -d ent0 -T download

  • as root, check current adapter microcode level :
# lsmcode -d ent0
The current microcode level for ent0
is 010522.

Efix IV16920s02 installation
  • upload the efix on VIO server :
padmin# ls efix
IV16920s02.120309.epkg.Z

  • as padmin install the efix with updateios command :
# updateios  -dev /home/padmin/efix -accept -install
*******************************************************************************
EFIX MANAGER PREVIEW START
*******************************************************************************

+-----------------------------------------------------------------------------+
Efix Manager Initialization
+-----------------------------------------------------------------------------+
Initializing log /var/adm/ras/emgr.log ...

+-----------------------------------------------------------------------------+
Processing Efix Package 1 of 1.
+-----------------------------------------------------------------------------+
Efix package file is: /home/padmin/efix/IV16920s02.120309.epkg.Z
MD5 generating command is /usr/bin/csum
MD5 checksum is 595d5782e9aac8f9c3aedf56493f3ce5
Accessing efix metadata ...
Processing efix label "IV16920s02" ...
Verifying efix control file ...

+-----------------------------------------------------------------------------+
Installp Prerequisite Verification
+-----------------------------------------------------------------------------+
Verifying prerequisite file ...
Checking prerequisites ...

Prerequisite Number: 1
   Fileset: devices.pciex.771000801410b003.rte
[..]
[..]
[..]

*** NOTICE ***
This efix package requires the target system to be rebooted after the current
operation is complete. It is recommended that you reboot the target system as
soon as possible after installation to avoid disruption of current functionality.

+-----------------------------------------------------------------------------+
Efix State
+-----------------------------------------------------------------------------+
Setting efix state to: REBOOT REQUIRED

+-----------------------------------------------------------------------------+
Boot Image Processing
+-----------------------------------------------------------------------------+
Rebuilding boot image ...
bosboot: Boot image is 49180 512 byte blocks.
Successfully rebuilt boot image.

+-----------------------------------------------------------------------------+
Operation Summary
+-----------------------------------------------------------------------------+
Log file is /var/adm/ras/emgr.log

EPKG NUMBER       LABEL               OPERATION              RESULT
1                 IV16920s02          INSTALL                SUCCESS
[..]

  • a reboot is required after efix installation :
padmin# shutdown -force -restart

Etherchannel 802.3 and SEA creation
  • after VIOS reboot recreate Etherchannel adapter and Shared Ethernet Adapter on VIO Server :
rmtcpip -all
mkvdev -lnagg ent0 ent1 ent2 ent3 -attr mode=8023ad
mkvdev -sea ent6 -vadapter ent5 -default ent5 -defaultid 999 -attr ha_mode=auto ctl_chan=ent4
mkvdev -vlan ent7 -tagid 1122
mktcpip -hostname diamond -inetaddr 10.240.122.82 -interface en8 -netmask 255.255.255.0 -gateway 10.240.122.254 -start

  • the only solution to check etherchannel 802.3ad consitency is to run an entstat on SEA adapter. Have look on those fields :
    • Link Status : Up or Down.
    • Media Speed Running : Currently running speed on adapter.
    • LACP Actor Activity : Is adapter on host sending LACP BPDU ? (Active or Passive)
    • LACP Actor Synchronization : Is adapter synchronized with its partner (switch) ?
    • LACP Partner Activity : Is switch sending LACP BPDU ? (Active or Passive)
    • LACP Partern Activity : Is switch synchronized with our host ?
  • The best solution is to have partner and actor activity set to active
  • IBM entstat is not really readable, it's a shame. Here is a little script (in attachement) to check LACP configuration on a host :
#./vio_lacp.sh
ent6 is an EtherChannel / IEEE 802.3ad Link Aggregation
There are 4 physical interfaces in ent6 EtherChannel 802.3ad Link Aggregation (ent0 ent1 ent2 ent3)
ent6 is part of ent7 Shared Ethernet Adapter


iface link  speed                device_type                                                            hw_mac               actor_a    actor_s    partner_a  partner_s
----  ----  -----                ----------                                                             ------               ------     ------     ---------  ---------
ent0| Up|    10 Gbps Full Duplex|  10 Gb Ethernet-SR PCI Express Dual Port Adapter (771000801410b003)|    00:21:5e:e2:56:a8|   Active|    IN_SYNC|   Active|    IN_SYNC|
ent1| Up|    10 Gbps Full Duplex|  10 Gb Ethernet-SR PCI Express Dual Port Adapter (771000801410b003)|    00:21:5e:e2:56:a8|   Active|    IN_SYNC|   Active|    IN_SYNC|
ent2| Up|    10 Gbps Full Duplex|  10 Gb Ethernet-SR PCI Express Dual Port Adapter (771000801410b003)|    00:21:5e:e2:56:a8|   Active|    IN_SYNC|   Active|    IN_SYNC|
ent3| Up|    10 Gbps Full Duplex|  10 Gb Ethernet-SR PCI Express Dual Port Adapter (771000801410b003)|    00:21:5e:e2:56:a8|   Active|    IN_SYNC|   Active|    IN_SYNC|

Etherchannel 802.3ad administration
  • remove adapter ent1 from Etherchannel 802.3ad adapter with ent5 SEA and ent4 Etherchannel :
# cfglnagg -rm -parent ent5 ent4 ent1

  • add adapter en1 from Etherchannel 802.3ad adapter with ent5 SEA and ent4 Etherchannel :
cfglnagg -add -parent ent5 ent4 ent1