lacp suspend-individual

When an end-host configured with port-channel LACP or NIC- or link-teaming, it requires the adjacent switch ports to receive LACP PDU to be able to start the negotiation and bring the port-channel to Up and Operational status. Without the end-host sending these LACP PDU, the switch ports will keep its status as down.

It sounds good to have this feature until you have a situation where the end-host needs to bring up just a single link while booting and without port-aggregation (LACP) it failed to send the LACP PDU to the switch. This will make the adjacent switch port to be down as part of its LACP negotiation.

This situation occurs when an end-host boots up with PXE boot to get an image but only enable a single link without LACP. The end-host’s LACP will only ever start (or to be configured) when the end-host received and installed the complete image.

In this scenario we can make the switch to bring the single interface up when the adjacent end-host is up even without sending LACP PDU. This can be achieved by telling the switch not to suspend the port if it didn’t receive LACP PDU.

lacp suspend-individual is a default configuration on Nexus switches to suspend the port if it didn’t receive LACP PDU.

The two N5K below connected to each other via port e1/1-2. Let’s assume that N5K-1 is the switch and N5K-2 is the end-host.

Ideally, both switches are configure using port-channel with LACP dynamic negotiation but for the purpose of this test, only N5K-1 is configured with LACP and N5K-2 pretend to be an end-host that just booted up with no port-channel configuration.

For testing, I’ve also put an SVI VLAN 400 in N5K1 with IP address 12.12.12.1/24 and N5K-2 with IP address 12.12.12.2/24. Configuration below.

! @ N5K-1
interface vlan 400
 ip address 12.12.12.1/24
 no shut
 
int e1/1-2
 switchport
 switchport mode access
 switchport access vlan 400
 channel-group 400 mode active
 no shut
 
int port 400
 switchport
 switchport mode access
 switchport access vlan 400
 no shut
 
N5K-1# sh port-chan sum
<omitted>
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
 Channel
--------------------------------------------------------------------------------
400 Po400(SD) Eth LACP Eth1/1(D) Eth1/2(D)

! @ N5K-2
int e1/1-2
 no switchport
 shut
 
int e1/1
 ip address 12.12.12.2/24

Now, let’s bring the N5K-2 (end-host) interface e1/1 up.

N5K-1# sh port-chan sum
<omitted>
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
 Channel
--------------------------------------------------------------------------------
400 Po400(SD) Eth LACP Eth1/1(s) Eth1/2(D)

N5K-1(config-if)# sh int status

--------------------------------------------------------------------------------
Port Name Status Vlan Duplex Speed Type
--------------------------------------------------------------------------------
Eth1/1 -- unknown e 400 full 10G SFP-H10GB-C
Eth1/2 -- notconnec 400 full 10G SFP-H10GB-C

N5K-1 (switch) port e1/1 will also come up with interface status as unknown and the port-channel status as suspended on port e1/1. Also watch that port-channel 400 status as SD - Switched-Down.

The LACP PDU counter also showing that N5K-1 (switch) is not receiving any from the end-host.

N5K-1(config-if)# show lacp counter
 LACPDUs Marker Marker Response LACPDUs
Port Sent Recv Sent Recv Sent Recv Pkts Err
---------------------------------------------------------------------
port-channel400
Ethernet1/1 165 0 0 0 0 0 0

Let’s disable the lacp suspend-individual and see what happens.

N5K-1(config-if)# int port 400
N5K-1(config-if)# no lacp suspend-individual 
ERROR: Cannot set/reset lacp suspend-individual for port-channel400 that is admin up
N5K-1(config-if)# shut
N5K-1(config-if)# no lacp suspend-individual 
Warning: !! Disable lacp suspend-individual only on port-channel with edge ports. Disabling this on network port port-channel could lead to loops.! 
N5K-1(config-if)# no shut

On a side note, you need to shut the port-channel before disabling the lacp suspend-individual command.

N5K-1# sh port-chan sum
<omitted>
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
 Channel
--------------------------------------------------------------------------------

400 Po400(SD) Eth LACP Eth1/1(I) Eth1/2(D) 
N5K-1# sh int status

--------------------------------------------------------------------------------
Port Name Status Vlan Duplex Speed Type
--------------------------------------------------------------------------------
Eth1/1 -- connected 400 full 10G SFP-H10GB-C
Eth1/2 -- notconnec 400 full 10G SFP-H10GB-C

See that the N5K1 port member e1/1 is now turns I - Individual and interface status shows as connected.

N5K2 (end-host) interface e1/1 status is connected and UP/UP.

N5K-2# sh ip int brief
IP Interface Status for VRF "default"(1)
Interface IP Address Interface Status
Po300 10.1.22.22 protocol-down/link-up/admin-up 
Eth1/1 12.12.12.2 protocol-up/link-up/admin-up 
N5K-2# sh int status

--------------------------------------------------------------------------------
Port Name Status Vlan Duplex Speed Type
--------------------------------------------------------------------------------
Eth1/1 -- connected routed full 10G SFP-H10GB-C
Eth1/2 -- disabled 1 full 10G SFP-H10GB-C

N5K2 (end-host) now has the L3 connectivity to reach the PXE server (wherever it might be) and ping test to N5K2 VLAN 400 IP address (gateway) also successful.

N5K-2# ping 12.12.12.1
PING 12.12.12.1 (12.12.12.1): 56 data bytes
36 bytes from 12.12.12.2: Destination Host Unreachable
Request 0 timed out
64 bytes from 12.12.12.1: icmp_seq=1 ttl=254 time=29.158 ms
64 bytes from 12.12.12.1: icmp_seq=2 ttl=254 time=2.044 ms
64 bytes from 12.12.12.1: icmp_seq=3 ttl=254 time=1.394 ms
64 bytes from 12.12.12.1: icmp_seq=4 ttl=254 time=1.397 ms

--- 12.12.12.1 ping statistics ---
5 packets transmitted, 4 packets received, 20.00% packet loss
round-trip min/avg/max = 1.394/8.498/29.158 ms