ISSUE: 802.3ad LACP does not Work.

[Voodoo]

Bonding multiple NIC’s with 802.3ad using LACP in Linux sets them with the “Transmit Hash Policy: layer2 (0)”. A hash based on the contents of the frame is computed and used to determine output port selection. The default hash policy only examines Layer 2 (source and destination MAC addresses) data in the frame. If all traffic is destined for a single-host or multiple hosts on a separate network connected via a router, only one link will be used when using Layer 2 hashing. For that reason, I would suggest using Layer2+3 or Layer3+4 hashing instead depending on your traffic pattern.

cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 1
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable

Slave Interface: eno1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: d8:9d:67:c6:66:11
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0

Slave Interface: enp1s0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: d8:9d:67:c6:66:10
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0

sudo nano /sys/class/net/bond0/bonding/xmit_hash_policy

change "layer2 0" to "layer3+4 1"

Reboot

Andy Gospodarek touched on this over at https://bugzilla.redhat.com/show_bug.cgi?id=586557

I would suggest adding the option:

“xmit_hash_policy=layer2+3″

to your bonding options.

More details about these options can be found in the bonding documentation about the xmit_hash_policy option:

xmit_hash_policy

Selects the transmit hash policy to use for slave selection in
balance-xor and 802.3ad modes. Possible values are:

layer2

Uses XOR of hardware MAC addresses to generate the
hash. The formula is

(source MAC XOR destination MAC) modulo slave count

This algorithm will place all traffic to a particular
network peer on the same slave.

This algorithm is 802.3ad compliant.

layer2+3

This policy uses a combination of layer2 and layer3
protocol information to generate the hash.

Uses XOR of hardware MAC addresses and IP addresses to
generate the hash. The formula is

(((source IP XOR dest IP) AND 0xffff) XOR
( source MAC XOR destination MAC ))
modulo slave count

This algorithm will place all traffic to a particular
network peer on the same slave. For non-IP traffic,
the formula is the same as for the layer2 transmit
hash policy.

This policy is intended to provide a more balanced
distribution of traffic than layer2 alone, especially
in environments where a layer3 gateway device is
required to reach most destinations.

This algorithm is 802.3ad compliant.

layer3+4

This policy uses upper layer protocol information,
when available, to generate the hash. This allows for
traffic to a particular network peer to span multiple
slaves, although a single connection will not span
multiple slaves.

The formula for unfragmented TCP and UDP packets is

((source port XOR dest port) XOR
((source IP XOR dest IP) AND 0xffff)
modulo slave count

For fragmented TCP or UDP packets and all other IP
protocol traffic, the source and destination port
information is omitted. For non-IP traffic, the
formula is the same as for the layer2 transmit hash
policy.

This policy is intended to mimic the behavior of
certain switches, notably Cisco switches with PFC2 as
well as some Foundry and IBM products.

This algorithm is not fully 802.3ad compliant. A
single TCP or UDP conversation containing both
fragmented and unfragmented packets will see packets
striped across two interfaces. This may result in out
of order delivery. Most traffic types will not meet
this criteria, as TCP rarely fragments traffic, and
most UDP traffic is not involved in extended
conversations. Other implementations of 802.3ad may
or may not tolerate this noncompliance.

The default value is layer2. This option was added in bonding
version 2.6.3. In earlier versions of bonding, this parameter
does not exist, and the layer2 policy is the only policy. The
layer2+3 value was added for bonding version 3.2.2.

I would also suggest adding these options to ifcfg-bond0:

BONDING_OPTS=”miimon=80 mode=4 xmit_hash_policy=layer2+3”

and removing this line and only this line from modprobe.conf:

options bond0 miimon=80 mode=4

Each time someone has asked about this issue, it turns out to be a configuration issue, so I am going to close this bug. Feel free to re-open it if you are still having problems after switching to layer2+3 or layer3+4 and provide more details about the traffic being transmitted by the bond.

REF: https://forums.unraid.net/topic/57064-how-to-set-8023ad-xmit_hash_policy-transmit-hash-policy/

• This topic was modified 5 months, 2 weeks ago by Voodoo.
• This topic was modified 5 months, 2 weeks ago by Voodoo.

[Voodoo]

Most people tend to just plug their network card in and expect it to go. However there are some of us who are always tinkering and trying to get the most out of something as we can. And in this post, I am going to touch on a few things that I have had to educate myself on as I went along.

This day and age we find ourselves looking at SMB 3.0 Multichannel so that a computer can transfer information over multiple network connections at once. In the old modem days, this method was called “shotgun” where you had 2 modems and 2 phone lines connected to an ISP that utilized the shotgun technology. This new SMB 3.0 Multichannel was released with Windows 8.1 and is supposed to be made even more easier in Win10. The drawback to this is your hardware has to have certain features and enough cpu cores to handlethe divided communications. This is on both the sending and the receiving computers. With my home lab having older workstations, they seem yo have the features but I have never had any luck with it. I have seen in quite a few placea that SAMBA can be setup to use the SMB 3.0 multichannel and it works well at least when the server is Linux and the client is Windows. When my quad Intel network cards get here for my PeppermintOS 10 Respin file sever, I may dig into this with the PeppermintOS 10 Respin clients…

Anyway, the project I am tinkering with right now is getting 802.3ad LACP aggregation working between 2 workstations (HP Z820 and a Asus P8P67 WS Revolution) along with a TP Link TL-SG2424 managed switch. They performed this task while they were both loaded with Win7 Pro, but M$ broke that functionality with Win10. With PeppermintOS 10 Respin, it is easy to bind the network interfaces. However, the simplified NetworkManager sets the bond at Transmit Hash Protocal to “layer2”. It seems that “layer2” only transmits over single network connection. So it seems that I have to change to “layer3+4” to get the Win7 802.3ad LACP aggregation back. I am also going to need to change thr LACP setting from slow to fast also. I saw ChrisTitusTech talk about netplan, but I also saw that the netplan on PeppermintOS 10 Respin turns over control to the simplified NetworkManager.

[Voodoo]

TEST #1
-HP Z820 which has the Intel 82574L and the Intel 82579LM
-Asus P8P67 WS Revolution which has the Intel 82574L and the Intel 82579V

Teaming/Bonding: Due to an error in Windows 10 with Intel ANS, these cannot be teamed.
SMB Multichannel: Both network cards have traffic in Windows 10 when transferring files between. However, only a single network card is used for a transfer at a time.

Test #1
2x Optiplex 990 DT with Dell TMGR6 (BCM 5719) Quad Port Gigabit NIC
TP Link TL-SG2424

Teaming/Bonding:
SMB Multichannel:
First, I disable the motherboard NIC. I then bond all 4 of the ports on the TMGR6 into a single bond setting the mode to “802.3ad”. I then setup the TL-SG2424 to bond those ports together in an LAG using LACP.

[Author]

Posts