ISSUE: 802.3ad LACP does not Work.

Home Forums Operating Systems Linux PeppermintOS ISSUE: 802.3ad LACP does not Work.

Viewing 3 posts - 1 through 3 (of 3 total)
  • Author
    Posts
  • #765
    Voodoo
    Keymaster

    Bonding multiple NIC’s with 802.3ad using LACP in Linux sets them with the “Transmit Hash Policy: layer2 (0)”. A hash based on the contents of the frame is computed and used to determine output port selection. The default hash policy only examines Layer 2 (source and destination MAC addresses) data in the frame. If all traffic is destined for a single-host or multiple hosts on a separate network connected via a router, only one link will be used when using Layer 2 hashing. For that reason, I would suggest using Layer2+3 or Layer3+4 hashing instead depending on your traffic pattern.

    cat /proc/net/bonding/bond0

    Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
    
    Bonding Mode: IEEE 802.3ad Dynamic link aggregation
    Transmit Hash Policy: layer2 (0)
    MII Status: up
    MII Polling Interval (ms): 1
    Up Delay (ms): 0
    Down Delay (ms): 0
    Peer Notification Delay (ms): 0
    
    802.3ad info
    LACP rate: slow
    Min links: 0
    Aggregator selection policy (ad_select): stable
    
    Slave Interface: eno1
    MII Status: up
    Speed: 1000 Mbps
    Duplex: full
    Link Failure Count: 0
    Permanent HW addr: d8:9d:67:c6:66:11
    Slave queue ID: 0
    Aggregator ID: 1
    Actor Churn State: none
    Partner Churn State: none
    Actor Churned Count: 0
    Partner Churned Count: 0
    
    Slave Interface: enp1s0
    MII Status: up
    Speed: 1000 Mbps
    Duplex: full
    Link Failure Count: 0
    Permanent HW addr: d8:9d:67:c6:66:10
    Slave queue ID: 0
    Aggregator ID: 1
    Actor Churn State: none
    Partner Churn State: none
    Actor Churned Count: 0
    Partner Churned Count: 0

    sudo nano /sys/class/net/bond0/bonding/xmit_hash_policy

    change "layer2 0" to "layer3+4 1"

    Reboot

    Andy Gospodarek touched on this over at https://bugzilla.redhat.com/show_bug.cgi?id=586557

    I would suggest adding the option:

    “xmit_hash_policy=layer2+3″

    to your bonding options.

    More details about these options can be found in the bonding documentation about the xmit_hash_policy option:

    xmit_hash_policy

    Selects the transmit hash policy to use for slave selection in
    balance-xor and 802.3ad modes. Possible values are:

    layer2

    Uses XOR of hardware MAC addresses to generate the
    hash. The formula is

    (source MAC XOR destination MAC) modulo slave count

    This algorithm will place all traffic to a particular
    network peer on the same slave.

    This algorithm is 802.3ad compliant.

    layer2+3

    This policy uses a combination of layer2 and layer3
    protocol information to generate the hash.

    Uses XOR of hardware MAC addresses and IP addresses to
    generate the hash. The formula is

    (((source IP XOR dest IP) AND 0xffff) XOR
    ( source MAC XOR destination MAC ))
    modulo slave count

    This algorithm will place all traffic to a particular
    network peer on the same slave. For non-IP traffic,
    the formula is the same as for the layer2 transmit
    hash policy.

    This policy is intended to provide a more balanced
    distribution of traffic than layer2 alone, especially
    in environments where a layer3 gateway device is
    required to reach most destinations.

    This algorithm is 802.3ad compliant.

    layer3+4

    This policy uses upper layer protocol information,
    when available, to generate the hash. This allows for
    traffic to a particular network peer to span multiple
    slaves, although a single connection will not span
    multiple slaves.

    The formula for unfragmented TCP and UDP packets is

    ((source port XOR dest port) XOR
    ((source IP XOR dest IP) AND 0xffff)
    modulo slave count

    For fragmented TCP or UDP packets and all other IP
    protocol traffic, the source and destination port
    information is omitted. For non-IP traffic, the
    formula is the same as for the layer2 transmit hash
    policy.

    This policy is intended to mimic the behavior of
    certain switches, notably Cisco switches with PFC2 as
    well as some Foundry and IBM products.

    This algorithm is not fully 802.3ad compliant. A
    single TCP or UDP conversation containing both
    fragmented and unfragmented packets will see packets
    striped across two interfaces. This may result in out
    of order delivery. Most traffic types will not meet
    this criteria, as TCP rarely fragments traffic, and
    most UDP traffic is not involved in extended
    conversations. Other implementations of 802.3ad may
    or may not tolerate this noncompliance.

    The default value is layer2. This option was added in bonding
    version 2.6.3. In earlier versions of bonding, this parameter
    does not exist, and the layer2 policy is the only policy. The
    layer2+3 value was added for bonding version 3.2.2.

    I would also suggest adding these options to ifcfg-bond0:

    BONDING_OPTS=”miimon=80 mode=4 xmit_hash_policy=layer2+3”

    and removing this line and only this line from modprobe.conf:

    options bond0 miimon=80 mode=4

    Each time someone has asked about this issue, it turns out to be a configuration issue, so I am going to close this bug. Feel free to re-open it if you are still having problems after switching to layer2+3 or layer3+4 and provide more details about the traffic being transmitted by the bond.

    REF: https://forums.unraid.net/topic/57064-how-to-set-8023ad-xmit_hash_policy-transmit-hash-policy/

    • This topic was modified 2 months, 1 week ago by Voodoo.
    • This topic was modified 2 months, 1 week ago by Voodoo.
    #768
    Voodoo
    Keymaster

    Most people tend to just plug their network card in and expect it to go. However there are some of us who are always tinkering and trying to get the most out of something as we can. And in this post, I am going to touch on a few things that I have had to educate myself on as I went along.

    This day and age we find ourselves looking at SMB 3.0 Multichannel so that a computer can transfer information over multiple network connections at once. In the old modem days, this method was called “shotgun” where you had 2 modems and 2 phone lines connected to an ISP that utilized the shotgun technology. This new SMB 3.0 Multichannel was released with Windows 8.1 and is supposed to be made even more easier in Win10. The drawback to this is your hardware has to have certain features and enough cpu cores to handlethe divided communications. This is on both the sending and the receiving computers. With my home lab having older workstations, they seem yo have the features but I have never had any luck with it. I have seen in quite a few placea that SAMBA can be setup to use the SMB 3.0 multichannel and it works well at least when the server is Linux and the client is Windows. When my quad Intel network cards get here for my PeppermintOS 10 Respin file sever, I may dig into this with the PeppermintOS 10 Respin clients…

    Anyway, the project I am tinkering with right now is getting 802.3ad LACP aggregation working between 2 workstations (HP Z820 and a Asus P8P67 WS Revolution) along with a TP Link TL-SG2424 managed switch. They performed this task while they were both loaded with Win7 Pro, but M$ broke that functionality with Win10. With PeppermintOS 10 Respin, it is easy to bind the network interfaces. However, the simplified NetworkManager sets the bond at Transmit Hash Protocal to “layer2”. It seems that “layer2” only transmits over single network connection. So it seems that I have to change to “layer3+4” to get the Win7 802.3ad LACP aggregation back. I am also going to need to change thr LACP setting from slow to fast also. I saw ChrisTitusTech talk about netplan, but I also saw that the netplan on PeppermintOS 10 Respin turns over control to the simplified NetworkManager.

    #770
    Voodoo
    Keymaster

    TEST #1
    -HP Z820 which has the Intel 82574L and the Intel 82579LM
    -Asus P8P67 WS Revolution which has the Intel 82574L and the Intel 82579V

    Teaming/Bonding: Due to an error in Windows 10 with Intel ANS, these cannot be teamed.
    SMB Multichannel: Both network cards have traffic in Windows 10 when transferring files between. However, only a single network card is used for a transfer at a time.

    Test #1
    2x Optiplex 990 DT with Dell TMGR6 (BCM 5719) Quad Port Gigabit NIC
    TP Link TL-SG2424

    Teaming/Bonding:
    SMB Multichannel:
    First, I disable the motherboard NIC. I then bond all 4 of the ports on the TMGR6 into a single bond setting the mode to “802.3ad”. I then setup the TL-SG2424 to bond those ports together in an LAG using LACP.

Viewing 3 posts - 1 through 3 (of 3 total)
  • You must be logged in to reply to this topic.