Lots of broadcast LLC traffic coming from access points

  • 4
  • Question
  • Updated 2 years ago
  • Answered
We now have close to 100 Aerohive access points on our network and in about 10 seconds you can see thousands of broadcast LLC (SNAP) packets coming from the Aerohive APs. What is this traffic and can it be minimized in some way?
Photo of Chip Andrews

Chip Andrews

  • 12 Posts
  • 0 Reply Likes
  • frustrated

Posted 4 years ago

  • 4
Photo of Chip Andrews

Chip Andrews

  • 12 Posts
  • 0 Reply Likes
Here's a screenshot of a typical Wireshark capture:

Photo of BJ

BJ, Champ

  • 374 Posts
  • 45 Reply Likes
Here's a nice link on 802.11 packets, including the use of LLC...
http://www.wildpackets.com/resources/compendium/wireless_lan/wlan_packets

Best,
BJ
Photo of Chip Andrews

Chip Andrews

  • 12 Posts
  • 0 Reply Likes
Unfortunately it doesn't answer the question of why.  The Aerohive's are emitting these packets on the managment VLAN and not the actual traffic VLAN where the clients are placed. 
Photo of Nick Lowe

Nick Lowe, Official Rep

  • 2491 Posts
  • 451 Reply Likes
Could you clarify what problem this is causing you?
Photo of Chip Andrews

Chip Andrews

  • 12 Posts
  • 0 Reply Likes
No negative effects at this point but once we have 200 or so the increase in broadcast traffic will be significant enough to make network diagnostics very difficult as well as decrease overall network throughput.  Again - there are broadcast packets sent to every single endpoint on the subnet.
Photo of BJ

BJ, Champ

  • 374 Posts
  • 45 Reply Likes
Are you using multiple vlans and is the LLC traffic traversing each vlan or just the mgmt vlan?
Photo of Chip Andrews

Chip Andrews

  • 12 Posts
  • 0 Reply Likes
Just mgmt so far - still need to check wireless vlan
Photo of BJ

BJ, Champ

  • 374 Posts
  • 45 Reply Likes
If it is indeed only on the mgmt vlan, and only mgmt traffic is traversing, I can't foresee any realized issues. Are you capturing this through the wired connection on the AP's or though the air?
Photo of Chip Andrews

Chip Andrews

  • 12 Posts
  • 0 Reply Likes
Wired on the APs and only on the MgMt interface.  I checked the wireless VLAN and there is no evidence of the LLC/SNAP broadcasts - wired or wireless.


The problem is that the Mgmt Interface is on the primary VLAN which means it is co-mingled with all the other normal network traffic - for which broadcast traffic has increased 10x since the introduction of the Aerohives.  Perhaps the solution is to segregate the Aerohive Mgmt traffic to a private VLAN.
Photo of BJ

BJ, Champ

  • 374 Posts
  • 45 Reply Likes
Yeah, segmenting mgmt traffic is typically good practice across the board, not just AH gear.
Photo of Chris Stuart

Chris Stuart

  • 3 Posts
  • 0 Reply Likes
I know this is an old thread, and maybe I should open a new one, but I will try here first. We have the exact same problem, except ours seems to be causing a DoS type of symptom . We do have an open support case about this issue. As you can see in the screen shot, every AP is sending the exact same packet twice. We have noticed that rebooting the APs seems to clear up the problem and allow clients to connect again without issues. This seems to last for roughly 6-8 hours, which we are a school system, so it lasts just long enough. This capture is from a mirrored port of an ap. We have set with supplemental cli to limit the data rate of multicast traffic at the eth0 interface to 50 kb/s on all APs at this site. Most of the APs are 121's running 6.4r1g which is recommended by aerohive for 121's. Any thoughts? 
Photo of Nick Lowe

Nick Lowe, Official Rep

  • 2491 Posts
  • 451 Reply Likes
If you are seeing a DoS-like symptom, have you checked for a broadcast storm in the network, typically caused by a switching loop?

https://en.wikipedia.org/wiki/Switching_loop
Photo of Chris Stuart

Chris Stuart

  • 3 Posts
  • 0 Reply Likes
We have searched, and came up empty on that. We have HP switches and we have turned on loop protection on all the edge ports with an administrative shutdown on the port if a loop is detected.
Photo of Tom Hulley

Tom Hulley

  • 1 Post
  • 0 Reply Likes
We have the same LLC broadcast traffic. Its not causing a problem yet, but it amounts to about 1 packet per second per AP. We have over 200 AP's and plan to add many more. Segmenting the management traffic only pushes the broadcast to another VLAN. Its still traffic on the wire that consumes bandwidth (especially on uplink ports with multiple VLAN's). I would like to know what purpose this traffic is for. It seems quite inefficient and perhaps unnecessary.
Photo of Chris Stuart

Chris Stuart

  • 3 Posts
  • 0 Reply Likes
We finally discovered that for whatever reason, band steering was turned on, but we did not turn this on. Also we have bonjour gateway on, and recently added about 50 more apple devices. We made an ACL on our switches to not let any traffic, bonjour included, able to communicate with any other vlan except for our server network. This seemed to decrease the LLC broadcast traffic. It got so bad that it was like a DoS and all our APs were at 100% cpu utilization. We also put broadcast and multicast limits on our switches. So far we have been good after these changes. I would say that the firmware has something to do with the changes we needed to make. we are running the following firmware for our APs:
120 - 6.2r1c
121 - 6.4r1g
230 - 6.8
330 - 6.5r4
340 - 6.5r4
Photo of Roberto Casula

Roberto Casula

  • 2 Posts
  • 3 Reply Likes
In general, this traffic is related to Aerohive's AMRP protocol which provides most of the distributed control-plane functions of HiveOS. There are various types of packets that are broadcast (at layer-2) by each AP, including hello packets, keepalives, link state updates, hive broadcasts, l3 roaming updates and client updates. Basically any update which needs to be received by all other APs in the management broadcast domain will be sent as L2 broadcasts packets using this LLC encapsulation.

The size and volume of some of these will be dependent on the configuration and environment. I can imagine that a particular problem in the environment, for example clients which are unnecessarily roaming between APs very frequently, could cause a dramatic increase in the volume of this traffic.

It is generally a good idea to keep the number of APs in each management broadcast domain as low as possible - a rule of thumb I used in the past was around 20 APs but wherever possible I tried to keep it even lower than this. You can still retain all the distributed capabilities, including L2 and L3 roaming, even when APs are on different management VLANs (e.g. a separate management VLAN for each floor in a building), either by configuring static AMRP neighbours or allowing AMRP communication over the wireless interface.

If the levels are causing a problem, the best thing to do would be to log a support ticket - there are debug logs that can be enabled to capture and log the AMRP traffic and hopefully identify if there is anything unexpected going on, but in some networks it may just be "normal" behaviour for the protocol and the best thing to do would be to optimise things by logically segmenting the management broadcast domains, e.g. separate AP management VLAN per building floor or per access switch/stack.