Dealing with Excessive Multicast Traffic

  • 2
  • Question
  • Updated 2 years ago
So we continue to tune our wireless network.  While tweaking several settings with support we noticed what seems to be a disproportionate amount of tx multicast traffic.   For example:


BHS-WAP-1103#show inter wifi0 count | include cast
        97236 rx unicast data frames
        1138 rx multicast data frames
        532 rx broadcast data frames
        230519 tx unicast data frames
        829935 tx multicast data frames
        527946 tx broadcast data frames
7% unicast data tx retry rate

Every access point is on a single management VLAN.   So I simply plugged a laptop with Wireshark into that VLAN and captured mulitcast traffic and can see that it is coming from all over the place.  This is traffic generated by clients connected to the wireless network... BNJP, CUPS, ICMPv6 and v4 multicast solicitation and listener report, piles of Broadcast LLC packets sourced from the Aerohive APs, mDNS, NBNS, etc.   Basically a myriad of traffic from clients and APs across the network... all on that VLAN.  

This is a high school with about 70 access points and several hundred connected clients at any given time.   Any suggestions regarding how best to mitigate the amount of multicast traffic getting propagated within this VLAN?

I can keep it from traversing WAN trunks by simply dropping multicast on those interfaces.  Maybe I need to do the same on each trunk from the PoE edge switches to the core?
Photo of Tony Andrews

Tony Andrews

  • 52 Posts
  • 5 Reply Likes

Posted 2 years ago

  • 2
Photo of Jonathan Hurtt

Jonathan Hurtt

  • 98 Posts
  • 48 Reply Likes
Tony,

You stated "Every access point is on a single management VLAN", but is this the same VLAN that clients are also on? 

If so then one way would be to segment your Wireless clients to another VLAN to keep the management traffic from having to be forwarded to the wireless interfaces. 

I am not a fan of dropping any traffic unless you know exactly what it is used for, some applications use non-unicast in many ways.

Hope that helps.
Photo of Tony Andrews

Tony Andrews

  • 52 Posts
  • 5 Reply Likes
Hi Johnathan...

No, clients associate with different VLANs based upon SSID.  Maybe we have something configured incorrectly in our policies?  

Our setup:

Every access point is connected to a trunk port on a Cisco PoE switch.  Each trunk port is set with Native VLAN 352 and allows vlans 351-354.   VLAN 352 is the 10.19.x.x network.

So we pre-assigned ip addresses to each access point by creating a reservation for each AP in our DHCP server and when the AP is connected it defaults to VLAN 352 and gets the pre-assigned ip address.

In our network policy we have three SSIDs, each associated with a different VLAN... 351, 353 and 354.  

However... even with this setup, multicast traffic would still be transmitted on each ap interface would it not?  
Photo of Jonathan Hurtt

Jonathan Hurtt

  • 98 Posts
  • 48 Reply Likes
Tony,

With your description, multicast/broadcast from VLAN352 should not be traversing the wireless medium since there are no clients on VLAN352 that need the non-unicast traffic. If you were sniffing traffic on VLAN 352 that is probably not the traffic that was represented by the "show inter wifi0 count | include cast" command. 

First have you taken a look at the traffic on VLAN 351, 353 or 354? I will say that you shouldn't see the Broadcast LLC from Aerohive APs on those VLANs. There might be other multicast traffic that you would want to focus on. I do agree that is odd to see that nearly 4x amount of multicast traffic.

Also i would clear the counters (clear interface wifi0 counter) and see if this is just a spike or a continuing trend. Also look at the sub interfaces (wifi0.1 or wifi0.2) and see if one sub interface (which is a  SSID or VLAN in your case) is more than others. 
Photo of Tony Andrews

Tony Andrews

  • 52 Posts
  • 5 Reply Likes
Thanks Jonathan...

I did my packet capture on a trunk port.  So the traffic I observed was traffic from all VLANs.

My guess... as you are suggesting ... is that the majority of the non unicast traffic is from just one VLAN.  I will check this out.
Photo of Tony Andrews

Tony Andrews

  • 52 Posts
  • 5 Reply Likes
I noted this morning one particular anomaly with respect to a certain subinterface:

BHS-WAP-2127#show inter wifi0.1 count | include multicast
10 rx multicast data frames
0 multicast echo
0 tx multicast data frames
28232 multicast/broadcast vlan mismatch
The vlan mismatch count is growing at a very high rate just on this subinterface.
Photo of Tony Andrews

Tony Andrews

  • 52 Posts
  • 5 Reply Likes
So after looking at subinterfaces I'm not sure I know much more.  I cleared counters and then issued the commands below about a second apart.  You can see that tx multicast frames  on wifi0 increment quickly.  Not so much on the subinterfaces. 

I am still a little perplexed on the multicast/broadcast vlan mismatch numbers.  Why so high for 0.1 and not 0.3?   I'm assuming it has to do with the fact that the AP is connected to a trunk port with a native vlan assigned to it.  But why one subinterface and not the other?

BHS-WAP-2129#show inter wifi0 counter | include multicast
3 rx multicast data frames
2027 tx multicast data frames
BHS-WAP-2129#show inter wifi0 counter | include multicast
23 rx multicast data frames
2498 tx multicast data frames
BHS-WAP-2129#show inter wifi0.1 counter | include multicast
0 rx multicast data frames
0 multicast echo
0 tx multicast data frames
4517 multicast/broadcast vlan mismatch
BHS-WAP-2129#show inter wifi0.1 counter | include multicast
0 rx multicast data frames
0 multicast echo
0 tx multicast data frames
4779 multicast/broadcast vlan mismatch
BHS-WAP-2129#show inter wifi0.2 counter | include multicast
0 rx multicast data frames
0 multicast echo
0 tx multicast data frames
0 multicast/broadcast vlan mismatch
BHS-WAP-2129#show inter wifi0.3 counter | include multicast
44 rx multicast data frames
0 multicast echo
0 tx multicast data frames
200 multicast/broadcast vlan mismatch
BHS-WAP-2129#show inter wifi0.3 counter | include multicast
54 rx multicast data frames
0 multicast echo
0 tx multicast data frames
220 multicast/broadcast vlan mismatch
Photo of Jonathan Hurtt

Jonathan Hurtt

  • 98 Posts
  • 48 Reply Likes
Tony, that is a very good point, i would suggest opening a support case to see if they can zero in on this. Please report back if you find a solution.
Photo of Tony Andrews

Tony Andrews

  • 52 Posts
  • 5 Reply Likes
I think that the solution in this case has everything to do with the wired network and not so much the wireless network.

Consider this...   In an effort to simplify management we created a VLAN for each SSID we created.  Each of these VLANs is trunked to every location in our WAN... 9 school buildings, layer 2 all the way.  

So when a Samsung phone at the high school starts blasting out mDNS frames, those frames are propagated everywhere that particular VLAN exists... across WAN trunks to every single AP in the enterprise.

To verify this, I simply killed multicast traffic on the WAN trunk port at one of our elementary buildings.  The result was a dramatic drop in tx multicast frames... from something like 100-200 a second to less than 10 a second.

So in our effort to be efficient we created a problem by creating too large a broadcast domain.   The solution is fairly simple...  we just add layer 3 at each building and stop pushing those VLANs (except the management VLAN) across the metro ethernet trunks.
Photo of Luke Harris

Luke Harris

  • 265 Posts
  • 18 Reply Likes
This has made for interesting reading - I've been noticing large amounts of broadcast traffic on the AP management VLAN. Much like your setup Tony, the wireless management VLAN exists across multiple sites. We have a number of access points at a particular site that seem to be having issues contacting the default gateway (IP tracking configuration) which is causing them to broadcast the _ac SSID - at least according to HiveManager.

Could this be caused by excessive multicast/broadcast traffic on the management VLAN?