L2 DoS - What to do about it?

  • 3
  • Question
  • Updated 3 months ago
  • Doesn't Need an Answer
I have been spammed with over 90 e-mails today from my APs at a single site (12 APs) due to "L2 DoS" conditions. In this case, "receiving probe req frames" over 1200 PPS, looks like it is all coming from a single MAC address.

This is all well and good to be notified that something bad is happening; the problem is that there doesn't seem to be anything I can *DO* about it.

There is no way to ban the MAC address, nothing I can do to tell the AP's to simply ignore it and stop e-mailing me about it, zilch. I already have the threshold on L2 DoS alerts set to only go every 5 minutes (so thankfully I'm not getting a message a minute), but it's still very irritating. I also seem to get a message from 3-5 APs simultaneously. Does this mean that the offending device is constantly moving around the site? Or just that it is within range of that number of APs, so they all feel the need to report it? (If it is the latter, I should only get a single e-mail - they are in the same Hive, they need to coordinate the notification rather than sending out separate messages.)

This really needs to be improved somehow because I'm sick of my phone exploding (e-mails) every time someone has a badly behaved Wifi device.
Photo of ZPrime

ZPrime

  • 4 Posts
  • 0 Reply Likes
  • annoyed

Posted 5 years ago

  • 3
Photo of Crowdie

Crowdie, Champ

  • 972 Posts
  • 272 Reply Likes
I had the same issue at a site and did a protocol analysis using Air Magnet WiFi Analyzer PRO. I found that a single client was the cause of the issue - in this case it was transmitting class three frames to access points it was not associated to. The client was an old Windows XP kiosk and when it was powered down the issue went away.

The reason you get multiple APs reporting an issue is that either multiple APs can detect the offending transmissions or multiple APs are being directly affected. It doesn't necessarily mean the client is moving around the site.

Aerohive APs have integrated packet capture capabilities and, if you don't own a product like Air Magnet WiFi Analyzer PRO, you can use these to capture the frames from one of the affected access points and display them in Wireshark.
Photo of Tash Hepting

Tash Hepting

  • 55 Posts
  • 29 Reply Likes
ZPrime,

I'll talk to product management and see what we're planning for enhancements in this area. Some sort of rollup or time-base suppression of notifications would help... Something along the lines of "Don't notify me more than x in y minutes for the same condition" seems like it would address the spam issue.

Many of these issues are occuring with clients who have not yet authenticated, or are experiencing issues at such a low level that there is nothing that we can do to prevent it. Probe are a great example of this - the client can send a probe any time he wants, regardless of his currently connected state. Anything we did to throttle that would end up throttling everyone. Your best bet in this situation is to find the offending client and remove it from the network. In addition to the notification spam, it is negatively impacting your network performance.

Regards,
Tash
Photo of Scott M.

Scott M., Sr. Support Engineer

  • 104 Posts
  • 8 Reply Likes
Banning a MAC address will not work to mitigate the L2DoS issue because attackers easily spoof MAC addresses. All you can really do is find the attacker or raise the threshold.

I like Tash"s idea of placing a limiter on the number of messages, but I think the message should also say something like... "this message repeated 90 times" so the admin is still aware of the frequency/duration of the attacks.
Photo of David Dippon

David Dippon

  • 19 Posts
  • 8 Reply Likes
ZPrime,

I have seen cases with customers who have successfully found "offending" clients
that have caused their AP to report L2 DoS. I know this can be like chasing a ghost or finding a needle in a hay stack.

I also had another customer who disabled the low data rates on his AP, and after which his L2 DoS messages disappeared.
Photo of Chris Kelly

Chris Kelly

  • 8 Posts
  • 0 Reply Likes
Just an update - this problem is still an issue.  I'm sitting here with 55 messages coming in from APs that appear to be close by a Samsung device that is spewing probe frames (probe req frames exceed 1200 PPM).  Same MAC, and the "client" MAC doesn't show up in the list of client devices, so I'm just deleting emails right and left.  probably from one of the Samsung WiFi apps that scan for wireless signals.  I've tested this with my personal Nexus tablet using something like the Nuts about Nets app.  Be nice if there was something in HiveManager that would let me just do a global block of an offending MAC.  I don't have time to hunt these things down, but if I break 'em they might find their way to the university Help Desk where they can be delt with easier.

Photo of Nick Lowe

Nick Lowe, Official Rep

  • 2491 Posts
  • 451 Reply Likes
Chris,

Assuming you have not already done so...

Are you able to disable all the 802.11b data rates in your environment?
If possible, could you try doing this and seeing if the issue reoccurs or at least improves?

Many universities have already done this for performance reasons.

You could well have bad acting clients that will continue regardless, but this is a good first step in the troubleshooting process. I have also observed L2 DoS issues to a greater extent where legacy, low data rates are offered.

I agree that it would be nice to see 'more options' here from reporting, triage and mitigation perspectives.

Nick





(Edited)
Photo of Chris Kelly

Chris Kelly

  • 8 Posts
  • 0 Reply Likes
Sniffing using the AP shows the traffic seems to be on the 5 ghz radio. Yes, we have dropped B data rates. The obnoxious client seems to be crying for a linksys router. The AP snswers with our SSIDs but it ignores that. Tried to trick it with a open linksys ssid but it would not connect. We tore the building apart looking for it and found zip. It was there Monday, 30 minutes Tuesday AM, all day Wednesday 8:27 to 5:20. Thinking that means it was outside in a car.
Photo of Loren

Loren

  • 48 Posts
  • 2 Reply Likes
This is what I found.  The problem is how the networks were added to Android device.  So you need to find the device and remove all the networks and then add them through the auto discovery.  With that being said you have to find the device! 



http://blog.taddong.com/2011/05/vulnerability-in-android-to-add-or-not.html






(Edited)
Photo of Chris Kelly

Chris Kelly

  • 8 Posts
  • 0 Reply Likes
We've had two snow days here. Ill expect the device to show up Monday. I took pictures of the cars in the lot out front facing the window with the main complainer AP. Lol - we've been going into offices and classrooms saying "Network police, show us your devices. "
Photo of Brad Nightingale

Brad Nightingale

  • 4 Posts
  • 0 Reply Likes
I dont get it, why would there be a feature like this that seems to provide no real value add, that you cant modify, and all you can do it turn it off?  Whats the point in having it?  Im getting the same stream of email, and its only 1ppm over the default threshold - I dont want to turn the notifications off in case there really is an issue - so whats the other option?
Photo of Crowdie

Crowdie, Champ

  • 972 Posts
  • 272 Reply Likes

We have a similar issue at a tertiary institution and it was almost impossible to track the offending device down.  In the end we blocked the device from associating to the wireless network and the owner logged a support call saying "they could connect yesterday but can't today".  This way we had the owner bring the offending device to us rather than having us chase it all over the campus.  Not a long term solution but useful when you have a few offending wireless clients.

Photo of Andrei Tunes

Andrei Tunes

  • 3 Posts
  • 0 Reply Likes
How did you block the device? Because I tried to block too, but I'm still receiving this messages.
Photo of Crowdie

Crowdie, Champ

  • 972 Posts
  • 272 Reply Likes
You will still get the errors as blocking a client will not stop it responding to probes.  The point was to get the owner of the offending device to identify themselves.  I have found that 99% of the time the user has absolutely no idea that their wireless device is causing an issue. 
Photo of Chris Kelly

Chris Kelly

  • 8 Posts
  • 0 Reply Likes
Update - tried the block trick, but no response. Since then I've seen many other devices cause similar issues. The lame solution here is to send all the notifications to an Outlook folder, glance over the subject lines, then just dump the folder. I dump between 50 and 100 entries.
Photo of Nick Lowe

Nick Lowe, Official Rep

  • 2491 Posts
  • 451 Reply Likes
Chris,

Are you not able to whip up some logic against the emails that you receive?

Were it me, I would handle the events represented by each email programatically to get a summation and actionable statistics.

I would not want the raw events being suppressed automatically as this would interfere with or preclude analysis.

Honestly, I do not see a problem with the currently implemented behaviour - playing devil's advocate perhaps, but should it not be incumbent on the administrator to find a way of processing those emails appropriately, they are just data points after all, based on what they want to see?

Would it not be fair to say that your issue is that you are not filtering/processing the data you are receiving not with the data itself?

Nick
Photo of Chris Kelly

Chris Kelly

  • 7 Posts
  • 0 Reply Likes
Haha, update after 4 years.  Basically gave up on this problem and it just shows up sometimes.  Found this thread again because I can actually see one of the offenders.

Event Type:                        L2DoS

Trap Type:                           Threshold Crossing

Device ID:                            4018B15CD0C0

Device Name:                    ESAP10-2N-1C14

Time:                     01-30-2018 14:46:16

Message:                            [wifi]: L2 DoS: interface wifi0(HSUGuest) station f40f:241d:48c4: receiving deauth frames exceed 120 PPM.

 Device: [wifi]: L2 DoS: interface wifi0(HSUGuest) station 4018:b15c:97d7: receiving deauth frames exceed 120 PPM

30 messages or so a day - station f40fxxx is a MacBook belonging to a professor in that building.  She apparently left it in her office yesterday because I got the error reports including that MAC address at 9:31 pm, 11:31pm 3:31am , 5:31, 9:31 so far.  It's walking around - unit has been in three building so far today.  Same L2Dos msg from each location on various APs in a particular building.  since she is Faculty, she should be logged in on "HSUWireless" so not sure why the device is pounding on HSUGuest.
(Edited)
Photo of Doug

Doug

  • 14 Posts
  • 0 Reply Likes
Chris thanks for the information. Hopefully you are able to find resolution as it is an annoyance. Hopefully once you can reach out to her you confirm she has removed the HSUGuest from her workstation.
Photo of Chris Kelly

Chris Kelly

  • 8 Posts
  • 0 Reply Likes
So how would you interpret this - all MACs are Aerohive, not clients.  The "b417" listed is wifi0.4 on WAP SRAP2 in the basement.  Rm106 - 1st floor, Rm230 - 2nd.  And when I see these it's always with "HSUGuest".  HSUGuest is the same config as HSUSecure - 802.1X/WPA2 Enterp[rise.  Just puts clients not classified as Fac/Staff on the Student VLAN.  Since Friday, I have 10 of these and not all in the same building.  Also noticed that messages with client MACs also complain about "HSUGuest".  NEvef about "HSUSecure" which is exactly the same thing, same config.  I may just jack the L2 Dos criteria up off the chart to get rid of these.  How about some programming that says alarm threshold zero = ignore.

Device: [wifi]: L2 DoS: interface wifi0(HSUGuest) station 4018:b15c:b417: receiving deauth frames exceed 120 PPM

 Event

 Event Type:                        L2DoS

Trap Type:                           Threshold Crossing

Device ID:                            4018B15C76C0

Device Name:                    SRAP9-Rm106-1D43

Time:                     02-24-2018 12:06:08

Message:                            [wifi]: L2 DoS: interface wifi0(HSUGuest) station 4018:b15c:b417: receiving deauth frames exceed 120 PPM.

 Device: [wifi]: L2 DoS: interface wifi0(HSUGuest) station 4018:b15c:b417: receiving deauth frames exceed 120 PPM

 

Event

 Event Type:                        L2DoS

Trap Type:                           Threshold Crossing

Device ID:                            4018B1C6BE00

Device Name:                    SRAP31-Rm230-1A34

Time:                     02-24-2018 12:06:09

Message:                            [wifi]: L2 DoS: interface wifi0(HSUGuest) station 4018:b15c:b417: receiving deauth frames exceed 120 PPM.
Photo of Crowdie

Crowdie, Champ

  • 972 Posts
  • 272 Reply Likes
Being able to limit a type of alert from a single MAC address to once every xxx time period (hour/day) would be really useful.  You don't want to stop the alerts all together as you can't tell if the wireless client is still causing an issue.
Photo of Chris Kelly

Chris Kelly

  • 7 Posts
  • 0 Reply Likes
The "solution" so far is to build a custom MAC Dos filter Disassociation and Deauth thresholds set way to the sky and apply same to all SSIDs.  Initial setting is 1200 and so far that has killed the noise.