APs dropping CAPWAP connection

  • 2
  • Question
  • Updated 1 year ago
Been having this problem a couple of days now. I have been having many APs lose CAPWAP connection to our hivemanager instance. When they drop, they will generally not come back up, and I will have to SSH into each AP and canually set the capwap client server name for them to come back up. I have seen this issue with 120s, 121s, 230s, and 330s. Problem isn't isolate to one building or network policy. I have roughly 560 APs that connect to an on prem HM instance, however, the traffic does traverse the WAN to go from APs to HM due to limitations. Any advice.
Photo of Brad

Brad

  • 8 Posts
  • 0 Reply Likes
  • Confused.

Posted 3 years ago

  • 2
Photo of Mike Kouri

Mike Kouri, Official Rep

  • 1030 Posts
  • 271 Reply Likes
Brad,
That's weird. When you do SSH into the AP, what is the value you see for the capwap server name? Is it correct, or is it just gone, or has it somehow been changed to an incorrect value? What version(s) of HiveOS are you running on these APs?

Has anything changed in your network topology around the time that this issue began occurring? 
Photo of Brad

Brad

  • 8 Posts
  • 0 Reply Likes
For some the value is gone, for some it is incorrect. I believe all the APs that I have seen doing this are running some version of 6.1. No topology changes were made when everything started going crazy.
Photo of Mike Kouri

Mike Kouri, Official Rep

  • 1030 Posts
  • 271 Reply Likes
Hmm. Okay, for the units where this has changed to an incorrect setting, is it consistently the same on each of the APs? Is there any possibility that someone is operating a "rogue" instance of HiveManager? Do you have a test lab with a private HiveManager instance like many of our larger enterprise customers do? If so, could the security gateway between that lab and rest of your organization have changed?
Photo of Brad

Brad

  • 8 Posts
  • 0 Reply Likes
The address that they connect to is actually an instance of HMOL from what I can tell. It is an AWS owned IP and directs to a HMOL auth page when I go to it. We do not have a separate test lab setup.
Photo of David Dippon

David Dippon

  • 19 Posts
  • 8 Reply Likes

Brad,

We would like to get more information, and have opened case 110515, and someone from our Aerohive Technical Assistance Center will be in touch with you.

Thanks

Photo of Matt Carasso

Matt Carasso

  • 1 Post
  • 0 Reply Likes
Has this been resolved Im currently having a similar issue with some Ap 230 Ap's Running 6.8R1 
Photo of Col Hawksworth

Col Hawksworth

  • 5 Posts
  • 1 Reply Like
I have always experienced this intermittent problem with CAPWAP, all my APs can suddenly go offline, then a few will come back online.  Then sometime later, the remainder will come back on... it's always the same CAPWAP error, 'The CAPWAP connection with HiveManager was lost'.  We use HMOL, with 120, 121, 130, 330 APs running 6.5r3 and 6.6r2
Photo of Ash

Ash

  • 2 Posts
  • 0 Reply Likes
We are having the same issues with AP121's and SR2024P switches using Hive Manager NG.  Devices for no reason will show as offline, yet they are still functioning as an AP in that clients are active on the network.  So far, the only option we have had is to SSH onto them and reboot or disable and re-enable the CAPWAP.

The worrying thing is that this issue is not a new one.  It looks to have been round for some years now yet I have found no cases here or on the web to show resolutions to the problem or potential fixes in the pipeline.

We are using the latest version of NG (cloud based) and the latest firmware for all the AP's and switches that have been affected.
Photo of Louis

Louis

  • 1 Post
  • 0 Reply Likes
We've over 650 AP250 on Hive Manager 6.8r7 which are randomly dropping and showing offline but are accessible and serving wireless clients. Issue raised with Aerohive, debugs provided and upgraded to 7.1r1 as advised but no difference at all. Can't say I feel the issue is being taken very seriously and given the investment we've made in Aerohive, this is rather disappointing. 
Photo of Roberto Minotti

Roberto Minotti, Employee

  • 51 Posts
  • 5 Reply Likes

Louis & all, I have several customer complaining about this behaviour and all the time they do SSL DPI Inspection at the firewall level. Once turned off, all APs get back online (capwap restart needed).

Photo of Ash

Ash

  • 2 Posts
  • 0 Reply Likes
Morning,

And after dong this, does the problem go away permanently or do they have to do this each time the CAPWAP connection fails?

Regards,

Ash
Photo of Roberto Minotti

Roberto Minotti, Employee

  • 51 Posts
  • 5 Reply Likes
the problem disappeared, of course
(Edited)
Photo of Szymon

Szymon

  • 15 Posts
  • 1 Reply Like
Hello,

I had customer with the same strange CAPWAP connection drops. After " dynamic airtime scheduling" has been turn off on policy all drops disappeared. All AP's has stable CAPWAP connection.

Thanks,
Szymon
Photo of Jimmy B

Jimmy B

  • 4 Posts
  • 0 Reply Likes
In my Enterprise environment,  There is no way I will get the OK to disable DPI inspection over SSL. I don't think anyone experiencing this issue believes this is a good idea. Are developers looking into this anomaly ? 
Photo of Nick Lowe

Nick Lowe, Official Rep

  • 2491 Posts
  • 451 Reply Likes
Hi,

You just need to exempt our traffic from being modified. Benign/passive inspection of the traffic does not cause a problem. For our TLS connections, they need to not be MITMd.

Thanks,

Nick
Photo of Stephen Phelps

Stephen Phelps

  • 1 Post
  • 0 Reply Likes
I have a similar problem with CAPWAP. When I am ssh'd into the AP, the CAPWAP client rotates between having the correct CAPWAP information to zero'd out HiveManager information. CAPWAP also rotates between UDP and HTTP. It seems to happen more with remote locations with Charter Internet service.
Photo of Hans

Hans

  • 68 Posts
  • 8 Reply Likes
Here the same issue, we checked everything but nothing incorrect to see. AP's CAPWAP connections goes down and up at random times (mostley just 5 AP's are online of the 90 AP's).
Photo of Dawn Douglass

Dawn Douglass

  • 67 Posts
  • 3 Reply Likes
We have experienced this issue off and on over the years.  It started again about a week ago.  The APs are not subject to DPI.  There have been no network topology changes in months.  No firmware changes to the HMOL/APs either.  The APs will show connected in HMOL, then when I refresh a minute or two later, they display that they now have CAPWAP connectivity but, now other APs do not.