Can't load configuration onto AP230s

  • 1
  • Question
  • Updated 3 years ago
We've got some new AP230s that I've added into our HMOL instance, but I can't load any configuration onto them. I've created profile with a simple WPA SSID with a plain PSK and when I upload it I get the errors:

Upload Configuration: Because the device is currently disconnected from HiveManager, the upload is staged and will resume after the device reconnects.
Update User Database: The upload operation was aborted because a previous process failed.

- I've ssh'd onto the AP and did a show capwap client, and it shows connected. Pings work OK, and the DNS server settings are fine.

Also, I can't push a new firmware version onto the device, I simply get a timeout.

Does anyone know what the problem could be or where to look next?

Thanks,
Kerry
Photo of Kerry Thompson

Kerry Thompson

  • 4 Posts
  • 3 Reply Likes

Posted 3 years ago

  • 1
Photo of AJ Nurcombe

AJ Nurcombe

  • 11 Posts
  • 1 Reply Like
If you console or ssh onto the AP and run the command capwap ping redirector.aerohive.com this will check if the capwap port is open to allow connectivity.


If you do not get a response then port 12222 is blocked denying the AP connecting to HMOL.


Also ensure when you push config that DNS setting are set correctly under additional settings in the guided configuration.
(Edited)
Photo of Kerry Thompson

Kerry Thompson

  • 4 Posts
  • 3 Reply Likes
Hi AJ

I've run the capwap ping, and it all looks fine :-

AH-0a84c0#capwap ping redirector.aerohive.com
CAPWAP ping parameters:
    Destination server: redirector.aerohive.com (54.172.0.252)
    Destination port: 12222
    Count: 5
    Size: 56(82) bytes
    Timeout: 5 seconds
--------------------------------------------------
CAPWAP ping result:
    82 bytes from 54.172.0.252 udp port 12222: seq=1 time=232.92 ms
    82 bytes from 54.172.0.252 udp port 12222: seq=2 time=231.746 ms
    82 bytes from 54.172.0.252 udp port 12222: seq=3 time=231.704 ms
    82 bytes from 54.172.0.252 udp port 12222: seq=4 time=230.980 ms
    82 bytes from 54.172.0.252 udp port 12222: seq=5 time=231.197 ms
    ------- redirector.aerohive.com CAPWAP ping statistics -------
    5 packets transmitted, 5 received, 0.00% packet loss, time 6682.783ms
    rtt min/avg/max = 230.980/231.543/232.92 ms
AH-0a84c0#

I've also tried changing the capwap client over to HTTP, port 80, and I still get the same problem with no config loading. From the firewall logs I can see the APs successfully connecting out to HMOL (over https when capwap client is HTTP and over ssh when capwap client is normal capwap).

Connectivity-wise everything looks fine.
Photo of Roberto Casula

Roberto Casula, Champ

  • 231 Posts
  • 111 Reply Likes
Are you pushing a full configuration the first time?

Does your firewall allow the APs' management interfaces to connect out to HMOL on port 22 (SSH)? SSH (well SCP) is required for bulk operations like firmware updates in addition to UDP port 12222 used for the normal CAPWAP communication.

Also, make sure the policy and AP configuration you are pushing isn't changing the management settings/IP/routing in a way that means the AP loses connectivity to HMOL when the config is applied. If this happens (for example if you've accidentally specified the wrong management/native VLANs in the policy, or if the policy assignes an incorrect DNS server, or if you've assigned an incorrect static IP address or default gateway in the AP setting), then the AP will try to apply your settings, fail to connect back to HMOL and then automatically roll back to recover.
Photo of Kerry Thompson

Kerry Thompson

  • 4 Posts
  • 3 Reply Likes
Hi Roberto

I've double-checked the firewall connectivity today, and run some more tests. The firewall logs show everything connecting just fine. but still no config is being pushed onto the APs.

We've got 2 APs running 6.2r1a and one on 6.4r1d. I've found those on 6.2r1d won't reboot when a reboot command is issued on the CLI over ssh, so that's one problem identified.

Here's a screenshot of the current error message of a full config load to an AP230 running 6.4r1d:


So I cleared the web cache and rebooted the AP and tried again. Same problem.
Time to place a support call I think.
Photo of Kerry Thompson

Kerry Thompson

  • 4 Posts
  • 3 Reply Likes
The problem has been resolved. Thanks to everyone who commented or otherwise scratched their heads over this one.

The issue was the firewall, but it wasn't a straightforward rule missing.
The problem was with path MTU discovery, and since the firewall's Internet interface was a PPPoE tunnel then full-sized IP packets weren't getting through from HMOL to the APs - for some reason path MTU discovery between the endpoints failed to detect and compensate.

We fixed the TCP max segment size to 1400 in the firewall and everything started working fine.

The symptoms were quite confusing and difficult to figure out - CAPWAP and HTTP management transports worked fine with their small packet size, but a config or firmware load would fail. It all makes sense once you know that the problem was.

Thanks again,
Kerry
Photo of Roberto Casula

Roberto Casula, Champ

  • 231 Posts
  • 111 Reply Likes
Hi Kerry,

Good catch. PMTUD is a pain. MSS Clamping is a clunky workaround but at least it works reliably - most firewalls/routers running PPPoE will do MSS clamping by default but some don't, so this is certainly something that others might run in to in future. Thanks for posting your findings.