AP370 access points will not update

  • 1
  • Question
  • Updated 2 years ago
I have AP370 access points running version 6.0r2, and HMOL version 6.5r1. When I try to update the APs the update times out and the status shows "Abort". This is the error message I receive when I hover over the update status:
 

Upload Configuration: The operation timed out. Update User Database: The upload operation was aborted because a previous process failed. Device Reboot: The upload operation was aborted because a previous process failed.  


My APs are broken up into two categories, Internal and External, which means that they are going through different networks. I have ruled out any netowrk issues like a firewall blocking connections, and like I said they are on different networks all together so the chances of it being a network issue is really unlikely. I will occasionally just update/reboot the APs on a whim, and it looks like I last rebooted the Internal APs 63 days ago, and the External APs 34 days ago. It would seem to me then that the issue arose sometime in the last 34 days. 

I consoled into the APs and ran a "capwap ping redirector.aerohive.com" command and it looks like network connectivity between the APs and the redirector on port 12222 are fine. I can't seem to paste the screenshot in here but these are the parameters:

Destination server: redirector.aerohive.com (54.172.0.252)
Destination port: 12222
Count: 5
Size: 56(82) bytes
Timeout: 5 seconds

5 packets transmitted, 5 received

I browsed to the redirector.aerohive.com site also and the connection status is showing that it is disconnected.
 

In the HiveManager GUI I looked at the logs for the APs under Utilities --> Diagnostics and this is the output from one of my External APs immediately after I tried to update them, right up to the point that the Update Status read "Abort":

2016-05-06 10:18:57 info    ah_cli: admin:<show logging buffered tail 500>
2016-05-06 10:18:10 debug   kernel: bgscan_check_load failed for vap wifi0
2016-05-06 10:17:48 info    capwap: CAPWAP: receive DCD send response to CAPWAP event!, length:1124
2016-05-06 10:17:48 info    capwap: receive event DCD send response to CAPWAP: eventid = 73: length = 1124
2016-05-06 10:17:48 info    capwap: CAPWAP: receive DCD send response to CAPWAP event!, length:1308
2016-05-06 10:17:48 info    capwap: receive event DCD send response to CAPWAP: eventid = 73: length = 1308
2016-05-06 10:17:48 info    capwap: CAPWAP: receive DCD send response to CAPWAP event!, length:1308
2016-05-06 10:17:48 info    capwap: receive event DCD send response to CAPWAP: eventid = 73: length = 1308
2016-05-06 10:17:48 info    capwap: CAPWAP: receive DCD send response to CAPWAP event!, length:1308
2016-05-06 10:17:48 info    capwap: receive event DCD send response to CAPWAP: eventid = 73: length = 1308
2016-05-06 10:17:48 info    capwap: CAPWAP: receive DCD send response to CAPWAP event!, length:1308
2016-05-06 10:17:48 info    capwap: receive event DCD send response to CAPWAP: eventid = 73: length = 1308
2016-05-06 10:17:48 info    capwap: CAPWAP: receive DCD send response to CAPWAP event!, length:1308
2016-05-06 10:17:48 info    capwap: receive event DCD send response to CAPWAP: eventid = 73: length = 1308
2016-05-06 10:17:00 info    kernel: [wifi]: wifi1: background scan done, num_bgscans 1545, num_req_bgscans 0
2016-05-06 10:16:37 info    kernel: [wifi]: wifi1: trigger background scan: chans scanned 1, cont periodic scan 
2016-05-06 10:16:12 info    kernel: [mesh]: set proxy : 649a:bee1:e5af e01c:4182:8200 n/a flag 0x1402
2016-05-06 10:16:12 info    amrp2: set proxy route: 649a:bee1:e5af -> e01c:4182:8200 ifp N/A upid 400 flag 0x1402 monitor(0/0) pkt/sec ok
2016-05-06 10:15:46 info    ah_cli: admin:<save server-files>
2016-05-06 10:15:46 notice  capwap: "scpuser" successfully scp 52.1.104.157 server
2016-05-06 10:15:41 info    kernel: [mpi]: socket is closed, pid(543), protocol(0)
2016-05-06 10:15:41 info    ah_cli: admin:<save web-page web-directory Guest-Wifi-1 scp://scpuser@52.1.104.157:/HiveManager/downloads/MPIUA/cwp/webpage/Guest-Wifi-1.tar.gz no-prompt _password *** >
2016-05-06 10:15:41 info    ah_cli: admin:<web-directory Guest-Wifi-1>
2016-05-06 10:15:41 info    ah_cli: admin:<no web-directory Guest-Wifi-1>



These two lines of output jumped out at me but I don't know what they mean:

2016-05-06 10:15:41 info    kernel: [mpi]: socket is closed, pid(543), protocol(0)

2016-05-06 10:18:10 debug   kernel: bgscan_check_load failed for vap wifi0

I have confirmed that my SSIDs are using the correct VLANs, and that the APs are using the correct subnet, IP routes, etc. I looked at the configurations on the APs and these are the DNS servers they are using:

208.67.222.222 
208.67.220.220

Does anyone have any idea why my APs are not updating?
Photo of Rick

Rick

  • 11 Posts
  • 0 Reply Likes

Posted 2 years ago

  • 1
Photo of Mike Kouri

Mike Kouri, Official Rep

  • 1030 Posts
  • 271 Reply Likes
HiveOS 6.0 had some issues where certain processes would not release memory when they are done with it, with the results that over time the device would have less and less free memory. It's quite possible that HiveOS could not allocate a large enough chunk of memory for the update, and thus the error you are seeing. Please try, on one of these devices, to reboot it first, then as soon as it is back up and running, perform your update. If that succeed then you have a possible explanation and strategy for the remainder of them.
Photo of Rick

Rick

  • 11 Posts
  • 0 Reply Likes
Thanks for the suggestion, Mike. I rebooted the AP first, and then tried to update afterwards but the update aborted again. Below is the memory on the AP. Is there anyway to see how much memeory the individual processes are using, or a log that shows why a process might fail?


Total Memory:       515072 KB
Free Memory:        263940 KB
Used Memory:        251132 KB
Photo of Rick

Rick

  • 11 Posts
  • 0 Reply Likes
I consoled into the AP and ran a "sho capwap client" command. The last two lines show that the packets are getting dropped due to connection loss and not a buffer shortage. I have obscured the "Virtual HiveManager Name" because it is my company's name...but I wonder if that is correct? The Hivemanager GUI shows my Virtual HiveManager Name starts in a VHM-12345 format (12345 aren't the real numbers) and then my company's name in parentheses. Should I see that same VHM number from the CLI of the AP? I was also wondering why my "HiveManager Backup Name" is blank? Does this information below look correct?


AH-82c680_Server_Room_External#sho capwap client
CAPWAP client:   Enabled
CAPWAP transport mode:  UDP
RUN state: Connected securely to the CAPWAP server
CAPWAP client IP:        10.1.10.14
CAPWAP server IP:        52.1.104.157
HiveManager Primary Name:hm-useast-204.aerohive.com
HiveManager Backup Name:
CAPWAP Default Server Name: redirector.aerohive.com
Virtual HiveManager Name: *****
Server destination Port: 12222
CAPWAP send event:       Enabled
CAPWAP DTLS state:       Enabled
CAPWAP DTLS negotiation: Disabled
     DTLS next connect status:   Enable
     DTLS always accept bootstrap passphrase: Enabled
     DTLS session status: Connected
     DTLS key type: passphrase
     DTLS session cut interval:     5 seconds
     DTLS handshake wait interval: 60 seconds
     DTLS Max retry count:          3
     DTLS authorize failed:         0
     DTLS reconnect count:          0
Discovery interval:      5 seconds
Heartbeat interval:     30 seconds
Max discovery interval: 10 seconds
Neighbor dead interval:105 seconds
Silent interval:        15 seconds
Wait join interval:     60 seconds
Discovery count:         0
Max discovery count:     3
Retransmit count:        0
Max retransmit count:    2
Keepalives lost/sent:    0/331
Event packet drop due to buffer shortage: 0
Event packet drop due to loss connection: 10
Photo of Knarf

Knarf

  • 18 Posts
  • 2 Reply Likes
Can you push a complete config to the AP? Also, there is an alternative way to update the HiveOS. You can use the netui, net configuration. Just open a web browser and type in the IP address, management IP, for the AP. You can then log in with admin and if a config has been pushed the password will be the password under home, device management, other global settings, unobscure the password. The default password is aerohive all lower case. You should see HiveOs update software on the left. You can grab the HiveOs image from support.aerohive.com
Photo of Rick

Rick

  • 11 Posts
  • 0 Reply Likes
Knarf,

Thanks for the info. I cannot push a complete config to the APs because the connection to HiveManager times out. The web browser method is good to know if I want to upgrade the Hive OS, but right now I'm just concerned with the fact that I can't upload any changes I make on the APs to the HiveManager. I want to determine why those updates are being aborted even though it is clear from the troubleshooting I've done thus far that the APs are able to connect to the HiveManager on port 12222?
Photo of Bill W.

Bill W.

  • 222 Posts
  • 35 Reply Likes
Make sure that you have port TCP 22 open in your firewall. This is the port used for HiveOS updates and config updates.

Also, you could try changing the CAPWAP port to 80. Use the command: capwap client transport http. For whatever reason, sometimes changing the CAPWAP port improves connectivity. I think this also changes the updates to use TCP 443 instead of TCP 22.

And you could also try updating your APs to the current HiveOS to see if that helps too.
Photo of Rick

Rick

  • 11 Posts
  • 0 Reply Likes
Thanks, Bill. I actually spoke to Aerohive Support and they also could not not update when they took an AP370 in their office and associated it to our HiveManager. They ended up moving us to a new server and the issue has been resolved. I do not know, though, what was occurring on the Aerohive server side that was preventing the updates from completing. 
Photo of Bill W.

Bill W.

  • 222 Posts
  • 35 Reply Likes
Glad to hear it got resolved.