Since upgrading Hivemanger to 6.8r5 we cannot push a complete upload to any AP121

  • 3
  • Question
  • Updated 1 year ago
since upgrading Hivemanger to 6.8r5 we cannot push complete upload to any AP121 but AP120 and AP130 are fine. I can upload a new image and upgrade the HiveOS but I still cannot push a complete upload, we get the following error: Upload Configuration: An error occurred when executing the following CLI command: save config scp://******@IPaddress:/HiveManager/downloads/home/script/new/4018B18E590C0_full_new.config current no-prompt _password *******.Unknown error
Update User Database: The upload operation was aborted because a previous process failed

Anyone else had this problem or have any ideas apart from rebooting them and upgrading the HiveOS?
Photo of Allan Cameron

Allan Cameron

  • 6 Posts
  • 0 Reply Likes

Posted 2 years ago

  • 3
Photo of Bill W.

Bill W.

  • 222 Posts
  • 35 Reply Likes
As weird as this sounds, try downgrading them to 6.4r1g.  Then once they are back up, push a full config with software upgrade.

I've encountered this issue sporadically. Sometimes just doing a retry will work.  Sometimes a reboot will work.  Sometimes waiting several hours will work.  But I have found that if I downgrade to 6.4r1g (chose this version because I remember it being the most stable version of the 6.4 line) and then push a full config, it works every time.

So now I just do a retry first, and if that fails I downgrade.

Also, I noticed this prior to 6.8r5.  But I didn't discover the downgrade trick until recently.

I hope this helps you.
Photo of Peter Powell

Peter Powell

  • 6 Posts
  • 3 Reply Likes
We had exactly the same issue and tech support was not able to get to the bottom of it. Didn't try the downgrade and upgrade mentioned. However we found this issue after we upgraded to HiveOS 6.5.r5 before we upgraded to the current 6.8 release of HM. We could not push a full config to most of our AP121. Also saw a certain precentage of DHCP problems with some APs so they would not get an address after reboot.

Downgraded the HiveOS again and everything is stable, Support was trying to tell us it was a CAPWAP issue but we didnt have it before the upgrade or after the downgrade and there were no network changes in the interim. Not the behavour desired from a "golden" release.
Photo of Allan Cameron

Allan Cameron

  • 6 Posts
  • 0 Reply Likes
Thanks Bill, I took your advice and downgraded an AP121 to HiveOS 6.4r1g and I was able to do a complete upload however after upgrading to 6.5r5 again I still cannot do a complete upload. I did notice I can do a successful delta upload to the AP121 without downgrading and after it the audit matches so I have asked an Aerohive account manager who commented on another recent post about upgrading Hivemanager to 6.8r5 if I could get away with just doing a delta. 

I don't know what the best version of HiveOS for an AP121 is at the moment but this is the only model that has a problem for us!
Photo of mr bee

mr bee

  • 77 Posts
  • 0 Reply Likes
Problem of full config push is already there from 6.5rx. It should be solved in 6.5r6, will be released soon.
Photo of Szymon

Szymon

  • 15 Posts
  • 1 Reply Like

Hi Allan,

I don't think so that 121's should work on 6.5r5 version of HiveOS. I had the same issue with some 121's and 120's AP and got below update from Aerohive support:

"Do NOT update any APs or devices to 6.5r5, as it is known to have issues including high CPU usage which can overload the APs (especially dangerous for 100-series devices), as well as potentially causing SSH issues for the units. It is generally recommended to keep the large majority of 100-series devices on that last version before the 6.5 build, as they are lower spec'd and may experience CPU overload due to the amount of processes and memory, though this generally occurs if you do not first reboot the units before upgrading the firmware. All other units (AP230, AP350, etc." are cleared to upgrade to 6.5r4, as that is currently the most stable golden release firmware active at the moment."

Thanks,

Szymon

Photo of Nick Lowe

Nick Lowe, Official Rep

  • 2491 Posts
  • 451 Reply Likes
For what it's worth, I would not agree with the quoted section above.

The SSH issue observed with 6.5r5 is an interop issue that was inadvertently missed between the built-in client used in HM6 after changes were made in HiveOS to remove the support for older ciphers. We are working on an update to HM6 to resolve this. It is not a regression in HiveOS.
Photo of Nick Lowe

Nick Lowe, Official Rep

  • 2491 Posts
  • 451 Reply Likes
Update: This is corrected in HiveManager 6.8r7 and is listed in the release notes.
Photo of Bill W.

Bill W.

  • 222 Posts
  • 35 Reply Likes
Yeah, I forgot to mention that I only upgrade to 6.5r4. I've had some RADIUS issues with 6.5r5 (and 6.8r1 for ac models).
Photo of Dawn Douglass

Dawn Douglass

  • 67 Posts
  • 3 Reply Likes
Bill,

Can you explain more about the RADIUS issues.  I am in the this situation now and support has been unable to resolve.  
Photo of Nick Lowe

Nick Lowe, Official Rep

  • 2491 Posts
  • 451 Reply Likes
Hi Dawn,

There is a known issue that pertains to certain forms of Class attributes in HiveOS 6.5r5 and HiveOS 7.1r1, experienced primarily with NPS, and an issue with RADIUS accounting retransmissions pertaining to the calculation of the Request Authenticator in HiveOS 6.5r5, HiveOS 6.8r1 and HiveOS 7.0rX. The can cause authentication issues.

These will be corrected with our upcoming releases.

My suggestion would be to disable RADIUS accounting temporarily for the RADIUS servers configured via HiveManager and to ensure that NPS does not generate RADIUS class attributes by adding the Generate-Class-Attribute pseudo-attribute in either Connection Request Policies (CRPs) or Network Policies (NPs) under Vendor Specific RADIUS Attributes, setting the value to False.
(Once added, due to a quirk in NPS, this will then appear under Standard RADIUS Attributes.)



Alternatively, use previous versions of HiveOS temporarily if you have a need for RADIUS accounting in the environment.

Thanks,

Nick
(Edited)
Photo of Szymon

Szymon

  • 15 Posts
  • 1 Reply Like
From my experience HiveOS 6.4r1g.2138 is working the best on 121's ... 
Photo of Allan Cameron

Allan Cameron

  • 6 Posts
  • 0 Reply Likes
All of our AP121s are on HiveOS 6.5r5 and I haven't had any issues yet, thats been since Friday but I will need to downgrade them if I want to perform a complete upload. I will try and get in touch with support and get them to confirm the best version of HiveOS to run on the different models we have, it would be nice if someone from Aerohive would comment to save us the the bother and the time we are all spending on this.
Photo of Nick Lowe

Nick Lowe, Official Rep

  • 2491 Posts
  • 451 Reply Likes
I would recommend using HiveOS 6.5r4 for the QCA-based APs where RADIUS is used and 6.5r5 otherwise, and newer in the 6.5 Honolulu branch as these become available in the future. Only if a regression occurs should an older version be used, hopefully only temporarily while the causal factors are addressed.
(Edited)
Photo of Dennis Hamman

Dennis Hamman

  • 29 Posts
  • 4 Reply Likes
I have had similar issues pushing complete uploads since 6.8r3 and on 6.8r4 as well.  Most of my AP121s are 6.4r1 and I have lots of issues with complete uploads.  I get the same error as the original poster.  I actually reset some Access points to factory default so i could push a complete config since they kept failing.  I have at least 4000 to update so if I figure out the trick I will let everyone know.  My goal is to adventually move all of my AP121s to 6.5r5 and all of my AP130s as well.  Has anyone actually seen the high CPU usage that is reported by support?  I myself have not and I was wondering if I am missing something.
Photo of Szymon

Szymon

  • 15 Posts
  • 1 Reply Like

I saw and fight with high cpu usage on 6.5r4 and 6.5r5 on 121's. With those firmware version couldn't push full config to AP's and AP's loose capwap connection randomly to HM for couple of minutes and then get back. Not all AP's been affected and different number of AP's played up together every day ....

After downgrade to 6.4r1g.2138 never heard about any issues from customer.

Photo of Dennis Hamman

Dennis Hamman

  • 29 Posts
  • 4 Reply Likes
I had a lot of High CPU usage at the beginning of the school year due to multicast traffic Specifically mDNS from all of the chromebooks we use.  How do you notice your high cpu utilization normally do you monitor it with SNMP?
Photo of Jerri Son

Jerri Son

  • 4 Posts
  • 0 Reply Likes
If you are using cacti this would help: http://forums.cacti.net/viewtopic.php?f=12&t=54736 . Monitoring CPU is not viable with SNMP (yet) afaik - which should hopefully be implemented asap :)
Photo of Dennis Hamman

Dennis Hamman

  • 29 Posts
  • 4 Reply Likes
I just spoke with support they said that they were working to address this issue in HIvemanager 6.8r6.  The other option would be to downgrade back to 6.6r3.
Photo of mr bee

mr bee

  • 77 Posts
  • 0 Reply Likes
These issues appear since 6.5 golden release. In every subversion it is there. It should be solved in 6.5r6, released this month.
Photo of Dennis Hamman

Dennis Hamman

  • 29 Posts
  • 4 Reply Likes
i have installed Hivemanager 6.8r6.  This did not resolve my issue I guess I will have to wait for the new HiveOS.  Which does not make much since because everything use to work fine and my AP121s are on 6.4r1 still all that has changed is the Hivemanager.
Photo of mr bee

mr bee

  • 77 Posts
  • 0 Reply Likes
yeah sorry, I meant HiveOS 6.5r6
Photo of Jerri Son

Jerri Son

  • 4 Posts
  • 0 Reply Likes
Had issues as well - one thing that mended the problem somewhat was to use TFTP instead of SSH, since I suspected a newer sshd service on the HM to be the cause of the issue. We only had trouble upgrading mutliple APs at once but HM let ́s you pick TFTP as an alternative (if service is enabled). As for 6.5r5 running bad on the 121s - haven ́t seen any issues so far and we ́ve been running that release for about 3 weeks now iirc.
Photo of Peter Powell

Peter Powell

  • 6 Posts
  • 3 Reply Likes
Has anyone gotten any further with this issue? It seems to still exists with HM 6.6r6 and HiveOS 6.5r5 on the AP121. Rolling back to HiveOS 6.4r1g seems to be the only thing that allows for a successful full update. The errors we see are exactly as reported in the original post above of being and SCP error. All communications between AP and HM pass all tests available.
Photo of Nick Lowe

Nick Lowe, Official Rep

  • 2491 Posts
  • 451 Reply Likes
Hi Bill,

I can confirm that there is a fix for an issue of this nature in the next release of HiveOS 6.5

Thanks,

Nick
Photo of Bill W.

Bill W.

  • 222 Posts
  • 35 Reply Likes
Thanks for the confirmation Nick!
Photo of Dennis Hamman

Dennis Hamman

  • 29 Posts
  • 4 Reply Likes
I have upgraded some AP121s to 6.5r6 and complete uploads work again!!!!!!!  Finally!!!!!  I have tested 30 access points with no failures.  The uploads take a little longer but they work so I can't really complain.  I am waiting a month or so before I upgrade my remaining 4000 AP121s just to make sure there are no new bugs.  
Photo of FPUAH

FPUAH

  • 8 Posts
  • 1 Reply Like
I've updated an AP121 to 6.5r6 as well and full config uploads are working again. Like Dennis I'm waiting a bit to see if other bugs surface. Strangely, I don't see this fix in the addressed issues in the release notes.
Photo of Bill W.

Bill W.

  • 222 Posts
  • 35 Reply Likes
Yeah, they forgot to put it in the release notes. I pointed it out in another thread, and Mike replied that they forgot to add it (CFD-2324).
https://community.aerohive.com/aerohi...
Photo of Dawn Douglass

Dawn Douglass

  • 67 Posts
  • 3 Reply Likes
We had this issue as well and downgraded to 6.5r4.  Since the downgrade everything is operating well.  
Photo of Dennis Hamman

Dennis Hamman

  • 29 Posts
  • 4 Reply Likes
Ap121 complete upload still takes way too long. can Aerohive fix this or what? this is ridiculous i have to spend all day upgrading 3000 access points.  are there any other access points on the market that don't takes as long?
Photo of Crowdie

Crowdie, Champ

  • 972 Posts
  • 272 Reply Likes
I have a number of AP121 deployments and I am slowly replacing them with AP130s.  The AP121 just doesn't seem to have the memory and, to a lesser degree, CPU to handle what some deployments require of them.