AP230 QoS Engine Issues

  • 3
  • Question
  • Updated 1 year ago
There seems to be a bug in the firmware on the AP230s that causes the QoS engine to stop all traffic outbound from an AP.

We are in a 1 to 1 school with over 1800 Mac OS X and iOS devices.  The issue seems to start around 30 days of uptime on APs in high traffic areas. Clients connecting to these APs will experience the following symptoms:
  • All clients will connect to the AP and obtain an IP address from our DHCP server.
  • Some or all clients will experience slowness browsing the internet, or some or all clients will not able to pass any traffic.
The workaround is to either reboot the AP or log into the CLI and issue the two commands of:
no qos enable
qos enable

This will resolve the issue, but if the AP stays up for another 30 days, then it will experience the issue again.

I have worked with support and they can not seem to find out what is causing this issue.  We have had custom HiveOS built and one of the firmware running 6.4r1b seem to have resolved the issue, but support suggested that we upgrade to 6.6r1b because it addresses the issue with the QoS engine.  Upgrading to the 6.6r1b on the AP230s we are still experiencing the issue.  We have only seem this on the AP230s.  When we had the AP330s, everything worked well and we did not have this issue.

Has anyone else in experienced this issue?
Photo of Montrey Jackson

Montrey Jackson

  • 8 Posts
  • 0 Reply Likes
  • frustrated

Posted 2 years ago

  • 3
Photo of Arison Mercado

Arison Mercado

  • 113 Posts
  • 8 Reply Likes
Yes I had this exact same issue, So in order to fix this I had to remove Firewall profile and QoS because it was causing max IP sessions @8000 and once that was removed the IP sessions settled at 100. As much as I love Aerohive, I believe its the extra profiles settings like QoS, and Firewall that cause these issues to happen. In a replacement for QoS, I just rate limited our guest network and settled for not having the QoS profile on our internal network.
Photo of Montrey Jackson

Montrey Jackson

  • 8 Posts
  • 0 Reply Likes
@Arision

Interesting....

I believe that we have removed the QoS profile, but looking in the Configuration, it is set in the User Profiles, but I do not see how to disable it.  Am I just overlooking something?
Photo of Arison Mercado

Arison Mercado

  • 113 Posts
  • 8 Reply Likes
Montrey,

I'm not aware of any setting to disabling it, I just removed it from the Marker Map settings and left the Rate Control & Queing Policy as default.. Also, look at the addition settings edit tab located in the "Configuration Interfaces and User Access" Menu  and their is another QoS settings where you might have added it and make sure they're blank. Furthermore, I would look to see if you have a Firewall Profile selected because that caused most of my problems.
(Edited)
Photo of Kevin Gee

Kevin Gee

  • 54 Posts
  • 4 Reply Likes
Hey Montrey/All,

I've glad you mentioned this as I think we have the same problem. I hadn't narrowed down the solution beyond a reboot, next time it happens I will try your qos disable idea first.

My feeling is that this problem started to appear after we upgraded the HiveOS to the 6.6 track (we are currently on 6.6r2a) but I may be wrong. I have been thinking about switching to 6.5 but as you may well be aware there are some bug fixes specific to AP230s that are documented as fixed in 6.6 but not mentioned in 6.5 (e.g. APs reporting the wrong channel Hive Manager).

What Hive OS are you running?

Regards, Kevin.
Photo of Montrey Jackson

Montrey Jackson

  • 8 Posts
  • 0 Reply Likes
Hi Kevin,

We are running 6.6r1b on the AP230s.  We are also running a custom build from Aerohive based on the 6.4r1 code.  Both versions of the Hive OS are experiencing the issue.  
Photo of Montrey Jackson

Montrey Jackson

  • 8 Posts
  • 0 Reply Likes
Arison

I just checking the QoS Settings under Configuration Interfaces and User Access.  Both the Classifier Map and Marker Map fields are blank.  We do have the user-def-qos enabled in the User Profile.  This is the default Rate Control & Queing Policy.

We do have a IP Firewall Policy in place for our Guest and student user profiles.  But this is something we do not want to remove from the two. 
Photo of Arison Mercado

Arison Mercado

  • 113 Posts
  • 8 Reply Likes
Montrey,

When we removed that it was a complete 180 change of performance and all those issues you have mentioned went away. So since we decided to remove the firewall policy we resulted in just using our actual Firewall for internet connection to do the content filtering and app blocks.
Photo of Brian Powers

Brian Powers, Champ

  • 396 Posts
  • 92 Reply Likes
This is Dynamic Airtime Scheduling that you are disabling/enabling.  It is under the Additional Settings - QOS Settings of the Network Policy as a whole.  There are other threads (I believe) on here that relate to that.  It is something I've ran across on multiple occasions.  It is something that I disable completely now at sites.
Photo of Montrey Jackson

Montrey Jackson

  • 8 Posts
  • 0 Reply Likes
Brian,

Thanks for this.  We had an AP up for 53 days experience the issue. After applying the configuration to disable Dynamic Airtime Scheduling on this AP, the issue no longer exist.  I have deployed the configuration on other APs that are nearing the 30 day mark.  I will see if this indeed resolves the issue.
Photo of Brian Lowrance

Brian Lowrance

  • 10 Posts
  • 0 Reply Likes
We have this issue as well on our AP230's (AP's allow connection, allow DHCP but then will not flow other traffic after being up for over 30 days).

HMOL 6.6.r3a
HiveOS 6.6r2a Irvine.2309

This thread is over a year old.  Has this not been addressed/resolved by Aerohive in firmware yet?
Photo of Bill W.

Bill W.

  • 222 Posts
  • 35 Reply Likes
I can't say for sure, but it's possible. You are not running the most current version of HM nor HiveOS.  The current version of HM is 6.8r7a.  And the current version of HiveOS for an AP230 is 6.5r6 (golden version) or 7.1r1 (latest version).  So you could try updating HM and then updating your APs to see if this resolves your problems.  You're quite a few revs out and many bugs have been fixed from the versions you're using.

The other option would be to schedule a reboot of the APs so that they reboot prior to 30 days.  You can do this using the supplemental CLI.
Photo of Brian Lowrance

Brian Lowrance

  • 10 Posts
  • 0 Reply Likes
I currently have a reminder set for every other Friday evening to reboot the 230's.

In HMOL, when I select to upgrade firmware on an AP230 to see what is available, it shows 6.6r2 (what I am running currently), 6.5r4, 6.5r3 and 6.1r6.

I intend to upgrade HMOL soon but our students are doing national and end of year testing so my configs are currently frozen until that's complete.
Photo of Montrey Jackson

Montrey Jackson

  • 8 Posts
  • 0 Reply Likes
The fix for us was to disable the Dynamic Airtime Scheduling.

As Bill stated, upgrading to HiveOS 6.5.r5 or 6.5r6 would also help, along with disabling the Dynamic Airtime Scheduling. 

We have seen issues with 7.1r1 were it would broadcast duplicate SSIDs (with PSK) and the duplicates would be an open SSID that does not pass traffic.  We have downgraded to 6.5.r5 to resolve this issue.
Photo of Bill W.

Bill W.

  • 222 Posts
  • 35 Reply Likes
Brian,

You won't see the newer HiveOS versions until you upgrade HM.

And by reminder, I hope that means you used the CLI Supplement (or manually) used the "reboot schedule" command to reboot the APs every other Friday evening.
Photo of Brian Lowrance

Brian Lowrance

  • 10 Posts
  • 0 Reply Likes
I had been logging into HMOL, selecting all of the AP230's (using a device filter) and then going to Utilities > Reboot.

I didn't know about the Supplemental Cli option.  I do now thanks to you and will be using that going forward!