"The system busy exceed" logs

  • 1
  • Question
  • Updated 2 years ago
Hi Guys,

One of our customer has " tracking IP" and "access console" features on under they wifi policy. From couple of  weeks they started to has some weird problems with some of AP's ( no wifi power and channel details for some of AP's under hive, users not able to connect to wifi at the same time ) I get some logs from those AP's and can see hundreds of below warning messages:

2016-03-14 09:14:53 warn    ah_top: PM: the system busy exceed 6540 second.2016-03-14 09:13:50 warn    ah_top: PM: the system busy exceed 6480 second.
2016-03-14 09:12:47 warn    ah_top: PM: the system busy exceed 6420 second.
2016-03-14 09:11:42 warn    ah_top: PM: the system busy exceed 6360 second.
2016-03-14 09:10:39 warn    ah_top: PM: the system busy exceed 6300 second.
2016-03-14 09:09:34 warn    ah_top: PM: the system busy exceed 6240 second.
2016-03-14 09:08:31 warn    ah_top: PM: the system busy exceed 6180 second.
2016-03-14 09:07:28 warn    ah_top: PM: the system busy exceed 6120 second.
2016-03-14 09:06:26 warn    ah_top: PM: the system busy exceed 6060 second.
2016-03-14 09:05:24 warn    ah_top: PM: the system busy exceed 6000 second.
2016-03-14 09:04:23 warn    ah_top: PM: the system busy exceed 5940 second.
2016-03-14 09:03:20 warn    ah_top: PM: the system busy exceed 5880 second.
2016-03-14 09:02:18 warn    ah_top: PM: the system busy exceed 5820 second.
2016-03-14 09:01:16 warn    ah_top: PM: the system busy exceed 5760 second.
2016-03-14 09:00:14 warn    ah_top: PM: the system busy exceed 5700 second.
2016-03-14 08:59:12 warn    ah_top: PM: the system busy exceed 5640 second.
2016-03-14 08:58:08 warn    ah_top: PM: the system busy exceed 5580 second.
2016-03-14 08:57:02 warn    ah_top: PM: the system busy exceed 5520 second.
2016-03-14 08:55:56 warn    ah_top: PM: the system busy exceed 5460 second.
2016-03-14 08:54:51 warn    ah_top: PM: the system busy exceed 5400 second.
2016-03-14 08:53:48 warn    ah_top: PM: the system busy exceed 5340 second.
2016-03-14 08:52:45 warn    ah_top: PM: the system busy exceed 5280 second.
2016-03-14 08:51:44 warn    ah_top: PM: the system busy exceed 5220 second.
2016-03-14 08:50:39 warn    ah_top: PM: the system busy exceed 5160 second.
2016-03-14 08:49:36 warn    ah_top: PM: the system busy exceed 5100 second.
2016-03-14 08:48:31 warn    ah_top: PM: the system busy exceed 5040 second.
2016-03-14 08:47:30 warn    ah_top: PM: the system busy exceed 4980 second.
2016-03-14 08:46:25 warn    ah_top: PM: the system busy exceed 4920 second.
2016-03-14 08:45:21 warn    ah_top: PM: the system busy exceed 4860 second.
2016-03-14 08:44:17 warn    ah_top: PM: the system busy exceed 4800 second.
2016-03-14 08:43:14 warn    ah_top: PM: the system busy exceed 4740 second.
2016-03-14 08:42:10 warn    ah_top: PM: the system busy exceed 4680 second.
2016-03-14 08:41:08 warn    ah_top: PM: the system busy exceed 4620 second.
2016-03-14 08:40:04 warn    ah_top: PM: the system busy exceed 4560 second.
2016-03-14 08:39:02 warn    ah_top: PM: the system busy exceed 4500 second.
2016-03-14 08:38:00 warn    ah_top: PM: the system busy exceed 4440 second.
2016-03-14 08:36:59 warn    ah_top: PM: the system busy exceed 4380 second.
2016-03-14 08:35:56 warn    ah_top: PM: the system busy exceed 4320 second.
2016-03-14 08:34:53 warn    ah_top: PM: the system busy exceed 4260 second.
2016-03-14 08:33:51 warn    ah_top: PM: the system busy exceed 4200 second.
2016-03-14 08:32:47 warn    ah_top: PM: the system busy exceed 4140 second.
2016-03-14 08:31:45 warn    ah_top: PM: the system busy exceed 4080 second.
2016-03-14 08:30:42 warn    ah_top: PM: the system busy exceed 4020 second.
2016-03-14 08:29:33 warn    ah_top: PM: the system busy exceed 3960 second.
2016-03-14 08:28:26 warn    ah_top: PM: the system busy exceed 3900 second.
2016-03-14 08:27:12 warn    ah_top: PM: the system busy exceed 3840 second.
2016-03-14 08:27:08 info    ah_monitor: Track IP: Turn off  ,access console, .
2016-03-14 08:26:45 info    ah_monitor: Track IP: Turn on  ,access console, .
2016-03-14 08:26:20 info    ah_monitor: Track IP: Turn off  ,access console, .
2016-03-14 08:26:02 info    ah_monitor: Track IP: Turn on  ,access console, .
2016-03-14 08:25:49 warn    ah_top: PM: the system busy exceed 3780 second.
2016-03-14 08:25:43 info    ah_monitor: Track IP: Turn off  ,access console, .
2016-03-14 08:25:31 info    ah_monitor: Track IP: Turn on  ,access console, .
2016-03-14 08:25:09 info    ah_monitor: Track IP: Turn off  ,access console, .
2016-03-14 08:24:18 warn    ah_top: PM: the system busy exceed 3720 second.

Due to "track IP" logs there I ask customer to remove "tracking ip" and "access console" from policy and this morning all logs seems to be all right.

I'm not to sure if "the system busy exceed" logs are related to CPU usage and why "tracking ip" causing this if it related at all ? Is this mean that those AP's are loosing connectivity and access console is on from time to time ? I can see " the system busy exceed"  logs literally all day long every minute or so with "tracking ip" and "access console" on.

Is anyone saw this kind of logs and how I should interpret this ?

Thanks for any help

Szymon
Photo of Szymon

Szymon

  • 16 Posts
  • 0 Reply Likes

Posted 2 years ago

  • 1
Photo of Gary Smith

Gary Smith, Official Rep

  • 299 Posts
  • 61 Reply Likes
Hi Szymon,

The logs "system busy" suggest that the AP is seeing high CPU. In your case, for quite some time. To diagnose, we would need to see some outputs from the AP at the time of the issue;

"show cpu detail"
"show system processes state"

These outputs would tell us what was cauing the high CPU. You might also want to see what traffic is leaving the switch port where the AP connects. The track-ip is tracking a particular address on the network, usually the default gateway if the AP is losing connectivity to this, you will see the track-ip messages.

Kind Regards,
Gary Smith
Photo of Nick Lowe

Nick Lowe, Official Rep

  • 2491 Posts
  • 451 Reply Likes
Could you also clarify the model(s) of AP experiencing this and the version of HiveOS?

One of AP100/AP120/AP121/AP141?
Photo of Szymon

Szymon

  • 16 Posts
  • 0 Reply Likes
Hi Guys,

Thanks for get back to me. it looks like all AP's affected are 120/121's working under 6.5r3a Honolulu 2530 firmware version. Customer is using mix of 120/121 and 320's.

Szymon