High CPU utilization (probably Bonjour)

  • 1
  • Question
  • Updated 1 year ago
Hi,

We have an AP121 running HiveOS 6.5r3a Honolulu.2530 which is at 100% CPU. 
When enabling debugging on the console, it gets flooded with bonjour gateway cancel browse service events (see snippet below). Bonjour gateway isn't enabled so I don't really know what is going on. I only have console access briefly after rebooting the AP (by shutting down the switch port). After rebooting CPU goes to 100% instantly. Since most CPU utilization goes to software interrupts, could it be some kind of traffic causing the issue?

KON-AP-5#show cpuCPU total utilization:                100.000%
CPU user utilization:                 3.649%
CPU system utilization:               8.029%

KON-AP-5#show cpu detail
CPU total utilization:                100.000%
CPU user utilization:                 9.405%
CPU system utilization:               13.861%
Number of interrupt in last second:   3980
Interrupt utilization:                0.000%
Soft interrupt utilization:           76.732%

KON-AP-5#show system processes state
Mem: 95764K used, 26140K free, 0K shrd, 2356K buff, 28348K cachedCPU:  5.6% usr  6.7% sys  0.0% nic  0.0% idle  0.0% io  0.0% irq 87.6% sirq
Load average: 30.66 25.94 17.65 51/235 4128
  PID  PPID USER     STAT   VSZ %MEM CPU %CPU COMMAND
 1639     1 root     S    30868 25.2   0 32.4 /sbin/mdnsd
 1607  1576 root     S     121m101.7   0 12.7 /opt/ah/bin/ah_dcd
 1605  1576 root     S     103m 86.7   0 11.0 /opt/ah/bin/ah_auth
    3     2 root     RW       0  0.0   0  8.0 [ksoftirqd/0]
 2280  1576 root     S     116m 97.1   0  7.8 /opt/ah/bin/capwap
 1610  1576 root     R    61552 50.3   0  5.4 /opt/ah/bin/amrp2
 1881     1 root     S    55684 45.5   0  2.4 /opt/ah/bin/ah_bgd
 1884     1 root     S    61156 50.0   0  2.2 /opt/ah/bin/ah_lcs
  846     1 root     S     6620  5.4   0  2.2 /opt/ah/bin/ah_syslogd
 4017  2680 root     R     3656  2.9   0  1.8 top
 2282  1576 root     S    53512 43.7   0  1.6 /opt/ah/bin/ah_nbr
 1882     1 root     S    74392 60.8   0  0.7 /opt/ah/bin/ah_vpn
 1576     1 root     R    73076 59.7   0  0.7 /opt/ah/bin/ah_top
  819     2 root     RWN      0  0.0   0  0.7 [jffs2_gcd_mtd6]
 1609  1576 root     S    93052 76.1   0  0.5 /opt/ah/bin/ah_scd
 2679  2650 admin    S    10428  8.5   0  0.5 sshd: admin@pts/0
  847     1 root     S     1984  1.6   0  0.5 /opt/ah/bin/ah_klogd
 1604  1576 root     S    43716 35.7   0  0.3 /opt/ah/bin/ah_capture
 1540     1 root     R    36368 29.7   0  0.3 /opt/ah/bin/ah_monitor
    4     2 root     RW       0  0.0   0  0.3 [events/0]
 1602  1576 root     S     153m128.7   0  0.1 /opt/ah/bin/ah_brd
 2279  1576 admin    S    92124 75.3   0  0.1 /opt/ah/bin/ah_webui
 1606  1576 root     S    62304 50.9   0  0.1 /opt/ah/bin/fed
 2281  1576 root     S    61788 50.5   0  0.1 /opt/ah/bin/capwap_srv
 2680  2679 root     S    29840 24.4   0  0.1 -ah_cli_ui
   76     2 root     SW       0  0.0   0  0.1 [bdi-default]
 1611  1576 root     S    78304 64.0   0  0.0 /opt/ah/bin/l7d
 1875  1576 root     S    62324 50.9   0  0.0 /opt/ah/bin/ah_ipfw
 1601  1576 root     S    54168 44.3   0  0.0 /opt/ah/bin/ah_dcm
 1883     1 root     S    53628 43.8   0  0.0 /opt/ah/bin/ah_ltr
 1608  1576 root     S    44552 36.4   0  0.0 /opt/ah/bin/ah_ibeacon
 1874  1576 root     S    38796 31.7   0  0.0 /opt/ah/bin/dhcpc
 2273  1576 root     S    35956 29.4   0  0.0 /opt/ah/bin/ah_cli
 1843     1 root     R    30108 24.6   0  0.0 /opt/ah/bin/ntpclient
 2553     1 root     S    11520  9.4   0  0.0 /usr/local/bin/php-cgi -b 127.0.0.1:2008
 2650  2635 root     S    10276  8.4   0  0.0 sshd: admin [priv]
 2537     1 admin    R     7828  6.4   0  0.0 /usr/local/sbin/hiawatha
 2635     1 root     S     7628  6.2   0  0.0 /sbin/sshd -f /etc/sshd_config
 2684     1 root     S     3540  2.9   0  0.0 /sbin/getty -L ttyS0 9600 vt100
    1     0 root     S     3328  2.7   0  0.0 init
 1582     1 root     S     2424  1.9   0  0.0 /opt/ah/bin/guard
   74     2 root     SW       0  0.0   0  0.0 [sync_supers]
    5     2 root     SW       0  0.0   0  0.0 [khelper]
    2     0 root     SW       0  0.0   0  0.0 [kthreadd]
    8     2 root     SW       0  0.0   0  0.0 [async/mgr]
   77     2 root     SW       0  0.0   0  0.0 [kblockd/0]
   86     2 root     SW       0  0.0   0  0.0 [khubd]
  107     2 root     SW       0  0.0   0  0.0 [kswapd0]
  108     2 root     SW       0  0.0   0  0.0 [aio/0]
  110     2 root     SW       0  0.0   0  0.0 [crypto/0]
  765     2 root     SW       0  0.0   0  0.0 [mtdblockd]
  939     2 root     SW       0  0.0   0  0.0 [ip_reconfig/0]



LOG Snippet:

2016-12-01 13:54:53 warn    mDNSResponder: MDNS:add virtual interface VLAN12016-12-01 13:54:53 info    amrp2: receive kevent KEVT_IF_CHG type IF_DOWN from wifi1.1 index 12
2016-12-01 13:54:53 warn    ah_dcd: Interface wifi1.1 is down.
2016-12-01 13:54:53 info    capwap: CAPWAP receive kevent KEVT_IF_CHG, eventid = 17, size = 36
2016-12-01 13:54:53 info    amrp2: amrp delete ifp wifi1.1(4018:b14e:1d28)
2016-12-01 13:54:53 info    kernel: [mac]: MAC learning: recv netdev notifier event 2 on wifi1.1
2016-12-01 13:54:53 info    kernel: [mpi]: wifi1.1 notify kevent KEVT_IF_CHG, type IF_DOWN(2)
2016-12-01 13:54:54 warn    ah_auth: ah_auth_bulk_group_gen_users: bulk(8096) to bulk(9999) can't be generated because the timestamp out of range.
2016-12-01 13:54:54 notice  ah_auth: ah_auth_bulk_group_gen_users: group(UG_BAN_1DAY) user_cb(gen=0, active=50)
2016-12-01 13:54:54 info    kernel: [fe]: access interface counter 5, portal interface counter 1
2016-12-01 13:54:54 info    ah_bgd: bonjour gateway cancel browse service type:_ni-logos._tcp.
2016-12-01 13:54:54 info    ah_bgd: bonjour gateway cancel browse service type:_munhttp._tcp.
2016-12-01 13:54:54 info    ah_bgd: bonjour gateway cancel browse service type:_jenkins._tcp.
2016-12-01 13:54:54 info    ah_bgd: bonjour gateway cancel browse service type:_airplay._tcp.
2016-12-01 13:54:54 info    ah_bgd: bonjour gateway cancel browse service type:_sftp-ssh._tcp.
2016-12-01 13:54:54 info    ah_bgd: bonjour gateway cancel browse service type:_sleep-proxy._udp.
2016-12-01 13:54:54 info    ah_bgd: bonjour gateway cancel browse service type:_hudson._tcp.
2016-12-01 13:54:54 info    ah_bgd: bonjour gateway cancel browse service type:_workstation._tcp.
2016-12-01 13:54:54 info    ah_bgd: bonjour gateway cancel browse service type:_mopnet100._tcp.
2016-12-01 13:54:54 info    ah_bgd: bonjour gateway cancel browse service type:_ssh._tcp.
2016-12-01 13:54:54 warn    mDNSResponder: interface change, mdnsd refresh interface list and service!!!
2016-12-01 13:54:54 info    amrp2: receive kevent KEVT_IF_CHG type IF_GOING_DOWN from wifi1.2 index 14
2016-12-01 13:54:54 info    capwap: CAPWAP receive kevent KEVT_IF_CHG, eventid = 17, size = 36
2016-12-01 13:54:55 info    kernel: [mpi]: wifi1.2 notify kevent KEVT_IF_CHG, type IF_GOING_DOWN(7)
2016-12-01 13:54:55 info    ah_bgd: bonjour gateway cancel browse service type:_mopnet300._tcp.
2016-12-01 13:54:55 info    ah_bgd: bonjour gateway cancel browse service type:_icc-service._tcp.
2016-12-01 13:54:55 info    ah_bgd: bonjour gateway cancel browse service type:_apple-mobdev2._tcp.
2016-12-01 13:54:55 info    ah_bgd: bonjour gateway cancel browse service type:_scan-target._tcp.
2016-12-01 13:54:55 info    ah_bgd: bonjour gateway cancel browse service type:_rfb._tcp.
2016-12-01 13:54:55 info    ah_bgd: bonjour gateway cancel browse service type:_teamviewer._tcp.
2016-12-01 13:54:55 info    ah_bgd: bonjour gateway cancel browse service type:_airdroid._tcp.
2016-12-01 13:54:55 warn    mDNSResponder: MDNS:add virtual interface VLAN1
2016-12-01 13:54:55 info    amrp2: receive kevent KEVT_IF_CHG type IF_DOWN from wifi1.2 index 14
2016-12-01 13:54:55 info    capwap: CAPWAP receive kevent KEVT_IF_CHG, eventid = 17, size = 36
Photo of YMEsko

YMEsko

  • 11 Posts
  • 1 Reply Like

Posted 2 years ago

  • 1
Photo of Gary Smith

Gary Smith, Official Rep

  • 299 Posts
  • 61 Reply Likes
Hi,

Is this a different deployment to the one mentioned here; High CPU utilization after upgrading to HiveOS 6.5r3a Honolulu.2530 

Kind Regards,
Gary Smith
Photo of YMEsko

YMEsko

  • 11 Posts
  • 1 Reply Like
It could be the same. However, this is on a completely different network than the previous issue. I'll set up the remote sniffer to see if I see similar traffic.
Photo of Gary Smith

Gary Smith, Official Rep

  • 299 Posts
  • 61 Reply Likes
Thanks. I would also suggest following the suggestions I made in the previous thread to see what is left. I'd be interested to see your findings.

Kind Regards,
Gary Smith
Photo of Luke Harris

Luke Harris

  • 265 Posts
  • 18 Reply Likes
YMEsko - I had a similar issue within my own deployment. One of the steps I took to reduce Bonjour traffic was to nominate a single AP per subnet (site) to act as a dedicated Bonjour Gateway and then ran the 'no bonjour-gateway' enable command on my APs to disable the service. 

I believe at the time I also disabled qos airtime scheduling as well. 
Photo of Eastman Rivai

Eastman Rivai, Official Rep

  • 146 Posts
  • 17 Reply Likes
We have a known issue on AP121 version 6 where the AP is flooding the network with MDNS traffic even though Bonjour is not enabled. The issue is addressed on 6.5r7. 
Photo of FPUAH

FPUAH

  • 8 Posts
  • 1 Reply Like
Any ETA on when 6.5r7 will be released?