Anyone else having issues with AP's losing connectivity to NG?

  • 1
  • Question
  • Updated 3 years ago
Every few minutes our AP's lose connectivity to NG then re-connect. On an AP330 we see the LED change from white to green then back again every few mins. It makes it quite difficult to push out policies as NG thinks one or more AP's are disconnected.

We don't have any Internet issues, so I am wondering if we are breaching a CAPWAP round-trip delay timer or something, but I can't remember how to check or set this.

Anyone have any ideas? The problem seems to have got worse recently, when we first moved onto NG everything was fine. I have tried configuring one of the AP's statically rather than using DHCP in case it was a network issue our end but it hasn't made any difference.

Cheers,

Paul
Photo of Paul

Paul

  • 19 Posts
  • 0 Reply Likes

Posted 3 years ago

  • 1
Photo of Paul

Paul

  • 19 Posts
  • 0 Reply Likes
Just managed to do a show capwap client while NG was saying it was disconnected...

8IC-AP330#show capwap client
CAPWAP client:   Enabled
CAPWAP transport mode:  UDP
DISCOVERY state: Sending Discovery packets to find the CAPWAP server
CAPWAP client IP:        192.168.0.2
CAPWAP server IP:        54.154.134.72
HiveManager Primary Name:54.154.134.72
HiveManager Backup Name: 54.76.207.46
CAPWAP Default Server Name: redirector.aerohive.com
Virtual HiveManager Name: VHM-MKKVQJEE
Server destination Port: 12222
CAPWAP send event:       Enabled
CAPWAP DTLS state:       Enabled
CAPWAP DTLS negotiation: Disabled
     DTLS next connect status:   Enable
     DTLS always accept bootstrap passphrase: Enabled
     DTLS session status: Disconnect
     DTLS key type: passphrase
     DTLS session cut interval:     5 seconds
     DTLS handshake wait interval: 60 seconds
     DTLS Max retry count:          3
     DTLS authorize failed:         0
     DTLS reconnect count:          0
Discovery interval:      5 seconds
Heartbeat interval:     30 seconds
Max discovery interval: 10 seconds
Neighbor dead interval:105 seconds
Silent interval:        15 seconds
Wait join interval:     60 seconds
Discovery count:         0
Max discovery count:     3
Retransmit count:        0
Max retransmit count:    2
Primary server tries:    1
Backup server tries:     0
Keepalives lost/sent:    433/7935
Event packet drop due to buffer shortage: 0
Event packet drop due to loss connection: 14

8IC-AP330#show capwap client
CAPWAP client:   Enabled
CAPWAP transport mode:  UDP
SULKING state: Pausing between two sets of DISCOVERY states
CAPWAP client IP:        192.168.0.2
CAPWAP server IP:        54.76.207.46
HiveManager Primary Name:54.154.134.72
HiveManager Backup Name: 54.76.207.46
CAPWAP Default Server Name: redirector.aerohive.com
Virtual HiveManager Name: VHM-MKKVQJEE
Server destination Port: 12222
CAPWAP send event:       Enabled
CAPWAP DTLS state:       Enabled
CAPWAP DTLS negotiation: Disabled
     DTLS next connect status:   Enable
     DTLS always accept bootstrap passphrase: Enabled
     DTLS session status: Disconnect
     DTLS key type: passphrase
     DTLS session cut interval:     5 seconds
     DTLS handshake wait interval: 60 seconds
     DTLS Max retry count:          3
     DTLS authorize failed:         0
     DTLS reconnect count:          0
Discovery interval:      5 seconds
Heartbeat interval:     30 seconds
Max discovery interval: 10 seconds
Neighbor dead interval:105 seconds
Silent interval:        15 seconds
Wait join interval:     60 seconds
Discovery count:         3
Max discovery count:     3
Retransmit count:        0
Max retransmit count:    2
Primary server tries:    0
Backup server tries:     1
Keepalives lost/sent:    436/7994
Event packet drop due to buffer shortage: 0
Event packet drop due to loss connection: 14
Photo of Paul

Paul

  • 19 Posts
  • 0 Reply Likes
I noticed the CAPWAP server IP is alternating between the two HiveManager IP's. tracert to either does not look too good from here in the UK...

8IC-AP330#tracert 54.154.134.72
traceroute to 54.154.134.72 (54.154.134.72), 30 hops max, 38 byte packets
 1  192.168.0.1 (192.168.0.1)  1.034 ms  0.647 ms  0.614 ms
 2  lo0.10.Central10.ptn-bng02.plus.net (195.166.130.190)  25.326 ms  24.748 ms  25.096 ms
 3  irb.10.PTW-CR01.plus.net (84.93.249.1)  25.353 ms  irb.10.PTW-CR02.plus.net (84.93.249.2)  25.320 ms  44.308 ms
 4  195.66.225.175 (195.66.225.175)  25.511 ms  195.66.237.175 (195.66.237.175)  25.152 ms  195.66.225.175 (195.66.225.175)  25.502 ms
 5  *  *  *
 6  176.32.106.36 (176.32.106.36)  36.606 ms  *  *
 7  176.32.106.34 (176.32.106.34)  36.685 ms  178.236.0.171 (178.236.0.171)  39.365 ms  176.32.106.34 (176.32.106.34)  40.395 ms
 8  178.236.0.210 (178.236.0.210)  37.423 ms  178.236.0.206 (178.236.0.206)  35.991 ms  178.236.0.191 (178.236.0.191)  39.914 ms
 9  *  178.236.0.210 (178.236.0.210)  35.695 ms  *
10  *  *  *
11  *  *  *


8IC-AP330#tracert 54.76.207.46
traceroute to 54.76.207.46 (54.76.207.46), 30 hops max, 38 byte packets
 1  192.168.0.1 (192.168.0.1)  1.082 ms  0.699 ms  0.669 ms
 2  lo0.10.Central10.ptn-bng02.plus.net (195.166.130.190)  25.923 ms  25.493 ms  25.327 ms
 3  irb.10.PTW-CR01.plus.net (84.93.249.1)  25.598 ms  25.760 ms  33.218 ms
 4  195.66.237.175 (195.66.237.175)  25.112 ms  25.783 ms  25.333 ms
 5  178.236.3.53 (178.236.3.53)  39.706 ms  178.236.3.57 (178.236.3.57)  40.443 ms  41.403 ms
 6  *  *  176.32.106.29 (176.32.106.29)  35.881 ms
 7  176.32.106.34 (176.32.106.34)  39.639 ms  178.236.1.139 (178.236.1.139)  36.495 ms  176.32.106.36 (176.32.106.36)  36.383 ms
 8  178.236.0.208 (178.236.0.208)  36.718 ms  178.236.1.139 (178.236.1.139)  36.215 ms  178.236.1.137 (178.236.1.137)  41.832 ms
 9  178.236.0.206 (178.236.0.206)  36.407 ms  178.236.0.208 (178.236.0.208)  36.505 ms  *
10  178.236.1.189 (178.236.1.189)  36.167 ms  *  *
11  *  *  *
12  *^C


Both seem to get stuck in Ireland on Amazon's network, 178.236.... Maybe some issue with NG running on AWS in Ireland?
(Edited)
Photo of Gary Smith

Gary Smith, Official Rep

  • 299 Posts
  • 61 Reply Likes
Hi Paul,

Can you configure the transport mode to HTTP port 80 and see if that stabilises the connectivity? I believe that AP's connecting to HMNG should use TCP rather than CAPWAP 12222 so although a CAPWAP connection is possible, the HM will be pushing for the AP to use TCP.

I assume HTTP port 80 traffic is allowed?

CLI commands;
"capwap client transport HTTP"
"capwap client server port 80"

Kind Regards,
Gary Smith
Photo of Paul

Paul

  • 19 Posts
  • 0 Reply Likes
Thanks, I did that but still seem to be having problems... these commands were run over a couple of minutes:

8IC-AP330#show capwap client
CAPWAP client: Enabled
CAPWAP transport mode: HTTP on TCP
RUN state: Connected securely to the CAPWAP server
CAPWAP client IP: 192.168.0.2
CAPWAP server IP: 54.154.134.72
HiveManager Primary Name:54.154.134.72
HiveManager Backup Name: 54.76.207.46
CAPWAP Default Server Name: redirector.aerohive.com
Virtual HiveManager Name: VHM-MKKVQJEE
Server destination Port: 80
CAPWAP send event: Enabled
CAPWAP DTLS state: Enabled
CAPWAP DTLS negotiation: Disabled
DTLS next connect status: Enable
DTLS always accept bootstrap passphrase: Enabled
DTLS session status: Connected
DTLS key type: passphrase
DTLS session cut interval: 5 seconds
DTLS handshake wait interval: 60 seconds
DTLS Max retry count: 3
DTLS authorize failed: 0
DTLS reconnect count: 0
Discovery interval: 5 seconds
Heartbeat interval: 30 seconds
Max discovery interval: 10 seconds
Neighbor dead interval:105 seconds
Silent interval: 15 seconds
Wait join interval: 60 seconds
Discovery count: 0
Max discovery count: 3
Retransmit count: 1
Max retransmit count: 2
Primary server tries: 0
Backup server tries: 0
Keepalives lost/sent: 886/15892
Event packet drop due to buffer shortage: 0
Event packet drop due to loss connection: 20

8IC-AP330#show capwap client
CAPWAP client: Enabled
CAPWAP transport mode: HTTP on TCP

CAPWAP client IP: 192.168.0.2
CAPWAP server IP: 54.154.134.72
HiveManager Primary Name:54.154.134.72
HiveManager Backup Name: 54.76.207.46
CAPWAP Default Server Name: redirector.aerohive.com
Virtual HiveManager Name: VHM-MKKVQJEE
Server destination Port: 80
CAPWAP send event: Enabled
CAPWAP DTLS state: Enabled
CAPWAP DTLS negotiation: Disabled
DTLS next connect status: Enable
DTLS always accept bootstrap passphrase: Enabled
DTLS session status: Connected
DTLS key type: passphrase
DTLS session cut interval: 5 seconds
DTLS handshake wait interval: 60 seconds
DTLS Max retry count: 3
DTLS authorize failed: 0
DTLS reconnect count: 0
Discovery interval: 5 seconds
Heartbeat interval: 30 seconds
Max discovery interval: 10 seconds
Neighbor dead interval:105 seconds
Silent interval: 15 seconds
Wait join interval: 60 seconds
Discovery count: 0
Max discovery count: 3
Retransmit count: 2
Max retransmit count: 2
Primary server tries: 0
Backup server tries: 0
Keepalives lost/sent: 888/15893
Event packet drop due to buffer shortage: 0
Event packet drop due to loss connection: 20

8IC-AP330#show capwap client
CAPWAP client: Enabled
CAPWAP transport mode: HTTP on TCP
IDLE state: Preparing to enter the DISCOVERY state
CAPWAP client IP: 192.168.0.2
CAPWAP server IP: 54.76.207.46
HiveManager Primary Name:54.154.134.72
HiveManager Backup Name: 54.76.207.46
CAPWAP Default Server Name: redirector.aerohive.com
Virtual HiveManager Name: VHM-MKKVQJEE
Server destination Port: 80
CAPWAP send event: Enabled
CAPWAP DTLS state: Enabled
CAPWAP DTLS negotiation: Disabled
DTLS next connect status: Enable
DTLS always accept bootstrap passphrase: Enabled
DTLS session status: Connected
DTLS key type: passphrase
DTLS session cut interval: 5 seconds
DTLS handshake wait interval: 60 seconds
DTLS Max retry count: 3
DTLS authorize failed: 0
DTLS reconnect count: 0
Discovery interval: 5 seconds
Heartbeat interval: 30 seconds
Max discovery interval: 10 seconds
Neighbor dead interval:105 seconds
Silent interval: 15 seconds
Wait join interval: 60 seconds
Discovery count: 0
Max discovery count: 3
Retransmit count: 0
Max retransmit count: 2
Primary server tries: 0
Backup server tries: 1
Keepalives lost/sent: 888/15893
Event packet drop due to buffer shortage: 0
Event packet drop due to loss connection: 20

8IC-AP330#show capwap client
CAPWAP client: Enabled
CAPWAP transport mode: HTTP on TCP
DISCOVERY state: Sending Discovery packets to find the CAPWAP server
CAPWAP client IP: 192.168.0.2
CAPWAP server IP: 54.76.207.46
HiveManager Primary Name:54.154.134.72
HiveManager Backup Name: 54.76.207.46
CAPWAP Default Server Name: redirector.aerohive.com
Virtual HiveManager Name: VHM-MKKVQJEE
Server destination Port: 80
CAPWAP send event: Enabled
CAPWAP DTLS state: Enabled
CAPWAP DTLS negotiation: Disabled
DTLS next connect status: Enable
DTLS always accept bootstrap passphrase: Enabled
DTLS session status: Disconnect
DTLS key type: passphrase
DTLS session cut interval: 5 seconds
DTLS handshake wait interval: 60 seconds
DTLS Max retry count: 3
DTLS authorize failed: 0
DTLS reconnect count: 0
Discovery interval: 5 seconds
Heartbeat interval: 30 seconds
Max discovery interval: 10 seconds
Neighbor dead interval:105 seconds
Silent interval: 15 seconds
Wait join interval: 60 seconds
Discovery count: 0
Max discovery count: 3
Retransmit count: 0
Max retransmit count: 2
Primary server tries: 0
Backup server tries: 1
Keepalives lost/sent: 888/15893
Event packet drop due to buffer shortage: 0
Event packet drop due to loss connection: 20

8IC-AP330#show capwap client
CAPWAP client: Enabled
CAPWAP transport mode: HTTP on TCP
DTLS state: Handshaking with CAPWAP server using DTLS
CAPWAP client IP: 192.168.0.2
CAPWAP server IP: 54.76.207.46
HiveManager Primary Name:54.154.134.72
HiveManager Backup Name: 54.76.207.46
CAPWAP Default Server Name: redirector.aerohive.com
Virtual HiveManager Name: VHM-MKKVQJEE
Server destination Port: 80
CAPWAP send event: Enabled
CAPWAP DTLS state: Enabled
CAPWAP DTLS negotiation: Disabled
DTLS next connect status: Enable
DTLS always accept bootstrap passphrase: Enabled
DTLS session status: Disconnect
DTLS key type: passphrase
DTLS session cut interval: 5 seconds
DTLS handshake wait interval: 60 seconds
DTLS Max retry count: 3
DTLS authorize failed: 0
DTLS reconnect count: 0
Discovery interval: 5 seconds
Heartbeat interval: 30 seconds
Max discovery interval: 10 seconds
Neighbor dead interval:105 seconds
Silent interval: 15 seconds
Wait join interval: 60 seconds
Discovery count: 0
Max discovery count: 3
Retransmit count: 0
Max retransmit count: 2
Primary server tries: 0
Backup server tries: 1
Keepalives lost/sent: 888/15893
Event packet drop due to buffer shortage: 0
Event packet drop due to loss connection: 20

8IC-AP330#show capwap client
CAPWAP client: Enabled
CAPWAP transport mode: HTTP on TCP

CAPWAP client IP: 192.168.0.2
CAPWAP server IP: 54.76.207.46
HiveManager Primary Name:54.154.134.72
HiveManager Backup Name: 54.76.207.46
CAPWAP Default Server Name: redirector.aerohive.com
Virtual HiveManager Name: VHM-MKKVQJEE
Server destination Port: 80
CAPWAP send event: Enabled
CAPWAP DTLS state: Enabled
CAPWAP DTLS negotiation: Disabled
DTLS next connect status: Enable
DTLS always accept bootstrap passphrase: Enabled
DTLS session status: Connected
DTLS key type: passphrase
DTLS session cut interval: 5 seconds
DTLS handshake wait interval: 60 seconds
DTLS Max retry count: 3
DTLS authorize failed: 0
DTLS reconnect count: 0
Discovery interval: 5 seconds
Heartbeat interval: 30 seconds
Max discovery interval: 10 seconds
Neighbor dead interval:105 seconds
Silent interval: 15 seconds
Wait join interval: 60 seconds
Discovery count: 0
Max discovery count: 3
Retransmit count: 0
Max retransmit count: 2
Primary server tries: 0
Backup server tries: 0
Keepalives lost/sent: 888/15894
Event packet drop due to buffer shortage: 0
Event packet drop due to loss connection: 20

8IC-AP330#show capwap client
CAPWAP client: Enabled
CAPWAP transport mode: HTTP on TCP

CAPWAP client IP: 192.168.0.2
CAPWAP server IP: 54.76.207.46
HiveManager Primary Name:54.154.134.72
HiveManager Backup Name: 54.76.207.46
CAPWAP Default Server Name: redirector.aerohive.com
Virtual HiveManager Name: VHM-MKKVQJEE
Server destination Port: 80
CAPWAP send event: Enabled
CAPWAP DTLS state: Enabled
CAPWAP DTLS negotiation: Disabled
DTLS next connect status: Enable
DTLS always accept bootstrap passphrase: Enabled
DTLS session status: Connected
DTLS key type: passphrase
DTLS session cut interval: 5 seconds
DTLS handshake wait interval: 60 seconds
DTLS Max retry count: 3
DTLS authorize failed: 0
DTLS reconnect count: 0
Discovery interval: 5 seconds
Heartbeat interval: 30 seconds
Max discovery interval: 10 seconds
Neighbor dead interval:105 seconds
Silent interval: 15 seconds
Wait join interval: 60 seconds
Discovery count: 0
Max discovery count: 3
Retransmit count: 0
Max retransmit count: 2
Primary server tries: 0
Backup server tries: 0
Keepalives lost/sent: 888/15894
Event packet drop due to buffer shortage: 0
Event packet drop due to loss connection: 20

8IC-AP330#show capwap client
CAPWAP client: Enabled
CAPWAP transport mode: HTTP on TCP
IDLE state: Preparing to enter the DISCOVERY state
CAPWAP client IP: 192.168.0.2
CAPWAP server IP: 54.154.134.72
HiveManager Primary Name:54.154.134.72
HiveManager Backup Name: 54.76.207.46
CAPWAP Default Server Name: redirector.aerohive.com
Virtual HiveManager Name: VHM-MKKVQJEE
Server destination Port: 80
CAPWAP send event: Enabled
CAPWAP DTLS state: Enabled
CAPWAP DTLS negotiation: Disabled
DTLS next connect status: Enable
DTLS always accept bootstrap passphrase: Enabled
DTLS session status: Connected
DTLS key type: passphrase
DTLS session cut interval: 5 seconds
DTLS handshake wait interval: 60 seconds
DTLS Max retry count: 3
DTLS authorize failed: 0
DTLS reconnect count: 0
Discovery interval: 5 seconds
Heartbeat interval: 30 seconds
Max discovery interval: 10 seconds
Neighbor dead interval:105 seconds
Silent interval: 15 seconds
Wait join interval: 60 seconds
Discovery count: 0
Max discovery count: 3
Retransmit count: 0
Max retransmit count: 2
Primary server tries: 1
Backup server tries: 0
Keepalives lost/sent: 888/15894
Event packet drop due to buffer shortage: 0
Event packet drop due to loss connection: 20

8IC-AP330#show capwap client
CAPWAP client: Enabled
CAPWAP transport mode: HTTP on TCP
DISCOVERY state: Sending Discovery packets to find the CAPWAP server
CAPWAP client IP: 192.168.0.2
CAPWAP server IP: 54.154.134.72
HiveManager Primary Name:54.154.134.72
HiveManager Backup Name: 54.76.207.46
CAPWAP Default Server Name: redirector.aerohive.com
Virtual HiveManager Name: VHM-MKKVQJEE
Server destination Port: 80
CAPWAP send event: Enabled
CAPWAP DTLS state: Enabled
CAPWAP DTLS negotiation: Disabled
DTLS next connect status: Enable
DTLS always accept bootstrap passphrase: Enabled
DTLS session status: Disconnect
DTLS key type: passphrase
DTLS session cut interval: 5 seconds
DTLS handshake wait interval: 60 seconds
DTLS Max retry count: 3
DTLS authorize failed: 0
DTLS reconnect count: 0
Discovery interval: 5 seconds
Heartbeat interval: 30 seconds
Max discovery interval: 10 seconds
Neighbor dead interval:105 seconds
Silent interval: 15 seconds
Wait join interval: 60 seconds
Discovery count: 0
Max discovery count: 3
Retransmit count: 0
Max retransmit count: 2
Primary server tries: 1
Backup server tries: 0
Keepalives lost/sent: 888/15894
Event packet drop due to buffer shortage: 0
Event packet drop due to loss connection: 20

8IC-AP330#show capwap client
CAPWAP client: Enabled
CAPWAP transport mode: HTTP on TCP
RUN state: Connected securely to the CAPWAP server
CAPWAP client IP: 192.168.0.2
CAPWAP server IP: 54.154.134.72
HiveManager Primary Name:54.154.134.72
HiveManager Backup Name: 54.76.207.46
CAPWAP Default Server Name: redirector.aerohive.com
Virtual HiveManager Name: VHM-MKKVQJEE
Server destination Port: 80
CAPWAP send event: Enabled
CAPWAP DTLS state: Enabled
CAPWAP DTLS negotiation: Disabled
DTLS next connect status: Enable
DTLS always accept bootstrap passphrase: Enabled
DTLS session status: Connected
DTLS key type: passphrase
DTLS session cut interval: 5 seconds
DTLS handshake wait interval: 60 seconds
DTLS Max retry count: 3
DTLS authorize failed: 0
DTLS reconnect count: 0
Discovery interval: 5 seconds
Heartbeat interval: 30 seconds
Max discovery interval: 10 seconds
Neighbor dead interval:105 seconds
Silent interval: 15 seconds
Wait join interval: 60 seconds
Discovery count: 0
Max discovery count: 3
Retransmit count: 0
Max retransmit count: 2
Primary server tries: 0
Backup server tries: 0
Keepalives lost/sent: 888/15894
Event packet drop due to buffer shortage: 0
Event packet drop due to loss connection: 20
8IC-AP330#
Photo of Nick Lowe

Nick Lowe, Official Rep

  • 2491 Posts
  • 451 Reply Likes
What do you get from:

debug console
_debug capwap all
Photo of Paul

Paul

  • 19 Posts
  • 0 Reply Likes
Hi Nick, I was just about to do this but for the last 50 mins the connection has been stable, before that it seemed to be losing connectivity every few minutes. I will keep an eye on it and try this if it starts again. Thanks.
Photo of Dan

Dan, Employee

  • 1 Post
  • 0 Reply Likes
Paul it looks more like a network connectivity issue somewhere between the AP and the NG data center in Ireland. A bit difficult to troubleshoot from NG side since it's on the receiving end. Do let us know if it happens again. NG does not use the native UDP CAPWAP on port 12222. It uses http/s instead.
Photo of Youping Liu

Youping Liu

  • 11 Posts
  • 0 Reply Likes
NG only supports TCP HTTP for capwap connections. Please use the following IP addresses for capwap servers 
54.86.52.73
54.152.60.81
Photo of Nick Lowe

Nick Lowe, Official Rep

  • 2491 Posts
  • 451 Reply Likes
NG is not setting "capwap client transport HTTP" meaning that UDP is tried first until it times out and HTTP is tried in fallback. Is this intended or a bug in config generation?
(Edited)
Photo of Paul

Paul

  • 19 Posts
  • 0 Reply Likes
Ok so it was still dropping the connection every 5 or 10 mins or so, I have updated the config to use the capwap servers in Youping's post above, I still can't traceroute to them but will see what happens with these.
Photo of Paul

Paul

  • 19 Posts
  • 0 Reply Likes
Also just echoing Nick's point, if NG doesn't use UDP 12222 any more then we'll need a way for all the AP's to figure this out rather than going around and having to manually update them all.
Photo of Paul

Paul

  • 19 Posts
  • 0 Reply Likes
AP is connected but NG GUI now shows it disconnected since I made the change (about 30 mins ago now).

8IC-AP330#show capwap clientCAPWAP client:   Enabled
CAPWAP transport mode:  HTTP on TCP
RUN state: Connected securely to the CAPWAP server
CAPWAP client IP:        192.168.0.2
CAPWAP server IP:        54.86.52.73
HiveManager Primary Name:hmng-prd-va-cwps-02.aerohive.com
HiveManager Backup Name: hmng-prd-va-cwpm-01.aerohive.com
CAPWAP Default Server Name: redirector.aerohive.com

It seems the AP is now connected to the "VA" cluster rather than "IE"?

When I go to cloud.aerohive.com I always get redirected to https://cloud-ie.aerohive.com/ so I suspect these "va" capwap servers are not configured on the Ireland cluster? If I try and log into cloud-va.aerohive.com I get permission denied.

So I'm going to revert my config back to:

hmng-prd-ie-cwps-01.aerohive.com
hmng-prd-ie-cwpm-01.aerohive.com

Are there any firewalls or IP policies set that are stopping me doing a traceroute or ICMP ping? That would be really handy as currently it never completes so it's hard to see if there is a connectivity issue.