All tunneled clients simultaneously disconnect from CVG randomly.

  • 1
  • Question
  • Updated 2 years ago
Hello,

We currently run a secure (802.1X) corporate network using the CVG ESXi virtual appliance across 60 locations. Clients are a mix of mainly IPads and some windows 7/10 laptops. Randomly all clients drop layer 3 connectivity, meaning they lose site of their gateway and any routing. They are still showing as being connected to the AP, but in the case of the windows machines end up with a bogus 169.x.x.x address. 

We have noticed also and may be totally unrelated, that the NTP on the virtual cvg has been off up to 2 minutes when this occurs. We have ntp configured and we have verified through packet caps that it is updated every hour but still somehow time is jumping around with in that. 

To address the randomness everything could run fine for a week, or 1 day we have seen both and everything in between. If the CLIENT is rebooted it will reconnect just fine. 

We have a ticket with support that is escalated to tier 3/ the dev team but was hoping that someone had seen this behavior in the past and right now any help is appreciated.

We are running:
HiveOS 6.6r2 Irvine.2309
HM version is 6.6r3

Thanks!
Photo of Bradley J

Bradley J

  • 2 Posts
  • 0 Reply Likes

Posted 2 years ago

  • 1
Photo of Mike Kouri

Mike Kouri, Official Rep

  • 1030 Posts
  • 271 Reply Likes
From your description it is definitely a clock-drift issue within the CVG, which causes the VPN tunnels to all be torn down once the variance from 'real' time gets too large.  This can happen with virtualized environments, the hypervisors cannot guarantee perfect time .

Where is your CVG getting it's time source? Perhaps you need to set up the CVG to update it's clock more frequently?
Photo of Bradley J

Bradley J

  • 2 Posts
  • 0 Reply Likes
Hi Mike,

Thanks for the reply, the ntp source is set to use our core nexus 7k switch, which in-turn uses the tock.navy clock. This is the setup we use for all non windows devices. We have the cvg set to its lowest configurable setting on the refresh which is 60 minutes would love to set it to once a minute. We have also confirmed through packet caps that it is successfully updating, doing show clock on the device and comparing it to the active time i have seen in 30 seconds off then 2 minutes later its current, when we had the latest outage today it was over a minute off. Seems to be jumping and always ahead in time.
(Edited)