Unable to push delta/complete configuration

  • 1
  • Question
  • Updated 2 years ago
Hi,

For a while now I've not been able to push any configs to (al lot) of my aerohive ap's (ap230).
It keeps saying: "Could not download the captive web portal file". I tried clearing the web directory of the AP via ssh, they still wouldn't update after that. I also deleted any firewall policies, and the actual captive web portal, but they still won't update.

We're gonna try restarting our Hivemanager, and I'll check again.
Photo of Dennis

Dennis

  • 29 Posts
  • 1 Reply Like

Posted 2 years ago

  • 1
Photo of Maurice van Zundert

Maurice van Zundert

  • 31 Posts
  • 1 Reply Like
Hi there! I have had a bit of the same issue, no issues with your firewall / port 22 is open?
I know you do not want to hear this but try to reset the AP to factory default and do a complete update, probably this helps.
Photo of Dennis

Dennis

  • 29 Posts
  • 1 Reply Like
Hi,

Yeah port 22 is open, I already tried resetting and AP to factory default, but that wouldn't work either. So now i have 1 AP that is without any configuration.
Now of course i could ssh into that AP and paste in the config manually, but that wouldn't solve my issue...
Photo of Dennis

Dennis

  • 29 Posts
  • 1 Reply Like
So far, still no luck.
We restarted the HM and it still won't work.
each day more and more of our AP's are having this issue.
forcing the AP to upgrade it's firmware also wouldn't work.
I pasted a config in there manually via ssh, and after that it still won't upgrade.

I'm officially at a loss..
Photo of Brian Powers

Brian Powers, Champ

  • 396 Posts
  • 92 Reply Likes
Dennis,

Is your HM on premise?  Mind if I ask what code is on the APs and HM platform?  

Have you attempted to use the traditional method of Update - Advanced - Update and Activate Configuration (assuming you're using the Legacy 6 platform and not HM NG)?  There, you can de-select the "Upload and activate captive web portal pages and the server key" checkbox to truly determine if it is related to some CWP object.  Seems unlikely, but it would be a way to possibly verify.

If on prem and using Legacy 6 platform and an AP is running 6.2+ code, see if you can exec ssh into the back end of your HM itself.  This would prove beyond reason that SSH is traversing between the AP VLAN and the HM VLAN.  Command is: exec ssh-client server <HM IP> user <username>

Have you attempted to push a new Network Policy to any single piece of gear (perhaps the one that you defaulted, so as to not impact more WLAN users)?  

Another option could be to port mirror the AP port and the HM port and dig through the packet captures to see if anything sticks out.  
Photo of Dennis

Dennis

  • 29 Posts
  • 1 Reply Like
Hi Brian,

Yes, our HM is on premise, the code we're running is 6.6r2a.
And our HM is running 11.9.3.3.
We are allowing SSH to the HM but are unable to make a connection, i will have to ask our server manager for the details.

I have attempted to push a new policy, but also that would not work.

I will do a port-mirror and wireshark this afternoon, i'll get back to you on that :)

Thanks for the helpfull suggestions!
Photo of Roberto Casula

Roberto Casula, Champ

  • 231 Posts
  • 111 Reply Likes
This is a really weird problem. A packet capture of the management traffic between AP and HiveManager would be useful to see exactly what's going on.

One outlandish possibility, assuming your HiveManager is on VMWare, might be a VMWare NIC driver issue (host driver, not guest driver). I have (once) seen an issue where a driver bug caused an issue with certain packets (mainly max-size TCP packets) not reaching a Linux guest. All Windows guests on the same host were unaffected. Was resolved with a VMWare driver upgrade.

Other than that, is there a firewall between the APs and HiveManager? Any ACLs on the LAN switches? Assuming again HM is on VMWare, how is your vSwitch configured in terms of NIC failover/load-balancing? What vendor are your LAN switches?
Photo of Dennis

Dennis

  • 29 Posts
  • 1 Reply Like
Hi Roberto,

I will make a wireshark capture today.

We have 2 physical firewalls inbetween our VM space and our AP's.
On both of which we are allowing the following ports, udp/12222, https, ssh.
I will ask our system manager to troubleshoot the VM.

Thank you as well, for the suggestions!
Photo of BJ

BJ, Champ

  • 374 Posts
  • 45 Reply Likes
Dennis,
Have you tried testing by creating a brand new, very basic configuration and pushing it as a complete config to one, preferably a test AP? Similar to what Brian recommended, but perhaps a bit more specific. I too would be curious which versions of HM and HOS are in use. 
Photo of Dennis

Dennis

  • 29 Posts
  • 1 Reply Like
So, I've done a wireshark capture, and honestly i'm not really surprised.
Our Hivemanager has a local IP, and is also being natted on our firewall.
Now whenever i update a device, it tries to answer to the HM's local IP. even though i specified the external IP in the AH config.

I've been having issues with this before, but i was always able to maintain connection to the AH and still be able to configure it. My guess would be that this is a bug in the latest firmware?
At least I hope so.. In the long run we're going to remove the NAT portion and just give it an external IP.

If anyone knows of a fix for this issue that would be great.
(Edited)
Photo of Brian Powers

Brian Powers, Champ

  • 396 Posts
  • 92 Reply Likes
Dennis,

Is there a reason that you are pointing the gear to the public facing IP vs the private IP for gear that also exist in the internal assumed RFC1918 address space?  If you changed that, what impact would it have?  I'm failing to see a reason why pointing gear from your internal LAN to the public IP is necessary.  
Photo of Dennis

Dennis

  • 29 Posts
  • 1 Reply Like
The Hivemanager is in a different network because of the merging of two companies.
We want to keep this seperate because we are looking to become a reseller and sell wifi services to our customers.
(Edited)
Photo of BJ

BJ, Champ

  • 374 Posts
  • 45 Reply Likes
A stab in the dark, but the Hivemanager redirector might help in your case since you're using NAT, particularly if you're losing capwap. Check with your account manager or support to get started with an account.
Photo of Dennis

Dennis

  • 29 Posts
  • 1 Reply Like
Hi Brian,

yes I've set up this option.
And the thing is, even though I've set up this option, some AH's when updating still revert to speaking to the local ip of the hivemanager.

which is weird of course because i specifically defined the option.
Photo of Brian Powers

Brian Powers, Champ

  • 396 Posts
  • 92 Reply Likes
Does there also happen to be a DNS record or DHCP option in place that is specifying the private IP?  Or are you able to determine how they are finding that address?
Photo of Dennis

Dennis

  • 29 Posts
  • 1 Reply Like
Yes we have 2 DNS Records in place, one at the isp section of our network for the private IP, and one in the office network for the external IP.
Photo of Brian Powers

Brian Powers, Champ

  • 396 Posts
  • 92 Reply Likes
I'd wager to say that this is somehow related to why the APs sometimes roll over to the private IP.  Unsure if this is also possibly what is causing the updates to fail, but if an SSH session fails to one IP and not the other, it could be.
Photo of Dennis

Dennis

  • 29 Posts
  • 1 Reply Like
The thing is the AP's are never able to use the DNS server in the ISP section.
Photo of Dennis

Dennis

  • 29 Posts
  • 1 Reply Like
Also note that we're using the Hivemanager NG version.
I probably should have mentioned that earlier..
Photo of Brian Powers

Brian Powers, Champ

  • 396 Posts
  • 92 Reply Likes
This, "And our HM is running 11.9.3.3." from an earlier comment told me what I needed to know about the NG platform or the legacy platform, so no worries there.
Photo of Dennis

Dennis

  • 29 Posts
  • 1 Reply Like
We are going to install a new server with a static WAN Ip.
Is there a way to copy the current config and restore it onto a new server?
Or do we need CLI access to the VHM?
Photo of Brian Powers

Brian Powers, Champ

  • 396 Posts
  • 92 Reply Likes
Could you not just change the IP of the HM NG via the vSphere software?
Photo of Dennis

Dennis

  • 29 Posts
  • 1 Reply Like
We don't have CLI access... i asked the company we bought it from for the credentials yesterday.
So lets hope they give them :)
Photo of Sjoerd de Jong

Sjoerd de Jong, Employee

  • 97 Posts
  • 20 Reply Likes
Hi Dennis (it's me, the guy from the company ;-) )

The IP settings are changed from the VCenter perspective, not on the cli of the HiveManager itself.

The thing is, the HM should be able to be reachable through NAT for monitoring and updates. Many customers have done this already. There seems to be a misconfig or bug since the AP's keep getting the local IP-address of the HM in the update procedure instead of the WAN address that has been set in the AP. I am looking into this with Aerohive right now.

By the way (i think this is clear but just to be sure): The AP itself is always initiating the connection for update process to happen. Would you like to share the Wireshark capture (part where it advertises it's internal IP) with my by e-mail?

Cheers,
Sjoerd
Photo of Dennis

Dennis

  • 29 Posts
  • 1 Reply Like
Ok so I've done some tests,
I can update an AP as long as i don't push a Captive Web Portal or Update the firmware on the device...

Any tips?

EDIT:

We may have tracked the problem down to a faulty DNS Server.
I'll keep you posted

EDIT2:

Yes, A faulty DNS was the cause of our issues. there used to be a DNS entry for the HM when we were still moving from one office to the other. The system manager told us he had removed the entry. But when we checked just now the entry was still there.

So guys, thank you for all the effort and thought you've put into this post!
This took way more time than it should have :)
(Edited)