Why aren't we seeing any syslog messages from our AP's?

  • 3
  • Question
  • Updated 2 months ago
We're running a small trial of AP550's, AP250's and a single AP150. We're running on current NG 11.28 and a mix of Hive OS 8.1r1, 8.1r2 and 8.1r2a.

I've configured syslog servers, set them to info and then debug, but the only messages coming through seem to be the same as the output of a "sh log messages" from the cli of an AP which is basically showing nothing of much use.

apname#sh log messages
<181>1 2017-12-06T10:05:30.009996+00:00 aerohive -ah_cli_ui: [security-5--ah_cli_ui-#104004]Admin "<admin>" successfully logged in
<133>1 2017-12-06T07:22:26.298185+00:00 aerohive ah_top: [system-5-ah_top-#106001]System is initialized

This can't possibly be all that we should be logging? I'd expect to be seeing reams of information constantly hitting the syslog servers. What am I missing?
Photo of Paul Smith

Paul Smith

  • 9 Posts
  • 0 Reply Likes
  • frustrated

Posted 6 months ago

  • 3
Photo of Nick Lowe

Nick Lowe, Official Rep

  • 2491 Posts
  • 451 Reply Likes
Hi Paul,

There is a redesign of Syslog in progress that started with HiveOS 8.1r1. You should see far more Syslog data with HiveOS 8.2r1 when this becomes available.

Thanks,

Nick
(Edited)
Photo of Paul Smith

Paul Smith

  • 9 Posts
  • 0 Reply Likes
Thanks for getting back to me Nick. Do we have an eta on that? It's a pretty major feature to be missing whilst we're evaluating different vendors.
Photo of Nick Lowe

Nick Lowe, Official Rep

  • 2491 Posts
  • 451 Reply Likes
Hi Paul,

Can you send me an email at nlowe {at} aerohive.com so that I can reply to you with additional information?

Thanks,

Nick
(Edited)
Photo of Edward Marshall

Edward Marshall

  • 7 Posts
  • 0 Reply Likes

Hello,

Having the same issue now with 8.2.r1. Do you know if this was fixed? If I downgrade the APs to version 6.5r6 or 6.5.r8b, the INFO syslog messages are much detailed and include the IP Address and Username of users during the authentication process. We use this for user identification for our firewall so is crucial for us.

I have wireshark captures from both firmware versions if someone wants to have a look.

Kind regards,

Edward

Photo of Nick Lowe

Nick Lowe, Official Rep

  • 2491 Posts
  • 451 Reply Likes
Hi Edward,

A redesign of Syslog created a split between developer, QA and user level logs as well as slightly changing the message format. This redesign was started via HiveOS 8.1r1. A second phase was delivered in HiveOS 8.2r1.

Prior to HiveOS 8.1r1, there was no such distinction and all of these were mixed together. Some customers have become accustomed to seeing developer and QA targeted internal data in the Syslog output.

In HiveOS 8.1r1 and going forward, it is required to use an undocumented command tree to get access to developer and QA level logs when debugging issues in conjunction with support. This undocumented command does not persist across a reboot, is expected and is not considered a defect.

Therefore, if you are expecting to see Dev and QA targeted logs in the normal Syslog output in HiveOS 8.2r1, this will not be the case. 

With HiveOS 8.2r1 you should, however, now see user level logs being generated which was not the case in 8.1r1. Is this what you observe or not?

The following is what I see from HiveOS 8.2r1 in the user level logs that I think gives you the information you are seeking:

<14>1 2018-01-31T13:36:56.369762+00:00 aerohive ah_auth: Station cc08:1234:5678 ip 192.168.1.19 username nlowe hostname iPhone OS Apple iPod, iPhone or iPad, flag = DHCP 

If you have a script interpreting the messages via a regular expression, etc., that would likely need updating to handle the new format if so.

Have you observed that this data is missing in the packet capture that you have taken? Are you expecting a different message? Not getting this one?

The product management team are reviewing this to decide what more, if anything, should be exposed in the user level logs going forward.

If you feel there if something missing from the user level logs in 8.2r1 that you would like to see, please let us know so that we can pass this feedback on to that team.

Cheers,

Nick
Photo of Edward Marshall

Edward Marshall

  • 7 Posts
  • 0 Reply Likes

Hi Nick,

Thanks for the reply and detailed explanation. Actually the user level logging that you're seeing from 8.2.r1 is exactly what we need and are missing.

The only messages we are receiving are like this:

ah_trapd: [aaa-6-ah_trapd-#103003]Station 6c70:9f1c:1111 was authenticated on 885b:dd60:2222 through SSID NETWORK vid 111.

We are using the Palo Alto User-ID agent which collects the messages and uses a regular expression to capture the username and IP mapping. From your output, I can see that we would need to indeed update this with the new syntax if we can get the messages onto the syslog server.

I'm assuming that our output is from the 'Non-HiveOS Syslog Facility' and somehow we need to get the 'User' messages sent to this as well? There are much less options than under 'HiveOS Syslog Facility' in the 'Syslog Server' settings on HM-NG.

Kind regards,

Edward

Photo of Nick Lowe

Nick Lowe, Official Rep

  • 2491 Posts
  • 451 Reply Likes
Hi Edward.

For Palo-Alto integration with NPS, can I suggest you look at this method of integration which does not use Syslog:

https://github.com/cesanetwan/scripts/wiki/CEFilter-UID-RADIUS-script
https://github.com/cesanetwan/uid-radius-script-ps/wiki

This will not work, however, where HiveOS is acting as the RADIUS server or a different third-party RADIUS server is used.

(It is often better to use NPS than the built-in RADIUS server for this type of deployment scenario.)

I will do some background investigation on the Syslog side of things.

Cheers,

Nick
(Edited)
Photo of Edward Marshall

Edward Marshall

  • 7 Posts
  • 0 Reply Likes

We're actually using a couple of Aerohive APs as RADIUS servers and not MS NPS. This has always worked well in the past but we're dependant on the other APs syslog messages to our syslog server to capture the Username/IP mapping. Thanks for looking into it, I've opened a support case as well so will post an update once I hear back.

Thanks again,

Edward

Photo of Nick Lowe

Nick Lowe, Official Rep

  • 2491 Posts
  • 451 Reply Likes
Hi Edward,

I am going to progress the Syslog issue with the product management team and potentially via a CFD. Stay tuned...

Regards,

Nick
(Edited)
Photo of Eric Geiger

Eric Geiger

  • 1 Post
  • 0 Reply Likes
Hi Nick,

can you give us an update regarding this issue ? is a patch already available ?

Thank you

Best Regards

Eric
Photo of Edward Marshall

Edward Marshall

  • 7 Posts
  • 0 Reply Likes

Hi Eric,

We have an SR open with Aerohive and we are waiting for a fix for this. This was the latest update from a couple weeks ago:

" Based on our discussions with the product management team, it is currently looking like a solution for this is most likely to come mid to late Q2 2018 with HiveOS releases that are due after 8.3r2, and the companion HMNG release that is due around that time. "

Not sure if Nick has any other info?

Kind regards,

Edward