Let me start off explaining what I'm trying to do:
We have an RMM tool installed on lots of Windows servers. It is able to send Windows Event Logs to a central store, but not in an efficient or reliable way. I'd like to use the native WEF on Windows Servers to send specific events to a central store parsed of any of the redundant noise (so just, what is the event id, source, and other details particular to the eventid/source like username attempted, workstation/originating IP). These servers are not on the same domain and are geographically dispersed across the globe. I know there are numerous platforms to do this (Splunk, for example) but they are typically overpriced and have become bloated nonsense for trying to perform this straightforward task.
My original idea was to configure WEF on servers and have them send the logs to a central server with subscriptions set up to listen for them, have the logs parsed there for the important details and then use something like logstash/filebeat/nxlog to push them to ELK so we could dashboard important events (failed logons, security logs being cleared, kerberos priv escalation exploits/golden tickets creations etc). The deeper I got the more I realized this is not how WEF/WinRM are intended to be used. They want you to have an on-premise server in the same domain to store the logs on. The closest I could find to this is this writeup: https://blogs.msdn.microsoft.com/canberrapfe/2015/09/21/diy-client-monitoring-setting-up-tiered-event-forwarding/ but it is targeted at multiple sites in AD belonging to the same domain. In our case the central log storing server will not be on any domain and needs to accept event logs from numerous other domains.
Before I spend multiple hours getting it set up I figured I'd ask here– is this the sort of thing I should just use filebeat to parse and push directly from the servers in question instead of even bothering with WEF at all? That's what it is feeling like now but I just wanted to reach out to make sure I'm not overlooking something obvious.
Best Answer
It's been a long time since I attempted native log-shipping on Windows, but I do remember it was way harder than I hoped. Can't imagine doing it outside of the domain, or even outside of the federation. This is going to be a logstash answer because that's what I know.
Your instincts are correct, the way to do this outside of a single domain is to use something like Winlogbeat to extract the Windows events out of the producers and transmit them somewhere you can process them. Depending on your fluency with Windows Events manipulation, it may be easier for you to do the filtering at the Windows Events level (send just the events you want to a same-server event-log just for that, and point WinlogBeat at that), rather than do it at the beats level.
Doing this at scale means thinking about tradeoffs. For a large fleet of Windows security-log producers you're looking at a high event-rate when it comes to ingestion. For larger setups its highly recommended to use a buffering queue of some kind between beats and the Logstash tier that does all the markup. There are a few options, but Redis and Kafka are supported if those are already in your infrastructure.
You can also send directly from beats into Elasticsearch, but that leaves a lot of the transformative power of Logstash on the table. Also, ES ingests data faster (more events-per-second) when there are a few nodes doing large bulk inserts rather than many nodes doing smaller bulk inserts. Again, you may already have the ElasticSearch infrastructure/expertise on site already where this may not be a problem.