I've been playing with Log Parser the past few days, we're trying to retain specific event IDs from the security logs of our Domain Controllers. As these are high traffic volume logs, the data in them tends to not persist more than between 15 and 30 minutes. Therefore, I need to run this query every 10 minutes or so as a scheduled task to make sure I don't lose any data.
I've (finally!) managed to get a query going that pulls the relevant data from the logs with the relevant event IDs, and write it to a SQL database. The problem I'm having is that the select statement fails with an "unknown field" error if I try to pull the EventRecordID property (which I can see in the XML for any event).
It does generate a RecordNumber; however, this number appears to relate to the query, not the event record itself. This means that I have a unique record number for each event every time I run the query, so if an event is still in the logs over multiple consecutive runs, it gets written to the database multiple times.
How can I get this to record only one instance of the vent in the database?
Best Answer
Ok so I don't know I there's actually an answer to this question. i.e. documentation/explanation of the RecordNumber field or why Log Parser fails to read the EventRecordID field.
However, one of the answers posted to this ServerFault question will allow me to accomplish what I need anyway. (I've upvoted that answer but the OP hasn't marked it as a solution).
The solution is to use a checkpoint file, which records the last Log Parser query run, and only reads new entries from the specified log. This not only ensures unique records will be read from the log (and more importantly written to the database), but also improves the efficiency of the query too.
There is some information about checkpointing buried deep in this old TechNet article, and a neat concise description of how to use it in this blog post too.