Iis – How to ensure no duplicates when using LogParser to load IIS logs into SQL Server every fifteen minutes

iislogginglogparser

I would like to set up a scheduled task to run every fifteen minutes on a work web server that will run LogParser on today's IIS log file and insert it into a SQL Server database table.

How would I ensure that I don't copy in any duplicate data but at the same time ensure that all the records have been copied?

Also how would I get LogParser to always look at todays log file without running expensive queries such as SELECT * FROM ex*.log and using a date and time condition?

What I have been playing with so far is:

SELECT *
FROM \\Path\To\Logs\ex*.log
WHERE date = SYSTEM_DATE()
AND time > SUB(SYSTEM_TIME(), TO_TIMESTAMP('00:30', 'hh:mm'))

However if I ran this every half an hour I'm sure to get duplicate entries. Also if it didn't work for whatever reason I would end up with missing data which I would eliminate by just overwriting the whole file for the previous day each morning.

Any tips?

Best Answer

Have you checked the ' -iCheckPoint' switch? It stores a timestamp of the last run and only accesses subsequent records.

Related Topic