Apache missing requests in access log

apache-2.4httplogging

we're on Apache/2.4.7 on Ubuntu 14.04.

We're trying to solve a mystery of http requests getting lost somewhere.

Our payments provider sends us notifications/request about payments made by our customers. Time to time it happens that they get no confirmation/response from our side.

This behavior is reproducible by sending a test/check notification from payments providers side. You can send e.g. 14 test notifications that all get confirmation in form of response from us with code 200. Then the 15-th notification would be shown as timed-out and there is no response from our side.

After verifying number of requests sent from our payments provider (they see 15) we could only see 14 in our access log. The number doesn't have to be 15 though, it happens arbitrarily.

Can you think about any circumstances, when a request wouldn't make it to access log (overloaded server, networking issues, crash in processing)? Can a http request simply get lost somewhere on it's way? Can you thing of any reason why something like this would be happening? Is there a way to debug this?

Thank you very much for your help!

Tomas

Best Answer

Thank you for your answers! I just got an email from our payments provider saying that after investigation they've just fixed an issue with one of the hops of one of the ISP's they are using. Problems on our side got fixed by this.. so the requests really were getting lost it seems.

Related Solutions

ColdFusion Server crash after thousands of HTTP requests

Have you monitored CPU, disk and bandwidth usage on your server? The first thing I'd do is make sure none of these resources are being overloaded. Perfmon is an obvious place to start and check out your CF logs.

Were you making 50K post requests (i.e. separate requests) one after the other or is this a single request for the 50K records in an XML file? If the former then why would you make 50K requests? Could it be done by requesting all 50K records at once as that would be just 1 request and 1 file transfer? If the 50K requests are sequential (i.e. the next request isn't made until the current one is complete) then that seems like a gentle way of making a large number of requests.

How To Tell Apache To Not Log Certain Requests In Its Access Log

The filters only need to be added to one config. Try removing what you added to apache2.conf and keep everything in /sites-available/default.

For example, if you want to filter out search bots from your logs, the end of your config file will look like:

    ErrorLog ${APACHE_LOG_DIR}/error.log

    # Possible values include: debug, info, notice, warn, error, crit,
    # alert, emerg.
    LogLevel warn

    # Filter all search bots from the access log.
    SetEnvIfNoCase User-Agent googlebot dontlog
    SetEnvIfNoCase User-Agent bingbot dontlog
    SetEnvIfNoCase User-Agent baiduspider dontlog

    # Custom Log location + options
    CustomLog ${APACHE_LOG_DIR}/access.log combined env=!dontlog
</VirtualHost>

Also, when you do apachectl graceful, it can take a while. If you're just testing the changes, delete the log files, do apachectl graceful, and when you see the log files get re-created, you'll know they are populated using the new filter options you specified.

Related Topic

Trouble filtering googlebot from apache access log