We have a three-server config with each server running nginx, and rails via Mongrel. Each rails installation has its own set of log files, in it's own 'log' folder, which is just a regular folder on the server. The log file i'm interested in is <RAILS APP>/log/production.log
Any request hitting the site is sent randomly to either of the three servers, and so the logging can be on any one of the three logs: effectively each log file has the log output for a third of the requests on the site.
Sometimes i need to look through the log for a specific request, to see how it was handled, and i always have to have all three open in less, and search for my request, which is a bit of a pita. What would be much nicer is to have a single log file which combines the three.
Now, you're probably thinking "Put them all in a shared folder and just have one log file". But, we don't want to do that, for fear of efficiency hits: we do use a shared tmp folder for certain things but it's a bit slow, and we don't want to have all of the logging going over http (which is what would happen if we used a shared folder). So, that isn't an option.
Any other suggestions? I'm thinking of a cron task which runs eg once a minute, copies the logs from server 1, 2 and 3 into the same place, and combines them IN A CLEVER WAY so that the ordering of the lines in the log is the same as if all three servers wrote out to the same log in the first place.
thanks, Max
EDIT – example log file output looks like this:
Feb 06 13:15:31 ec2-rails rails[21419]: ESC[4;36;1mUser Update (3.3ms)ESC[0m ESC[0;1mUPDATE `users` SET `custom_signup_fields` = NULL, `all_license_sku_ids` = '--- \n- 1\n',
`rank_criteria` = '--- \n:has_attached_a_doc_to_a_lesson?: false\n:has_downloaded_a_premade_lesson?: false\n:has_logged_into_lesson_viewer?: false\n:has_downloaded_a_custom_l
esson?: false\n:has_logged_in?: true\n:has_created_pupil_access?: false\n:has_created_a_lesson?: false\n:has_favorited_a_lesson?: false\n:has_viewed_pupil_usage_data?: false\n
:has_viewed_a_lesson?: true\n', `preferences` = '--- \n/: \n current-scheme-step: 5931\n current_product_id: 21\n' WHERE `id` = 52331ESC[0m
Feb 06 13:15:31 ec2-rails rails[21779]: ESC[4;36;1mInstrumentFamily Load (2.6ms)ESC[0m ESC[0;1mSELECT `instrument_families`.id FROM `instrument_families` INNER JOIN users_in
strument_families ON instrument_families.id = users_instrument_families.instrument_family_id WHERE ((`users_instrument_families`.user_id = 51668)) ESC[0m
Feb 06 13:15:31 ec2-rails rails[21419]: ferret_create/update: User : 52331
Feb 06 13:15:31 ec2-rails rails[21419]: creating doc for class: User, id: 52331
One thing to note here is the number in square braces – this is the pid of the mongrel running rails – there are 16 of these on each server.
Best Answer
Whilst I understand that you are looking for a quick way to get 3 log files..."merged" in a way, you might be best just setting yourself up with a Log server and do more cool things with your logs. Like you stated, you wanted to check things in a log, no problem with a central logging server.
The logs can also monitor your server health.
I use Graylog2, it's sleek, simple to use and really effective to search in. (Graylog2 is a free, open-source alternative to Splunk).
This would then accept log pushes from your server and you would see requests easily. (They even have an easy install script!)
Provided that you still want to do your original suggestion, then I recommend you use sort on the timestamps every minute. (Provided you have timestamps!)