Ubuntu – Munin Aggregate Graphs from several servers

Here is documentation on the munin capabilities to change the sample periods: Keep more data with custom retention plans

It depends on which version of munin are you are using. 1.4 has some limited ability to store higher precision data, but what you really want is what will be in 2.0 when it is released. The second alpha was released around a month ago, so it'll likely be a while before it's out and packaged for your use.

Wow! How are you measuring load times? As far as I knew nginx would only report request response times ($request_time) which is something completely different.

I've not had a good look for a few months, but last time I checked there was very little available for analysing response times. PastMon looks promising. And there are commercial tools like Client Vantage (rather expensive).

I ended up writing my own - its not that hard really to create a simple awk script to report all hits which are over a threshold - but remember that you'll need to go back and check to see how the URL behaves the rest of the time. e.g.

# looking for URLs matching 'example.com/interesting' 
# with URL in $6 and $request_time in $8

BEGIN {}
$6==/example.com\/interesting/ {
  if ( $8>0.3) {
     n[$6]+=1;       # no of hits by URL
     t[$6]+=$8;      # sum of times by url
     s[$6]+=$8 * $8; # sum of sq of times by url
     if (m[$6]<$8) m[$6]=$8; # max time for url
  }
}
END {
   print "url, n, avg, stddev, max";
   for (x in n) {
     print x ", " n[x] ", " t[x]/n[x] ", " sqrt(s[x]-t[x]*t[x])/(n[x]-1) ", " m[x]; 
   }
}

If you are measuring the response times on the proxy, then you're also measuring the time taken to deliver the request across the network - i.e. your application may be behaving consistently but the spikes are introduced by changes on the internet / client. If you want to see what your application is really doing then you need to look at your webserver logs.