Nginx & php-fpm very slow on first access

I have a nginx 1.4.5 webserver with php-fpm 5.5.7 running on my server, but if my server stays idle for a while, accessing it is very slow. It takes about 10-20 seconds to get the response (first byte), after that its fast again.
I dont know why it takes ages to respond on first access after a while. I tried both unix sockets and tcp, no effect. Heres my config:

NGINX:

user  nginx;
worker_processes  1;
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1000;
    multi_accept on;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    keepalive_timeout  65s;
    keepalive_requests 1000;
    gzip off;
    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

my php standard config:

include=etc/fpm.d/*.conf
[global]
error_log = log/php-fpm.log
log_level = notice
emergency_restart_threshold = 10
emergency_restart_interval = 1m
process_control_timeout = 10s

now the pool which takes ages to respond (first byte):

[myweb]
;listen = /usr/local/etc/sockets/myweb.socket
listen = 127.0.0.1:9001
listen.owner = max
listen.group = max
listen.mode = 0666
user = max
group = max
listen.allowed_clients = 127.0.0.1
pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.status_path = /status
pm.min_spare_servers = 2
pm.max_spare_servers = 4
pm.max_requests = 200
slowlog = /home/max/logs/max.slow.log
pm.status_path = /status
request_terminate_timeout = 120s
rlimit_files = 131072
rlimit_core = unlimited
catch_workers_output = yes
php_admin_value[default_charset] = UTF-8
php_value[error_reporting] = 2147483647
php_admin_flag[display_errors] = On
php_admin_flag[display_startup_errors] = On
php_admin_value[error_log] = "/var/tmp/php.max.error.log"
php_admin_value[log_level] = DEBUG
php_admin_value[upload_max_filesize] = 10M
php_admin_value[post_max_size] = 10M
php_admin_value[max_execution_time] = 60

Server tech details:

AMD Athlon(tm) 64 X2 Dual Core Processor 3400+
4GB RAM
500 GB RAID1 HDD
website runs on MySQL 5.5.35
nginx 1.4.5
php-fpm 5.5.7

Timeline pic
The browser waited 22 seconds for first byte. Receiving the content took only 86ms.
So this must be the server (nginx/php-fpm):

The website runs in a users home folder, the user is max. Everything works fine. No slow log files generated, no php errors, no nginx errors.

Why in the world its slow on first byte?

UPDATE 2:

Setting nginx's error log to DEBUG, gave me only some infos about the match.
But right after i called the url, i got 3 log entries printed while watching it with "tail -f", so nginx is doing its right as usual, right after its redirecting to the php script, it takes about 15-20 seconds before i get my response and more logs printed. so either its php-fpm, or mysql.
I would like to know how can i "reset" my server so i can easily reproduce that long first-byte response. I guess its caching the result or something and using it after the first call, how to kill that cache so i can test. Because i can test this only once in 3 or 4 hours, then i have to wait before i can try again getting the slow first-byte response.