I've got an Apache server hosting several json APIs.
This server is running Ubuntu 16.04.4 LTS with Apache and Mysql.
Response times for all API are very good except for one! It takes about 5s to get a json response from the particular API whereas it takes milliseconds for the other APIs.
I checked multiple things without success :
1) the SQL queries ran almost instantly (with and without cache – I tried both)
2) our developper added a custom header to find out how long the API takes to generate the json response : 2-3 ms
3) calling the API from a remote computer or locally on the server with wget gives the same result : it takes 5s to get the reply.
According to the browser developper mode, the 5s are spent "waiting". Google Chrome says "Waiting for the first byte". Then data transfer seems fast according to the dev mode (a few ms)
4) apache doesn’t seem to be overwhelmed by https requests, see screenshot displaying server-status page
Server status
5) access to the https API is done through multiple firewalls but no reverse proxy is present
As part of the debugging process, I tried to improve the server performances :
-
I moved the vm to a our least busy VMware host
-
I moved this vm to an SSD based datastore
Do you have any idea why Apache is so slow to reply ? Do you know any way to troubleshoot this issue ?
Thank you for your help
Thomas
Best Answer
Days ago i found the following thread : Apache has a long lag before responding This problem looked like mine but it sounded related to IPv6. As IPv6 is disabled on my servers, I thought the solution wasn’t suitable to solve my issue.
In a last desperate attempt to solve this issue, I tried the described solution anyway : I just added “options single-request-reopen” to /etc/resolv.conf and then the 5s delay disappeared.
At this moment, I didn’t really understand why this setting was solving my performance issue as IPv6 is disabled on my servers.
Moreover hostname lookups is disabled in Apache.
After a closer investigation ….
1) This new Json API uses Google Firebase for authentication, which means that DNS resolution is required to access Google services. I forgot this in my former investigation… I thought that DNS couldn't be a problem here. I was wrong.
2) IPv6 queries on a IPv4-only server
Basically a Linux server relies on getaddrinfo for dns resolution. According the man page (gai.conf which is getaddrinfo configuration file) :
According to the RFC 3484 :
Back to “single-request-reopen” :
I think that my server was waiting on the second reply … which took 5 seconds. “single-request-reopen” allowed me a bypass this issue.
3) Prioritize IPv4 dns queries :
I also changed getaddinfo behavior as well to ensure the IPv4 communications are preferred, see /etc/gai.conf
Thomas