Long time lurker, first time poster on the StackExchange network.
I have been racking my brain on this for the past couple of days and have read a lot of threads and tried a lot of things, but have seen absolutely no movement. Here's the situation:
I set up a free tier EC2 box as a staging machine on Ubuntu 14.04 with LEMP, specifically Nginx and PHP7 with WordPress. We've built the site and everything works fine. It has a TTFB of 505 milliseconds.
I setup a free tier Cloud Compute box as a production machine with the same configuration, but the TTFB is 14 seconds. Know that the specs on the Google box are slightly better than those on EC2. g1-small (1 vCPU, 1.7 GB memory) vs. t2-micro (1 vCPU, 1 GB memory). SSDs on both sides.
I've tried a lot of things including much of the methods for diagnosing TTFB issues that are in this answer https://serverfault.com/a/350422. I've also tried upping the memory for PHP and WordPress. I've tried installing packages and plugins that people have suggesting for perf optimization. I'm specifically struggling with the fact that my staging instance needed none of this. Before you ask "why don't you just do it all on AWS," know that using Google Cloud Compute for production is a requirement for this project.
In the discussions I've had with people, someone said that Google Cloud doesn't have output flushing to serve content before the response is completed. Can anyone confirm?
Thanks in advance for any insight you can offer in pointing me in the right direction to solve this. Also, let me know any other details I can offer to make this easier to solve.
EDIT ONE: Answering questions below. Thank you so much for offering to help.
Are the two servers in the same geographic location? How big is the difference?
The EC2 instance is in US East or Virginia.
The GCE instance is in US-West1a which I believe is in Oregon.
I am in New York City, so not enough of a geographic difference to justify a 14 second TTFB. Also, tools that I've used have geographic positions such as Dallas (a reasonable mid-point between the two cities) and report the 14s TTFB as well.
First thing to work out is where the delay is. Please edit your question to >include a curl of the Google website, the matching Nginx access log entry, any >matching Nginx error log entry, and the PHP access / error logs. PHP access >logs need to be enabled. Also watch "top" while the curl is happening, and take >a representative screenshot. Finally, please share webpagetest.org tests of >both environments to demonstrate the problem, obfuscated if your domain names >are secret – web crawlers find all domains anyway. – Tim 3 hours ago
-
cURL
time_namelookup: 0.000n
time_connect: 0.078n
time_appconnect: 0.000n
time_pretransfer: 0.078n
time_redirect: 0.000n
time_starttransfer: 13.469n
time_total: 13.469n
I don't have enough reputation points to post an image or a link to another image, but when I run top it php-fpm7.0 and mysqld popup.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24625 www-data 20 0 374680 43352 30376 S 0.7 2.5 0:00.90 php-fpm7.0
21244 mysql 20 0 870388 71088 11012 S 0.3 4.1 0:10.57 mysqld
-
Nginx Access log entry
[07/May/2017:06:01:58 +0000] "GET / HTTP/1.1" 200 58528 "-" "curl/7.35.0"
-
Nginx Error log entry
Nothing new here with that request
- PHP Access log entry
Nothing in the PHP-FPM logs.
- PHP Error log entry
Nothing from this request, but I did do a browser request prior to it and here's what is in the slow logging (I put the window at 5 seconds):
[07-May-2017 06:01:48] [pool www] pid 24625
script_filename = /var/www/html/wordpress/index.php
[0x00007fad55e12810] mysqli_real_connect() /var/www/html/wordpress/wp-
includes/wp-db.php:1540
[0x00007fad55e126f0] db_connect() /var/www/html/wordpress/wp-includes/wp-db.php:658
[0x00007fad55e12620] __construct() /var/www/html/wordpress/wp-content/themes/xxxx/inc/artist-products.php:7
[0x00007fad55e12590] edb_db_init() /var/www/html/wordpress/wp-content/themes/xxxx/inc/db/items.php:258
[0x00007fad55e124d0] edb_get_product_link() /var/www/html/wordpress/wp-content/themes/xxxx/inc/artist-products.php:23
[0x00007fad55e123c0] edb_display_frontpage_items() /var/www/html/wordpress/wp-content/themes/xxxx/page-templates/home.php:95
[0x00007fad55e121e0] [INCLUDE_OR_EVAL]() /var/www/html/wordpress/wp-includes/template-loader.php:74
[0x00007fad55e12140] [INCLUDE_OR_EVAL]() /var/www/html/wordpress/wp-blog-header.php:19
[0x00007fad55e120a0] [INCLUDE_OR_EVAL]() /var/www/html/wordpress/index.php:17
- EC2 Timeline
https://www.screencast.com/t/vxHTCcyf
- GCE Timeline
https://www.screencast.com/t/22OXgA7T
The most obvious thing to try would be to write a barebones hello world php >script and measure ttfb of that, if you haven't. If it's anywhere near 14s then >you have a problem that has nothing to do with wordpress or the database. >"Google Cloud doesn't have output flushing" applies to Google App Engine HTTP >responses, which are returned en bloc — so this is not applicable to compute >instances.
Yep, I've done this. Hello World and other static files respond quickly. PHPInfo also responds quickly. If I recall correctly, all simple and static files were around 700 ms TTFB. Thanks for the clarification on the output flushing only being relevant to App Engine. At least I know there is a simple solution that I'm just missing.
The only thing I've been seeing that gives me anything to work with is the PHP slow logging.
[07-May-2017 00:56:39] [pool www] pid 24793
script_filename = /var/www/html/wordpress/index.php
[0x00007fad55e14810] mysqli_real_connect() /var/www/html/wordpress/wp-includes/wp-db.php:1540
[0x00007fad55e146f0] db_connect() /var/www/html/wordpress/wp-includes/wp-db.php:658
[0x00007fad55e14620] __construct() /var/www/html/wordpress/wp-content/themes/xxxx/inc/artist-products.php:7
[0x00007fad55e14590] edb_db_init() /var/www/html/wordpress/wp-content/themes/xxxx/inc/db/items.php:258
[0x00007fad55e144d0] edb_get_product_link() /var/www/html/wordpress/wp-content/themes/xxxx/inc/artist-products$
[0x00007fad55e143c0] edb_display_frontpage_items() /var/www/html/wordpress/wp-content/themes/xxxx/page-templat$
[0x00007fad55e141e0] [INCLUDE_OR_EVAL]() /var/www/html/wordpress/wp-includes/template-loader.php:74
[0x00007fad55e14140] [INCLUDE_OR_EVAL]() /var/www/html/wordpress/wp-blog-header.php:19
[0x00007fad55e140a0] [INCLUDE_OR_EVAL]() /var/www/html/wordpress/index.php:1
This made me think that the database queries are slow, but I installed the query monitoring plugin for WP and all of the queries are showing as executing fast as well.
Best Answer
The most likely cause for the delay is the underlying infrastructure difference between AWS and GCP - in the cloud world, spec-sheets aren't always a true reflection on real-world performance, and using the cheapest set of instance types is never a good idea if you want to benchmark performance.
Almost every performance angle in GCP is linked to the number of CPUs - 2Gb of network per CPU core for example. Also, on Google Cloud, a
g1-microis a shared-core machine. From the link:On AWS, a
t2.microis classed as burstable - which on the face of it seems similar to Google's shared-core model but is subtly different:In order to perform a true like-for-like comparison, I would strongly suggest using higher-performing instance types on both sides to guarantee resource availability, for example an Amazon
m3.mediumand Googlen1-standard-1- both providing 1 CPU and 3.75GB of RAM.To remove geography from your possible list of culprits, I would also locate your AWS instance in the us-west-2, which is also in Oregon - it may be further from your location, but at least you'll be testing two instances which are a similar distance away, rather than one near and one far.
I would also try re-creating the instances multiple times to guard against "noisy neighbours" and hardware quirks - if you're using configuration management (and if you're running in the cloud, you should be) this should be a doddle.