Magento with Turpentine, Varnish, and Nginx – Performance Optimization

cachefull-page-cachemagento-1.9performancevarnish

We have a very slow Magento store while most of the things have been cached and using Redis and others like OpCache but no best results as we need to make pages appear in less than 5 seconds atleast, so we are recommended to use Varnish full page cache which should give us better performance.

I was trying to follow this guide https://blog.nexcess.net/2012/05/21/installing-varnish-for-your-magento-store-on-centos-6/ but realized that it is too old written years ago back in 2012. So, I went to Turpentine github https://github.com/nexcess/magento-turpentine/wiki/Installation and followed the steps from there. Now, extension is successfully installed without any error and I can see it under admin panel now I need to configure it for which I looked into https://github.com/nexcess/magento-turpentine/wiki/Configuration section but it appears to me as they are just describing things instead of kind of steps needs to be done, so i am getting confused here.

If I check the blog again after git, and follow it from step 4 that also does not apply to me since I am using Nginx and also some variables in that blog are very old. Has someone gone through with this before and got a good idea for what action needs now to be taken place? should I simply change Nginx listening port to 8080 and Varnish to 80, if yes than where exactly? Following are my current files

/etc/varnish/default.vcl

# This is an example VCL file for Varnish.
#
# It does not do anything by default, delegating control to the
# builtin VCL. The builtin VCL is called when there is no explicit
# return statement.
#
# See the VCL chapters in the Users Guide at https://www.varnish-cache.org/docs/
# and http://varnish-cache.org/trac/wiki/VCLExamples for more examples.

# Marker to tell the VCL compiler that this VCL has been adapted to the
# new 4.0 format.
vcl 4.0;

# Default backend definition. Set this to point to your content server.
backend default {
    .host = "127.0.0.1";
    .port = "8080";
}

sub vcl_recv {
    # Happens before we check if we have this in cache already.
    #
    # Typically you clean up the request here, removing cookies you don't need,
    # rewriting the request, etc.
}

sub vcl_backend_response {
    # Happens after we have read the response headers from the backend.
    #
    # Here you clean the response headers, removing silly Set-Cookie headers
    # and other mistakes your backend does.
}

sub vcl_deliver {
    # Happens when we have all the pieces we need, and are about to send the
    # response to the client.
    #
    # You can do accounting or modifying the final object here.
}

Do I need to change .host IP to my site public IP and .port to 80 here? and some other settings?

/etc/varnish/varnish.params

# Varnish environment configuration description. This was derived from
# the old style sysconfig/defaults settings

# Set this to 1 to make systemd reload try to switch vcl without restart.
RELOAD_VCL=1

# Main configuration file. You probably want to change it.
VARNISH_VCL_CONF=/etc/varnish/default.vcl

# Default address and port to bind to. Blank address means all IPv4
# and IPv6 interfaces, otherwise specify a host name, an IPv4 dotted
# quad, or an IPv6 address in brackets.
# VARNISH_LISTEN_ADDRESS=192.168.1.5
VARNISH_LISTEN_PORT=6081

# Admin interface listen address and port
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=6082

# Shared secret file for admin interface
VARNISH_SECRET_FILE=/etc/varnish/secret

# Backend storage specification, see Storage Types in the varnishd(5)
# man page for details.
VARNISH_STORAGE="malloc,1G"

# Default TTL used when the backend does not specify one
VARNISH_TTL=120

# User and group for the varnishd worker processes
VARNISH_USER=varnish
VARNISH_GROUP=varnish

# Other options, see the man page varnishd(1)
DAEMON_OPTS="-p thread_pool_min=5 -p thread_pool_max=500 -p thread_pool_timeout=300 -p cli_buffer=16384 -p feature=+esi_ignore_other_elements -p vcc_allow_inline_c=on"

do I also need to uncomment and change the VARNISH_LISTEN_ADDRESS=192.168.1.5 and
VARNISH_LISTEN_PORT=6081 to public IP address of my website and port to 80? and what about VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1 and VARNISH_ADMIN_LISTEN_PORT=6082 also just to make you aware I did configure DAEMON_OPTS= options according to as described in section 2 of https://github.com/nexcess/magento-turpentine/wiki/Installation

/etc/nginx/conf.d/example.org.conf

server {
    listen 155.27.132.123:80;
    listen 155.27.132.123:443 ssl;
    server_name example.org *.example.org;
    root /home/www/vhosts/example.org/httpdocs;

    ssl_certificate      /etc/nginx/ssl/example.org.com.crt;
    ssl_certificate_key  /etc/nginx/ssl/example.org.com.key;
    ssl_protocols        TLSv1 TLSv1.1 TLSv1.2;

    location / {
        index index.html index.php;
        try_files $uri $uri/ @handler;
        expires 30d;
    }

Do I need to change the listen port to 8080 here? i tried to do that but it was keep showing the Nginx default page.

Admin > System > Cache Management
Admin > System > Cache Management

is this fine or do I need to make any change here?

Admin > System > Configuration > Varnish Options > Servers
enter image description here
Do i need to make any change here? i think I need to enter Varnish Authentication Key from /etc/varnish/secret file? Also, Config File Location & Custom VCL File Location are not there where path is defined. Will these be generated when I press Admin > System > Cache Management > Varnish Management > Apply Varnish Config. If that is the that VCL settings are being generated from here case than would I still require to edit /etc/varnish/default.vcl & /etc/varnish/varnish.params or these will be overwritten by Turpentine Magento admin section? also why the paths here in above image why the paths are defined as {{root_dir}}/var/default.vcl & {{root_dir}}/app/code/community/Nexcessnet/Turpentine/misc/custom_include.vcl
finally what to do with Admin > System > Configuration > Caching Options > Backend
enter image description here

Do I need to define my server public interface IP in Backend Host and port 80 in Backend Port

Finally, thanks for your time to read this and I hope that I will be able to overcome this varnish cache issue and this post will help many more.

Best Answer

Yikes lots of questions here. BTW, feel free to open issues on our github as well to get help. I know lots of folks there use nginx, while I personally have little experience with it.

Do I need to change .host IP to my site public IP and .port to 80 here?

The backend should point to the IP and host where Magento is running with nginx.

do I also need to uncomment and change the VARNISH_LISTEN_ADDRESS=192.168.1.5 and VARNISH_LISTEN_PORT=6081 to public IP address of my website and port to 80?

Yes, I believe so.

and what about VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1 and VARNISH_ADMIN_LISTEN_PORT=6082 also just to make you aware I did configure DAEMON_OPTS= options according to as described in section 2 of https://github.com/nexcess/magento-turpentine/wiki/Installation

This will depend on how you're setting things up in your environment. These should be accessible to Turpentine, so that it can communicate with Varnish to do things like apply VCL changes and ban/purge items from the cache.

Admin > System > Cache Management: is this fine or do I need to make any change here?

We recommend that people turn off all other caches, and leave on the two Varnish related caches. Once things are working you can tweak these settings.

Admin > System > Configuration > Varnish Options > Servers : need to make any change here? i think I need to enter Varnish Authentication Key from /etc/varnish/secret file?

Yes, unless you've set up Varnish to not use a key.

Also, Config File Location & Custom VCL File Location are not there where path is defined. Will these be generated when I press Admin > System > Cache Management > Varnish Management > Apply Varnish Config.

No, when you click 'Apply Varnish Config' Turpentine will generate a VCL file and attempt to load it into the running Varnish instance. It will also save the generated VCL file in the location you specify under Config File Location - which may not matter if your Varnish is not set up to read that file when starting up. Custom VCL File Location is a way to add custom VCL code to the generated VCL file.

finally what to do with Admin > System > Configuration > Caching Options > Backend

This is where you tell Turpentine how to communicate with the Varnish admin interface (IP, port) so it can apply VCL, ban/purge content etc.

You may find our wiki configuration page helpful when setting things up.

Related Topic