AWS – EFS Replication Guide

CPU utilization (%) is not the key, key is CPU load average (queue) and networking metrics, apache metrics, buffers, etc. Load balancers are very simple devices, problems, where LB's are involved in architecture usually are not related to the ELB's, but to the nature of how rest of the things work.

To see where is the problem, you most go through following steps:

• Check if apache is responding to the local requests, if not - problem is NOT the ELB
• Check states of apache workers (i.e. mod_status), tune MPM settings accordingly
• Check CPU load average, if load average grows above CPU count and iowait grows - you have trouble with IO
• Check if connection persistency is enabled and if it is really really required, if you really use sessions on webservers which require access to the same web instance
• Check keepalive settings for apache, disable it or set very low timeout value
• Check if you have iptables enabled on the instance and if nf_conntrack_max and nf_conntrack_count kernel parameters are configured with higher values. If you don't need it - disable and do not load modules at all
• Stress test single instances wit http requests (hint: ab, jmeter)

• Check and tune kernel parameters accordingly:

  net.core.wmem_max
  net.core.rmem_max
  net.core.netdev_max_backlog
  
  net.core.somaxconn
  net.ipv4.tcp_rmem
  net.ipv4.tcp_wmem
  net.ipv4.tcp_no_metrics_save
  net.ipv4.tcp_timestamps
  net.ipv4.tcp_fin_timeout
  net.ipv4.tcp_max_tw_buckets
  net.ipv4.tcp_tw_recycle
  net.ipv4.tcp_synack_retries
  net.ipv4.tcp_keepalive_time
  
  net.netfilter.nf_conntrack_acct
  net.netfilter.nf_conntrack_generic_timeout
  net.netfilter.nf_conntrack_tcp_timeout_syn_sent
  net.netfilter.nf_conntrack_tcp_timeout_syn_recv
  net.netfilter.nf_conntrack_tcp_timeout_established
  net.netfilter.nf_conntrack_tcp_timeout_fin_wait
  net.netfilter.nf_conntrack_tcp_timeout_close_wait
  net.netfilter.nf_conntrack_tcp_timeout_last_ack
  net.netfilter.nf_conntrack_tcp_timeout_time_wait
  net.netfilter.nf_conntrack_tcp_timeout_close
  net.netfilter.nf_conntrack_tcp_timeout_max_retrans
  net.netfilter.nf_conntrack_tcp_timeout_unacknowledged
  net.netfilter.nf_conntrack_icmp_timeout
  net.netfilter.nf_conntrack_events_retry_timeout
  net.ipv4.netfilter.ip_conntrack_generic_timeout
  net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent
  net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent2
  net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv
  net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait
  net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait
  net.ipv4.netfilter.ip_conntrack_tcp_timeout_last_ack
  net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait
  net.ipv4.netfilter.ip_conntrack_tcp_timeout_close
  net.ipv4.netfilter.ip_conntrack_tcp_timeout_max_retrans
  net.ipv4.netfilter.ip_conntrack_icmp_timeout
  net.netfilter.nf_conntrack_tcp_loose
  net.netfilter.nf_conntrack_max net.nf_conntrack_max
  net.netfilter.nf_conntrack_count
  

Apache is not responding after that? It's not the fault of ELB at all.

Important updates:

In October, 2018, AWS expanded the capabilities of the network technology underpinning EFS so that it now natively works across managed VPN connections and cross-region VPC peering, without resorting to the proxy workaround detailed below.

https://aws.amazon.com/about-aws/whats-new/2018/10/amazon-efs-now-supports-aws-vpn-and-inter-region-vpc-peering/

EFS added support for connectivity via AWS Direct Connect circuits in late 2016.

https://aws.amazon.com/blogs/aws/amazon-efs-update-on-premises-access-via-direct-connect-vpc/

Comments have raised some interesting issues, since in my initial reading of the question, I may have assumed more familiarity with EFS than you may have.

So, first, a bit of background:

The "Elastic" in Elastic File System refers primarily to the automatic scaling of storage space and throughput -- not external access flexibility.

EFS does not seem to have any meaningful limits on the amount of data you can store. The documented maximum size of any single file on an EFS volume is 52,673,613,135,872 bytes (52 TiB). Most of the other limits are similarly generous.

EFS is particularly "elastic" in the way it is billed. Unlike filesystems on EBS volumes, space is not preallocated on EFS, and you only pay for what you store on an hourly average basis. Your charges grow and shrink (they're "elastic") based on how much you've stored. When you delete files, you stop paying for the space they occupied within an hour. If you store 1 GB for 750 hours (≅1 month) and then delete it, or if you store 375 GB for 2 hours and then delete it, your monthly bill would be the same... $0.30. This is of course quite different than EBS, which will happily bill you $37.50 for storing 375 GB of 0x00 for the remaining hours in the month.

S3's storage pricing model much the same as EFS, as billing for storage stops as soon as you delete an object, and the cost is ~1/10 the cost of EFS, but as I and others have mentioned many times, S3 is not a filesystem. Utilities like s3fs-fuse attempt to provide an "impedance bridge" but there are inherent difficulties in trying to treat something that isn't truly a filesystem as though it were (eventual consistency for overwrites being not the least of them). So, if a real "filesystem" is what you need, and it's for an application where access needs to be shared, or the storage needs space required is difficult to determine or you want it to scale on demand, EFS may be useful.

And, it looks cool when you have 8.0 EiB of free space.

$ df -h | egrep '^Filesystem|efs'
Filesystem                                            Size  Used Avail Use% Mounted on
us-west-2a.fs-5ca1ab1e.efs.us-west-2.amazonaws.com:/  8.0E  121G  8.0E   1% /srv/efs/fs-5ca1ab1e
us-west-2a.fs-acce55ed.efs.us-west-2.amazonaws.com:/  8.0E  7.2G  8.0E   1% /srv/efs/fs-acce55ed

But it is, of course, important to use the storage service most appropriate to your applications. Each of the options has its valid use cases. EFS is probably the most specialized of the storage solutions offered by AWS, having a narrower set of use cases than EBS or S3.

But can you use it from outside the VPC?

The official answer is No:

Mounting a file system over VPC private connectivity mechanisms such as a VPN connection, VPC peering, and AWS Direct Connect is not supported.

— http://docs.aws.amazon.com/efs/latest/ug/limits.html

EFS is currently limited to only EC2 Linux access only. That too within the VPC. More features would be added soon. You can keep an eye on AWS announcements for new features launched.

— https://forums.aws.amazon.com/thread.jspa?messageID=732749

However, the practical answer is Yes, even though this isn't a an officially supported configuration. To make it work, some special steps are required.

Each EFS filesystem is assigned endpoint IP addresses in your VPC using elastic network interfaces (ENI), typically one per availability zone, and you want to be sure you mount the one in the availability zone matching the instance, not only for performance reasons, but also because bandwidth charges apply when transporting data across availability zone boundaries.

The interesting thing about these ENIs is that they do not appear to use the route tables for the subnets to which they are attached. They seem to be able to respond only to instances inside the VPC, regardless of security group settings (each EFS filesystem has its own security group to control access).

Since no external routes are accessible, I can't access the EFS endpoints directly over my hardware VPN... so I turned to my old pal HAProxy, which indeed (as @Tim predicted) is necessary to make this work. It's a straightforward configuration, since EFS uses only TCP port 2049.

I'm using HAProxy on a t2.nano (HAProxy is very efficient), with a configuration that looks something like this:

listen fs-8d06f00d-us-east-1
    bind :2049
    mode tcp
    option tcplog
    timeout tunnel 300000 
    server fs-8d06f00d-us-east-1b us-east-1b.fs-8d06f00d.efs.us-east-1.amazonaws.com:2049 check inter 60000 fastinter 15000 downinter 5000
    server fs-8d06f00d-us-east-1c us-east-1c.fs-8d06f00d.efs.us-east-1.amazonaws.com:2049 check inter 60000 fastinter 15000 downinter 5000 backup
    server fs-8d06f00d-us-east-1d us-east-1d.fs-8d06f00d.efs.us-east-1.amazonaws.com:2049 check inter 60000 fastinter 15000 downinter 5000 backup

This server is in us-east-1b so it uses the us-east-1b endpoint as primary, the other two as backups if the endpoint in 1b ever fails a health check.

If you have a VPN into your VPC, you then mount the volume using the IP address of this proxy instance as the target (instead of using the EFS endpoint directly), and voilà you have mounted the EFS filesystem from outside the VPC.

I've mounted it successfully on external Ubuntu machines as well as Solaris¹ servers (where EFS has proven very handy for hastening their decommissioning by making it easier to migrate services away from them).

For certain situations, like moving data into AWS or running legacy and cloud systems in parallel on specific data during a migration, EFS seems like a winner.

Of course, the legacy systems, having higher round-trip times, will not perform as well as EC2 instances, but that's to be expected -- there aren't exceptions to the laws of physics. In spite of that, EFS and the HAProxy gateway seem to be a stable solution for making it work externally.

If you don't have a VPN, then a pair of HAProxy machines, one in AWS and one in your data center, can also tunnel EFS over TLS, establishing an individual TCP connection with the payload wrapped in TLS for transport of each individual EFS connection across the Internet. Not technically a VPN, but encrypted tunneling of connections. This also seems to perform quite well.

¹Solaris 10 is (not surprisingly) somewhat broken by default -- initially, root didn't appear to have have special privileges -- files on the EFS volume created by root are owned by root but can't be chowned to another user from the Solaris machine (Operation not permitted), even though everything works as expected from Ubuntu clients. The solution, in this case, is to defeat the NFS ID mapping daemon on the Solaris machine using svcadm disable svc:/network/nfs/mapid:default. Stopping this service makes everything work as expected. Additionally, the invocation of /usr/sbin/quota on each login needs to be disabled in /etc/profile. There may be better or more correct solutions, but it's Solaris, so I'm not curious enough to investigate.