Docker – How to restart all Tasks of a Service

You don't need a load-balancer. You need to adjust how your application deals with items in the queue.

1. Benchmark your application on various instance sizes to determine the optimal/maximum number of tasks it can handle simultaneously. For the purpose of illustration, let's say that number is 20.

2. Change your application to pull a maximum of 20 items from the queue, never more. When one of your 20 sub-tasks is complete, then it can go back to the queue for more.

At this point, your single worker server should not "overload". It will simply take a while to work through a large queue of items.

You need to have such a limit to prevent your "severe load during launching of the containers".

3. Use CloudWatch alerts to add more worker instances as your queue gets too big and to terminate instances as the queue gets smaller. Each instance will process 20 at a time.

Once you have this, you can make some optimizations:

1. Do you need to spawn new containers every time, or can you re-use them? This would save some CPU cycles when processing a sub-task.

2. You may be able to use EC2 instances in a stopped state rather than spawning fresh instances. But you'd have to do custom handling of your CloudWatch alert for this, but it could be done. You'd want to have a listener on SNS waiting for the alerts at which point it can start and stop EC2 instances as needed.

3. Benchmark your application using various instance sizes. It's possible that using two c4.xlarge is better than using one c4.2xlarge. Experiment and try different combinations.

After a discussion with the support team, it turns out that ECS cannot support our current use case.

There is a workaround that solves one of the issues we are facing. That workaround is to create a separate, essential, health-check container and in the same ECS task as the actual application container. The purpose of the health-check container is to monitor the application container to determine when the application has been started completely. If it detects that the application has failed to start, it exits, causing the ECS service to cycle the task. The ELB is then configured to perform its health checks against the health-check container, which will always report that it is up via the relevant port. This workaround will prevent the ECS service from cycling the ECS task due to failed health checks.

However, the ELB will begin routing traffic to the application container immediately. It will do so, even if the application container is not yet ready to receive traffic (for example, because it is still waiting for a cache to load). Currently, there is no way to delay the ELB from sending traffic to the application container, as the ECS service provides no support a grace period. We have managed to workaround this issue by providing messages to our application containers via SQS and only having them pull from the queue when their caches are fully loaded. However, we have future use cases (such as serving web requests) where this is not a feasible option. To this end, I intend to raise a feature request for the grace period.

As an aside, both Kubernetes (http://kubernetes.io/v1.0/docs/user-guide/walkthrough/k8s201.html#application-health-checking) and Marathon (https://mesosphere.github.io/marathon/docs/health-checks.html) already support this option for health checking, if someone reading this is happy not to use a managed service.