How to Update Minimum & Maximum Running Tasks in ECS

There are a number of considerations to take into account here. Firstly, to resolve the most limiting problem - the lack of SSH access.

Since your previous launch config did not specify a keypair, you will have no valid credentials with which to access the instance. Unfortunately, the initial keypair cannot be added after the instance is launched.

To remedy this, you must create a new launch config, passing the --key and --group parameters, in addition to all the parameters you passed previously. --key takes the name of the keypair you want to use, while --group takes the security group name (if not in VPC) or ID.

In cases where you can't access your instance, the console log can help you to verify the instance has in fact successfully booted. A common issue is boot failure due to missing volumes (especially trying to mount ephemeral volumes that exist only on the larger instance types, when booting a smaller instance type).

An important point of mention is that an AMI is not updated if you change a running instance. You must explicitly create the new image. As such, if you try to launch a new instance using the same AMI you are currently using on a customized instance, there is a good chance that you will simply be launching one of the default AMIs, and not one with your customizations on it.

Use ec2-describe-images to determine the block device mapping of your image - and the snapshot the volume is based on - this will verify that you will be mounting an EBS volume that has your customizations built into it.

If you do not have an up to date AMI to use for autoscaling:

• Create a snapshot of your EBS volume
• Create an AMI with ec2-register -n IMAGE_NAME -s SNAPSHOT_ID
  • If you have additional EBS volumes to attach, specify those by adding the --block-device-mapping (-b) parameter (e.g. -b /dev/xvdf=SNAP_ID)
• Verify that you have the correct block device mappings with ec2-describe-images

Once you have an up to date AMI, you need to create a new launch config that will use this AMI. If desired, you can pass additional block device mappings to the command. Use as-create-launch-config, passing it your new AMI and all the parameters you used previously.

Finally, you must update your autoscaling group. This group is associated with a particular launch config - the new launch config will not be automatically detected and has no effect on the autoscaling group until you explicitly associate it. Use as-update-auto-scaling-group GROUP_NAME --launch-configuration CONFIG_NAME to make this change.

Once the changes have been made, you can simulate an autoscaling event using the as-execute-policy command.

Remember to give your instances a few minutes to boot up - if your ELB is showing instances as unhealthy, you may want to increase the --grace-period of your autoscaling group.

I have created a Lambda function to update the instance agent in all my ECS clusters:

var AWS = require('aws-sdk');
AWS.config.update({ region: 'sa-east-1' });

exports.handler = async(event, context) => {
    var ecs = new AWS.ECS();

    var responseArray = [];

    const clusters = await ecs.listClusters({}).promise();

    for (var i = 0; i < clusters.clusterArns.length; i++) {
        const clusterArn = clusters.clusterArns[i];

        const clusterInstances = await ecs.listContainerInstances({
            cluster: clusterArn
        }).promise();

        for (var j = 0; j < clusterInstances.containerInstanceArns.length; j++) {
            const containerInstanceArn = clusterInstances.containerInstanceArns[j];

            try {
                const response = await ecs.updateContainerAgent({
                    containerInstance: containerInstanceArn,
                    cluster: clusterArn
                }).promise();

                responseArray.push({
                    cluster: clusterArn,
                    containerInstance: containerInstanceArn,
                    response: response
                });
            }
            catch (e) {
                responseArray.push({
                    cluster: clusterArn,
                    containerInstance: containerInstanceArn,
                    response: e
                });
            }
        }
    }

    return responseArray;
};

Then I created a CloudWatch event rule to execute lambda function daily. Works good for me.