Cron – Running crontab on AWS auto scaling group

amazon-web-servicesautoscalingcron

I have the following setup:
1 server with application X, that runs a crontab once an hour. The crontab connects to the database and runs some heavy calculations and exports data to a special data file and restarts the application. The export can run anywhere from 10 minutes to 40 minutes.

I want to:

  • Move that server to AWS and use auto-scaling group.

  • The crontab to run on only one server, do the calculation export the data and somehow sync it to all other live servers.

  • All servers should automatically detect the new data and restart themselves safely (not while syncing for example).

  • New servers that starts from the auto-scaling groups show automatically fetch the data files on startup before starting the actual application.

I don't have a "simple" idea on how to do it or any AWS specific solution.

This is my idea:

  • Run one server outside auto-scaling group. Execute crontab only on that server. All data files will be uploaded to S3.

  • All autoscaling servers will have a crontab that runs every minute and check for an unique file "please_download_me_TIMESTAMP"

  • Once the files are downloaded the script will restart the service.

  • If a new server is started, on startup it will automatically fetch all files from S3.

Would you think this would work?

Best Answer

This is a common issue in AWS EC2 and has been solved. See https://gist.github.com/kixorz/5209217 for an example with implementation.

Related Topic