I have the following setup:
1 server with application X, that runs a crontab once an hour. The crontab connects to the database and runs some heavy calculations and exports data to a special data file and restarts the application. The export can run anywhere from 10 minutes to 40 minutes.
I want to:
-
Move that server to AWS and use auto-scaling group.
-
The crontab to run on only one server, do the calculation export the data and somehow sync it to all other live servers.
-
All servers should automatically detect the new data and restart themselves safely (not while syncing for example).
-
New servers that starts from the auto-scaling groups show automatically fetch the data files on startup before starting the actual application.
I don't have a "simple" idea on how to do it or any AWS specific solution.
This is my idea:
-
Run one server outside auto-scaling group. Execute crontab only on that server. All data files will be uploaded to S3.
-
All autoscaling servers will have a crontab that runs every minute and check for an unique file "please_download_me_TIMESTAMP"
-
Once the files are downloaded the script will restart the service.
-
If a new server is started, on startup it will automatically fetch all files from S3.
Would you think this would work?
Best Answer
This is a common issue in AWS EC2 and has been solved. See https://gist.github.com/kixorz/5209217 for an example with implementation.