I have this schema to receive http requests and do some processing on files under a Windows Server EC2 instance.
API Gateway -> Lambda -> EC2 (Node.js + Express)
For now, there are a few requests per day, so the instance is stopped until a request comes. When a requests comes, the Lambda function starts the instance but the first request is lost. If the instance is running, the request is forwarded.
I want a solution in which I do not lose the first request. I have several approachs but none of them seems to be the right one:
1) Save that request into ElastiCache (Redis/memcached). In the startup of the Express server, check if there is any requests saved.
2) Use Amazon SQS to store all requests. Then switch the Express API for a worker to check the SQS for new messages. I don't like this solution because I will be checking all the time for a SQS that most of the time will be empty and I suppose this can be expensive.
3) Re-invoke the Lambda function with a delay. Just do another call to the Lambda function with the same request in 1 minute (for example), when the instance is ready. This is the best solution right now in my opinion, simple and effective, but I don't know how to implement it. I know you can schedule Lambda executions but there are like "execute this function all days at 3 p.m.", and I only want to execute the function once with a delay.
I'm stuck with this, hope someone can clarify my ideas.
Best Answer
Use the SQS solution.
When your Lambda function executes, store the request into SQS.
Use Auto Scaling to launch and terminate your EC2 instance as requests are added to the queue.
On your EC2 instance, poll the SQS queue for work. You can minimize SQS costs by minimizing requests:
SQS requests are not that expensive. Polling once a minute would yield 43,200 requests a month. This is well below the 1 million requests you get for free each month. Even if not covered by the free tier, the first million requests are only $0.50.