Ubuntu – Create an Amazon EC2 instance with automatic access to private S3

(Strange post, so hopefully this won't be as strange a reply).

On the subject of security flaws: it is considered general bad practice to store cgi-bin scripts within the document root of the web server. Even the W3C eludes to it under "Are compiled languages such as C safer ..." in their World Wide Web Security FAQ:

Consider the following scenario. For convenience's sake, you've decided to identify CGI scripts to the server using the .cgi extension. Later on, you need to make a small change to an interpreted CGI script. You open it up with the Emacs text editor and modify the script. Unfortunately the edit leaves a backup copy of the script source code lying around in the document tree. Although the remote user can't obtain the source code by fetching the script itself, he can now obtain the backup copy by blindly requesting the URL:

    http://your-site/a/path/your_script.cgi~

(This is another good reason to limit CGI scripts to cgi-bin and to make sure that cgi-bin is separate from the document root.)

This is not as significant a threat as the ability to write a file within the document root. However, an attacker could obtain the source code of the cgi, devise a directed attack against it, and use it as a stepping stone into the server.

To mitigate this, you can add the following lines to the lighttpd.conf (or some variation therein) to direct cgi-bin to a directory separate from the /var/www/lighttpd document root.

$HTTP["url"] =~ "/cgi-bin/" { cgi.assign = ( "" => "" ) }
alias.url = ( "/cgi-bin/" => "/usr/lib/cgi-bin/" )

This requires both the cgi and alias modules for lighttpd.

The basic premise of your setup seems fine, however, there are a few items that you may want to factor in.

Firstly, EC2 network (and I/O) bandwidth is dependant on instance type. If you are hoping to use t1.micro instances do not expect 'super fast internet connectivity' - even with an m1.small, you may not see the performance you are looking for. Also, keep in mind that you pay for bandwidth used on EC2 (and not just for instance time).

With regard to your first point, there should be no real difficulty in setting up Python on an EC2 instance. However, the potential difficulty arises from coordinating your instances. For example, if you have 2 instances running, how will you split the task between them? How will each instances 'know' what the other has done (presuming you aren't going to manually partition a list of URLs). Moreover, if you are launching an instance, will one of the EC2 instances be responsible for handling that or will your local machine deal with it (if it is one of the EC2 instances, how do you determine which instance will be responsible for the task (i.e. to prevent the 'launch' task being executed by every instance) and how do you redistribute the tasks to include the new instance? How do you determine which instance(s) to terminate automatically?

Undoubtedly, all of the above are possible (corosync/heartbeat, pacemaker, auto-scaling, etc.) but easy to overlook initially. Regardless, if you are looking for the 'best price' you will probably want to go with spot instances (as opposed to on-demand), however, for that to work, you do need a fairly robust architecture. (It is worth noting that the spot prices fluctuates significantly - at times exceeding the on-demand price; depending on the time-scale over which you are working, you will either want to set a low upper spot price, or determine the best approach (spot/on-demand) on a regular (hourly) basis to minimize your costs.) Although, I can't confirm it at the moment, the simplest (and cheapest) option may be AWS' auto-scaling. You need to setup Cloudwatch alarms (but Cloudwatch does provide 10 free alarms) and auto-scaling itself does not have a cost associated with it (other than the cost of new instances and the Cloudwatch costs).

Given that I really have no idea of the scope of your undertaking, I might ask why not simply use EC2 for the parsing and processing. Especially if the parsing is complex, the pages can be fetched faster than they can be processed, and you have a large number of pages (presumable, otherwise you wouldn't be going to through the effort of setting up AWS), it might be more efficient to simply process the pages on EC2, and when everything is done, download a dump of the database. Arguably, this might simplify things a bit - have one instance running MySQL (with the data stored on an EBS volume), each instance queries the MySQL instance for the next set of records (and perhaps marks those as reserved), fetches and processes, and saves the data to MySQL.

If you are not going to run MySQL on EC2, you can either store your HTML files on S3, as you have mentioned, or can go with saving them on an EBS volume. The advantage of S3 is that you don't need to pre-allocate storage (especially useful if you don't know the size of the data you are dealing with) - you pay for PUTs/GETs and storage; the downside is speed - S3 is not meant to be used as a filesystem, and (even though you can mount it as a filesystem) it would be fairly inefficient to be saving each individual file to S3 (as in you will want to accumulate a few pages and them upload them to S3). Additionally, if you have a large volume of files (tens of thousands) the processing of fetching all the filenames, etc. can be slow. EBS volumes are meant to be used as storage attached to an instance - the advantage is in speed - both transfer rates and the fact it has a 'filesystem' (so reading a list of files, etc. is quick) - EBS volumes persist beyond instance termination (except for EBS root volumes, which do not by default (but can be made to)). The downsides of EBS volumes are that you have to pre-allocate a quantity of storage (which cannot be modified on the fly) - and you pay for that amount of storage (regardless of whether all of it is in use); you also pay for I/O operations (also, the performance of EBS volumes is dependent on network speed - so larger instances get better EBS performance). The other advantage of EBS is that, being a filesystem, you can perform a task like gzipping the files very easily (and I imagine that if you are downloading a lot of html pages you will not want to be fetching individual files of S3 later on).

I am not really going to speculate on the possibilities (keeping in mind that at a very large scale, something like map-reduce/hadoop would be used to manage this kind of task), but as long as you have an approach for partitioning the task (e.g. MySQL instance) and managing the scaling of instances (e.g. auto-scaling), the idea you have should work fine.