First, don't assume that a RDMS isn't going to scale. It might not be the right solution, but saying it won't scale doesn't make sense unless you've considered how your data is going to come into the system, how it's going to be queried and what you eventually want to see from those queries.
Recording raw page hits may or may not be a large dataset. If you do this in a simple naive way, recording a row for every single hit, it may not scale, but this isn't necessarily the smartest way to record these things. You are likely going to be working from server logs, which will then distill them into an aggregated form.
Path tracking is likely to be the largest dataset here since you'll need the breadcrumbs from each individual user, but the querying part is important here. To do this in a sophisticated way, you'll likely be using some application logic, not a raw query.
Unless you have a large number of users, a single RDMS should be able to handle these queries. The general idea is to keep both the aggregate data and the raw, fine grained level data in different tables. The aggregates provide fast queries with indexes, etc. and the fine grained data can be used to be build new metrics.
Some databases and some BI solutions provide automated ways to do this. Oracle has aggregate persistence for example, but in my work I've found myself writing batch jobs to build aggregates.
Longer term, you'll want to learn about modeling your data dimensionally rather than relationally. Dimensional models and star schemas are more extensible than a relational model you replicate from a production system and provide a better way to manage the granularity of the data and cached aggregates.
If you have really large datasets, then you'll need to start thinking about using distributed processing, map/reduce, etc. But you'll save yourself a lot of time if you can manage to use a traditional database in an efficient manner. Performing complex analytics (i.e., more than simple aggregates such as SUM or AVE) requires a lot of more thinking and expertise in the map/reduce framework than in SQL.
I would use autonomous servers. Each server hosts the uploading frontend, the encoder and the download service. This way you don't have to transfer files around. To scale, simply add more servers- it sounds like you don't have any obstacles doing that.
Research if you can stream the process; start encoding the file while it is still uploading, download the result while it is still being encoded. This won't reduce the cost of any of the operations, but the end user will perceive a significant benefit.
Offer alternatives to HTTP upload if your files are large. If the upload stops for whatever reason, the user must restart the upload from scratch.
Best Answer
The main problem as described is that Producers are faster than the consumers. This reminds me a lot of http://ferd.ca/queues-don-t-fix-overload.html . Reactive streams are an initiative that I noticed recently to provide some solution to this kind of setting.
You can have a look at any Queue oriented software products like:
to see how you can deal with overachieving producers.
However the main question remains how you want to deal with it from your business point of view? Since your consumers (3rd party) are limited in the amount of messages they can handle, your approach to batch import your buffered messages seems reasonable. Aggregating or even dropping messages might also be viable depending on your scenario.
No matter how you want to react to overflow, your queue should be made aware of this fact ( i.e. introducing back-pressure) and then you can apply your own strategy which will depend on your requirements.
In my last project I ended up pulling messages from a consumer through the queue system:
I hope this provides some handles that allow you to find your own solution.