A mobile back-end I am building should receive a larger amount of data (some sensor recordings) from an Android phone. The phone does not have much use of it so it was most efficient to store the data into plain files (SQLite can get quite slow with larger amounts). Now when we upload the data to a REST service backed with a distributed database, we have another issue. Storing to it takes some time, since it is really a lot. I thought of uploading the file to the web service as fast as possible, leave the phone alone, cache the file somewhere on the server and then some long-running workers will pick the data up and chunk it into databases. We have some other mechanisms to verify the data is properly stored and so on, but that is not important.
I would like to know is the REST->file->database approach valid? My concern is where to store the file? Disk, some in-memory database, a cache? Web servers fail (we can request a re-upload, but I would rather mitigate the risk earlier), or the server can get cluttered. I'm afraid the local storage on the web servers does not scale really well, if we have multiple workers and servers.
Thank you in advance.
Best Answer
It's perfectly valid, and is usually done in situations like the one you have. Others have suggested message queues, which are nice but they'll either take up memory or wind up writing the data to a backing store anyway, so now you've got another layer of software doing what you could do directly. That said, it may be easier and cleaner to use a message queue's API instead of rolling your own solution; it all depends on your environment and performance requirements. I would imagine the message queue would provide a cleaner separation of concerns, but you'll have to determine for yourself if there's going to be a performance hit.