I'm exploring different ways of storing user-uploaded files (all are MS Office documents or alikes) on our high load web site. It's currently designed to store documents as files and have a SQL database store all metadata for those files. I'm concerned about growing out of the storage server and SQL server performance when number of documents reaches hundreds of millions. I was reading a lot of good information about CouchDB including its built-in scalability and performance, but I'm not sure how storing files as attachments in CouchDB would compare to storing files on a file system in terms of performance.
Anybody used CouchDB clusters for storing LARGE amounts of documents and in high load environment?
Best Answer
In reply to Redmumba. The CouchDB dev team would be interested in the crashes you are seeing.
On top of that: CouchDB's whole architecture is based on the fail-early principle. All subsystems as well as the main server are designed in a way to terminate and recover immediately when an error occurs. "crashes" are just part of normal operation, it makes for a lot more reliable software (ironically, but that's the whole Erlang philosophy).
As for the question, CouchDB will fit the requirements good enough. CouchDB's attachment streaming is definitely IO bound very near filesystem speed. CouchDB documents give you all the space you need for metadata and document attachments keep the binary data close by. No need to use different systems for that.