Imagine a scenario as follows: Lets say you have a central computer which generates a lot of data. This data must go through some processing, which unfortunately takes longer than to generate. In order for the processing to catch up with real time, we plug in more slave computers.
Further, we must take into account the possibility of slaves dropping out of the network mid-job as well as additional slaves being added. The central computer should ensure that all jobs are finished to its satisfaction, and that jobs dropped by a slave are retasked to another.
The main question is: What approach should I use to achieve this?
But perhaps the following would help me arrive at an answer:
Is there a name or design pattern to what I am trying to do?
What domain of knowledge do I need to achieve the goal of getting these computers to talk to each other? (eg. will a database, which I have some knowledge of, be enough or will this involve sockets, which I have yet to have knowledge of?)
Are there any examples of such a system? The main question is a bit general so it would be good to have a starting point/reference point.
Note I am assuming constraints of c++ and windows so solutions pointing in that direction would be appreciated.
Best Answer
Yes. This pattern is known as distributed computing(or distributed programming or whatever cool word you want to put after distributed). My suggestion will be not to build this in-house before looking at other solutions. You can look at this stack overflow question for various options. And then take calculated decision.