You are quite close to answering your own question. :)
In the Observable/Observer pattern (note the flip), there are three things to bear in mind:
- Generally, the notification of the change, i.e. 'payload', is in the observable.
- The observable exists.
- The observers must be known to the existing observable (or else they have nothing to observe on).
By combining these points, what is implied is that the observable knows what its downstream components, i.e. the observers are. The data flow is inherently driven from the observable - observers merely 'live and die' by what they are observing on.
In the Producer/Consumer pattern, you get a very different interaction:
- Generally, the payload exists independently of the producer responsible for producing it.
- Producers do not know how or when consumers are active.
- Consumers need not need to know the payload's producer.
The data flow is now completely severed between a producer and a consumer - all the producer knows is that it has an output, and all the consumer knows is that it has an input. Importantly, this means that producers and consumers can exist entirely without the presence of the other.
Another not-so-subtle difference is that multiple observers on the same observable usually gets the same payload (unless there is an unconventional implementation), whereas multiple consumers off the same producer may not. This depends if the intermediary is a queue-like or topic-like approach. The former passes a different message for each consumer, while the latter ensures (or attempts to) that all consumers processes on a per-message basis.
To fit them into your application:
- In the Observable/Observer pattern, whenever your watching thread is initializing, it must know how to inform the controller. As the observer, the controller is likely waiting for a notification from the watching thread before it lets the threads handle the change.
- In the Producer/Consumer pattern, your watching thread only needs to know the presence of the event queue, and interacts solely with that. As the consumer, the controller then polls the event queue, and once it gets a new payload, it lets the threads handle it.
Therefore, to answer your question directly: if you want to maintain some level of separation between your watching thread and your controller such that you can operate them independently, you should tend towards the Producer/Consumer pattern.
Yes you should.
It not only makes your back end re-usable but allows for more security and better design. If you write your backend as part of a single system, you're making a monolithic design that's never easy to extend, replace or enhance.
One area where this is popular at the moment is in Microservices. Where the backend is split into many little (or even large) services that each provide an API that the client system consumes. If you imagine using many 3rd party sources of data in your application you realise you might be doing this already.
One other benefit is that the construction and maintenance of each service can be handed off to a different team, they can add features to it that do not affect any other team producing product. Only when they are done and release their service do you them start to add features to your product to consume them. This can make development much smoother (though potentially slower overall, you would tend to get better quality and understandable)
Edit: OK I see your problem. You think of the API as a remote library. It's not. Think of the service as more of a data providing service. You call the service to get data and then perform operations on that data locally. To determine if a user is logged on you would call "GetUser" and then look at the 'logged on' value, for example. (YMMV with that example, of course).
Your example for bulk user creation is just making excuses - there is no difference here, whatever you could have done in a monolithic system can still be done in a service architecture (e.g. you would have passed an array of users to bulk-create, or a single one to create. You can still do exactly the same with services).
MVC is already based around the concept of isolated services, only the MVC frameworks bundle them into a single project. That doesn't mean you lose anything except the bundled helpers that your framework is giving you. Use a different framework and you'll have to use different helpers. Or, in this case, rolling your own (or adding them directly using a library).
Debugging is easy too - you can thoroughly test the API in isolation so you don't need to debug into it (and you can debug end-to-end, Visual Studio can attach to several processes simultaneously).
Things like extra work implementing security is a good thing. Currently, if you bundle all the code into your website, if a hacker gains access to it, they also gain access to everything, DB included. If you split it into an API the hacker can do very little with your code unless they also hack the API layer too - which will be incredibly difficult for them (ever wondered how attackers gain vast lists of all website's users or cc details? It's because they hacked the OS or the web server and it had a direct connection to the DB where they could run "select * from users" with ease).
I'll say that I have seen many web sites (and client-server applications) written like this. When I worked in the financial services industry, nobody would ever write a website all-in-one, partly as it's too much of a security risk, and partly because much development is pretty GUIs over stable (i.e. legacy) back-end data processing systems. It's easy to expose the DP system as a website using a service style architecture.
2nd Edit: Some links on the subject (for the OP):
Note that when talking about these in context of a website, the web server should be considered the presentation layer, because it is the client that calls the other tiers, and also because it constructs the UI views that are sent to the browser for rendering. It's a big subject, and there are many ways to design your application - data-centric or domain-centric (I typically consider domain centric to be 'purer', but YMMV), but it all comes down to sticking a logic tier in between your client and your DB. It's a little like MVC if you consider the middle, API, tier to be equivalent to your Model, only the model is not a simple wrapper for the DB, it's richer and can do much more (e.g. aggregate data from 2 data sources, post-process the data to fit the API, cache the data, etc.):
Best Answer
The decision on what technology to use is largely going to be about the skills in your team, and the audience to which your SDK is aimed, and to some extent whether or not your API will in the longer term benefit from being a provider of observable streams.
Rx.NET is a very powerful and flexible way of doing things, and I would recommend any .NET developer who needs to handle concurrency and/or streams of events to look into it.
(I assume that by "IObservable" you mean Rx.Net. Or are you looking into Akka Streams or some such?)
However, getting into the IObservable way of thinking can be a steep learning curve.
There is a lot of overlap between Task and Rx.NET. One explanation of the overlap is here but in my opinion is biased in favour of Task.
If you anticipate that your API will be dealing with asynchronous streams, such as returning long lists of data, or driving a push model, or wanting to combine multiple streams, then look into Rx.NET
If you anticipate that your API will benefit from LINQ over streams (with Rx.NET you can easily query IObservables as they are the 'push' version of IEnumerable, the 'pull' version), then look into Rx.NET. For example, look into .Where .Select .SelectMany .Buffer .Zip .Scan operators in Rx.NET and decide if you will have many uses for them.
If you see that your API will eventually become a push-model stream service and if you have the skills on hand, you might want to seriously consider Rx.NET
If it's just the small problem you described in your original post, it might not be worth the overhead of learning the Rx way unless you already have skills on hand. In essence, your chain of tasks (see Attached child tasks here ) is a dumbed down Rx IObservable. The advantage is that this code is intuitive for many developers. You don't need to view it as IObservable vs Callback, a Task is not a callback, and your child tasks (intermediate steps) can be returned as a collection of tasks (eg: you could compose your tasks as a list like final task + intermediate task, and then wait for When Any or some such, maybe attach the intermediate as child tasks).
Update: The concern you raise in your update that the operation completion message will be raised with a null value during intermediate steps makes no sense. Whichever approach you take, intermediate steps will not describe the operation completion reason, unless of course that intermediate step is post-hoc actualized as the final step. In the 1st scenario (Tasks): your intermediate action will not raise a completion reason (null). In the 2nd scenario (Observables): your intermediate actions will not raise a completion reason. What is the difference? What you should do in the case of the Observable is declare it an observable of "action results". This is semantically equivalent to Task (a future result). Declare X types of result message. Eg:
You can then implement Task or IObservable as you wish.
If you go the route of Observable. You can offer
.Finallyeither in the subscriber or the observable to handle the FinalResultMessage.In short, do not make the API an IObservable but IObservable if you go that route, and apply the same thinking for Task too.