How to structure the data that is sent from the server to the user?
Use the messaging pattern. Well, you're already using a messaging protocol, but I mean structure the changes as messages... specifically events. When server side changes, that results in business events. In your scenario, your client views are interested in these events. The events should contain all data relevant to that change (not necessarily all view data). The client page should then update the parts of view it is maintaining with the event data.
For instance, if you were updating a stock ticker and AAPL changed, you wouldn't want to push all stock prices down or even all the data about AAPL (name, description, etc). You would only push AAPL, the delta, and the new price. On the client, you would then update only that stock price on the view.
Should I only send events like "this resource has been updated and you should reload it via an AJAX call" or push the updated data and replace previous data loaded via initial AJAX calls?
I would say neither. If you are sending the event, go ahead and send relevant data with it (not the whole object's data). Give it a name for the kind of event it is. (The naming and what data is relevant to that event is beyond the scope of the mechanical workings of the system. This has more to do with how the business logic is modeled.) Your view updaters need to know how to translate each specific event into a precise view change (i.e. only update what changed).
How to define a coherent and scalable skeleton to data sent? is this a model update message or "there was an error with blahblahblah" message
I would say this is a large, open-ended question that should be broken up into several other questions and posted separately.
In general though, your back end system should create and dispatch events for important happenings to your business. Those could come in from external feeds or from activity in the back-end itself.
How not to send data about everything from anywhere in the backend?
Use the publish/subscribe pattern. When your SPA loads a new page which is interested in receiving real-time updates, the page should subscribe to only those events it can use, and call the view update logic as those events come in. You will probably need pub/sub logic on the server to reduce the network load. Libraries exist for Websocket pub/sub, but I'm not sure what those are in the Rails ecosystem.
How to reduce the business logic duplication both on the server and the client side?
It sounds like you are having to update the view data on both the client and server. My guess is you need the server-side view data so that you have a snapshot to get the real-time client started. Being that there are two languages/platforms involved (Ruby and Javascript), the view update logic will have to be written in both. Aside from transpiling (which has its own issues), I don't see a way around that.
Technical point: Data manipulation (view update) is not business logic. If you mean use case validation, then that seems unavoidable since the client's validations are necessary for good user experience, but cannot ultimately be trusted by the server.
Here is how I see such a thing structured well.
Client Views:
- Requests a view snapshot and the view's last seen event number
- This will prepopulate the view so the client doesn't have to build from scratch.
- Could be over HTTP GET for simplicity
- Makes a websocket connection and subscribes to specific events, starting from the view's last event number.
- Receives events over websocket and updates its view based on event type/data.
Client Commands:
- Request data change (HTTP PUT/POST/DELETE)
- Response is only success or failure + error
- (The event(s) generated by the change will come over websocket and trigger a view update.)
The server side could actually be broken up into several components with limited responsibilities. One that just processes the incoming requests and creates events. Another could manage client subscriptions, listen for events (say in-process) and forward appropriate events to subscribers. You could have a third that listens for events and updates server-side views -- maybe this even happens before subscribers receive the events.
What I have described is a form of CQRS + Messaging, and a typical strategy to address of the kind of issues you are facing.
I didn't bring Event Sourcing into this description as I'm not sure if it's something you want to take on or if you need it necessarily. But it is a related pattern.
Best Answer
People think of kafka as a message broker, but it's also sort of a database that stores and retrieves messages in order, and tracks every consumer's place in that list. This is an extremely nice fit to store something like chat messages, especially if the
websocketbox on your diagram is actually multiple websockets on multiple machines, where the bottomputMessagearrow in that box goes through kafka as well.In a typical kafka application, the DB in your diagram would be more for aggregate purposes like search or analytics, not for queries like "get me the new messages since the last time I polled."