I'm developing a NodeJS API that will be consumed, for now, by an Android app.
I need 2 important things here: a News Feed and a Notification system. And I need it to be scalable.
I'm using MongoDB with Express for this project.
Notification System
About the notification system I'm thinking about a document that would hold:
- actor: id refering to an User object
- verb: a String defining the type of action: add friend, comment, etc.
- object: an id refering to: User, Activity, whatever, depending on the type of verb. I would get the correct object depending on the value of the verb.
My biggest doubt about this structure is: How would I return all this data to my Android app. Is it correct to return a JSON with the notification unpopulated (only ID's), and the Android app would use the ID's and query the API to build the notification. I mean, the client would make a call to: /getUser/:id to get the username, then /getActivity/:id to get the activity details, you get the point… Wouldn't that be to much calls just for a notification?
Because, imagine a user getting 50 notifications, that would be ~100 requests only to get the notification details. If that happens to 1K users, it would be 10K requests to get notification details.
News Feed
What's the best way to my client to get the latest activity? Just query the database for the latest inserted data that matches the criteria?
I shouldn't be dwelling too much on that because the application will start small, but I would like to start with a not so bad design. That's my first application of this type so I don't know what to expect.
Thanks!
Best Answer
Notification Systems
Typically, notification systems use a message queue rather than joins between modelled properties.
In this model:
A message queue is just a list you agree to append to.
Every user has a unique message queue.
Messages for that user are appended to the end of the queue.
Calls for new notifications fetch all items in a queue after a specific index, typically in batches of N.
Calls for notifications can also include requests between any two indices. This allows you to fetch older notifications.
Notification systems typically require persistence. i.e. you don't want older messages to not be retrievable. So events are never removed from the queue.
You're not wrong to model the users of your system with IDs or the type of interactions they can have as strings in MongDB.
The only problem is you're attempting to do a massive outer join to fetch every new event. This, as you've correctly predicted, leads to a large number of requests being made.
There is nothing preventing you from having your backend create new Event objects out of users and interaction types, storing them in a list that's uniquely paired to a user, and consuming them from the queue. This reduces the number of calls you need to make to fetch new notifications significantly.
You might want to look at Apache Kafka, which is an immutable log system that has inbuilt persistence guarantees, rather than a document-oriented database like MongoDB to handle your message queues.
News Feeds
A news feed is just a queue that contains a different event type.
You can reuse message queues here.