I am working on developing a personal temperature logging viewer based on my rasppi curl'ing data into my web server's api. Temperatures are taken every 2 seconds and I can have several temperature sensors posting data. Needless to say I will have a lot of data to handle even within the scope of an hour. I have implemented a very simple paging api from the server so the server doesn't timeout and is currently only returning data in 1000 units per call, then paging through the data. I had the idea to intially show say the last 20 minutes of data from a sensor (or all sensors depending on user choices), then allowing the user to select other timeframes from which to show data. The issue comes in when you want to view all sensors or an extended time period (say 24 hours).
Is there a best practice of handling this large amount of data?
Would it be useful to load those first 20 minutes into the live view and then cache into local storage something like the last 24 hours?
I haven't been able to find a decent idea of this in use yet even though there are a lot of ways to take this problem. I am just looking for some suggestions as to what might provide a good balance between good performance and not caching the entire data set on the client side (as beyond a week of data this might not be feasible).
Best Answer
Not sure about "best practice", but I think would be a good idea to clearly define what you need to do with the data. What will the user actually need to see at any one time? Instinctively, 24 hours of data points every 2 seconds doesn't seem that possible to handle, cognitively. Maybe you'll have a per-5-minute average, that you'll need to graph? In that case, I would create an API that requests these averages so the browser doesn't need to deal with all the data. You could cache these on the server if needs-be, for other requests. If the user will see one-screen of fine-grained temperatures at any one time, then create an API that requests one-screen's worth of data at a time (similar to your paging API).
Using one-screen's worth of information at a time is also a good strategy if you have lots of Angular's bindings: if you end up with 1000s of them then interface performance will drop.
It's always a bit hard to decide how to optimize before you know exactly where the bottlenecks are. Keeping-it-simple for the first design is often a good idea. Don't build your own caching layer, at least at first: use $http's built-in cache, and maybe also an aggressive caching strategy using http headers, so the browser can cache the Ajax requests as well.