C# – In-memory data collection object(s) vs database

cdatabasememorysqlwpf

I have a couple of questions about the practicality of developing a WPF application that loads data into memory from a text file and then manipulates the resulting object(s) instead of transactions with a database such as SQLite. Multiple views would have bindings on the resulting object(s) in memory. Persisting the updates to the text file on disk might only happen at app shutdown.

The data is a ham radio logbook stored as .adi format. ADI is a textual format that stores tabular data in strings, including column names and their values. Rows of data are terminated with an tag. There are numerous apps in existence that parse these .adi files line by line into a database table where it is read and edited by the app. The entire database can then be exported back to an .adi file whereby each table record is serialized into a line of text in the .adi file. Most of them are utilizing some form of database like SQL.

The largest logbook that I'm aware of is about 120,000 records, but a typical user might reach 60,000 to 70,000 in their lifetime. Each record of data could have as many as 30-40 columns including a mixture of string, integer and datetime values.

I'm interested in developing an app that can handle say 100,000 such records without the use of a database solution such as SQLite or MS SQL. An .adi text file of this magnitude should be under 50mb in size on disk. When the app first loads, it would read the text file into an object for binding to a grid. Edits would be applied to the object(s) until app shutdown when they are persisted to the file on disk.

  1. Is an object based on a text file approximately 50mb in size likely to be problematic if the app is running on a semi-modern Windows machine with 4gb-8gb of system memory?

  2. The ADI standard includes many enumerations. If I were converting this file standard to a relational database I would have numerous lookup tables representing the enumerations. But given the goal of developing without the need for an external database solution, I am thinking of storing the enumerations as class objects in a library. Am I correct that a hard-coded dictionary or enumeration does not occupy any memory space until it is called?

Thanks for your help.

Best Answer

Is an object based on a text file approximately 50mb in size likely to be problematic if the app is running on a semi-modern Windows machine with 4gb-8gb of system memory?

Not the object itself. I'm more concerned with the way you plan on implementing the UI. The grid design you propose will require many, many times the amount of memory that holding the actual ADIF will require, and it won't exactly be user-friendly.

Am I correct that a hard-coded dictionary or enumeration does not occupy any memory space until it is called?

You can certainly make such objects lazy-loading. However, I've had a look at the enumerations, and the amount of memory required for them is almost certainly inconsequential anyway.

...without the use of a database solution such as SQLite or MS SQL

There are many compelling reasons why those other guys built their apps on top of a database system; you get loads of capabilities for free. Having all that log data in a database suddenly makes it useful for all sorts of purposes, not the least of which is filtering, searching, sorting and querying the resulting data.

Related Topic