Python – Parsing scripts that use curly braces

Data is now (after parsing) stored in just a list, searching for a specific peace of data is not very well optimized. Are there other structures I could use, which I can search using file(string) - level(integer) - sub-level(integer) so I can quickly get a specific data object.

Assuming that you want to search by File, Level, Sub-Level only, you have a clear hierarchical structure in that description, it sounds like you can divide one ArrayList into multiple steps. You could even make each hierarchy level a class itself to make it more clear, for example:

class SomeFileData {
   List<LevelData> levels;
}

class LevelData {
   List<SubLevelData> sublevels;
}

class SubLevelData {
   // Probably similar to your current `FileData` implementation
}

And if you need to save multiple files at once, you can use a Map with the file name as key.

class LotsOfFileData {
    Map<String, SomeFileData> files; 
}

Using this hierarchy, you can easily get file "mydata.dat", level 3, sublevel 4.

LotsOfFileData allData;
allData.getFile("mydata.dat").getLevel(3).getSublevel(4)

Each step in the get process looks at the current class' list or map to retrieve the data.

By the way, declare variables according to their interface, not their implementation. Declare your ArrayLists as List to allow for easy change of implementation.

To answer your question, you really should benchmark (i.e. measure the execution time of several variants of your program). I guess it might depend on how many small files you need (10 thousand files is not the same as 10 billion files), and what file system you are using. You could use tmpfs file systems. It also obviously depends on the hardware (SSD disks are faster).

I would also suggest to avoid putting a big lot of files in the same directory. So prefer dir01/file001.txt ... dir01/file999.txt dir02/file001.txt ... to file00001.txt ... file99999.txt ie have directories with e.g. at most a thousand files.

I would also advise to avoid having a big lot of tiny files (e.g. files with less than a hundred bytes of data each): they make a lot of filesystems unhappy (since a file needs at least its inode).

However, you should perhaps consider other alternatives, like using a database (which might be as simple as Sqlite ...) or using some indexed file (like gdbm ...)

Regarding RAM, the kernel tries quite hard to keep file data in RAM. See e.g. linuxatemyram.com; read about posix_fadvise(2), fsync(2), readahead(2), ...

BTW, Python code will ultimately call C code and use the same (kernel provided) syscalls(2). Most file system related processing happens inside the Linux kernel. So it won't be faster (unless Python adds it own user-space buffering to e.g. read(2) data in megabyte chunks, hence lowering the number of executed syscalls).

Notice that every Linux system is able to deal with a lot of disk data, either with a single huge file (much bigger than available RAM: you could have a 50Gbyte file on your laptop, and a terabyte file on your desktop!) or in many files.