C# – Sync one thousand vector3 across network with minimal bytes

At the scales you mentioned, the concerns are totally different from those you have mentioned.

Cache locality

• There are two related concepts:
  1. Locality, the reuse of data on the same cache line (spatial locality) that was recently visited (temporal locality)
  2. Automatic cache prefetching (streaming).
• At the scales you mentioned (hundred MBs to gigabytes, in 4MB chunks), the two factors have more to do with your data element access pattern than the memory layout.
• My (clueless) prediction is that statistically there might not be much performance difference than a giant contiguous memory allocation. No gain, no loss.

Data element access pattern

• This article visually illustrates how memory access patterns will affect performance.
• In short, just keep in mind that if your algorithm is already bottlenecked by memory bandwidth, the only way to improve performance is to do more useful work with the data that is already loaded into the cache.
• In other words, even if YourList[k] and YourList[k+1] have a high probability of being consecutive (one in four-million chance of being not), that fact will not help performance if you access your list completely randomly, or in large unpredictable strides e.g. while { index += random.Next(1024); DoStuff(YourList[index]); }

Interaction with the GC system

• In my opinion, this is where you should focus on most.
• At the minimum, understand how your design will interact with:
  • Generational garbage collection,
  • Large object heap
• I am not knowledgeable in these topics so I will leave others to contribute.

Overhead of address offset calculations

• Typical C# code is already doing a lot of address offset calculations, so the additional overhead from your scheme wouldn't be any worse than the typical C# code working on a single array.
  • Remember that C# code also does array range checking; and this fact does not prevent C# from reaching comparable array processing performance with C++ code.
  • The reason is that performance is mostly bottlenecked by memory bandwidth.
  • The trick to maximizing utility from memory bandwidth is to use SIMD instructions for memory read/write operations. Neither typical C# nor typical C++ does this; you have to resort to libraries or language add-ons.

To illustrate why:

• Do address calculation
• (In OP's case, load chunk base address (which is already in cache) and then do more address calculation)
• Read from / write to the element address

The last step still takes the lion's share of time.

Personal suggestion

• You can provide a CopyRange function, which would behave like Array.Copy function but would operate between two instances of your NonContiguousByteArray, or between one instance and another normal byte[]. These functions can make use of SIMD code (C++ or C#) for maximizing memory bandwidth utilization, and then your C# code can operate on the copied range without the overhead of multiple dereferencing or address calculation.

Usability and interoperability concerns

• Apparently you cannot use this NonContiguousByteArray with any C#, C++ or foreign-language libraries that expect contiguous byte arrays, or byte arrays that can be pinned.
• However, if you write your own C++ acceleration library (with P/Invoke or C++/CLI), you can pass in a list of base addresses of several 4MB blocks into the underlying code.
  • For example, if you need to give access to elements starting at (3 * 1024 * 1024) and ending at (5 * 1024 * 1024 - 1), this means the access will span across chunk[0] and chunk[1]. You can then construct an array (size 2) of byte arrays (size 4M), pin these chunk addresses and pass them to the underlying code.
• Another usability concern is that you will not be able to implement the IList<byte> interface efficiently: Insert and Remove will just take too long to process because they will require O(N) time.
  • In fact, it looks like you can't implement anything other than IEnumerable<byte>, i.e. it can be scanned sequentially and that's it.

should one load only the needed amount as shown above ?

You should load into the buffer all that you can feasibly expect to process with the code that follows. In the DataReader documentation example, they read the entire stream into the buffer, because they are going to process it all immediately.

The reason for the buffer is that IO is slow (usually). So the amount of data you specify is loaded with asynchronous IO into the memory buffer up front. Then you can subsequently read it without waiting for IO on every read. That's a good thing for performance. IO is batched which will improve perf on many devices (e.g. mechanical hard drives). Your code's execution is suspended (due to async/await) until IO is finished, so it isn't tying up CPU cycles.

or should one load a constant size (e.g. 65536 bytes), keep track of reading position then possibly pre-fetch more on larger requests ? (basically wrapping a DataReader in a helper class)

Sometimes the size of the data will be too large to load in memory all at once. .NET itself sets a memory limit of 2GB per object (well, sortof). So if the data you are reading is close to 2GB, then you will definitely want to keep track of stream's read position, and read only part of the file into the buffer. Then once you get to the end of the buffer, fill it back up again from the next read position and continue processing. Repeating as necessary until you've processed the whole file.