What formats of executables and libraries does Embox require, and how can I tell?
Embox is a configurable OS and one can create separate kernel image and user application. But since MCU has rather small memory size, most embedded RTOSes including Embox link all software (kernel, apps etc.) to a single image by default.
How does one compile a C program into an executable/library format (again, perhaps EXE/SO) that is compatible with Embox?
You should setup make and a cross-compiler. Then you have to configure Embox (choose a desired template) with make confload-xxx. And finally build the image with a command make.
After that you can find the image in a file ./build/base/bin/embox. Here is a short [getting started][1] page.
How does one deploy the executables to the MCU such that the RTOS can run the 2 apps at startup?
Although Embox creates a single image you can have several apps in it. Unlike most RTOSes in Embox you can have several main routines. This feature is enabled by using a special build system. Embox consists of set of modules and each module is described using a DSL language. There is an 'application' type of module. If a module is marked as an 'application' the build system strips main symbol away and replaces it with <package-name>_<module-name>_main. Also the build system adds the new symbol and the application name to a special applications registry, so that you can run your application from the Embox shell or in some other way.
How does a user interact with the RTOS? Shell/SSH/console?
Embox has a built-in shell and you can access it through UART (COM port), for example. Embox also has telnet and httpd apps which you can interact through. Also Embox has a port of sshd (with dropbear project), but I think it's not the easiest way to communicate with MCU.
My question is if the CPU hits 100% usage, are the 10 instances still being parallel processed?
Yes.
Or Does the CPU start sequencing them instead of parallel processing them?
Your processor probably doesn't have 10 CPU cores, so yes, it is doing swapping of them in some sequence, too. But it does not suddenly run one thread to completion when it gets to 100%. It is using swapping to give each thread a time slice; and it is doing this whether it is at 100% or not.
Am I better off simply reducing the number of instances till the CPU usage is less than 100%?
No, you want 100% CPU usage, or else you're wasting cycles!
However, that is not even close to the whole story, and, 100% CPU is not a great measurement of the efficiency of the CPU cycles that are being used.
You should probably run as many threads as your processor has CPU cores. (If you are using single threaded processes, then count those as threads.) When you run more than that, then the operating system swaps the threads into and out of the CPU cores in a balance of fairness and other policies that gives each thread a time slice here and there.
Swapping causes various kinds of overheads: direct overhead as cycles are wasted doing swapping instead of your work, and indirect overhead, which has to do with the operation of the caches — the highest speed memories that are directly on the processor die. Caches form one level of the memory hierarchy, which also includes main memory (your 16+ GB of RAM) and ultimately also includes the disc as a form of memory. Using the memory hierarchy well is the key to performance throughput.
As each thread runs for a while, it will take over part of the cache for it's own purposes. In some sense, the cache is warming up to that thread, and this is good.
When the operating system can, it will give consecutive time slice to the same thread on the same processor, and the that thread will have a warm cache to start.
Too many threads at once will cause undo cache pollution as due to their being swapped in: when a thread is first swapped in, the cache will be cold for it, and it will have to warm up before you're getting to best performance.
Yet still, processing a large data set efficiently requires not just keeping the cores busy but also broadly getting better use of the memory hierarchy.
One approach to finding a good way to slice the work, which you can tweak and refine by timing, is something like this:
Divide the whole work into chunk size that are about the size of the large processor cache (L3, maybe).
Then divide each chunk of work into sections of as many processor cores as the processor has (e.g. 2, 4), and run those. Then as those sections complete, run subsequent sections of the next chunk.
A lot has to do with the workload, and the cache architecture. Smaller L1 caches are likely to be replicated and each dedicated to a particular CPU core; larger L2 and L3 caches may be shared by all cores, though potentially with restriction such that one CPU core doesn't starve the others.
It goes to how your specific workload runs on the specific cache architecture on the machine in question. So, subdividing the chunks by the L3 size may work well, but chunks of the L2 size may work better.
One application/workload may access its section over and over in a more or less random way, while a different application/workload may only run serially thru the section start to finish. These different workloads have different efficiency aspects in the memory hierarchy, so breaking them up in some different way might help or hurt one or the other workload.
There are also better measurements you might want to take using various tooling.
Best Answer
Profile it!
It's the only way to know exactly what is happening and what is using what resources and where it's being limited at.