If you're using SoftwareSerial, I think you have no chance of detecting serial overflow, because the docs say you can't use Serial.available. But even if you were using the hardware serial library, I don't think you can detect if bytes are getting dropped. The closest thing you could do is use Serial.available to see when you're about to overrun the buffer.
But this is really what hardware and XON/XOFF flow control are for. When the serial driver supports this functionality, you can prevent buffer overflows because the host will tell the client when no more data can be accepted over the serial line, and vice versa.
I don't think Arduino supports any kind of flow control, especially since begin() only takes a serial port speed as a parameter.
Every RTOS for a PIC which does not have a software-addressable stack generally requires that all but one of the tasks must have its work divided into uninterruptable pieces which begin and end at the top stack level; the "task-yield" operation does not use a function call, but rather a sequence like
// This code is part of task C (assume for this example, there are tasks
// called A, B, C
movlw JumpC4 & 255
goto TASK_SWITCH_FROM_C
TargetC4:
Elsewhere in the code would be some code like:
TASK_SWITCH_FROM_A:
movwf nextJumpA // Save state of task C
// Now dispatch next instruction for task A
movlw TaskB_Table >> 8
movwf PCLATH
movf nextJumpB,w
movwf PCL
TASK_SWITCH_FROM_B:
movwf nextJumpB // Save state of task C
// Now dispatch next instruction for task A
movlw TaskC_Table >> 8
movwf PCLATH
movf nextJumpC,w
movwf PCL
TASK_SWITCH_FROM_C:
movwf nextJumpC // Save state of task C
// Now dispatch next instruction for task A
movlw TaskA_Table >> 8
movwf PCLATH
movf nextJumpA,w
movwf PCL
At the end of the code, for each task, there would be a jump table; each table would have to fit within a 256-word page (and could thus have a maximum of 256 jumps)
TaskC_Table:
JumpC0 : goto TargetC0
JumpC1 : goto TargetC1
JumpC2 : goto TargetC2
JumpC3 : goto TargetC3
JumpC4 : goto TargetC4
...etc.
Effectively, the movlw at the start of the task-switch sequence loads the W register with the LSB of the address of the instruction at JumpC4. The code at TASK_SWITCH_FROM_C would stash that value someplace, and then dispatch the code for task A. Later, after TASK_SWITCH_FROM_B is executed, the stored JumpC4 address would be reloaded into W and the system would jump to the instruction pointed to thereby. That instruction would be a goto TargetC4, which would in turn resume execution at the instruction following the task-switch sequence. Note that task switching doesn't use the stack at all.
If one wanted to do a task switch within a called function, it might be possible to do so if that function's call and return were handled in a manner similar to the above (one would probably have to wrap the function call in a special macro to force the proper code to be generated). Note that the compiler itself wouldn't be capable of generating code like the above. Instead, macros in the source code would generate directives in the assembly-language file. A program supplied by the RTOS vendor would read the assembly-language file, look for those directives, and generate the appropriate vectoring code.
Best Answer
Your question can be read two ways, DarkCoffee:
If a particular Arduino based device can be induced to overflow its stack, can it be exploited?
Yes, it is possible to exploit a stack overflow on an Arduino.
One possible attack is the return oriented programming method, which requires sufficiently complex firmware. So, one defense here is to keep your firmware as simple as possible. It is highly unlikely that the Arduino "hello world" sketch is vulnerable. That shouldn't be much comfort to you, though, because an LED blinker isn't terribly useful. Useful firmware will have more functions, and therefore more function tails to harvest for use in an abstract machine.
The Arduino also has a boot loader, which inherently has the power to overwrite firmware. It may be possible to exploit it to overwrite existing benign but vulnerable firmware with malign firmware.
My reading of the first page of the INRIA attack paper leads me to believe it combines both approaches: return oriented programming to execute enough code to activate the self-flashing ability of the microcontroller so that arbitrary code can be permanently loaded.
Are there stack overflow attacks on Arduinos in general?
I'm not aware of any attack that works on all Arduino based devices. Again, the "hello world" LED blinker sketch is probably invulnerable, simply because it is too trivial to be vulnerable.
The more code you write, the more likely it is that you will create a vulnerability.
Note that writing 5,000 lines of code then replacing 2 kLOC with 1,000 new lines isn't a net savings of 1 kLOC from a security standpoint. If those 5 kLOC were secure and you messed up while writing some of the new 1 kLOC, it's still a vulnerability. The moral of the story is that the most secure code tends to be that which is stared at the longest, and that means keeping it unchanged as long as possible.
Obviously, every vulnerability should be patched ASAP. This is no argument for keeping old vulnerabilities around. It's an argument against adding features thoughtlessly to code you believe secure though inspection, audit, and analysis.