I haven't had much personal experience with RTOS's other than QNX (which is great on the whole but it's not cheap and I have had a really bad experience with a particular board vendor and QNX's we-don't-care attitude for systems other than their most common) which is too large for PICs and MSP430's.
Where you will benefit from an RTOS is in areas such as
- thread management/scheduling
- inter-thread communications + synchronization
- I/O on systems with stdin/stdout/stderr or serial ports or ethernet support or a filesystem (not an MSP430 or PIC for the most part, except for the serial ports)
For peripherals of a PIC or MSP430: for serial ports I'd use a ring buffer + interrupts... something I write once per system and just reuse; other peripherals I don't think you'd find much support from an RTOS, as they are so vendor-specific.
If you need timing that is rock-solid to the microsecond, an RTOS probably won't help -- RTOS's have bounded timing, but typically do have timing jitter in their scheduling due to context switching delays... QNX running on a PXA270 had jitter in the tens of microseconds typical, 100-200us maximum, so I wouldn't use it for stuff that has to run faster than about 100Hz or which needs timing much more accurate than about 500us. For that kind of stuff you probably will have to implement your own interrupt handling. Some RTOS's will play nicely with that, and others will make it a royal pain: your timing and their timing may not be able to coexist well.
If the timing/scheduling is not too complex, you may be better off using a well-designed state machine. I would highly recommend reading Practical Statecharts in C/C++ if you haven't already. We've used this approach in some of our projects where I work, and it's got some real advantages over traditional state machines for managing complexity.... which is really the only reason you need an RTOS.
Yes, C++ is still useful in embedded systems. As everyone else has said, it still depends on the system itself, like an 8-bit uC would probably be a no-no in my book even though there is a compiler out there and some people do it (shudder). There's still an advantage to using C++ even when you scale it down to something like "C+" even in a 8-bit micro world. What do I mean by "C+"? I mean don't use new/delete, avoid exceptions, avoid virtual classes with inheritance, possibly avoid inheritance all together, be very careful with templates, use inline functions instead of macros, and use const variables instead of #defines.
I've been working both in C and C++ in embedded systems for well over a decade now, and some of my youthful enthusiasm for C++ has definitely worn off due to some real world problems that shake one's naivete. I have seen the worst of C++ in an embedded systems which I would like to refer to as "CS programmers gone wild in an EE world." In fact, that is something I'm working on with my client to improve this one codebase they have among others.
The danger of C++ is because it's a very very powerful tool much like a two-edged sword that can cut both your arm and leg off if not educated and disciplined properly in it's language and general programming itself. C is more like a single-edged sword, but still just as sharp. With C++ it's too easy to get very high-levels of abstraction and create obfuscated interfaces that become meaningless in the long-term, and that's partly due to C++ flexibility in solving the same problem with many different language features(templates, OOP, procedural, RTTI, OOP+templates, overloading, inlining).
I finished a two 4-hour seminars on Embedded Software in C++ by the C++ guru, Scott Meyers. He pointed out some things about templates that I never considered before and how much more they can help creating safety-critical code. The jist of it is, you can't have dead code in software that has to meet stringent safety-critical code requirements. Templates can help you accomplish this, since the compiler only creates the code it needs when instantiating templates. However, one must become more thoroughly educated in their use to design correctly for this feature which is harder to accomplish in C because linkers don't always optimize dead code. He also demonstrated a feature of templates that could only be accomplished in C++ and would have kept the Mars Climate Observer from crashing had NASA implemented a similar system to protect units of measurement in the calculations.
Scott Meyers is a very big proponent on templates and judicious use of inlining, and I must say I'm still skeptical on being gung ho about templates. I tend to shy away from them, even though he says they should only be applied where they become the best tool. He also makes the point that C++ gives you the tools to make really good interfaces that are easy to use right and make it hard to use wrong. Again, that's the hard part. One must come to a level of mastery in C++ before you can know how to apply these features in most efficient way to be the best design solution.
The same goes for OOP. In the embedded world, you must familiarize yourself with what kind of code the compiler is going to spit out to know if you can handle the run-time costs of run-time polymorphism. You need to be willing to make measurements as well to prove your design is going to meet your deadline requirements. Is that new InterruptManager class going to make my interrupt latency too long? There are other forms of polymorphism that may fit your problem better such as link-time polymorphism which C can do as well, but C++ can do through the Pimpl design pattern (Opaque pointer).
I say that all to say, that C++ has its place in the embedded world. You can hate it all you want, but it's not going away. It can be written in a very efficient manner, but it's harder to learn how to do it correctly than with C. It can sometimes work better than C at solving a problem and sometimes expressing a better interface, but again, you've got to educate yourself and not be afraid to learn how.
Best Answer
I've had this problem occasionally. As you say, the "ready to go" WiFi modules are expensive and usually very limited in what they let you do. Most try to do too much, like give you a UART interface to a single TCP stream. That's fine if that's all you want, but I usually have multiple TCP servers or clients, or need more out of band control than what a dumb stream gives me.
What you want is a WiFi module that interfaces at the raw network packet transport layer instead of higher level protocols. This is sortof like the MAC layer for wired ethernet. Now you can run your own network stack and do whatever you want above that.
We have done exactly this with the Microchip ZeroG WiFi modules. We implemented a ZeroG packet layer driver in our PIC 18 network stack. The rest of the stack doesn't know how packet I/O is performed, so you can set up all the TCP, UDP and other connections as you want.
The source code for our network stack is included in the free PIC Development Tools release at http://www.embedinc.com/pic/dload.htm. Of course the stack exists within the context of our firmware framework, but using all that is a good idea anyway. All the stuff I've mentioned here is available for free.