C++ – Design choices when doing multithreading in C++

cmultithreading

OK, so, background:

I have a little "server" app/module that is written in C++. This is kindof a tunnel application that provides a standardized API via TCP/IP[a] and on the other hand talks to the "real" server via TCP/IP[a] via it's own interface.

It doesn't even officially support multiple clients, but due to the inherently multithreaded nature of the processing that is done by the server and by the client (plus asynchronous callbacks) the tunneling app itself is multithreaded and runs pretty well during normal operations.

The application uses a number of simple mutexes (Win32 CRITICAL_SECTIONS) to lock down the code paths that would create problems when executed concurrently and this does work pretty well in practice.

A little picture:

[proprietary server]        [tunneling app]        [client]
                     <--1-                  <--1-
                              (upcalls)    |<--2-
                     <--3-                  <--3-
                                           | -2-->
                      --1->    (returns)     -1-->
                      --3->                  -3-->
                      -a-->    (callbacks)   -a1->
                      -b--> |                -a2->

So pretty much everything goes with regards to call chains.

We have now identified a number of problems in fringe cases like calls during initialization or shutdown of the server and/or tunneling app and I'm struggling to properly fix these as they are mostly related to multithreaded object and thread lifetimes and just throwing more mutexes or more locks at the problem really doesn't cut it.

As for the Question:

What I am looking for is advice and guideline on how to write multithreaded (C++) code that would work in all corner cases from the start and doesn't involve just locking everything down. Possibly with a specific focus on the context of heavy multithreading in a network communication scenario.

Any insights welcome.

Notes:

[a] : It's a CORBA communication interface. But The problems I see are really not related to the actual messaging mechanism used, as the messaging part on the wire works very well.

Best Answer

Running multiple threads in a shared memory space and trying to figure out where to put locks does not work, as you discovered. You need to approach the problem from the opposite end: threads really should share nothing and communicate with each other via messages. See this article on Erlang-style concurrency.

Related Topic