I've been developing concurrent
systems for several years now, and I
have a pretty good grasp on the
subject despite my lack of formal
training (i.e. no degree).
Many of best programmers I know didn't finish the University.
As for me I studied Philosophy.
C/C++, C#, Java, etc.). In particular,
it can be near impossible to recreate
conditions that happen readily on one
system in your development
environment.
yes
How do you figure out what can be made concurrent vs. what has to be
sequential?
we usually start with a 1000 miles high metaphor to clarify our architecture to ourselves (firstly) and to others (secondly).
When we faced that problem, we always found a way to limiting the visibility of concurrent objects to non concurrent ones.
Lately I discovered Actors in scala and I saw that my old solutions were a kind of "miniactors", much less powerful than scala ones. So my suggestion is to start from there.
Another suggestion is to skip as many problems as possible: for example we use centralised cache (terracotta) instead of keeping maps in memory, using inner class callbacks instead of synchronised methods, sending messages instead of writing shared memory etc.
With scala it's all much easier anyway.
How do you reproduce error conditions and view what is happening
as the application executes?
No real answer here. We have some unit test for concurrency and we have a load test suite to stress the application as much as we can.
How do you visualize the interactions between the different
concurrent parts of the application?
Again no real answer: we design our Metaphor on the whiteboard and we try to make sure there are no conflicts on the architectural side.
For Arch here I mean the Neal Ford's definition: Sw Architecture is everything that will be very hard to change later.
programming leads me to believe you
need a different mindset than you do
with sequential programming.
Maybe but for me it's simply impossible to think in a parallel way, so better design our software in a way that doesn't require parallel thinking and with clear guardrails to avoid crashes between concurrency lanes.
You might also want to consider looking up the subject of "ber tlv"
http://en.wikipedia.org/wiki/Type-length-value
This is the data stream format used by the worlds financial processing network, and is the backbone behind Chip & Pin terminals, ATM's and many other banking systems.
The best part about it is it's very simple to use.
The T,L,V part stands for
The Tag is a numerical identifier that specify's the type of message, in your case if your just making use of the format, you can most likely define your own tags for this, but if you where actually handling real financial data you'd have to use the tag numbers defined in the standard.
The Length part is again simple, it specify's how many bytes of data the value part is, like any value the more bytes you allocate to the length, the longer your payload can be.
The Value part is then just the payload for your actual message, and how you format the bytes for this is entirely up to you.
Beacuse it's such a simple to decode/encode protocol (You can do it entirely using byte arrays) you can easily sent it over UDP using a very small, very fast packet size.
It's also suitable for streaming, and multiple messages can be sent back to back in a continuous non closing connection, which is quite ideal if you decide you must use TCP.
It would in theory work well over HTTP too using a web socket, but I've never actually tried that so I couldn't comment on how well.
As for libraries supporting it, last time I looked "Craig's Utility Library"
https://github.com/JaCraig/Craig-s-Utility-Library
had very good support for TLV based structures in it, as do many of the smartcard libraries that are floating around (TLV is used on a lot of cards too)
If TLV is not your thing, then I'd definately back up what others have said and take a good close look at "Protobuf"
Update
I don't know what language your working in, or even if what I'm about to post will be of any use to you :-) , but here goes anyway.
This is a TLV decoder (Note it has no encoding capabilities, but it should be easy to reverse it) I wrote back in 2008(ish) using C# , it was designed for decoding ber-tlv packets coming off a smart card in a payment terminal, but it might serve as a starting point for you to hack it into a more useful shape.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
namespace Card_Analyzer
{
public class tlv
{
public int tag = 0;
public int length = 0;
public byte tagClass = 0;
public byte constructed = 0;
public List<byte> data = new List<byte>();
}
public class tlvparser
{
// List of found TLV structures
public List<tlv> tlvList = new List<tlv>();
// Constructor
public tlvparser(byte[] data)
{
if (data != null)
{
this.doParse(data);
}
}
// Main parsing function
public void doParse(byte[] data)
{
int fulltag = 0;
int tlvlen = 0;
int dptr = 0;
while (dptr < data.Length)
{
byte temp = data[dptr];
int iclass = temp & 0xC0;
int dobj = temp & 0x20;
int tag = temp & 0x1F;
if (tag >= 31) // Using extracted vars, decide if tag is a 2 byte tag
{
fulltag = (temp << 8) + data[dptr + 1];
tlvlen = data[dptr + 2];
dptr += 3;
}
else
{
fulltag = temp;
tlvlen = data[dptr + 1];
dptr += 2;
}// End if tag 16 bit
if ((tlvlen & 128) == 128)
{
tlvlen = (tlvlen << 8) + data[dptr];
dptr++;
}
tlv myTlv = new tlv();
myTlv.tag = fulltag;
myTlv.length = tlvlen;
myTlv.tagClass = Convert.ToByte(iclass >> 6);
myTlv.constructed = Convert.ToByte(dobj >> 5);
for (int i = 0; i < tlvlen; i++)
{
if(dptr < data.Length)
myTlv.data.Add(data[dptr++]);
}
if (myTlv.constructed == 1)
this.doParse(myTlv.data.ToArray());
tlvList.Add(myTlv);
}// End main while loop
}// End constructor
}// end class tlvparser
}// end namespace
If it's of no use, then feel free to just ignore it.
Best Answer
For ad-hoc checks, just use a standard hexdump and learn to eyeball it.
If you want to tool up for a proper investigation, I usually write a separate decoder in something like Python - ideally this will be driven directly from a message spec document or IDL, and be as automated as possible (so there's no chance of manually introducing the same bug in both decoders).
Lastly, don't forget you should be writing unit tests for your decoder, using known-correct canned input.