The key is how a quadrature encoding works: two signals are out of phase, so you can detect direction by which signal follows the other one. Combined, they have 4 states they pass through, but they will do so in opposite order for the opposite direction. I.e. 00-01-11-10- for right, 00-10-11-01- for left. As you see, they'll pass both the 01 and 10 states you're looking for - and the only way to know which way is by checking the next or previous state.
Given that you can guarantee only one encoder rotates at any time, the scaling of the quadrature decoder isn't really an issue. You can start by finding where the port changed and then decode only that transition.
Otherwise, we have the interesting challenge of finding a parallel algorithm for quadrature decoding applicable to microprocessors. A fundamentally parallel operation most of them have is bitwise operations on wider registers. Let's start by finding each channel where a change has happened, given the port arrangement a1b1a2b2 etc, i.e. every 2-bit group belongs to one channel.
If we do ((value&0xaa)>>1)^(value&0x55)) we get a parity value. This can then be xored with the previous parity value, and presto, we have a step signal. Next comes direction.
Let's set up a Karnaugh map, using inputs a, b, a' and b' (where ' means prior):
phase diagram ___/"""\___/""" a
_/"""\___/"""\_ b
a=0 a=1
b=0 b=1 b=1 b=0 1 means right, 0 means left, x don't care
a'=0 b'=0 x 1 x 0
a'=0 b'=1 0 x 1 x
a'=1 b'=1 x 0 x 1
a'=1 b'=0 1 x 0 x
We have a diagonal pattern, which tends to occur with xor functions. We also have a margin of values that should not be counted (meaning either no step or a missed step). We already found the step function to eliminate those. In essense, all we need is to find the diagonal with 0s in it, so we can invert step to get direction. It looks like the remaining discrimination can be done with b^a':
b^a' a=0 a=1
b=0 b=1 b=1 b=0
a'=0 b'=0 0 1 1 0
a'=0 b'=1 0 1 1 0
a'=1 b'=1 1 0 0 1
a'=1 b'=0 1 0 0 1
So, given that we need a'^b' for step and a' for direction, we can save those two bits from the previous step. Our functions are step=a'^b'^a^b, dir=step&(b^a').
old_a_axb = ((oldpin&0xaa)>>1) ^ oldpin
# This has a serious bug, in that the ROL step actually used B from
# the next channel over. Let's fix it.
#b_axb = ROL(pin)^(pin&0x55)
b_axb = ((pin&0xaa)>>1)^(pin&0x55)|((pin&0x55)<<1)
dir_step = old_a_axb ^ b_axb
# Rewrite since the selections get messy
old_al = oldpin&0xaa
old_ar = old_al>>1
old_br = oldpin&0x55
al = pin&0xaa
ar = al>>1
br = pin&0x55
bl = br<<1
axb_r = ar^br
axb_l = axb_r<<1
old_a_axb = oldpin ^ old_ar
b_axb = bl | axb_r = br*3^ar
dir_step = old_a_axb ^ b_axb
next_old_a_axb = axb_l^b_axb
It might be possible to optimize the a^b operation to occur only once, but given that I needed either a or b in the other bits I leave that to someone else. Also, this method doesn't discriminate between channels at all; use another mask and finding set bits to detect which channels actually stepped.
Addendum: The algorithm actually gets a lot cleaner if we do not pair the signals in adjacent bits, but use matching positions of separate variables:
# assume, for instance, a[3:0] in pin[7:4] and b[3:0] in pin[3:0]
a=pin>>4
b=pin&0x0f # Separate signals into individual variables
axb=a^b
step=oldaxb^axb
dir=olda^b
olda=a
oldaxb=axb
So, for one register width count of quadrature decoders, it takes two stored variables, three xor operations, and one extra temporary register (which rarely matters).
Best Answer
If you look at the ALPS datasheet for your part, you will see the following diagram:
Here, they show you a test circuit on the left side. You are supposed to be able to find that kind of thing in a datasheet. It's a small skill. But one you will need to develop. (You will also need to develop the skill required to find the datasheet in the first place, of course.)
Note that they include two resistors, attached to terminals A and B. These are "pull-up" resistors. On the right side, you can see that they specify (implied as a recommendation for you to consider) a value of \$5\:\textrm{k}\Omega\$. Also, you will notice that on the left side terminal C is grounded and that they show you an internal behavioral model of the device, showing two switches inside of the dashed-line rectangle there. Finally, their example shows a \$+5\:\textrm{V}\$ supply rail. This is almost certainly not a requirement. But taken together with the resistor values, it suggests that if you use a different rail voltage then you probably want approximately \$\frac{5\:\textrm{V}}{5\:\textrm{k}\Omega}\approx 1\:\textrm{mA}\$ as the current that these switches encounter during use. If you stay close to that approach, you will probably be fine. Taken together, this tells you pretty much how this thing is supposed to be designed for proper use and wired up.
It also includes, on the right, a few details about "chatter" and "bounce." You need to take notice of these, as well. In the middle, they show a "masking time" with hashed bands to emphasize these details. You need to either develop a hardware circuit or else do something in software (or both) to accommodate this transitioning behavior. Although there are some solid arguments for an external, analog hardware circuit before your software gets a chance to see anything, most designers would go cheap here and instead depend entirely on software sampling processed by a state machine or some other software method to get the job done without any additional external parts. (Zero added space, free is cheaper, no added long term or thermal drift to worry about, etc.)
If you don't know how to handle the chatter/bounce, you are going to be in a world of hurt already. But we've not yet discussed Gray code and it appears you don't know what that is, either. You should definitely check out the Wiki page on Gray codes. (The subject is actually much more interesting than the Wiki page even suggests, involving encryption, snake-in-the-box, coil-in-the-box, Hamiltonian walks, and error detection and correction just as starters -- you can make a life's work out of this area.)
There is a definitely good reason for choosing a Gray coding here. But you can read the Wiki and get most of that worked out on your own, I think.
In this case, though, it's probably a lot easier to just focus on the fact that this is a quadrature arrangement. With the circuit laid out as shown above, with those resistors, you should see two square waves that are offset from each other as shown in the diagram you included because that is how the internal "switches" are arranged to operate under rotation of the dial.
It's actually pretty nice because you can tell which direction the dial is turning. Just first use your eyes to move from left to right across your diagram and see the order in which the values change and then use your eyes to move in the opposite direction, from right to left, and see how those transitions are uniquely different.
The main thing for you to know is that you will need two resistors and a voltage source in order to finish the circuit around the rotary switch. You will also need to wire the A and B terminals over to your micro, as well. You will need debouncing software and you will need decoding software, as well. With all that in hand, you should be able to make some use of the switch.
One remaining problem you may have is that a human looking at the knob might imagine that it is pointing to the left (let's say) and that your software should already know this fact when it powers up. But without some kind of "home" pulse to examine (which would still require the operator to twirl the knob for you to see) or without an absolute encoding (which this doesn't provide), there isn't a way to be certain. All this device can do is provide you details about the fact that rotation is occurring and in which direction. It's a relative encoder, in other words, and not an absolute encoder. Whether or not that helps you? No idea. Depends on your application.
If interested, a particularly interesting area of absolute position Gray encoders is called Single Track Gray Code. I'm not sure who is commercially producing these, as a patent was issued in 1994 and if the patent is off it has only been a few years since. But the concept is interesting to read about, regardless.