There's not just an IR transmitter/receiver pair, there are two of them. Between transmitters and receivers there's a slotted wheel, which, when rotating, causes a pulse train in the receiver. (The light from the transmitter is blocked, can pass, blocked again, and so on.)
The trick is how the two receivers are placed, namely in quadrature.
This means that the pulses of one receiver precede the pulses of the other by a number of degrees (ideally 90°). If the wheel turn the other way the same pulses now lag the others.
Notice that on a rising edge of channel A the B channel is at a high level when turning one way, and low when turning the other way.
edit (about absolute encoders)
I wasn't completely satisfied with my reply to JGord's comment (some inaccuracies), hence this reprise
The system described above is known as an incremental encoder because it detects relative changes, from one position to the next. Over a full rotation the codes are repeated a number of times, so you can't know your absolute position just by looking at the code.
To overcome this there exist absolute encoders. Instead of two channels in quadrature they have a lot more channels creating a unique pattern for each rotation position. A 10 channel encoder can tell \$2^{10}\$ or 1024 different positions apart. Shaft encoders in robots are even more accurate.
The specific pattern is typical of Gray coding.
about Gray coding
Ordinary binary has the disadvantage that code transitions may create erroneous codes during the transition. Take for instance the change from 0111 (7) to 1000 (8). If the leftmost bit is a bit faster than the others you will see for a moment 1111 (15), which is totally off.
Gray code overcomes this by rearranging the codes so that there's only 1 bit changing at a time.
Absolute encoders won't help you to find the absolute mouse position, however, because the wheel rotates several times while you're moving the mouse. The "unique" pattern will repeat every few mm and isn't so unique after all. Besides, it's always possible to move the mouse when the computer is off, or you can lift the mouse and put it down again a bit further. Both actions will go undetected.
Further reading
"Control Shaft Encoders" \$-\$ Circuit Cellar issue 250, May 2011, p.28 ff
There will be a gap in each of the two pots near where they (ideally) instantaneously switch from 95% to 5% or vice-versa depending on the direction of rotation.
I suggest using the center range of each- switching from one to the other when the current one goes out of the center of the range, in this case it would be 50% +/- 22.5%. There will tend to be a smallish glitch at the transition. If you need to eliminate that you could blend the two readings together with a function that weights the two readings in a variable fashion. You still want to ignore the output that is too close to the transition, but to blend the two inputs in the intermediate range.
Best Answer
finding absolute position on a sphere
finding absolute position on a plane
Yes, there are several such patterns used to determine absolute position on a plane:
When a digital pen is used on digital paper, two key steps in the algorithm are (a) decoding the part of the pattern visible from the pen into bits, and (b) decoding those bits into an absolute location, which indicates which page in the notebook and a coarse position on that page. Or in other words, when designing the system and deciding where the ink goes on the paper, the system designer first (b) chooses some pattern of bits on a grid, perhaps the cyclic single-track gray code used in the Anoto address carpet or a de Bruijn torus or a variety of other patterns of bits-on-a-grid. Then the system designer (a) chooses one of a variety of ways to represent each bit as ink on paper.
Aboufadel, Smietana, and Armstrong; "Position Coding" give a detailed description of a few of the possible choices for (a) and (b). If I understand correctly, the Anoto address carpet uses a sequence closely related to a de Bruijn sequence, but it is technically not a a de Bruijn torus.
Hecht; "Printed Embedded Data Graphical User Interfaces" give a detailed description of DataGlyphs, which he uses for (a), and the "carpet code" he uses for (b). Jeff Breidenbach and I use the phrase "a DataGlyph address carpet" for that particular combination of (a) and (b).
adapting a flat pattern to a sphere
There's several ways of projecting such "flat" patterns onto a sphere; perhaps using a projection such as the quadrilateralized spherical cube or some other geodesic grid technique for pixelizing the sphere. All such projections have one kind of distortion (lots of seams) or another (a lot of stretching in some regions compared to others) or some sort of compromise (a few seams, and somewhat less stretching); hopefully you can find one that works OK with your hardware.
Native spherical patterns
De Bruijn torus
It almost works.
A de Bruijn torus works fine as the grid of bits in an address carpet for finding absolute position on a plane. The most popular address carpet systems use other bit patterns that are easier to decode than a de Bruijn torus, even though they require the sensor to view slightly more modules.
A de Bruijn torus would also work fine as the grid of bits in an address carpet for finding absolute position on a torus or a section of a cylinder -- with careful design, the pattern could be designed to "match up properly" traveling the full circle around the circumference of a cylinder, and to "match up properly" traveling both full circles around both circumferences of a torus.
Starting near (for example) the city of Accra, with careful design one could design a pattern based on a de Bruijn torus that completely covered a narrow band near the equator and "matched up properly" traveling the full circle around the equator (but left the poles uncovered). Alternatively, starting near the same city, with careful design one could use print exactly the same pattern near the city of Accra, but extend it along the the prime meridian and the international date line, covering both poles. Those two approaches would print exactly the same pattern on the globe near Accra, but alas, a de Bruijn torus gives conflicting patterns at Accra's antipode on the international date line near the equator.
Is there some other pattern similar to a de Bruijn torus that "matches up properly" in both directions at the same time, when printed to completely cover a sphere or even a cube? So far no one seems to be able to find one, and so we are forced to use systems that can tolerate "seams" in the pattern, in the same way that mapmakers are forced to cut at least one seam in a globe in order to project a whole globe on to a flat piece of paper.
Once you allow seams, then again patterns based on the de Bruijn torus would work fine -- cut one or more large chunks out of that pattern, and use the chunk(s) as "adapting a flat pattern to a sphere", trimming off overlap; even though the pattern won't "match up properly" at the seams.