Electronic – How does a ball mouse know the direction

encodermouse

Foreach axis on a mechanical (ball) mouse there is an IR transmitter and receiver pair with a wheel that looks like having only simple slits.

How is the direction calculated?

There's not just an IR transmitter/receiver pair, there are two of them. Between transmitters and receivers there's a slotted wheel, which, when rotating, causes a pulse train in the receiver. (The light from the transmitter is blocked, can pass, blocked again, and so on.)

The trick is how the two receivers are placed, namely in quadrature.

This means that the pulses of one receiver precede the pulses of the other by a number of degrees (ideally 90°). If the wheel turn the other way the same pulses now lag the others.

Notice that on a rising edge of channel A the B channel is at a high level when turning one way, and low when turning the other way.

I wasn't completely satisfied with my reply to JGord's comment (some inaccuracies), hence this reprise
The system described above is known as an incremental encoder because it detects relative changes, from one position to the next. Over a full rotation the codes are repeated a number of times, so you can't know your absolute position just by looking at the code.
To overcome this there exist absolute encoders. Instead of two channels in quadrature they have a lot more channels creating a unique pattern for each rotation position. A 10 channel encoder can tell \\$2^{10}\\$ or 1024 different positions apart. Shaft encoders in robots are even more accurate.

The specific pattern is typical of Gray coding.

Ordinary binary has the disadvantage that code transitions may create erroneous codes during the transition. Take for instance the change from `0111` (7) to `1000` (8). If the leftmost bit is a bit faster than the others you will see for a moment `1111` (15), which is totally off.