Electronic – Poor man large linear encoder

This isn't the usual rotary encoder pattern, but it should still work. See this wikipedia section for the normal gray-code implementation.

Basically, you will need to:

1. Store the current state of the encoders to a register
2. Wait for some length of time. (This must be short enough that only one transition occurs)
3. Compare the new encoder state with the old encoder state to see if they are different.
4. Based on your previous state, and the new state, decode the direction. (This step/algorithm would be easier if you used a typical implementation.)

The disadvantage of your setup is that it takes two transmitters, probably has a more complicated pattern drawn, and there's no way to identify if the encoder has gone 2 steps instead of 1. Additionally, only one bit changes with the gray code count, so the comparison routine is easier and not a look-up table.

[...]count some amount of pulses that are in the same direction before registering "ok, the rotation has started"

A kind of hysteresis is proposed by the O.P. If there is a slow but real motion, your actual input will be slow and smooth, but it would be perceived as a series of steps.

First thing that come to mind is low-pass filtering your signal in software. Boxcar average (sliding average), for example, is nice and simple.

Second thing that comes to mind is isolating the encoder from unwanted vibration.

P.S.

There are several philosophies.

Philosophy #1. Attempt to solve optical problems with optics, mechanical problems with mechanics, analog problems with analog, digital problems with digital, software problems with software. In that order or precedence (roughly).

Philosophy #2. Try to solve every problem with software, because software changes are more expedient than hardware changes. If it can't be helped with software, try digital solution. Then try analog... Notice that the order is reversed w.r.t. Philosophy #1.