Measuring rotations problem with motor encoder

You left out the all-important information about how fast the pulses will be coming in, so I'll just assume the worst cases pulses are still slow enough for properly written firmware to catch.

I have found a few tricks to doing quadrature decoding in firmware:

1. Don't try to catch individual changes, poll at regular intervals instead. Assume that the encoder can get stuck right on the transition for any one edge, and that edge can therefore rapidly dither. This would make a mess of any interrupt on change technique.

   With regular polling, you can tune the system carefully for how much time you want to spend in the interrupt routine. The upper limit of encoder speed is also nicely known, and isn't really any slower than the worst case with the interrupt on change method anyway. The polling method is guaranteed to work correctly up to some maximum motion speed you can easily calculate. The interrupt on change method degrades unpredictably as there is bouncing on transitions.

2. Always decode the full 4x position. Don't try to get cutesy and do stupid stuff like count +1 when A transitions while B is high, etc. These "shortcuts" are actually harder to implement and have corner cases that get you into trouble. Even if upper levels only want to know position to a single fully cycle, you keep yourself out of trouble by decoding the new position from every change of state.

3. Decode using a lookup table. Each interrupt, you take a snapshot of the current state of the two encoder lines. This together with the snapshot from the previous interrupt is 4 bits of data. From the previous state and the new state you can tell whether the 4x count stayed the same, or went up or down 1. Since there are only 16 possible cases, you can use these to dispatch from a table. There are 4 cases for each of staying the same, +1, and -1.

   There are also the 4 cases of the count having changed by 2 that require special handling. A change of 2 means that one transition was missed and the encoder is moving rapidly. From looking at the old and new state alone, you don't know whether it went forward 1/2 cycle or backward. Note that 1/2 cycle is a count of +2 or -2 in this system. The way I deal with this is to keep a last known direction flag. The code that does +1 and -1 sets this flag according to the known direction. When you get a step of 2, you assume it's going in the same direction as the last step of 1. That's a pretty good assumption considering inertia of real mechanical systems.

   In reality therefore you really have 5 bits of information to process each interrupt, the 2 previous encoder line levels, the new encoder line levels, and the last known direction. So for the ultimate in speed you make this a 32 entry table and handle each possible case with explicit code. The code for each of the cases adds from -2 to +2 to the current position and possibly updates the known direction flag.

As you can see, only a few instuctions are needed each polling interrupt. Your PIC (16F1825) can run at up to 32 MHz clock rate, which is 8 MHz instruction rate. Let's say you set up a interrupt every 50 instructions, which is well more than needed to do what I described above. That means you can poll at 160 kHz. Since the encoder can change by up to 1/2 cycle per interrupt, this supports up to 80 kHz encoder full cycle rate.

[...]count some amount of pulses that are in the same direction before registering "ok, the rotation has started"

A kind of hysteresis is proposed by the O.P. If there is a slow but real motion, your actual input will be slow and smooth, but it would be perceived as a series of steps.

First thing that come to mind is low-pass filtering your signal in software. Boxcar average (sliding average), for example, is nice and simple.

Second thing that comes to mind is isolating the encoder from unwanted vibration.

P.S.

There are several philosophies.

Philosophy #1. Attempt to solve optical problems with optics, mechanical problems with mechanics, analog problems with analog, digital problems with digital, software problems with software. In that order or precedence (roughly).

Philosophy #2. Try to solve every problem with software, because software changes are more expedient than hardware changes. If it can't be helped with software, try digital solution. Then try analog... Notice that the order is reversed w.r.t. Philosophy #1.