You need to decide if it is mechanical or optical.
If it is mechanical, you will probably get patterns of pin-to-pin continuity which change as you rotate the shaft. The fourth pin could be unconnected, could be the other side of the contacts, or could be a push-button.
If it is optical, you may get nothing with the continuity tester, depending on the voltage it applies. An experimental LED power supply made of a resistor and voltage source should show an LED-scale diode voltage drop across some pair of pins. Subsequent to that, you may at any given shaft position be able to get one or both of the other pins to sink current supplied through a resistor. You will have to imagine possible internal circuits - for example, a ground pin, an LED anode with the cathode to ground, and two NPN phototransistors with their emitters grounded and collectors exposed.
You can also browse distributor catalogs for a part which looks similar, and see if the indicated data might be consistent in some respects with the part you have in your hand.
EDIT: Just web searching on CTR and encoder finds this very similar appearing part. No data sheet, but it should be enough to figure out the rest with a meter: http://www.tme.eu/at/details/ps1010-20/drehschalter/ctr/ps1010-20-kq15a60-000/
(note: please do not edit to "prettify" this link - there's information contained in the raw URL itself which may be useful if the link breaks)
If you have the circuit it was obtained from, that can provide clues as well - easier to analyze something like this while it is still connected to whatever supplies it needs.
Detents are slight "clicks" the encoder will make when you turn it. Think of what it feels like when you scroll your mouse wheel (hopefully mouse wheels you've used have detents :P). They provide some sort of feedback (audible/touch) mechanism for the user to what a discrete step in the encoder is.
More information: Wikipedia - Detent
Best Answer
It is normal for a rotary encoder to go through the following sequence:
From a decoding perspective, one may either use the states marked with dashes or with asterisks. If one uses the states marked with dashes, then when the encoder is near a switching threshold, it may appear to jitter back and forth between two positions. If the decoder is designed correctly the overall rotation should be correct, since repeated jitters will cancel out, but a +/- 1 unit jitter may be annoying. If one uses the states marked with asterisks, apparent jitter will be eliminated (once an input changes, all future changes on that input will be ignored until the other input changes) but backlash will be increased by a tick.
If the decoding logic isn't implemented properly, it's possible for the jitter to cause unwanted counts in such a fashion that they don't cancel, but the solution in that case is to fix the decoding logic.