You need to decide if it is mechanical or optical.
If it is mechanical, you will probably get patterns of pin-to-pin continuity which change as you rotate the shaft. The fourth pin could be unconnected, could be the other side of the contacts, or could be a push-button.
If it is optical, you may get nothing with the continuity tester, depending on the voltage it applies. An experimental LED power supply made of a resistor and voltage source should show an LED-scale diode voltage drop across some pair of pins. Subsequent to that, you may at any given shaft position be able to get one or both of the other pins to sink current supplied through a resistor. You will have to imagine possible internal circuits - for example, a ground pin, an LED anode with the cathode to ground, and two NPN phototransistors with their emitters grounded and collectors exposed.
You can also browse distributor catalogs for a part which looks similar, and see if the indicated data might be consistent in some respects with the part you have in your hand.
EDIT: Just web searching on CTR and encoder finds this very similar appearing part. No data sheet, but it should be enough to figure out the rest with a meter: http://www.tme.eu/at/details/ps1010-20/drehschalter/ctr/ps1010-20-kq15a60-000/
(note: please do not edit to "prettify" this link - there's information contained in the raw URL itself which may be useful if the link breaks)
If you have the circuit it was obtained from, that can provide clues as well - easier to analyze something like this while it is still connected to whatever supplies it needs.
Detents are slight "clicks" the encoder will make when you turn it. Think of what it feels like when you scroll your mouse wheel (hopefully mouse wheels you've used have detents :P). They provide some sort of feedback (audible/touch) mechanism for the user to what a discrete step in the encoder is.
More information: Wikipedia - Detent
Best Answer
As per documentation, connections would be following:
A — A line of encoder, use pull-up with controller input pin.
B — B line of encoder, use pull-up with controller input pin.
C — GND of encoder part
5 — VCC for LEDs and pushbutton
3 — pushbutton - one could use 10k pulldown since it seems it is switched to VCC
1, 2, 4 — LEDs
The LEDs have an "always present" VCC on pin 5 - so you would light them up by pulling their respective pin to GND with appropriate series resistor (470R - 1k2).