I duplicated your circuit in Logisim (as an opportunity to do something in Logisim). There's nothing wrong with your circuit. There is something about Logisim I don't understand.
First off, the red lines are not lines in a high state; they are errors. One would expect this sort of error if two outputs were tied together. I did a bunch of breaking the circuit and tying lines high or low, and eventually, all the errors were "flushed out" and reconnecting the circuit normally produced the toggling it was designed to do.
Specifically, break the upper leftmost wire, the one that connects Q' to D, then connect D to a high or low source ("pull resistor" works well here), and toggle it until it's all green. Then, reconnect the feedback, and it will all work. Note that high and low are represented by green and dark green (?).
Pressing "Reset Simulation" will bring all the errors back. My guess is, that somewhere in the logic of the program, it has an "undefined state". These undefined states propagate through the gates to the extent that they don't "sort themselves out" the way real electronics do. Undef AND 0 should result in 0, not Undef. Same goes for 1 OR Undef.
Just in case this has been addressed in a later version, I'll note this Logisim is 2.7.1
Update: I "fixed" the problem (within the scope of this simulator, anyway) by inserting a NOR gate in the feedback path. Then connect a pushbutton to the other input. I replaced the original button with a clock signal (found under "wiring"). Now, pressing the button clears the error. (Resetting the logic brings the error back).
First, personally I would refer to the circuit you show as a set-reset flip-flop with enable, also called a latch. I reserve the words register and clock for an edge-activated two stage memory element.
The instable situation you sketch does exist and is called metastablility. It occurs when both of the set-rest flip-flop inputs were 1, and both switched to 0 at the same time. This causes a time period in which the outputs can show weird behavior, like oscillation or values half-way between 0 and 1. Eventually the flip-flop will settle to a stable situation, with one output 1 and the other 0. This metastable period is short, but IIRC its length follows some statistical distribution, so it has a non-zero (but very very small) chance of extending to any given length.
Current-day chips are generally synchronous internally, which avoid the metastable problem. It can still occur at the edges (external inputs), where it is usually eliminated (to a very high probability) with two flip-flop stages in series, where the second one is enabled only after the metastable period of the first is (very probably) over.
Best Answer
The easiest flip-flop to understand is the SR (Set-Reset) flip-flop:
Normally both inputs are high. When you pull the \$/S\$ input low, output \$Q\$ will go high regardless of the other input. Since the other NAND gate sees now a high level on both its inputs the \$/Q\$ output will be low. Now, even when \$/S\$ goes high again, the other input will be low, so the output \$Q\$ retains its state. That's the most easy way to make logic "remember" something.
Starting from the SR flip-flop you can make more complicated registered logic, where the D flip-flop is the most used.
This circuit is sometimes presented as an edge-triggered D -flipflop, but it's really level triggered, where \$CLK\$ is used to gate the \$D\$ input. If \$CLK\$ is low both inputs of the SR flip-flop are high, and it retains its output state. When \$CLK\$ goes high the \$D\$ input decides whether \$/S\$ or \$/R\$ goes low, and the output will set accordingly, thus remembering the state of \$D\$ when \$CLK\$ went high. The difference with a real edge-triggered D -flip-flop is that the output will change with the input as long as \$CLK\$ is high. To make it an edge-triggered flip-flop you'll have to include some feedback that makes the \$CLK\$ go low again immediately after going high. The D-type latch, as it's called, will remember the input state at the time the \$CLK\$ input goes low; i.e. the output will stop changing after the \$CLK\$ goes low again.
This is an edge-triggered D flip-flop: