No, an XOR gate will not help. In this type of state diagram, it's implicit that in any state, there are three possible inputs: no button, button A or button B. When there's no button press, you always stay in the current state, but for either button press, you need to advance to some other state that depends on which button it is.
As far as implementation with muxes, it's actually quite straightforward. I would just use four 2:1 muxes to turn the JK FFs into D FFs (drive the K input with the negation of the J input), and then use four 64:1 muxes to implement the state transition table directly — i.e., create a 64×4 ROM out of them — where the select inputs to the muxes are Q3, Q2, Q1, Q0, B and A.
I'm not going to go through your state machine because it's too complex to have instructive purpose (and also, it isn't quite right with respect to the question you were asked!!), but will do a truth table for a relatively quick example.
From http://d3i5bpxkxvwmz.cloudfront.net/resized/images/remote/http_s.eeweb.com/quizzes/2010/12/29/finite-state-machine-1293606984_299_253.png (which I'm using as an example of a state machine to simply write a logic table for):
There are three states. That means you need at least two logic bits to support them
State Q1(N) Q2(N)
S1 0 0
S2 0 1
S3 1 0
Q1 and Q2 are the outputs that map back to these states at time N. Note that you can accomodate a fourth state without changing the number of output bits (that would be 1 1).
Now, we add the inputs, and the state that the input brings the machine to at time N+1.
State Q1(N) Q2(N) Input NextState Q1(N+1) Q2(N+1)
S1 0 0 0 S3 1 0
S1 0 0 1 S2 0 1
S2 0 1 0 S1 0 0
S2 0 1 1 S2 0 1
S3 1 0 0 S2 0 1
S3 1 0 1 S3 1 0
You can also accommodate outputs (e.g., alarms) Lets say for example, you want to assert a bit, but only when transitioning from S3 to S2 (this doesn't appear in the original state machine, but I'm adding this to be thorough)
State Q1(N) Q2(N) Input NextState Q1(N+1) Q2(N+1) Output
S1 0 0 0 S3 1 0 0
S1 0 0 1 S2 0 1 0
S2 0 1 0 S1 0 0 0
S2 0 1 1 S2 0 1 0
S3 1 0 0 S2 0 1 1
S3 1 0 1 S3 1 0 0
The columns State and NextState aren't really part of the truth table -- I just put them in for clarity. Note also that there are no UNIQUE solution to problems like this.
You have 10 states in your case. For the moment, lets assume that this is correct. That means you need at least4 bits to represent them, and you'd have a much bigger table. Your A and B would map as the Input (0 and 1). You would need two outputs (one for your alarm, and one to unlock the safe)
Tedious. You bet. Now go to it!!
If you're stopping w/ a Truth Table, you're done at this step. If you need to implement this as a D-flip flop circuit, as in your original question, the next step would be to find any redundant states and eliminate them. After that, you would make a Karnough map and reduce the truth table down to logical expressions. After that, you would implement your logical expressions with the outputs of D flip flops (as required by your problem) serving as Q1 and Q2. All these steps are more than I'm willing to instruct on in this community, but the links I put in the answer to your original question should get you through.
Your Karnough map will be even more tedious than the truth table. You will have a bunch of "Don't Care's", though, which will help.
All that said, this is sort of the "official" path to these problems. For the simple problems, the process could get easier. It's hard to describe how, but sometimes you just "grok" the problem and see the flip-flop implementation you need, and then clean up a bit. Other times, you think you see this approach, try it, then realize you just wasted a bunch of time because one of your state transitions won't work the way you expect it to, and that you should have formalized your approach.
Lastly, for a "real" implementation, you would need to think about some initialization procedure. Flip-flops are not guaranteed to come up in any particular state, and you need a way to guarantee you start in the initial state.
Best Answer
Without answering your assignment, I'd probably start by drawing a state machine. This forces you to walk through the logic in a really visual way, until you really understand the question.
Once you have the state machine, you can write the truth table, and scan the truth table to find redundant states and get rid of them-- then maybe redraw the truth table.
There are a number of good sources on doing this -- https://www.youtube.com/watch?v=i0FK4fNTH9c for example.
Form there, you MIGHT be able to see your implementation, or you might not.
If not, the formulaic way to flip flop implementation is to make a Karnaugh Map, reduce it to logic, and then implement the logic w/ flip flops. I suggest poking around at http://www2.elo.utfsm.cl/~lsb/elo211/aplicaciones/katz/chapter6/chapter06.doc3.html
If your state machine is big, this can be quite a tedious exercise