Electronic – How to make a 4×4 switch out of 2×2 switches

I want to design a 4×4 switch with 2×2 switches.

The 2×2 Switch

The 2×2 switches have a one-to-one relationship between inputs and outputs. As such, the select signal for the 2×2 switch is only a single bit because there are only two options: (1) The 2×2 switch is in "bar" broadcast. The inputs go horizontally to the outputs (In0 -> Out0 and In1-> Out1). (2) The 2×2 switch is in "cross" broadcast mode. The inputs cross each other as they go to the outputs (In0 -> Out1 and In1 -> Out0).

The switch is unicast. No input can drive more than one output. Also, no output can be driven by more than one input.

The 4×4 Switch

I hope my reasoning is right. I think the 4×4 switch would have 4!=24 combinations given the unicast limitations. That is, if In0 can go to any 4 outputs, then In1 has three choices left. In1 can go to any 3 leaving 2 choices for In2 and then leaving 1 choice for In3.
So: 4*3*2*1 = 24 combinations.

Since my select signals are bits I would need 5 of them to cover all the combinations. 2^5 is 32 which is enough to cover 24 combinations.

This leads me to believe that there should be 5 2×2 switches inside the 4×4 switch, but I can't see a way to design it without running into a conflict between inputs going to outputs. Any help please?

Here is the VHDL for the 2×2 switch if it helps:

library IEEE;
use IEEE.STD_LOGIC_1164.ALL;

entity SW2x2 is
port (
    In0: in std_logic_vector(3 downto 0);
    In1: in std_logic_vector(3 downto 0);
    Out0: out std_logic_vector(3 downto 0);
    Out1: out std_logic_vector(3 downto 0);
    Sel: in std_logic);
end SW2x2;

architecture Behavioral of SW2x2 is

begin
    Out0 <= In0 when Sel='0' else
            In1 when Sel='1' else
            "0000";
    Out1 <= In1 when Sel='0' else
            In0 when Sel='1' else
            "0000";

end Behavioral;

FYI: The question is related to the design, not the VHDL code. VHDL is just given as reference.