Electronic – What does “Cycle \$\mu\$s” mean in this context

Given the goals of the class, I think the TTL approach is fine, and I say this as an "FPGA guy". FPGAs are a sea of logic and you can do all sorts of fun stuff with them, but there's only so much that's humanly possible to do in a semester.

Looking at your syllabus, your class is a mix of the logic design and "machine structures" courses I took in undergrad. (Plus, it's for CS majors. I'm all for CS majors having to face real hardware--letting them get away with writing code seems like a step back.) At this introductory level, where you're going over how assembly instructions are broken down, I see no real benefit to having students do things in code versus by hand. Doing HDL means learning the HDL, learning how to write synthesizable HDL, and learning the IDE. This is a lot more conceptual complexity and re-abstraction. Plus you have to deal with software issues.

Generally the point of a course that uses FPGAs is to practice creating logic that is useful--useful for talking to peripherals, serial comms, RAM, video generators, etc. This is valuable knowledge to have, but it seems very much out of the scope of your course. More advanced classes in computer architecture have students implement sophisticated CPUs in FPGAs, but again, this seems out of the scope of your course.

I would at the very least devote a lecture to FPGAs. Run through a few demos with a dev board and show them the workflow. Since you're at Mills, perhaps you could contact the folks at Berkeley who run CS150/152 and go see how they do things.

Your code simulates two multiplexers. These are actually asynchronous components. The fact that Verilog requires data1_temp and data2_temp to be declared as reg's is a quirk of Verilog syntax and your choice of coding style, and doesn't mean these signals would be the outputs of storage elements in a physical implementation.

If you want to capture these values in actual registers, you need to add those explicitly:

reg [7:0] data1, data2;
always @(posedge someclock) begin
    data1 <= data1_tmp;
    data2 <= data2_tmp;
end

But I would like to know what this mini register file would be made of in hardware. Particularly, the 4x8 bit array consisting of k0,k1,k2,k3.

You haven't shown how these variables are assigned, so it's not possible to say how they are implemented. As your code showed, just declaring them as reg doesn't guarantee they are implemented with actual storage elements. If you assign them inside a block that begins always @(posedge clk) then very likely they are flip-flops, but there are ways you could code them that would make them synthesize differently.

I thought when it came to registers and arrays like this, you need a clock to read out data, like RAM?

You need a clock to update a (physical) register. You can read it out at any time. For example:

wire [8:0] sum;
assign sum = k0 + k1; 

is perfectly valid code. sum will change whenever any of its inputs changes. If k0 and k1 are the outputs of flip-flops, their values will only change when there is a clock edge.

For another example, you could equally well describe your multiplexers with code like this:

reg [7:0] k0, k1, k2, k3;
wire [7:0] data1_tmp;
reg [1:0] reg1;
// k<n> and reg1 are assigned elsewhere.
assign data1_tmp = (reg1 == 0) ? k0 :
                   (reg1 == 1) ? k1 :
                   (reg1 == 2) ? k2 : k3;

how do I read from this tag_array and do the comparison all within the same clock cycle?

Let me repeat a key point for emphasis: You need to use a clock to assign a new value to a register (an actual hardware register or group of flip-flops). It's output is available at any time.

RAMs are different and how you access the contents of a RAM will depend on details of the type of RAM you use.

I got confused because frankly I don't know enough about memory hardware and how that's possible.

Another key strategy: When you are learning digital logic, I recommend you learn about the physical hardware first, and then work out or study how to simulate it in HDL second. So first, learn what a physical flip-flop is, then learn the standard Verilog methods of describing a flip-flop. Especially if you are trying to write HDL for synthesis, trying to write good code before you learn the capabilities of the underlying hardware will lead you down a lot of dead-end paths.