Well, it wasn't a bad scope or loose wiring or a floating ground... it was the PS/2 cable I used to connect the keyboard connector to the rest of the equipment! It's a case of not knowing your salvaged equipment very well, along with some buried surprises in the cabling itself.
Thanks for all the discussion and suggestions, though it was ultimately a non-member who caught the similarity to diode behavior and made me suspect something unexpected was actually present within this cabling.
I had an old cable from a CueCat. There's some interesting teardown info here, though that's not exactly what I was trying to do (I don't even have the 'Cat anymore... just the wiring. Figured someday there'd be a website I could post to after I screwed up while using it. :-)
Thinking incorrectly that these connectors were wired in parallel, I diligently tested the bare wiring what once connected to the 'Cat against the male plug (it was handy.. had I had ANY suspicion I'd have tested the correct connector and none of this would've happened. Assumptions...)
I accounted for the correct pin numbering and got everything wired just right based on that... except neither DATA nor CLK actually feed through to the female connector (I just figured the two extra wires were unused since they didn't probe to anything). So Vcc and Gnd were fine everywhere, but the actual DATA and CLK from the keyboard were not wired to anything!
So, by rights I should've gotten nothing useful at all on the output side, since those wires weren't even connected (they were snipped short and a little staggered. However, I clearly got two non-trivial and distinct (but greatly attenuated) signals, which I diligently read and which people here tried in vain to understand. It dawned on me they might indicate a reverse-biased diode was doing the attenuation, but I'm not sure (I actually did throw a diode into the mix after it was working and it had a similar effect). I think there's some circuitry in the male connector (which has two wire bundles, one from the female PS/2 plug and one from the CueCat side). If it were simple crosstalk I'd expect the signals I saw to be even smaller, and for these signals to be jumbled together, but they were VERY distinct (had they simply been dead I would've eventually realized I needed to test the female plug and would have found my error, but the apparent presence of protection diodes in that intermediate plug threw everything off. I'll post a followup if I manage to nail that detail down - I may even cut open the connector to do it.
Edit: I just cut it open... Vcc and Gnd were indeed bonded to all three targets as expected, while DATA and CLK were run over distinct lines from the female and male connectors to the endpoint. However, NO active or passive components were present, debunking that theory! It must've been just enough cross-talk. Given how they were crammed into the connector it's possible they were juuuuust close enough to their respective wires to bleed only their signals, though it's weird that I was getting attenuated though distinct clock and data signals despite the fact their wires were terminating on unused ping and their wires weren't even routed to the keyboard (that part of the cord didn't even have these two wires I was reading!) Bizarre...
I posted this thinking it was a very simple cable... clearly it was not and I used the wrong lines but this is a good example of what can happen when there are active components hidden in an apparently innocuous connector or cable (if anyone remembers APC's serial cables, they had hidden surprises in the form of special resistors to keep you from just using any old serial cable for communicating with your UPS... and the old iPod cables did that as well. Proprietary Cabling 101!) That's why the signals I saw rang a bell. Hopefully this helps others who are troubleshooting weird results with an apparently normal cable assembly.
tl;dr - Always keep the possibility of hidden components in mind when using salvaged connectors (especially if you get bizarre but strangely repeatable and well-isolated signals), and test the pins you'll actually be using, not just the ones you think are wired through!
Here's the output I got in the end. It was identical whether the keyboard was plugged into the microcontroller, with or without pull-up resistors, or even without anything at all (just straight into the scope). It should be a nice 5V p-p signal straight out... I was wrong about these open-collector outputs having any bearing on the signal strength or shape. They were just red herrings thanks to whatever is in that splitter jack...
Best Answer
I would start by disconnecting the square wave generator and replicating exactly what was done in the video. If you can make the sine wave appear, at least you know the rest of the VI is working.
You can also wrap the whole VI in a while loop and let it execute perpetually while you're fiddling with the DAQ view, instead of pressing play over and over again.