I had this exact problem on a (tower) server just like you explain, and it was easy:
smartctl will output the serial number of the drive
Vendors sometimes ship their own specific tools, like hdparm, that will do the same.
So output the serial of the bad drive, and then use a dentist's mirror and a flashlight to find the drive.
On a rackmount you'll usually have indicator lights like other people have said, but I bet the same would apply.
Funny enough, I have one of these particular boat anchors sitting on the floor right here, right now. (I need to get it packed up and posted on eBay. Anybody want a PowerEdge 2600? I'll autograph it... heh heh...)
I put some thought into making it a "useful" machine, but realisticly there's no internal storage option that works well since you'd have to butcher the hot-swap backplane and, very probably, eliminate any hot-swap capability in so doing.
The box is extremely power inefficient compared to a modern box in terms of compute operations per watt. You can make a great business case for chucking it and buying a new machine that is much more energy efficient and will use SATA hard disk drives.
Edit:
If you want to put SATA drives into the sleds that come with the box you're going to have to remove part or all of the factory SCSI backplane. In the world that I work in a server computer that's had factory parts forcibly removed from it to add new "features" isn't a reliable server computer anymore.
You could get a controller with something like eSATA ports. If that controller had BIOS and RAID functionality you could certainly use it. You might have to disable the option ROM on one or the other, but you could boot off of either (assuming you can find an eSATA controller w/ RAID functionality and a bootable option ROM). (AFAIK that server doesn't have PCI Express slots. My box has things stacked on top of it right now and I cannot be bothered to look. That box is vintage 2003 - 2004, though, and I'd be shocked if it had PCI Express slots...)
Get a PCI Firewire card and stick some external disks on it if you absolutely have to keep using the machine.
Best Answer
type man hdparm