Use Arduino(5V) to switch a 3.3V

1) The connection between the relay/pushbutton and Arduino ground isn't necessary to make the circuit work at you describe. You don't tell us anything about the PCB that the pushbutton is mounted to, so this ground connection may be required for the rest of your system to work or it may cause undesirable ground loops. Again, you don't share enough to comment, but I'd start without it.

2) The relay wiring may or may not be correct. You're connecting it the same way that the light bulb example uses the relay, so if the light bulb example works, your approach will work. However, the light bulb example is in conflict with the silkscreen on the board, which suggests that the relay contacts that you want (normally open) are the top leftmost terminal and the 3rd from the left (which you're currently using). The silkscreen does not show any connection to the 2nd from the left terminal, which you're currently using.

3) Andino Ghosh is absolutely right that contact bounce will likely be an issue. Depending on the design of the PCB that has the pushbutton on it, a small capacitor in parallel with the relay contacts may help (~1uF), but again without details on this half of your design it is tough to say for sure.

Are you trying to emulate a power button press to turn on a computer, or do you just want to turn on the power supply by itself?

For emulating a button press, I would recommend either a relay or an optoisolator. I don't think there are any standards for how the power button on the motherboard is set up, so it would not be advisable to rely on, say, one of the pins being ground.

For turning on the power supply, all you need to do is tie the PS_ON signal to ground. This signal must be held low to keep the power supply on. Since this is a well-defined standard, you can use a single NPN transistor to pull that pin low under the control of a microcontroller or similar.