The resulting error message indicates, that your PC is unable to find the ST Link altogether, can you try to disconnect your board from the CN2 connector and try again?
You should get something like "No MCU found" then, if you still get "Cannot connect to ST-Link" it would seem like your ST-Link is damaged or your USB cable is broken (that happens sometimes).
Your connections seem fine - check if you have no solder bridges on your board, if there is a short somewhere maybe that prevents the ST-Link from working correctly.
First of all, you are right, if your board has already got a supply voltage source you do not have to connect ST-Link's VDD pin.
The second thing I would recommend you is to open the STM32F411 Nucleo board's reference manual and look at the schematics. Especially the part where the ST-Link is connected to the controller on the board.
By ST Microelectronics, SWCLK, SWDIO, NRST and SWO (and GND of course) are connected to the target MCU. The additional SWO pin is used for debug purposes, you can access printed data with the printf
function through this pin with ST-Link Utility.
The Printf via SWO Viewer displays the printf data sent from the target through SWO.
So I can recommend you to connect SWO as well, can be useful later. Connect your MCU to ST-Link just like they have connected the Nucleo's MCU to it.
As for boot configurations, there are three selectable options, easiest it to stick with the Main Flash and tie Boot0 to GND, but I do not know your requirements so it is up to you to choose.
Best Answer
To 'unbrick' STM32F4, connect BOOT0 pin to 3V pin and then you can erase or flash it.