Electrical – PCB wire connectors

I can't answer the 8051 specific questions, but I can help with your first problem.

The standard method for doing this in industry (at least as far as I can tell) is to use pogo pins:
_{(source: sparkfun.com)}

The gold part sticks through some protoboard or another PCB. You connect your programmer to this circuit. This piece of PCB (or something thicker and sturdier like masonite or acrylic if you're doing hundreds of boards) also has larger pins which go through holes in your target board, to precisely locate your target board above the pogo pins. Light down pressure on the board makes contact between the spring-loaded pogo pins and the test points on the target PCB.

The company I work at uses this kind of pins to program every one of the millions of circuit boards they make every year, and every trace has a tiny test point which is probed by a tester through these pins.

If you want a debug header, there are other standard methods. I'm totally unfamiliar with this chip, so I can't answer that for you. 8, 14, and 20 pin headers are common for other devices.

EDIT: I took a quick look at the datasheet, and found on page 10:

3. In-System Programming (ISP) Function
The SM5964 can generate flash control signal by internal hardware circuit. User utilize flash control register, flash address register and flash data register to perform the ISP function without removing the SM5964 from the system. The SM5964 provides internal flash control signals which can do flash program/chip erase/page erase/protect functions. User need to design and use any kind of interface which SM5964 can input data. User then utilize ISP service program to perform the flash program/chip erase/page erase/protect functions.
3.1 ISP Service Program
The ISP service program is a user developed firmware program which resides in the ISP service program space. After user developed the ISP service program, user then determine the size of the ISP service program. User need to program the ISP service program in the SM5964 for the ISP purpose. The ISP service program were developed by user so that it should includes any features which relates to the flash memory programming function as well as communication protocol between SM5964 and host device which output data to the SM5964. For example, if user utilize UART interface to receive/transmit data between SM5964 and host device, the ISP service program should include baud rate, checksum or parity check or any error-checking mechanism to avoid data transmission error. The ISP service program can be initiated under SM5964 active or idle mode. It can not be initiated under power down mode.

So, it looks like you can provide a bootloader program to read in the data from any interface you like. This can be some GPIO or the UART that you access with pogo pins, or go to an existing connector on the board. How do you get this program to run?

3.4 Initiate ISP Service Program
To initiate the ISP service program is to load the program counter (PC) with start address of ISP service program and execute it. There are two ways to do so:
(1) Blank reset. Hardware reset with first flash address blank ($0000=#FFH) will load the PC with start address of ISP service program.
(2) Execute jump instruction can load the start address of the ISP service program to PC.

However, I have no idea how you're supposed to get the ISP program on the blank chip. This page has a few programs and programmers, the MSM9042 looks closer to what you want for in-circuit (not socketed) programming.

Have look into these suppliers: TE connectivity (used to be AMP), Deutsch connectors, Amphenol, Hypertronics.

Also, circular MIL connectors may be useful for you. Many manufacturers make them.