Run SCVMM 2012 and SCDPM 2012 on the same server

It should be fine unless you're using the performance and resource optimization features, but I would not consider it a "supported" configuration.

Is there any reason you cannot remove them from the original SCVMM and then add them with the external server, with no period of overlap? If you're just trying to save time, near 100% of the capabilities of SCVMM can be accessed, although less efficiently, through a combination of the Hyper-V Manager and Failover Cluster Manager, in the case of clustered scenarios.

This one saved the day:

http://social.technet.microsoft.com/Forums/en-US/dataprotectionmanager/thread/913e1de9-4b10-4479-9c6a-95c5b504e1d8/

It turns out that I had been a bit too eager when restricting Windows Firewall. After inspecting MSDPMAgentInstall.log I noticed this error:

ConfigureFirewall method return hr =0x80004005

A Google search took me to the link above.

I had deleted some default rules in Windows Firewall and the installer failed when it tried to modify those rules, even if the firewall was off. Copying the answer in here in case the referenced forum topic disappears.

For those looking for the quick solution to “why does setdpmserver.exe fail” you need to first figure out if it’s the Windows Firewall rule set causing the problem. Note: even if you have the firewall turned off you still need specific rules in the rule set in order for setdpmserver.exe to succeed. Run these netsh commands:

netsh advfirewall firewall set rule group=\"@FirewallAPI.dll,-29502\" new enable=yes
netsh advfirewall firewall set rule group=\"@FirewallAPI.dll,-34251\" new enable=yes
netsh advfirewall firewall add rule name=dpmra dir=in program=\"%PROGRAMFILES%\\Microsoft Data Protection Manager\\DPM\\bin\\DPMRA.exe\" profile=Any action=allow
netsh advfirewall firewall add rule name=DPMRA_DCOM_135 dir=in action=allow  protocol=TCP localport=135 profile=Any

If the first two commands result in errors you need to re-add some default FW rules. Copy below to a .reg file and run to add them back. Then restart the Windows Firewall service and try the above netsh again. If they work then setdpmserver.exe should work.

Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy\FirewallRules]
"RemoteSvcAdmin-In-TCP"="v2.0|Action=Allow|Active=TRUE|Dir=In|Protocol=6|LPort=RPC|App=%SystemRoot%\\system32\\services.exe|Name=@FirewallAPI.dll,-29503|Desc=@FirewallAPI.dll,-29506|EmbedCtxt=@FirewallAPI.dll,-29502|Edge=FALSE|"
"RemoteSvcAdmin-NP-In-TCP"="v2.0|Action=Allow|Active=TRUE|Dir=In|Protocol=6|LPort=445|App=System|Name=@FirewallAPI.dll,-29507|Desc=@FirewallAPI.dll,-29510|EmbedCtxt=@FirewallAPI.dll,-29502|Edge=FALSE|"
"RemoteSvcAdmin-RPCSS-In-TCP"="v2.0|Action=Allow|Active=TRUE|Dir=In|Protocol=6|LPort=RPC-EPMap|App=%SystemRoot%\\system32\\svchost.exe|Svc=RPCSS|Name=@FirewallAPI.dll,-29515|Desc=@FirewallAPI.dll,-29518|EmbedCtxt=@FirewallAPI.dll,-29502|Edge=FALSE|"
"WMI-RPCSS-In-TCP"="v2.0|Action=Allow|Active=TRUE|Dir=In|Protocol=6|LPort=135|App=%SystemRoot%\\system32\\svchost.exe|Svc=rpcss|Name=@FirewallAPI.dll,-34252|Desc=@FirewallAPI.dll,-34253|EmbedCtxt=@FirewallAPI.dll,-34251|Edge=FALSE|"
"WMI-WINMGMT-In-TCP"="v2.0|Action=Allow|Active=TRUE|Dir=In|Protocol=6|App=%SystemRoot%\\system32\\svchost.exe|Svc=winmgmt|Name=@FirewallAPI.dll,-34254|Desc=@FirewallAPI.dll,-34255|EmbedCtxt=@FirewallAPI.dll,-34251|Edge=FALSE|"
"WMI-WINMGMT-Out-TCP"="v2.0|Action=Allow|Active=TRUE|Dir=Out|Protocol=6|App=%SystemRoot%\\system32\\svchost.exe|Svc=winmgmt|Name=@FirewallAPI.dll,-34258|Desc=@FirewallAPI.dll,-34259|EmbedCtxt=@FirewallAPI.dll,-34251|Edge=FALSE|"
"WMI-ASYNC-In-TCP"="v2.0|Action=Allow|Active=TRUE|Dir=In|Protocol=6|App=%systemroot%\\system32\\wbem\\unsecapp.exe|Name=@FirewallAPI.dll,-34256|Desc=@FirewallAPI.dll,-34257|EmbedCtxt=@FirewallAPI.dll,-34251|Edge=FALSE|"

Thanks to sonicbum @ social.technet.microsoft.com