When running Enter-PSSession COMPUTERNAME with Enable-PSWSManCombinedTrace, I see the following relevant messages in the Windows Remote Management Operational log:
WSMan operation Get failed, error code 2150859046
WinRM cannot complete the operation. Verify that the specified computer name is valid, that the computer is accessible over the network, and that a firewall exception for the WinRM service is enabled and allows access from this computer. By default, the WinRM firewall exception for public profiles limits access to remote computers within the same local subnet.
The WinRM protocol operation failed due to the following error: The metadata failed to be retrieved from the server, due to the following error: WinRM cannot complete the operation. Verify that the specified computer name is valid, that the computer is accessible over the network, and that a firewall exception for the WinRM service is enabled and allows access from this computer. By default, the WinRM firewall exception for public profiles limits access to remote computers within the same local subnet. .
And sometimes:
The client got a timeout from the network layer (ERROR_WINHTTP_TIMEOUT)
COMPUTERNAME is a 2012 R2 Core server on the domain, under the same group policies as many other server for which Remote PowerShell, Server Manager, et al are working fine. I can RDP to this system, I can get WMI data from it (e.g. Get-WmiObject -ComputerName COMPUTERNAME -Class Win32_OperatingSystem returns what it should), and in every other way, it seems to be running just fine.
While it's set via Group Policy already, I've tried (numerous times an ways) to enable WinRM and Remote PowerShell, such as Enable-PSRemoting, or invoking the attendant steps this command executes individually.
I've changed to a different network interface, I've ensured other systems on the same network segment don't exhibit these symptoms, I've followed the advice of Get-Help about_Remote_Troubleshooting, and I've sacrified the requisite goat to Baal. Nothing helps.
These symptoms are reproducible from any domain client to this server, or if you contact the server by IP (after putting it in TrustedHosts). No other server exhibits this issue. There is no software or configuration (all the way down to FW rules enabled and features installed) that aren't on at least 2 other servers in my environment.
Any ideas?
MOST RECENT FINDINGS:
netsh http show iplist returns 127.0.0.1 on the non-working system, but returns nothing on the working systems.
As @out-null correctly pointed out in the comments, the fact that 5985 is listening on 127.0.0.1 is a problem. I've since excluded this system from the GPO configuring our WinRM settings and manually created the listener:
winrm create winrm/config/Listener?Address=*+Transport=HTTP
However, the result in netstat is the same. Note the output of winrm e below, where the IP is listed as a listener.
Still stumped on this one…
Original evidences/sanity checks
$> winrm e winrm/config/listener
Listener [Source="GPO"]
Address = *
Transport = HTTP
Port = 5985
Hostname
Enabled = true
URLPrefix = wsman
CertificateThumbprint
ListeningOn = 10.11.10.117, 127.0.0.1, 169.254.34.30, 169.254.47.200, 169.254.236.165, ::1, fe80::5efe:10.115.63.10 7%16, fe80::5efe:169.254.34.30%45, fe80::28b8:be74:53c:2fc8%12, fe80::69a9:e404:12bd:63c0%15, fe80::7cf2:ec84:332f:221e%14, fe80::cdc6:5ca0:6ae2:eca5%13
$> netsh winhttp show proxy
Current WinHTTP proxy settings:
Direct access (no proxy server).
$> Get-NetFirewallRule WINRM-HTTP-In-TCP | fl *
Name : WINRM-HTTP-In-TCP
ID : WINRM-HTTP-In-TCP
Group : @FirewallAPI.dll,-30267
Platform : {}
LSM : False
DisplayName : Windows Remote Management (HTTP-In)
Enabled : True
Profile : Domain, Private
Direction : Inbound
Action : Allow
EdgeTraversalPolicy : Block
PrimaryStatus : OK
Status : The rule was parsed successfully from the store. (65536)
EnforcementStatus : NotApplicable
PolicyStoreSourceType : Local
Caption :
Description : Inbound rule for Windows Remote Management via WS-Management. [TCP 5985]
ElementName : @FirewallAPI.dll,-30253
InstanceID : WINRM-HTTP-In-TCP
CommonName :
PolicyKeywords :
PolicyDecisionStrategy : 2
PolicyRoles :
ConditionListType : 3
CreationClassName : MSFT|FW|FirewallRule|WINRM-HTTP-In-TCP
ExecutionStrategy : 2
Mandatory :
PolicyRuleName :
Priority :
RuleUsage :
SequencedActions : 3
SystemCreationClassName :
SystemName :
DisplayGroup : Windows Remote Management
LocalOnlyMapping : False
LooseSourceMapping : False
Owner :
Platforms : {}
PolicyStoreSource : PersistentStore
Profiles : 3
RuleGroup : @FirewallAPI.dll,-30267
StatusCode : 65536
PSComputerName :
CimClass : root/standardcimv2:MSFT_NetFirewallRule
CimInstanceProperties : {Caption, Description, ElementName, InstanceID...}
CimSystemProperties : Microsoft.Management.Infrastructure.CimSystemProperties
COMPUTERNAME$> netstat -anp tcp
Active Connections
Proto Local Address Foreign Address State
TCP 0.0.0.0:135 0.0.0.0:0 LISTENING
TCP 0.0.0.0:445 0.0.0.0:0 LISTENING
TCP 0.0.0.0:3389 0.0.0.0:0 LISTENING
TCP 0.0.0.0:49152 0.0.0.0:0 LISTENING
TCP 0.0.0.0:49153 0.0.0.0:0 LISTENING
TCP 0.0.0.0:49154 0.0.0.0:0 LISTENING
TCP 0.0.0.0:49155 0.0.0.0:0 LISTENING
TCP 0.0.0.0:49174 0.0.0.0:0 LISTENING
TCP 0.0.0.0:49178 0.0.0.0:0 LISTENING
TCP 0.0.0.0:49191 0.0.0.0:0 LISTENING
TCP 10.11.10.117:135 192.168.5.71:64570 ESTABLISHED
TCP 10.11.10.117:135 192.168.5.71:64571 ESTABLISHED
TCP 10.11.10.117:135 192.168.5.71:64572 ESTABLISHED
TCP 10.11.10.117:139 0.0.0.0:0 LISTENING
TCP 10.11.10.117:3389 10.1.1.2:57970 ESTABLISHED
TCP 10.11.10.117:49153 10.1.1.2:58100 ESTABLISHED
TCP 10.11.10.117:50601 192.168.5.111:8014 ESTABLISHED
TCP 10.11.10.117:56508 192.168.5.177:445 ESTABLISHED
TCP 127.0.0.1:5985 0.0.0.0:0 LISTENING
TCP 127.0.0.1:47001 0.0.0.0:0 LISTENING
TCP 169.254.34.30:139 0.0.0.0:0 LISTENING
SOME-WORKING-COMPUTER$> netstat -anp tcp
Active Connections
Proto Local Address Foreign Address State
TCP 0.0.0.0:135 0.0.0.0:0 LISTENING
TCP 0.0.0.0:445 0.0.0.0:0 LISTENING
TCP 0.0.0.0:5985 0.0.0.0:0 LISTENING
TCP 0.0.0.0:47001 0.0.0.0:0 LISTENING
TCP 0.0.0.0:49152 0.0.0.0:0 LISTENING
TCP 0.0.0.0:49153 0.0.0.0:0 LISTENING
TCP 0.0.0.0:49154 0.0.0.0:0 LISTENING
TCP 0.0.0.0:49155 0.0.0.0:0 LISTENING
TCP 0.0.0.0:49158 0.0.0.0:0 LISTENING
TCP 0.0.0.0:49187 0.0.0.0:0 LISTENING
TCP 0.0.0.0:49192 0.0.0.0:0 LISTENING
TCP 0.0.0.0:49199 0.0.0.0:0 LISTENING
TCP 0.0.0.0:49213 0.0.0.0:0 LISTENING
TCP 192.168.5.11:139 0.0.0.0:0 LISTENING
TCP 192.168.5.11:5985 10.1.1.2:58153 ESTABLISHED
TCP 192.168.5.11:5985 10.1.1.2:58154 ESTABLISHED
TCP 192.168.5.11:5985 10.1.1.2:58156 ESTABLISHED
TCP 192.168.5.11:49203 192.168.5.177:49210 ESTABLISHED
TCP 192.168.5.11:49213 192.168.5.177:52784 ESTABLISHED
TCP 192.168.5.11:49213 192.168.5.177:54507 ESTABLISHED
TCP 192.168.5.11:49213 192.168.5.177:59034 ESTABLISHED
TCP 192.168.5.11:52905 192.168.5.177:49210 TIME_WAIT
TCP 192.168.5.11:52906 192.168.5.177:49210 TIME_WAIT
TCP 192.168.5.11:52907 192.168.5.111:8014 ESTABLISHED
TCP 192.168.5.11:52910 192.168.5.177:49210 TIME_WAIT
TCP 192.168.5.11:52915 192.168.5.177:49210 TIME_WAIT
TCP 192.168.5.11:52918 192.168.5.177:49210 TIME_WAIT
TCP 192.168.5.11:52920 192.168.5.177:49210 TIME_WAIT
TCP 192.168.5.11:52922 192.168.5.177:49210 ESTABLISHED
TCP 192.168.5.11:52923 192.168.5.177:49210 ESTABLISHED
TCP 192.168.5.11:52924 192.168.5.177:49210 ESTABLISHED
TCP 192.168.5.11:52925 192.168.5.177:49210 ESTABLISHED
TCP 192.168.5.11:52926 192.168.5.177:49210 ESTABLISHED
TCP 192.168.5.11:52927 192.168.5.177:49210 ESTABLISHED
TCP 192.168.5.11:54938 192.168.6.8:49157 ESTABLISHED
TCP 192.168.5.11:62632 192.168.5.177:49210 ESTABLISHED
TCP 192.168.5.11:64307 192.168.6.8:389 ESTABLISHED
Best Answer
Finally solved it, helped by the evidence I recently added to the question:
On systems where this was working, that list was empty. This seemed rather counter-intuitive to me at first. Nevertheless, I gave this a go:
Immediately after, I notice this output from
netstat:And indeed, WinRM works like it should.
I surmise, via testing, that if no HTTP listener is configured, then all HTTP listeners will bind to the default entity:
0.0.0.0. Because a loopback address was configured as a listener address, the listener was binding to this address instead.At some point, I must have taken some action that caused this configuration, though I'm not sure how. At any rate, it's working fine now. Thanks all.