Linux – Connecting to Hyper-V Virtual Machines Running CentOS 7 via SSH is Slow

centoslinuxssh

I'm running Windows 10. I have the Hyper-V enabled. I've created three Hyper-V virtual machines running Centos 7; specifically:

Linux version 3.10.0-693.21.1.el7.x86_64
(builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat
4.8.5-16) (GCC) ) #1 SMP Wed Mar 7 19:03:37 UTC 2018

I have a NAT network setup. All three VM's are on that network. When I ssh from the host (my Windows 10 machine) to one of the VM's, it is PAINFULLY slow.

After issuing the ssh command, I wait for over 15 seconds for a password prompt. I then enter the password and wait for over 15-seconds again:

root@192.168.40.5's password:
Connection to 192.168.40.5 closed by remote host.
Connection to 192.168.40.5 closed.

When I establish the ssh connection with the -vvv switch, this is what I see:

debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
...waits...
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: pkalg rsa-sha2-512 blen 535
debug2: input_userauth_pk_ok: fp SHA256:3+l2I9j+Gv/Y2i3ksm00X3lIBjFEepj7uOCN4kQuy2A
debug3: sign_and_send_pubkey: RSA SHA256:3+l2I9j+Gv/Y2i3ksm00X3lIBjFEepj7uOCN4kQuy2A
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to ***.***.***.7 ([***.***.***.7]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
...waits...
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: receive packet: type 91
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x10
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug3: send packet: type 98
debug1: Sending environment.
debug3: Ignored env SHELL
debug3: Ignored env TERM
debug3: Ignored env USER
debug3: Ignored env NAME
debug3: Ignored env LS_COLORS
debug3: Ignored env HOSTTYPE
debug3: Ignored env PATH
debug3: Ignored env PWD
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env SHLVL
debug3: Ignored env HOME
debug3: Ignored env LOGNAME
debug3: Ignored env LESSOPEN
debug3: Ignored env LESSCLOSE
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
...waits...
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0

My machine's sshd_config:

HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
SyslogFacility AUTHPRIV
AuthorizedKeysFile      .ssh/authorized_keys
PasswordAuthentication yes
ChallengeResponseAuthentication no
GSSAPIAuthentication no
GSSAPICleanupCredentials no
UsePAM yes
X11Forwarding yes
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
Subsystem       sftp    /usr/libexec/openssh/sftp-server
  • I experience the slowness when connecting to any of the CentOS VMs
  • The described slowness occurs when connecting from the host (Windows 10 box) or one of the other CentOS VMs
  • The VMs are on the same subnet
  • The VMs are assigned static IP addresses

I want to resolve the slowness issue. Any tips or ideas?

Adding details of one of the CentOS VM's:

ParentCheckpointId                  : a76e4644-66c3-4d8a-b252-1e671f2c3c7c
ParentCheckpointName                : sbx-misc-app01 - (5/3/2018 - 12:20:00 AM)
VMName                              : sbx-misc-app01
VMId                                : a283e141-b32b-428d-8d3b-a64cf08b1ee8
CheckpointFileLocation              : C:\Hyper-V\Virtual Machines
ConfigurationLocation               : C:\Hyper-V\Virtual Machines
SmartPagingFileInUse                : False
SmartPagingFilePath                 : C:\Hyper-V\Virtual Machines
SnapshotFileLocation                : C:\Hyper-V\Virtual Machines
AutomaticStartAction                : StartIfRunning
AutomaticStartDelay                 : 0
AutomaticStopAction                 : Save
AutomaticCriticalErrorAction        : Pause
AutomaticCriticalErrorActionTimeout : 30
AutomaticCheckpointsEnabled         : True
CPUUsage                            : 0
MemoryAssigned                      : 1612709888
MemoryDemand                        : 1031798784
MemoryStatus                        : OK
NumaAligned                         : False
NumaNodesCount                      : 1
NumaSocketCount                     : 1
Heartbeat                           : OkApplicationsUnknown
IntegrationServicesState            : 
IntegrationServicesVersion          : 0.0
Uptime                              : 18:23:29.4550000
OperationalStatus                   : {Ok}
PrimaryOperationalStatus            : Ok
SecondaryOperationalStatus          : 
StatusDescriptions                  : {Operating normally}
PrimaryStatusDescription            : Operating normally
SecondaryStatusDescription          : 
Status                              : Operating normally
ReplicationHealth                   : NotApplicable
ReplicationMode                     : None
ReplicationState                    : Disabled
ResourceMeteringEnabled             : False
CheckpointType                      : Standard
EnhancedSessionTransportType        : VMBus
Groups                              : {}
Version                             : 8.3
VirtualMachineType                  : RealizedVirtualMachine
VirtualMachineSubType               : Generation2
Notes                               : 
State                               : Running
ComPort1                            : VMComPort (Name = 'COM 1', VMName = 'sbx-misc-app01') [Id = 'Microsoft:A283E141-B32B-428D-8D3B-A64CF08B1EE8\8E3A359F-559A-4B6A-98A9-1690A6100ED7\0', VMId = 
                                      'a283e141-b32b-428d-8d3b-a64cf08b1ee8']
ComPort2                            : VMComPort (Name = 'COM 2', VMName = 'sbx-misc-app01') [Id = 'Microsoft:A283E141-B32B-428D-8D3B-A64CF08B1EE8\8E3A359F-559A-4B6A-98A9-1690A6100ED7\1', VMId = 
                                      'a283e141-b32b-428d-8d3b-a64cf08b1ee8']
DVDDrives                           : {DVD Drive on SCSI controller number 0 at location 1}
FibreChannelHostBusAdapters         : {}
FloppyDrive                         : 
HardDrives                          : {Hard Drive on SCSI controller number 0 at location 0}
RemoteFxAdapter                     : 
VMIntegrationService                : {Guest Service Interface, Heartbeat, Key-Value Pair Exchange, Shutdown...}
DynamicMemoryEnabled                : True
MemoryMaximum                       : 8589934592
MemoryMinimum                       : 536870912
MemoryStartup                       : 8589934592
ProcessorCount                      : 2
BatteryPassthroughEnabled           : True
Generation                          : 2
IsClustered                         : False
ParentSnapshotId                    : a76e4644-66c3-4d8a-b252-1e671f2c3c7c
ParentSnapshotName                  : sbx-misc-app01 - (5/3/2018 - 12:20:00 AM)
Path                                : C:\Hyper-V\Virtual Machines
SizeOfSystemFiles                   : 69632
GuestControlledCacheTypes           : False
LowMemoryMappedIoSpace              : 134217728
HighMemoryMappedIoSpace             : 536870912
LockOnDisconnect                    : Off
CreationTime                        : 5/2/2018 10:23:06 PM
Id                                  : a283e141-b32b-428d-8d3b-a64cf08b1ee8
Name                                : sbx-misc-app01
NetworkAdapters                     : {Network Adapter}
CimSession                          : CimSession: .
ComputerName                        : Thinger
IsDeleted                           : False

Here is the network setup code…

$switch_name = 'sbx-ext01'
$nat_name = 'net-sbx-ext01'
$nat_subnet_prefix_length = 24
$gateway_ip_address = '192.168.40.0'

New-VMSwitch -SwitchName $switch_name -SwitchType Internal

$interface_index = $(Get-NetAdapter -Name "*${switch_name}*").ifIndex

New-NetIPAddress `
    -IPAddress 192.168.40.1 `
    -PrefixLength 24 `
    -InterfaceIndex $interface_index

New-NetNat -Name $nat_name `
    -InternalIPInterfaceAddressPrefix $gateway_ip_address/$nat_subnet_prefix_length

Best Answer

It is not clear what "painfully slow" means. Do you mean that once you are connected via ssh that it is slow? Or do you mean that, once you enter you password and hit enter that it takes a long time, 30 seconds, to get a prompt?

If it is that it takes you 30-60 seconds to get a prompt this may be due to the SSH server attempting to reverse resolve the IP address of the connecting client. I have seen this happen when there is an issue with the DNS servers configured in resolv.conf on the server, e.g., they timeout.

One solution is to set the following in ssd_config on each of the machines:

UseDNS no

This is enabled by default.

Another solution is to correct the nameserver entries in resolv.conf

If this does not fix your issue would you please add more detail about what "painfully slow" means?