SSH ForwardAgent is receiving “Connection closed by remote host”

I'm trying to connect to a remote server using SSH ForwardAgent but I'm facing all the time the same issue :

ssh_exchange_identification: Connection closed by remote host

I've setup my ~/.ssh/config as follow :

#Proxy
Host my.remote.proxy.com
Hostname IPProxy.IP.IP.IP
User my.user
IdentityFile ~/.ssh/id_rsa
ForwardAgent yes

#Remote server
Host my.remote.server.com
Hostname IPRemote.IP.IP.IP
User my.user
IdentityFile ~/.ssh/id_rsa
ProxyCommand ssh my.user@my.remote.proxy.com  nc -w 10 %h %p 2> /dev/null

I'm able to ssh correctly into my.remote.proxy.com. I'm able to ssh correctly from my.remote.proxy.com to my.remote.server.com using ssh (and the private on test purpose).

My problem is that I'm not able to ssh from my host to my.remote.server.com using forwardagent.

I've setup /etc/ssh/sshd_config for both my.remote.proxy.com and my.remote.server.com as follow :

AllowAgentForwarding yes
AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes

I checked and both server are using a working version of openssl with forwardagent :

openssh-7.4p1-13.el7_4.x86_64
openssh-clients-7.4p1-13.el7_4.x86_64
openssh-server-7.4p1-13.el7_4.x86_64

The /var/log/secure from my.remote.proxy.com are returning :

Nov 15 04:39:07 [localhost] sshd[7866]: Accepted publickey for my.user
Nov 15 04:39:07 [localhost] sshd[7866]: pam_unix(sshd:session): session opened for user my.user by (uid=0)
Nov 15 04:39:08 [localhost] sshd[7869]: Received disconnect from IPPUBLIC port 61378:11: disconnected by user
Nov 15 04:39:08 [localhost] sshd[7869]: Disconnected from IPPUBLIC port 61378
Nov 15 04:39:08 [localhost] sshd[7866]: pam_unix(sshd:session): session closed for user my.user

Nothing is showing up from /var/log/secure on my.remote.server.com.