SSH X11 forwarding does not work. Why

This is a debugging question. When you ask for clarification please make sure it is not already covered below.

I have 4 machines: Z, A, N, and M.

To get to A you have to log into Z first.

To get to M you have to log into N first.

The following works:

ssh -X Z xclock
ssh -X Z ssh -X Z xclock
ssh -X Z ssh -X A xclock
ssh -X N xclock
ssh -X N ssh -X N xclock

But this does not:

ssh -X N ssh -X M xclock
Error: Can't open display: 

The $DISPLAY is clearly not set when logging in to M. The question is why?

Z and A share same NFS-homedir. N and M share the same NFS-homedir. N's sshd runs on a non standard port.

$ grep X11 <(ssh Z cat /etc/ssh/ssh_config) 
ForwardX11 yes
# ForwardX11Trusted yes

$ grep X11 <(ssh N cat /etc/ssh/ssh_config) 
ForwardX11 yes
# ForwardX11Trusted yes

N:/etc/ssh/ssh_config == Z:/etc/ssh/ssh_config and
M:/etc/ssh/ssh_config == A:/etc/ssh/ssh_config

/etc/ssh/sshd_config is the same for all 4 machines (apart from Port and login permissions for certain groups).

If I forward M's ssh port to my local machine it still does not work:

terminal1$ ssh -L 8888:M:22 N
terminal2$ ssh -X -p 8888 localhost xclock
Error: Can't open display:

A:.Xauthority contains A, but M:.Xauthority does not contain M.

xauth is installed in /usr/bin/xauth on both A and M.

xauth is being run when logging in to A but not when logging in to M.

ssh -vvv does not complain about X11 or xauth when logging in to A and M. Both say:

debug2: x11_get_proto: /usr/bin/xauth  list :0 2>/dev/null
debug1: Requesting X11 forwarding with authentication spoofing.
debug2: channel 0: request x11-req confirm 0
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.

I have a feeling the problem may be related to M missing in M:.Xauthority (caused by xauth not being run) or that $DISPLAY is somehow being disabled by a login script, but I cannot figure out what is wrong.

— update 20110628

I did not know about sshrc so that was a good guess. But alas, not the problem here. It does not exist on any of the 4 machines:

$ ls ~/.ssh/rc /etc/ssh/sshrc
ls: cannot access /home/tange/.ssh/rc: No such file or directory
ls: cannot access /etc/ssh/sshrc: No such file or directory

As mentioned the $DISPLAY variable is not set on M, but is fine on A:

$ ssh -X N ssh -X M 'echo \$DISPLAY'
<<empty>>
$ ssh -X Z ssh -X A 'echo \$DISPLAY'
localhost:14.0

The difference in output from a working session and a non-working session (Note: There are no warnings about X forwarding or xauth in the non-working session):

$ stdout ssh -X Z ssh -vX A 'echo \$DISPLAY' >/tmp/a
$ stdout ssh -X N ssh -vX M 'echo \$DISPLAY' >/tmp/b
$ diff /tmp/a /tmp/b
4c4
< debug1: Connecting to A [1.1.1.5] port 22.
---
> debug1: Connecting to M [1.1.3.3] port 22.
23,24c23,24
< debug1: Host 'A' is known and matches the RSA host key.
< debug1: Found key in /home/tange/.ssh/known_hosts:35
---
> debug1: Host 'M' is known and matches the RSA host key.
> debug1: Found key in /home/tange/.ssh/known_hosts:1
43d42
< debug1: Sending env LC_ALL = en_US.UTF-8
46c45
< localhost:14.0
---
> 
53,54c52,53
< Transferred: sent 2384, received 2312 bytes, in 0.2 seconds
< Bytes per second: sent 10714.8, received 10391.2
---
> Transferred: sent 2336, received 2296 bytes, in 0.0 seconds
> Bytes per second: sent 54629.1, received 53693.7

Instaling lsh-server instead of openssh-server on M fixes the X-forwarding, but is an unacceptable solution.