Ubuntu – Seeking clean resolution for zombie who caused by git fetch / ubuntu motd

gitJenkinsUbuntuzombie

I have jenkins and gitosis running on the same [Ubuntu 10.04] server. Jenkins fetches git commits from a repo served by gitosis (also running on the same server). Many repos, actually, since there are many jobs in Jenkins. Every now and then I end up with a zombie who process, where the root cause appears to be some Ubuntu-specific scripts to update /etc/motd. Here's what (part of) the process table looked like earlier today:

jenkins  30042     1  0 Mar19 ?        00:00:00 git fetch -t gitosis@example.com:testRepo.git +refs/heads/*:refs/remotes/origin/*
jenkins  30060 30042  0 Mar19 ?        00:00:00 ssh gitosis@example.com git-upload-pack 'testRepo.git'
root     30066   703  0 Mar19 ?        00:00:00 sshd: gitosis [priv]
root     30096 30066  0 Mar19 ?        00:00:00 sh -c /usr/bin/env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin /bin/run-parts --lsbsysinit /etc/update-motd.d > /var/run/motd.new
root     30097 30096  0 Mar19 ?        00:00:00 /bin/run-parts --lsbsysinit /etc/update-motd.d
root     30129 30097  0 Mar19 ?        00:00:00 /bin/sh /etc/update-motd.d/50-landscape-sysinfo
root     30148 30129  0 Mar19 ?        00:00:00 /usr/bin/python /usr/bin/landscape-sysinfo
root     30451 30148  0 Mar19 ?        00:00:00 [who] <defunct>

To clear it up I basically just killed every parent of the zombie who, then the zombie disappeared.

Any ideas on how to prevent this from happening or gracefully deal with it when it does?

Is there some way to disable these motd scripts sometimes? Like, only when Jenkins is doing git fetch? Actually, what would be perfect is if I could somehow disable the motd scripts whenever the gitosis user tries to SSH in (e.g. whenever any git client/server interaction takes place).

Maybe there's a straightforward way with Bash to kill any git process left running for over an hour? Ugh.

I generally like the handy motd provided by these scripts so I don't want to completely disable them. Sure would be nice if they'd timeout on their own. Why the heck is the motd updated on every login? Once a day seems like plenty.

I don't really want to switch to gitolite or a different git server, gitosis otherwise works fine. I don't want to switch to another protocol besides SSH; gitosis relies on SSH. Since gitosis is running on the same server, I should be able to refer to the git repositories using file:/// syntax. I'll give that a shot now. Update… nope, that won't work since that bypasses gitolite and won't allow jenkins to push tags back upstream. I could give jenkins more ACLs, but I'd rather stick with gitolite.

Side note: this won't help since it appears the problem occurs after the SSH connection is established.

Best Answer

If I'm reading this right, and I like to think that I am..

Then you're asking for a very roundabout way to disable update-motd on ubuntu.

Have a look in /etc/cron.d for /etc/cron.d/update-motd, and if you've got it, mv it to /tmp. That should stop it.

I don't think you should kill long-running processes. I do think that no harm can come out of disabling update-motd though.

You might want to have a look inside /etc/update-motd.d

/usr/sbin/update-motd uses run-parts to execute each script in /etc/update-motd.d in lexigraphic order, concatenating the results with the message-of-the-day header, /etc/motd.tail.

In this way, users, or even other packages can drop scripts into /etc/update-motd.d to affect the MOTD. (Lifted from https://wiki.ubuntu.com/UpdateMotd)

Here's a blogpost from someone else who hates update-motd as much as you ;)

Related Topic