I have five CentOS 6 linux systems at work, and encountered a rather strange issue that only seems to happen with my userid across all the linux systems I have… This is an example of the problem from entries I excepted from the last
command…
mpenning pts/19 Fri Nov 16 10:32 - 10:35 (00:03)
mpenning pts/17 Fri Nov 16 10:21 - 10:42 (00:21)
bill pts/15 sol-bill.local Fri Nov 16 10:19 - 10:36 (00:16)
mpenning pts/1 192.0.2.91 Fri Nov 16 10:17 - 10:49 (12+00:31)
kkim14 pts/14 192.0.2.225 Thu Nov 15 18:02 - 15:17 (4+21:15)
gduarte pts/10 192.0.2.135 Thu Nov 15 12:33 - 08:10 (11+19:36)
gduarte pts/9 192.0.2.135 Thu Nov 15 12:31 - 08:10 (11+19:38)
kkim14 pts/0 :0.0 Thu Nov 15 12:27 - 15:17 (5+02:49)
gduarte pts/6 192.0.2.135 Thu Nov 15 11:44 - 08:10 (11+20:25)
kkim14 pts/13 192.0.2.225 Thu Nov 15 09:56 - 15:17 (5+05:20)
kkim14 pts/12 192.0.2.225 Thu Nov 15 08:28 - 15:17 (5+06:49)
kkim14 pts/11 192.0.2.225 Thu Nov 15 08:26 - 15:17 (5+06:50)
dspencer pts/8 192.0.2.130 Wed Nov 14 18:24 still logged in
mpenning pts/18 alpha-console-1. Mon Nov 12 14:41 - 14:46 (00:04)
You can see two of my pts login entries above that do not have a source IP address associated with them. My CentOS machines have as many as six other users that share the systems. Approximately 10% of my logins see this issue, but no other usernames exhibit this behavior. There is no entry in /var/log/secure
for the entries without a source IP address.
Questions
Given the kind of scripts I keep on these systems (which control much of our network infrastructure), I'm a little spooked by this and would like to understand what would cause my logins to occasionally miss source addresses.
- Why does
last -i
show0.0.0.0
for pts line entries (also see this answer) - Is there anything (other than malicious activity) that would reasonably explain the behavior?
- Other than bash history timestamping, are there other things I can do to track the issue down?
Informational
Since this started happening, I enabled bash
history time-stamping (i.e. HISTTIMEFORMAT="%y-%m-%d %T "
in .bash_profile
) and also added a few other bash history hacks; however, that does not give clues to what happened during the previous occurrences.
All the systems run CentOS 6.3…
[mpenning@typo ~]$ uname -a
Linux typo.local 2.6.32-279.9.1.el6.x86_64 #1 SMP Tue Sep 25 21:43:11 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
[mpenning@typo ~]$
EDIT
If I use last -i mpenning
, I see entries like this…
mpenning pts/19 0.0.0.0 Fri Nov 16 10:32 - 10:35 (00:03)
mpenning pts/17 0.0.0.0 Fri Nov 16 10:21 - 10:42 (00:21)
Note for those trying to answer: I have not logged in with the screen
command or the GUI. All my logins are from SSH; to receive the bounty award, you must cite authoritative references to explain the last -i
0.0.0.0
entries sourced only via SSH.
EDIT 2 (for ewwhite's questions)
/etc/resolv.conf
(note that I used .local
addrs in last
output above to hide my company's info)
[mpenning@sasmars network]$ cat /etc/resolv.conf
nameserver 192.0.2.40
nameserver 192.0.2.60
domain mycompany.com
search mycompany.com
[mpenning@sasmars network]$
/etc/hosts
info (note that this customized hosts file only exists on one of the machines that has these problems)
[mpenning@sasmars network]$ cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
192.0.2.44 sasmars.mycompany.com sasmars
::1 localhost6.localdomain6 localhost6
## Temporary kludge until I add reverse hostname mappings...
## Firewalls
192.0.2.254 a2-inet-fw1
192.0.2.253 a2-inet-fw2
192.0.2.254 a2-wan-fw1
192.0.2.253 a2-wan-fw2
192.0.2.201 a2-fab-fw1
192.0.2.202 a2-fab-fw2
192.0.2.203 t1-eds-fw1
192.0.2.42 sasvpn
192.0.2.246 sasasa1
192.0.2.10 sasoutfw1
## Wireless
192.0.2.6 saswcs1
192.0.2.2 l2wlc3
192.0.2.4 l2wlc4
192.0.2.12 f2wlc5
192.0.2.16 f2wlc6
192.0.2.14 f2wlc1
192.0.2.8 f2wlc2
[mpenning@sasmars network]$
sftp
Output from /var/log/secure
*
Dec 26 10:36:37 sasmars sshd[26016]: pam_sm_authenticate: called (pam_tacplus v1.3.7)
Dec 26 10:36:37 sasmars sshd[26016]: pam_sm_authenticate: user [mpenning] obtained
Dec 26 10:36:37 sasmars sshd[26016]: tacacs_get_password: called
Dec 26 10:36:37 sasmars sshd[26016]: tacacs_get_password: obtained password
Dec 26 10:36:37 sasmars sshd[26016]: pam_sm_authenticate: password obtained
Dec 26 10:36:37 sasmars sshd[26016]: pam_sm_authenticate: tty [ssh] obtained
Dec 26 10:36:37 sasmars sshd[26016]: pam_sm_authenticate: rhost [192.0.2.91] obtained
Dec 26 10:36:37 sasmars sshd[26016]: pam_sm_authenticate: trying srv 0
Dec 26 10:36:38 sasmars sshd[26016]: Accepted password for mpenning from 192.0.2.91 port 55118 ssh2
Dec 26 10:36:38 sasmars sshd[26016]: pam_sm_setcred: called (pam_tacplus v1.3.7)
Dec 26 10:36:38 sasmars sshd[26016]: pam_unix(sshd:session): session opened for user mpenning by (uid=0)
Dec 26 10:36:38 sasmars sshd[26018]: pam_sm_setcred: called (pam_tacplus v1.3.7)
Dec 26 10:36:38 sasmars sshd[26018]: subsystem request for sftp
Dec 26 10:37:20 sasmars sshd[26016]: pam_unix(sshd:session): session closed for user mpenning
Dec 26 10:37:20 sasmars sshd[26016]: pam_sm_setcred: called (pam_tacplus v1.3.7)
FINAL RESOLUTION
See my answer below
Best Answer
script
behavior differences between RedHat and DebianLinked Libraries
CentOS 6.3 - script (util-linux-ng 2.17.2)
Ubuntu 12.04 - script (util-linux 2.20.1)
PTY
Base on upstream source code,
script
from both versions do open new pty. Following is test.Ubuntu 12.04
Ubuntu 12.04
script
did open a new pts(2). It just did not update/var/log/wtmp
.CentOS 6
I am skipping the test as we already know that
script
do open pty and register with wtmp.libutemper
So the main difference seems to be the extra library(
libutempter.so.0
) CentOSscript
linked with.Test with Ubuntu 12.04
Compiling
script
with libutempterTesting
Before running
script
Within
script
After
script
endThe root cause of emtpy hostname
And yes,
script.c
do create thewtmp
entry with empty hostname. Look at following code block inutil-linux-2.20.1/term-utils/script.c
Line:245-247Base on
libutempter-1.1.5/utempter.h
So
script.c
is actually passing empty hostname intoutempter_add_record
.RedHat Backport
The interesting thing is, upstream
util-linux-ng-2.17.2
actually does not supportlibutempter
. Seems Redhat decided to added that support back.The above command return empty result.
Conclusion
So the difference in behavior between the two distros is not a bug, but a choice. RedHat decided to support that feature, while Debian skipped it.