Is there a reason why /var/log/lastlog is a huge sparse file (1.1TB)?

Question

I have read some question, that ask advice how to rsync sparse files efficiently mentioning the files /var/log/lastlog and /var/log/faillog. Indeed I myself have stumpled over those files being an "issue" as their being backup via rsync turns them to become "unsparse".

What I hence wonder is, what is the need/backgrounding motivation to have those files as sparse, huge files (in my case it was 1.1TB)?

Also in relationship to this a follow up: Since I was assuming them to be logfiles I do not care about excesively I truncated those files, did I corrupt anything with truncating those files ?

Solution 1

What I hence wonder is, what is the need/backgrounding motivation to have those files as sparse, huge files (in my case it was 1.1TB)?

This is how it's supposed to be.

/var/log/lastlog is not a log file like /var/log/syslog, and its name should be read as "last logins list" rather than "last logfile".

It's maintained by the pam_lastlog(8) module, and it's basically an array like this:

struct lastlog {
    time_t  ll_time;    // 4
    char    ll_line[UT_LINESIZE];   // 32
    char    ll_host[UT_HOSTSIZE];   // 256
} entry[UINT_MAX];

Sizes of the fields on a typical x86-64 machine are in comments; an entry should be 4 + 32 + 256 = 292 bytes.

Every time a program using the pam_lastlog(8) pam module is logging a user in, it will seek to uid * sizeof(struct lastlog) and overwrite the entry corresponding to that user.

did I corrupt anything with truncating those files ?

You did corrupt the output of the lastlog(1) command, which nobody is using anyway ;-)

Solution 2

I have seen this behaviour after we have introduced some (large) LDAP UID numbers co-existing with "normal" UNIX UID numbers in a linux (RHEL 7) machine.

Also, in the "CAVEATS" section of the manual page for "lastlog" it states that

  *"Large gaps in UID numbers will cause the lastlog program to run longer with no output to the screen"*

"lastlog" will appear to hang as it processes entries with the intermediate unused UIDs

Maybe this could be also related to the observed problem, as FreeIPA LDAP UID numbers assigned were much larger than the ones from Linux

Comments

[humanityANDpeace]

So then I must have gotten myself into an endless loop, since my cure for when those files have grown excessively huge has since been to truncate them back to size (which as if I read correctly your answer is supposed to be the original cause of the huge size in and on itself?). I am puzzled how this however reaches sizes up to 1TB since, I would supposed imparative reboots for kernel updates would have interrupted any logging processes anyway. I want to believe you, but I am struggling to really grasp the particularities of this, i..e. how a file becomes sparse + huge, can you enlight me?

[ilkkachu]

An apparent size larger than the amount of data stored is the very definition of a sparse file.

[ilkkachu]

How do you know the files ended up being sparse because of truncating without restarting the programs? Give some reasoning for that claim.

[ilkkachu]

Actually, writing in append mode doesn't create a sparse file like that, the writes go to the end of the file, where ever it is at that point. (A write in normal mode would work as you described though.) But anyway, you've described how a file becomes sparse if it's truncated while being open, but where does it say that they truncated the file without restarting the relevant processes? Also, given that the man page for lastlog explicitly says that the file is sparse, are you sure it's used like a regular log file to begin with?

[mosvy]

lastlog has nothing to do with log files. Your answer is completely off. Try man lastlog and man pam_lastlog.

[mosvy]

@ilkkachu lastlog is not a log file like /var/log/messages. It's normal to be a big sparse binary file, since it's basically an array of structs indexed by user id, and the user ids are not contiguous.

[humanityANDpeace]

thank you! I understand you are quite new to this stackexchange community. Most likely you meant very well providing some insight to this issue, but as @ilkkachu suggest the very question mentions the fact that we deal with sparse files hence I knew that they actually use less space on disk. Do not be too discouraged by the downvotes.

[Julie in Austin]

That's one of those things which hadn't aged well. The code was written for much smaller machines, many, many years ago. One of these days I should join the maintenance team. Or re-do a bunch of that code from scratch.

[Julie in Austin]

it's actually /var/log/faillog, which isn't actually a log file, in the sense of being continually appended to.