BASH history truncated to 500 lines on each login

Question

For some reason, I cannot get my system to keep my BASH history after a reboot. Here are the relevant sections of my ~/.bashrc:

shopt -s histappend
PROMPT_COMMAND='history -a; updateWindowTitle'
export HISTCONTROL=ignoredups
export HISTSIZE=9999
export HISTFILESIZE=999999
export HISTFILE="$HOME/.bash_history"

As far as I can tell those are all the necessary options (I know I used to be able to keep history across multiple reboots without all of these in the past). However, despite having added these options several reboots ago, I still loose most of my history after a reboot. It is not empty, but it does not have the 9999 lines I had before rebooting.

Before anyone complains, yes I have read these questions. I have implemented some of their suggestions as listed above, the rest were either unhelpful or not relevant:

• Bash history loss when using histappend
• How to prevent Bash from altering history?
• What determines what shows up in the bash history command?
• How do I keep my bash history across sessions?
• save bash history, regularly

On the off chance that there may be other relevant commands in there, you can view my entire ~/.bashrc here.

So, what am I missing? Why is my history not saved? If anyone thinks another file may be relevant let me know and I'll post it. I checked by running grep -i hist \.* in my $HOME which showed that the only relevant . file containing the string hist or HIST was .bashrc.

I am running Linux Mint Debian Edition, GNU bash, version 4.2.36(1)-release (x86_64-pc-linux-gnu) and my favorite terminal emulator (in case that's relevant) is terminator.

---

UPDATE:

Following @mpy's suggestion in the comments, I changed my ~/.bashrc to set HISTFILE=~/bash_history as opposed to the default ~/.bash_history and that seems to solve the problem for interactive shells. Login shells still display the same behavior, with the history truncated at 500 lines. However, there are no HIST related variables set in the relevant files:

$ for f in /etc/profile ~/.profile ~/.bash_profile ~/.bash_login; do \
   echo -ne "$f :"; echo `grep HIST $f`; \
done
/etc/profile :
/home/terdon/.profile :grep: /home/terdon/.profile: No such file or directory
/home/terdon/.bash_profile :grep: /home/terdon/.bash_profile: No such file or directory
/home/terdon/.bash_login :grep: /home/terdon/.bash_login: No such file or directory
$ grep -r HIST /etc/profile.d/  <-- returns nothing

So, why is setting HISTSIZE and HISTFILESIZE in the ~/.bashrc not enough unless I explicitly set the $HISTFILE to something other than the default ~/.bash_history?

Solution 1

The problem actually boils down to the different behavior of login and non-login shells. I had set the variables that control history in my ~/.bahsrc. This file is not read when one starts a login shell, it's only read by interactive, non-login shells (from man bash):

When bash is invoked as an interactive login shell, or as a non-interactive shell with the --login option, it first reads and executes commands from the file /etc/profile, if that file exists. After reading that file, it looks for ~/.bash_profile, ~/.bash_login, and ~/.profile, in that order, and reads and executes commands from the first one that exists and is readable. The --noprofile option may be used when the shell is started to inhibit this behavior.

[. . . ]

When an interactive shell that is not a login shell is started, bash reads and executes commands from ~/.bashrc, if that file exists. This may be inhibited by using the --norc option. The --rcfile file option will force bash to read and execute commands from file instead of ~/.bashrc.

Therefore, each time I logged in, or dropped to a tty, or used ssh, the .history file was getting truncated because I had not set it to the right size in ~/.profile as well. I finally realized this and simply set the variables in ~/.profile where they belong, instead of ~/.bashrc

So, the reason my ~/.history was getting truncated was because I had only set the HISTORY variables in a file read by interactive, non-login shells and therefore every time I ran a different type of shell the variables would be ignored and the file would be cut accordingly.

Solution 2

My suggestion is to use another file as HISTFILE, not the default ~/.bash_history.

Although I have no analytical explanation, I'll try to outline what led me to this suggestion: If you use bash as your default (login) shell and also use X (which both is very probable) you have a running bash instance right after the (graphical) login:

systemd
 ...
  |-login
  |   `-bash      <<====
  |       `-slim
  |           |-X -nolisten tcp vt07 -auth /var/run/slim.auth
  |           |  `-{X}
  |           `-fluxbox
  |               `-xterm -bg black -fg white
  |                   `-bash
 ...

I think this instance is a login shell, so it doesn't read your ~/.bashrc and hence won't know anything about the histappend option:

man bash(1): When an interactive shell that is not a login shell is started, bash reads and executes commands from /etc/bash.bashrc and ~/.bashrc, if these files exist. (...)

As long as this "parent shell" runs, everything is fine, but upon its termination (i.e. system halt) it will override ~/.bash_history (because that's the default value) and messes up your history or clips it on system start to (again default) 500 lines. (Or perhaps both...)

It strikes me, too, that it's not sufficient to include the history configuration in ~/.bashrc, as this shouldn't be such a uncommon setup. I have no explanation for that.

---

Concerning your problem, that "Login shells still display the same behavior", you can try to include the history config also in ~/.bash_profile:

man bash(1): When bash is invoked as an interactive login shell, or as a non-interactive shell with the --login option, it first reads and executes commands from the file /etc/profile, if that file exists. After reading that file, it looks for ~/.bash_profile, (...)

Unfortunately I can't post a more justified explanation with details from my own bash config, as I'm a zsh guy...

Solution 3

Since all your settings are in order according to the man page, and since the history file is not restricted by size (bytes), the only possible explanation I can think of. It has to do with how the shell dies.

According to the online reference, the graceful exit (history saved) occurs only when the shell receives SIGHUP. I can't really explain how your system propagates signals when rebooted, but I suspect that your shell quits with SIGKILL or SIGPWR.

It could be because your WM runs asynchronously (wait) and the terminal emulator spawned from the WM where bash is gets an exit-forcing signal other than SIGHUP. It could also be that the OS is to quick to send the "final kill" to all processes before the initial graceful SIGHUP manages to get to the shell via X -> WM -> xterm, possibly because X or WM takes longer to exit than it takes for the OS to be ready to go down.

I'm on deep waters with this stuff, but I think something along those lines causes the erratic behavior. I've had this issue before, and the most solid remedy is exit in bash where you want to keep history.

I noticed history -a in your question, and I can't think of why that would not be sufficient to preserve the history.

You could troubleshoot the issue by figuring out what really kills your bash and moving on to figuring out where the signal originates and fixing the issue there, or simply flush the history when you know which signal is the last (assuming disks are still online by then):

trap "echo got 1  >/tmp/sig1;  exit" SIGHUP
trap "echo got 2  >/tmp/sig2;  exit" SIGINT
trap "echo got 15 >/tmp/sig15; exit" SIGTERM
 .. and so on...

The included screenshot illustrates what I'm talking about in the second and third paragraphs. The sequence there is I shell in from left, kill the left shell from right and cat the history.

man bash

On startup, (...) The file named by the value of HISTFILE is truncated, if necessary, to contain no more than the number of lines specified by the value of HISTFILESIZE (+ default 500).

If the histappend shell option is enabled (+ default on here), the lines are appended to the history file, otherwise the history file is overwritten.

online reference

3.7.6 Signals

When Bash is interactive, in the absence of any traps, it ignores SIGTERM (so that ‘kill 0’ does not kill an interactive shell), and SIGINT is caught and handled (so that the wait builtin is interruptible). When Bash receives a SIGINT, it breaks out of any executing loops. In all cases, Bash ignores SIGQUIT. If job control is in effect (see Job Control), Bash ignores SIGTTIN, SIGTTOU, and SIGTSTP.

Non-builtin commands started by Bash have signal handlers set to the values inherited by the shell from its parent. When job control is not in effect, asynchronous commands ignore SIGINT and SIGQUIT in addition to these inherited handlers. Commands run as a result of command substitution ignore the keyboard-generated job control signals SIGTTIN, SIGTTOU, and SIGTSTP.

The shell exits by default upon receipt of a SIGHUP. Before exiting, an interactive shell resends the SIGHUP to all jobs, running or stopped. Stopped jobs are sent SIGCONT to ensure that they receive the SIGHUP. To prevent the shell from sending the SIGHUP signal to a particular job, it should be removed from the jobs table with the disown builtin (see Job Control Builtins) or marked to not receive SIGHUP using disown -h.

If the huponexit shell option has been set with shopt (see The Shopt Builtin), Bash sends a SIGHUP to all jobs when an interactive login shell exits.

If Bash is waiting for a command to complete and receives a signal for which a trap has been set, the trap will not be executed until the command completes. When Bash is waiting for an asynchronous command via the wait builtin, the reception of a signal for which a trap has been set will cause the wait builtin to return immediately with an exit status greater than 128, immediately after which the trap is executed.

demonstrative screen shot

Comments

[Adnan Bhatti]

Are you the owner of .bash_history or root? Do ls -l .bash_history

[terdon]

@MSStp yes, it is owned by me. Thanks for the suggestion but I don't see how it could be a permissions issue anyway, either I have read/write access to it or I don't. Since some history is saved, I clearly do. If bash had a setting that caused problems when this file is not owned by the user, it would either complain or the entire history functionality would not work.

[MelBurslan]

when you execute history command, the output you see is identical to what you see, running cat .bash_history, other than the the line numbers ? I mean does the history command list time stamps or other information ? The reason I am asking is, if you see these esoteric stuff, it means, there is another module/function/program, which is messing with the shell history and a wrong or buggy version of whatever it is, might be causing you the grief.

[terdon]

@Mel_Burslan yes it's the same, the only difference is the line numbers.

[mpy]

Ok, a somewhat weird suggestion, but it's a weired problem, too ;): Try another file as HISTFILE, not the default ~/.bash_history. Very constructed explanation: I assume bash is your default shell, so upon system start a non-interactive shell is the parent of your X session (I also assume you use X), which will know nothing about histappend option (as .bashrc is only read by interactive shells), so as long this parent shell runs everything is fine, but upon termination (i.e. system halt) it will override ~/.bash_history (which is default) and messes up your history...

[terdon]

hmmm, interesting @mpy, I'll give it a try.

[depquid]

Also, I'd suggest checking your history file, then exiting a shell, and checking the file to see if there's any immediate effect. Also, anything in ~/.bash_logout

[terdon]

@mpy why don't you post that as an answer, I think you have something there, see mu updated question. Changing the hist file seems to do the trick for interactive shells. Any ideas why?

[terdon]

@mpy do post your answer, changing the name of the default history file seems to have solved it, you should get the bounty.

[mpy]

@terdon: I'm glad my suggestion works. I didn't post an answer in first place, as I felt I should do some more research to give a bulletproof explanation. Not only my intuition with lots of "possible", "probably" etc. phrases. But you convinced me ;)

[terdon]

I don't really get what you are doing in that screenshot. What is the /tmp/pso that you're killing? I see your point about the different kill signals (though as you say, I thought that is what history -a is there to deal with). I will test that for a while and report back.

[Ярослав Рахматуллин]

$ cat tmp/ps* \n ps -o pid= |head -n1 > ~/tmp/pso \n 17201 \n

[terdon]

Thanks, but grep -r HIST /etc/profile.d/ returns nothing, and I have already checked /etc/profile.

[terdon]

Explicitly setting the history options in my ~/.bash_profile solved the problem. I am now using ~/.bash_history as my history file but have simply added all the lines from ~/.bashrc shown in my question to ~/.bash_profile. Still not sure who was screwing up the interactive shells, but it seems to work now, thanks!

[terdon]

Sorry to unaccept but I'd forgotten to post an answer explaining what was going on. I figured it out with some help from @Gilles a while ago and finally got round to posting an answer. I wanted to accept mine because it actually explains the issue instead of offering a (very good) workaround and could help future visitors.

[mpy]

Very good point, thanks for sharing! However, I do not agree with: This file is not read when one starts a login shell, it's only read by interactive shells. Because a login shell can be interactive, too. Otherwise the whole history mechanism would make no sense. IMHO .bashrc is not read by (non-interactive OR login shells).

[terdon]

@mpy Indeed, sorry, I meant interactive, non-login shells. Answer edited.

[Noah Spurrier]

@terdon, the common idiom is that interactive shell options don't go in ~/.profile or ~/.bash_profile. Interactive shell options are put into ~/.bashrc. To avoid having to maintain settings in two places the following commands may be put at the top of a ~/.bash_profile: export BASH_ENV=~/.bashrc ; if [ -f ~/.bashrc ]; then . ~/.bashrc; fi ... and at the top of the ~/.bashrc put a check to make sure you really are running interactively: [ -z "$PS1" ] && return... Of course, it's just an idiom.

[terdon]

@NoahSpurrier BASH_ENV is not relevant, it only affects non-interactive shells. As for the "idiom", it is something that Debian started and with which I personally don't agree. I often have graphical options (xset and the like) in my .bashrc and I don't want those active when I run login shells either from a tty or through ssh. I want my .profile and .bashrc separate. Many (though not all) login managers source .profile when you log in so global variables are best set there where they will only be read once and not each time you open a terminal.

[Wilson F]

I read through the links you provided and I'm unclear about one thing: do settings in ~/.profile override (or, at least, are they applied later than) settings in /etc/bash.bashrc? I have a feeling that I probably shouldn't really be editing things like HISTSIZE and HISTFILESIZE in the global file /etc/bash.bashrc, since that affects all users, not just me (even though in this case I am actually the only user of this particular computer).

[terdon]

@WilsonF yes. The files are read sequentially and the personal files (~/.profile or ~/.bashrc) are read last. Anything set in those will take precedence over the global settings. You're quite right, you shouldn't be setting those variables in /etc/bash.bashrc. Use ~/.profile (or ~/.bash_profile) instead.