Why do I lose my ZSH history?

Question

Every once in a while I discover my zsh history has been truncated (or maybe lost entirely, difficult to tell), and I have to restore it from backup.

For example, today:

ls -lh ~/.zsh_history
-rw-------  1 stripe  staff    32K 21 Feb 10:20 /Users/stripe/.zsh_history

But in my backup from a few days ago:

-rw-------@  1 stripe  staff  203K 17 Feb 22:36 /Volumes/Time Machine Backups/.../Users/stripe/.zsh_history

I have configured zsh to save lots of history so it shouldn't be a matter of the shell intentionally trimming the file.

unsetopt share_history
setopt inc_append_history
setopt hist_ignore_all_dups
HISTSIZE=500000
SAVEHIST=$HISTSIZE

Has anyone else experienced this and found a way to mitigate it? Is there some zsh spring cleaning feature I'm unaware of?

Solution 1

ZSH

History file can be truncated/lost/cleaned for multiple reasons those can be:

• Corruption of the zsh history file (because of a power-cut/system-fail while a shell is opened, in this case fsck need to be setup to run when the system fail)
• Zsh config file is not loaded (for example if $HOME env variable is not defined)
• Unsupported character on history file can make zsh reset the history
• Cleaning tools like bleachbit
• Zsh misconfiguration
• Sharing the $HISTFILE among shells with a more restrictive $HISTSIZE (even implicitly, for example if you run a bash shell under zsh without a $HISTFILE in your bashrc, the subshell will use the inherited variable from zsh and will apply the $HISTSIZE defined in bashrc)
• etc.

Notes

History available setup options

HISTFILE="$HOME/.zsh_history"
HISTSIZE=500000
SAVEHIST=500000
setopt BANG_HIST                 # Treat the '!' character specially during expansion.
setopt EXTENDED_HISTORY          # Write the history file in the ":start:elapsed;command" format.
setopt INC_APPEND_HISTORY        # Write to the history file immediately, not when the shell exits.
setopt SHARE_HISTORY             # Share history between all sessions.
setopt HIST_EXPIRE_DUPS_FIRST    # Expire duplicate entries first when trimming history.
setopt HIST_IGNORE_DUPS          # Don't record an entry that was just recorded again.
setopt HIST_IGNORE_ALL_DUPS      # Delete old recorded entry if new entry is a duplicate.
setopt HIST_FIND_NO_DUPS         # Do not display a line previously found.
setopt HIST_IGNORE_SPACE         # Don't record an entry starting with a space.
setopt HIST_SAVE_NO_DUPS         # Don't write duplicate entries in the history file.
setopt HIST_REDUCE_BLANKS        # Remove superfluous blanks before recording entry.
setopt HIST_VERIFY               # Don't execute immediately upon history expansion.
setopt HIST_BEEP                 # Beep when accessing nonexistent history.

Answer

The following configuration is recommended for this situation (to be setup on ~/.zshrc file)

HISTFILE=/specify/a/fixed/and/different/location/.history
HISTSIZE=500000
SAVEHIST=500000
setopt appendhistory
setopt INC_APPEND_HISTORY  
setopt SHARE_HISTORY

Alternative

You can use a little script that check the history file size and do restore it from backup when necessary (in ~/.zshrc)

if [ /home/my/zsh/hist/file -lt 64000 ]; then
    echo "History file is lower than 64 kbytes, restoring backup..."
    cp -f /mybackup/histfile /home/my/zsh/hist/file
fi

Links

Additional infos are available on this and this questions.

Solution 2

I don't think any zsh option is going to save your precious .zsh_history.

My .zsh_history has been randomly truncated over the years, and I still don't know why. I've tried every option I could find on StackExchange, and obviously tried the config from oh-my-zsh.

Automated backups

In order to not care next time my history gets truncated, I added this line to crontab -e:

30 14 * * * cp /home/my_user/.zsh_history /backup/folder/zsh_history_$(date +\%Y_\%m_\%d).bak

Feel free to use anacron, rsync or any other tool. The goal is to have a collection of .zsh_history files somewhere safe, with at least some of them containing the desired information.

Restore history

When you need to restore one complete .zsh_history from your possibly truncated backups, you can use this command:

cat zsh_history*.bak | awk -v date="WILL_NOT_APPEAR$(date +"%s")" '{if (sub(/\\$/,date)) printf "%s", $0; else print $0}' | LC_ALL=C sort -u | awk -v date="WILL_NOT_APPEAR$(date +"%s")" '{gsub('date',"\\\n"); print $0}' > .merged_zsh_history

Which comes from this excellent article ("Combining zsh history files").

It merges the history files, sorts the command, removes duplicates and doesn't break multiline commands.

A few weeks later

As planned, my .zsh_history got truncated for no apparent reason.

My backups worked fine. Some of them still had duplicate commands in the same files. The awk code above only recognizes exact duplicates (time+duration+command) between files, but will leave them if, for example, ls has been called at different times. So I wrote this small Ruby script:

#! /usr/bin/env ruby
# Ruby script to merge zsh histories. In case of duplicates, it removes the old timestamps.
# It should do fine with multi-line commands.
# Make backups of your backups before running this script!
#
# ./merge_zsh_histories.rb zsh_history_*.bak ~/.zsh_history > merged_zsh_history

MULTILINE_COMMAND = "TO_BE_REMOVED_#{Time.now.to_i}"

commands = Hash.new([0,0])

ARGV.sort.each do |hist|
  $stderr.puts "Parsing '#{hist}'"
  content = File.read(hist)
  content.scrub!("#")
  content.gsub!(/\\\n(?!:\s*\d{10,})/, MULTILINE_COMMAND)
  should_be_empty = content.each_line.grep_v(/^:/) + content.each_line.grep(/(?<!^): \d{10,}/)
  raise "Problem with those lines : #{should_be_empty}" unless should_be_empty.empty?
  content.each_line do |line|
    description, command = line.split(';', 2)
    _, time, duration = description.split(':').map(&:to_i)
    old_time, _old_duration = commands[command]
    if time > old_time
      commands[command] = [time, duration]
    end
  end
end

commands.sort_by{|_, time_duration| time_duration}.each{|command, (time, duration)|
  puts ':%11d:%d;%s' % [time, duration, command.gsub(MULTILINE_COMMAND, "\\\n")]
}

It worked fine, and returned a valid zsh_history file, which contained all my commands, and wasn't much larger than the largest backup.

Solution 3

I have found an alternative solution: I installed a custom command history tool.

I'm using McFly. Beyond its neat features for smarter history search, the key in this context is that in order to work the tool must maintain its own history database.

Augments your shell history to track command exit status, timestamp, and execution directory in a SQLite database.

My hope is that even if my .zsh_history keeps getting corrupted, McFly's sqlite database will remain as a backstop.

In fact, McFly automatically updates the .zsh_history for compatibility reasons and so far, after about a month and a half, I've not seen it lose any data. Both the sqlite and the .zsh_history files are growing steadily. My .zsh_history is 370K now which I believe is a new record.

Solution 4

I had a similar issue and just figured it out today.

I use oh-my-zsh which sets SAVEHIST=10000. My history was over 10,000 lines, but I think the oldest commands are removed to make room for the new ones that come in, and 10k was working okay. However, I had another issue with oh-my-zsh which I was debugging, and at one point completely removed . $ZSH/oh-my-zsh.sh from my zshrc, this resulted in reseting SAVEHIST to the default value of 1000, and zsh promptly removed the other 9000 lines from my history file. If you have another terminal window open when this happens, you can still run history from that window to see your full history, but once you close that sessions, it's gone forever (unless of course you have backups).

Solution 5

If it's not too late to reply. When my zhistory file gets corrupted, it's always due to an abrupt shutdown. When I reboot, I just edit the ~/.zhistory file and delete the line that will be obviously corrupt. Usually the last line but sometimes the line next to last. Then I'm ok. My zhistory file is almost 1meg and spans many years.

View more solutions

Comments

[Admin]

It happened me today for the very first time. I had thousands of commands in my history, and suddenly today morning, there were only 40.

[shivams]

Thank you for the comprehensive answer. I am suspecting that in my case it could probably have some unsupported character that messed up the history. Is there any workaround for such situations? Further, do you have any reference for the file corruption possibility?

[intika]

@shivams as suggested here the ^@^@^@^@ characters are nulls, and they frequently show up when two programs try writing to the same file at the same time, and aren't using atomic writes (eg. they're buffered writes) or locking. Or it can be caused by filesystem corruption, bad restoration from backup, or similar. setopt INC_APPEND_HISTORY and setopt SHARE_HISTORY may probably help here... you can report that bug here sourceforge.net/p/zsh/bugs if it's not solved with the suggested options.

[shivams]

Thank you @Daniel for reporting another edge case.

[Daniel Marks]

@shivams Gotta get that stack exchange reputation up haha. The above happened to me a few times before I really dug into why it was happening. It is pretty tragic to loose the convenience of recalling old esoteric, useful commands. One might thing that I'd start putting them into scripts by now. Anyways, in the event that you still have a terminal window open with still has the full history cached, you can run fc -W history_backup to save that cached history to a backup file, then merge that with your history file, and nothing is lost.

[shivams]

Thank you. This is some useful advice.

[Eric Duminil]

@shivams: You're welcome. I was getting really frustrated to regularly lose my history, without apparent reason. When I notice, it's usually much too late, and only because I need a long forgotten command. Next time, I'll be prepared.

[shivams]

Whoa! Neural-network driven Ctrl-r! Will check this one out.

[Eric Duminil]

Cool, it looks interesting indeed.

[shivams]

Wow! This was very helpful. I am now using the automated backup solution, but had I known this before, it'd have helped me recover my corrupted files before I started backing up. Definitely a handy tip for the future. Thanks!

[Eric Duminil]

Good for you. But when my zsh history file gets corrupted, the whole file is usually truncated. I'd be happy with a few corrupt lines. Could you please share your system type, your zsh version and your history configuration, please?

[Gavin Gilmour]

Thanks for the cron suggestion. This issue is so annoying, lost megs and megs of precious history!

[Eric Duminil]

@GavinGilmour: Indeed. Every month or so, I notice that commands are missing from the history. I run wc -l ~/.zsh_history and notice that the file has 50 lines instead of 14000. :-/ I don't care anymore, though, and simply run my ruby script to merge all the backups.

[Eric Duminil]

@GavinGilmour: Do you have any idea what could have gone wrong? Hard reset, crashed program, crashed terminal? At least on my laptop, history truncation seems to be really random.

[Gavin Gilmour]

@EricDuminil Totally random from what I can gather unfortunately. But lost tons of useful commands from over the months so yeah really frustrating. :( Thanks again for the cron suggestion.