Understanding log after crash

Question

I have a new system on my laptop (Ubuntu Gnome 16.04) and first week or so it was great, but then it started to (seemingly randomly) freeze every now and then (about once a day, sometimes more; it's a complete system freeze, where the only option so far was to force shutdown). I was looking at some system logs (kern.log, syslog and dmesg) and found this in /var/log/syslog at the timestamp of the most recent freeze:

The complete syslog is here, the weird line is at 14582.

Does anybody have any hints on what could be the culprit of the freezes? Should I look at other logs? I tried to search for some advice on how to debug system crashes, but the information I found was rather scarce and not very helpful. For instance the Ubuntu wiki guide tells me to replicate the issue on CLI, but I don't know how since I don't know what is crashing. I hoped to find something in the logs but the ones recommended for checking show nothing of interest to my non-expert eye.

I want to try the SysRq method described in the Ubuntu wiki, however it rather conflicts with the Wikipedia article on this topic. This is why I hesitated to use it so far. If anybody has any advice on this, it would be also greatly apreciated.

Here is my system info:

$ uname -a
Linux ultrabook 4.13.0-31-generic #34~16.04.1-Ubuntu SMP Fri Jan 19 17:11:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Update

Today it freezed again, todays log with just errors (outputed by grep -i Error* /var/log/syslog), as suggested by Elder Geek can be found here. The timestamp of the crash is before 11:30:24, which is the time of new booting.

Thanks for any help.

Solution

It might be hard to confirm from your past case, but I believe this was the reason, and others will be able to upvote if this worked for them.

I found out there are a certain number of NVME SSD disks that don't play well with linux because of power related matters.

After many months of troubleshooting, I found the reason Ubuntu was suddenly freezing for no reason was because of that.

The solution was to add the following kernel parameter:

nvme_core.default_ps_max_latency_us=0

To do so, modify /etc/default/grub, and add the above to your GRUB_CMDLINE_LINUX_DEFAULT= string, e.g. mine looked like that:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash nvme_core.default_ps_max_latency_us=0"

Then save, and run update-grub.

And reboot the system.

I have had a 100% success rate since, e.g. 0 crashes, not even an odd crash once in a while.

@Jena, if you still have your original NVME SSD disk that was causing the issue, it would be nice to have a full circle confirmation.

References:

Official kernel bug report: https://bugzilla.kernel.org/show_bug.cgi?id=195039

Comments

[mikewhatever]

Can you describe the problem in some more details. Is it a crash or a freeze, or both? When do they happen? Also, if you want to show us a particular piece of text, copy/paste it, instead of taking a screenshot of a window of a program with that text inside, and then uploding it.

[jena]

It's a complete system freeze, only option is to force shutdown (I want to try SysRq next time it happens). I didn't notice any pattern, so far it seems random to me. I uploaded a screenshot because the weird part is more obvious in it (furthermore, copy-pasting changed the symbols). The full log as copied from gedit editor is in the link provided.

[Zanna]

Not much help, but I have had this problem. In my case it was fixed by blacklisting a driver, btsdio. I can't give any advice whatsoever on figuring out which driver, if any, could be the culprit in your case; I had read that this driver should be blacklisted on my hardware (which is a basketcase) for a completely different minor reason, and incidentally figured out it was to blame for these terrible freezes. Try testing with different kernels you have installed

[ravery]

The character you see is a text rendering of a control character. In your case it is the NULL character ( 00 ).

[Elder Geek]

Regarding your syslog: TL;DR replacing the thousands of lines of log file provided with the output of grep Error* /var/log/syslog would likely be more direct and to the point.

[jena]

@ElderGeek Ok i could do that, but in every documantation manual it's advised to post the full logs because context can be important.

[Elder Geek]

Have you tried booting a previous kernel?

[jena]

I had no previous kernel to boot from (I like to clean my SSD with apt autoremove once in a while) and I got really frustrated by the freezes, so I reinstalled the whole system (first to same version, then to Ubuntu 17.10, as the LTS had issues, as if my ~/.config folder survived disk formating & reinstall, weird). Now the freezes are gone, even though I run the same kernel as before (because new HWE policies). Of course I didn't install so much software yet, as I still suspect it was one of the programs I installed that caused all this..

[jena]

daaamn it happened again :( there was a kernel update last week and i probably rebooted and it freezed again yesterday :( nothing in the logs again, all just stop reporting before the freeze.. now i have two kernels: 4.13.0-32 and -21, the older one is probably ok, because there was no freeze for a week. But now i don't know how to boot into it, holding shift during boot does nothing on my Dell Latitude 6430u :/ and i'm not really comfortable with editing some system config files - any other option?

[jena]

so it's also freezing on the older kernel (4.13.0-21) - i'm lost now :(

[jena]

I upgraded last week to 18.04 LTS (with 4.15.0-20 kernel) and the freezes still occur, I'm sending the laptop to repair service. I will probably include this thread into the report.

[Wadih M.]

@jena I'm having those freezes as well, luckily I found your post because my syslog also contains @^@^@^@^@^@^@^@^@^@^@^@ at the moment of the freeze. Did you finally find a culprit by any chance ?

[jena]

@WadihM. well the service replaced my ssd and the freezes mostly stopped (I experienced only 3 since, always under heavy load) so I think I had a HW issue..

[Wadih M.]

@jena Thanks so much! I will also test my RAM, perhaps it's related to that. Because I added RAM that had different speeed recently, perhaps it has to do with that.

[Wadih M.]

As an update to everyone, the issue stopped after I disconnected the slower ram and moved the remaining RAM to another RAM slot, I also unplugged and re-plugged the disk. It has happened only once since after a few months, big improvement from once every night before! But I'm still wondering why it happened that one extra time after though, seems I might still be missing part of the picture. I'm starting to think it might have to do with swap on disk getting corrupt.

[jena]

I was also wondering about the 3 freezes, as the load wasn't that heavy (well there was gnome and lot of apps). But since the laptop was actually refurbished (Dell ultrabook for quite a cheap buck) I guess it will never be as new. Switching to elementaryOS helped with performance a lot and I didn't get any freeze since, for about a year now.

[Wadih M.]

@Jena Believe I found the reason, confident enough to post it as an answer, because I undid my previously mentioned attempts to fix it to go back to square one, and the kernel parameter alone was the end-all solution, going from a daily crash for months to no crashing at all. Now been a few months without crash, so coming back to post the solution. Cheers!