"No space left on device", then black screen on reboot

Question

I am using Ubuntu 18.04. I ran into a serious problem this morning. I remotely connect to my Ubuntu desktop in the lab via Teamviewer to finish training a machine learning model and present some results to my supervisor.

At one point, my script should pickle dump some large file of list of arrays (about 3GB), so I left the script running overnight. This morning, I remotely connect to check, and noticed the script stopped running with a No space left on device message.

I suspected the system dumped too many tmp files, so I executed reboot via terminal, forcing the system to reboot (my bad).

Unfortunately, after reboot, all I can see it black screen (with cursor blinking, see image below). Maybe system hung and unable to reach the login stage, so I cannot see the input login screen to enter password.

Is there a way I can safely restart my system? Any help please?

$ df -h
Filesystem    Size    Used    Avail    Use%    Mounted on
udev          7.8G       0      7.8G      0%     /dev
tmpfs          1.6G    1.4M     1.6G      1%     /run
/dev/sda1     422B      401G    0        100%    /
tmpfs         7.8G       0      7.8G      0%     /dev/shm
tmpfs         5.0M      4.0K    5.0M      1%     /run/lock
tmpfs         7.8G       0      7.8G      0%     /sys/fs/cgroup
/dev/loop2    256K      256       0     100%    /snap/gtk2-common-themes/9
/dev/loop1    67M       67M       0     100%    /snap/p8x-onenote/66
/dev/loop0    92M       92M       0     100%    /snap/core/8689
.
.
/dev/loop14   256K     256K       0     100%    /snap/gtk2-common-themes/5
tmpfs         1.6G        0     1.6G      0%    /run/user/1000

---

$ ls -alS /tmp
total 192
drwxrwxrt  14 root root 139264 mar 8 16:39  .
drwxr-xr-x 25 root root   4096 feb 19 06:34 ..
drwxrwxrwt 2  root root   4096 mar 8 15:23 .font-unix
drwxwxrwt  2  root root   4096 mar 8 15:23 .ICE-unix
drwx-----  2  root root   4096 mar 8 15:24 doise-PKdhtxMeriun

---

$ ls -al /home
total 20
drwxr-xr-x  5  root      root      4096  jul 17 2018  .
drwxr-xr-x 25  root      root      4096  feb  19  06:34 ..
drwxr-xr-x 59  akil      akil      4096  mar 8 08:32 akil
drwxr-xr-x 48   it         it      4096  feb 11 10:24 it
drwxr-xr-x 23 josepintor jpsepinto 4096 jul 17 2018 josepintor

---

ls -al /boot
total 102224
drwxr-xr-x 4  root root   4096  feb 20 06:08
drwxr-xr-x 25 root root   4096  feb 19 06:34
-rw-r--r-- 1  root root 217495  jan 17 13:59 config-4.15.0-76-generic
-rw-r--r-- 1  root root 217495  feb 11 18:53 config-4.15.0-88-generic
drwr-xr-x  5  root root   4096  feb 20 06:02 grub
drwxr-xe-x 4  root root   4096  feb 21 2017 grub.bak
-rw--r--r--1  root root   39379455 feb 6 06:52 initrd.img-4.15.0-76-generic
-rw-r--r-- 1 root root 39885351  feb 19 06:52 initrd.img-4-15.0-88-generic
-rw-r--r-- 1 root root 182704  jan 28 2016 memtest86+.bin
-rw-r--r-- 1 root root 184380 jan 28 2016 memtest86+.elf
-rw-r--r-- 1 root root 184840 jan 28  2016 memtest86+_multboot.bin
-rw------- 1 root root 40688355 jan 17 3:59 System.map-4.15.0-76-generic
-rw------- 1 root root 4063510 feb 11 18:53 System.map-4.15.0-88-generic
-rw------- 1 root root 8367768 jan 17 vmlinuz-415.0-76-generic
-rw------- 1 root root 8371864 feb 11 18:57 vmlinuz-4.15.0-88-generic

---

$ ls -alS /var/log
total 6084
-rw-r----  1 syslog   adm     1528831  mar  8 00:07 syslog.1
-rw-r----  1 syslog   adm      962690  mar  7 23:29 kern.log.1
-rw-r--r-- 1  root    root   717322    mar  8 15:24 boot.log
-rw-r--r-- 1  root    root   352885    agu 19 2019 vnetlib
-rw-rw-r-- 1  root    utmp   292876    mar  8 00:05  auth.log.1

Now I have access to the GRUB menu. I selected the Drop to root shell prompt option. But for like 5 minutes, the message I have is as follows (with cursor blinking)

Starting stop unreadahead data collection...
[ OK ] Started stop unreadahead data collection.
[ TIME ] Timed out waiting for device dev-di...04\x2dbef2\x2deed7474e2fbd.devce.
[ DEPEND ] dependency failed for /dev/disk/by-...0c431-1324-4004-bef2-eed7474e2fdb.
-

$ sudo blkid
/dev/sda1: UUID="6c987832-52fb-49e2-86fd-9cf24992511c" TYPE="ext4" PARTUUID="e9461ec-01"
/dev/sda2: UUID="a2c7d786-bda1-4b83-99f1-fca3b4c05bcb" TYPE="swap" PARTUUID="e946b1ec-02"

---

$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a 
# device; 
#....
# <file system> <mount point> <type> <options> <dump> <pass>
UUID=6c987832-52fb-49e2-86fd-9cf24992511c / ext4  errors=remount -ro 0 1
# swap was on /dev/sda2 during installation
UUID=a2c7d786-bda1-4b83-99f1-fca3b4c05bcb none swap sw 0 0
/dev/fd0  /media/floppy0 auto rw,user,noauto,exec,utf8 0 0

---

$ dpkg -l *readahead*
Desired=Unkonwn/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
| / Err?=(none)/Reinst-required (Status, Err: uppercase=bad)
||/ Name         Version       Architecture      Description
+++==============================================================
un readahead     <none>         <none>        (no description available)
ii unreadahead  0.100.0-21       amd64      Read required files in advance

Landed here, about the reboot.

---

In step-3 of the answer:

dpkg -l linux-* | grep -e ii | grep -e "-76"
ii linux-headers-4.4.15-76  4.15-76.86 all Header files related to Linux kernel version 4.15.0
ii linux-headers-4.15.0-76-generic  4.15.0-76.86 amd64 Linux kernel headers for version 4.15.0 on 64 bit x86 SMP
ii linux-image-4.15.0-76-generic 4.15.0-76.86 amd64 Signed kernel image generic
ii linux-modules-4.15.0-76-generic 4.15.0-76.86 amd64 Linux kernel extra modules for version 4.15.0-76.86 SMP
ii linux-modules-extra-4.15.0-76-generic 4.15.9-76.86 amd64 Linux kernel extra modules for version 4.15.0 on 64 bit x86 SMP

---

$ df -h
Filesystem     Size     Used     Avail   Use% Mounted on
udev           7.8G       0      7.8G      0    /dev
tmpfs          1.6G     1.1M     1.6G      1%   /run
/dev/sda1      422G     402G        0     100%   /
tmpfs          7.8G       0       7.8G     0%   /dev/shm
tmpfs          5.0M      4,0K    5.0M      1%   /run/lock
tmpfs          7.8G       0      7.8G     0%    /sys/fs/cgroup

---

$ ls -al /boot
total 102224
drwxr-xr-x    4    root    root     4096    feb   20 06:03
drwxr-xr-x   25    root    root     4096    mar    9 14:10
-rw-r--r--    1   root    root    217495    jan  17   13:59 config-4.15.0-76-generic
-rw-r--r--    1   root    root    217495    feb  11 18:53 config-4.15.0-88-generic

drwxr-xr-x    5   root    root     4096     feb   20   06:02  grub
drwxr-xr-x    5   root    root     4096     feb   21   2017  grub.bak
-rw-r--r--    1   root    root   39379455   feb    6   06:53 initrd.img-4.15.0-76-generic
....
......

Solution

Step #1:

In Recovery Mode, at the # prompt, do this...

sudo pico /etc/fstab # edit this file

Change this:

/dev/fd0  /media/floppy0 auto rw,user,noauto,exec,utf8 0 0

To this:

#/dev/fd0  /media/floppy0 auto rw,user,noauto,exec,utf8 0 0

Control+o then Enter # to write out the file

Control+x # to exit pico editor

reboot

Step #2:

Either TeamViewer in, or boot to a Ubuntu Live DVD/USB, and start the Disk Usage Analyzer application (this may be called something different on Lubuntu). A few clicks, and you can determine what's eating up your hard disk, and we can take the next step.

Disk Usage Analyzer is accessed from the GUI, but on Lubuntu it may not be installed by default, and you don't have the disk space to install it. The Ubuntu Live DVD/USB is what you used to install Lubuntu originally. If you don't have it any more, download the Ubuntu (not Lubuntu) image and create a Ubuntu Live DVD/USB which will have Disk Usage Anayzer already installed on it.

Step #3:

To try an clear a little disk space...

In the terminal...

dpkg -l linux-* | grep -e ii | grep -e "-76"

You should end up with 4-5 filenames. If it's more, then stop and tell me what you see. Then do...

sudo dpkg -P put_one_filename_here

and repeat this for the 4-5 filenames shown before.

You may be able to boot normally now. Cleanup continues.

Step #4:

Let's just make sure that you don't also have file system problems...

• boot to a Ubuntu Live DVD/USB
• open a terminal window by pressing Ctrl+Alt+T
• type sudo fdisk -l
• identify the /dev/sdXX device name for your "Linux Filesystem"
• type sudo fsck -f /dev/sdXX, replacing sdXX with the number you found earlier
• repeat the fsck command if there were errors
• type reboot

Comments

[MelcomX]

Do you have SSH-access to the machine?

[arilwan]

No, right now I can connect to it via TeamVeawer, but all I see is the black screen. Connection is successful, but no login screen, only the cursor blicking.

[arilwan]

@MelcomX I updated my Question with the screenshot of the situation now. Now I can login but I seems I lost access to my account. I can login with the admin credentials and tha's all I can see in the image.

[Boris Hamanov]

Edit your question (not into the comments please) and type df -h and see how much disk space is used/avail. Then do ls -al /boot and ls -alS /tmp (first 5 lines is enough), and ls -alS /var/log (first 5 lines is enough), and ls -al /home. Start comments to me with @heynnema or I may miss them.

[arilwan]

@heynnema I edited the question with said output.

[Boris Hamanov]

Let's do two of the commands again... ls -al /boot (all of it please) and ls -alS /var/log | more (first 5 lines). The boot to a Ubuntu Live DVD/USB and start the Disk Usage Analyzer and see if you can determine where your sda1 space went.

[arilwan]

@heynnema Edit to added. I don't really understand your last sentence, but I do not have Live DVD/USB. Besides, currently I access the ubuntu machine via TeamViewer.

[Boris Hamanov]

Via Teamviewer, start the Disk Usage Analyzer app, and try to figure out where your disk space went.

[Boris Hamanov]

When you turn on the computer, can you get to the GRUB menu?

[arilwan]

@heynnema not sure how to access GRUB menu

[Boris Hamanov]

Depending on the computer, hit/hold down the SHIFT or ESC key when you see the vendor splash screen. Timing is everything. Try it a few times until you can get the GRUB screen.

[arilwan]

@heynnema I can access the GRUB menu now, and have selected the "Drop to root shell prompt". Please see edit-3 in question and advise.

[Boris Hamanov]

Good job! Now we can get something done. At the # prompt, type sudo blkid and cat /etc/fstab and dpkg -l *readahead*. Try to capture the output as text for copy/paste, rather than screenshots. Tell me what version Ubuntu.

[Jeff]

Restart your system? Do you mean factory reset, or reboot it?

[arilwan]

@heynnema Edit 4 shows the commands output

[arilwan]

@heynnema display goes black, after a second reboot, it goes into the menu shown in figure 3 (question edit).

[Boris Hamanov]

Your last screenshot doesn't show the very ending statements, does it? Or does it not go any further? See Step #2 in my answer.

[arilwan]

Done so and rebooted. But after showing the manufacturer splash screen, immediately it shows the lubuntu logo, everything goes black. System working, nothing is displayed.

[arilwan]

Disk Usage Analyzer ? How do I access it, through Recovery menu? About Ubuntu Live DVD/USB... do you mean I should download new ubuntu image?

[Boris Hamanov]

Disk Usage Analyzer is accessed from the GUI, but on Lubuntu it may not be installed by default, and you don't have the disk space to install it. The Ubuntu Live DVD/USB is what you used to install Lubuntu originally. If you don't have it any more, download the Ubuntu (not Lubuntu) image and create a Ubuntu Live DVD/USB which will have Disk Usage Anayzer already installed on it.

[arilwan]

OK, will download it now

[s1mmel]

Can you try Ctrl-Alt-f1 e.g. to get a console?

[Boris Hamanov]

@arilwan Please see Step #3 in my answer.

[arilwan]

@heynnema in Edit-6 I add the output of first command in Step 3. Should I go ahead with the second command?

[Boris Hamanov]

@arilwan Good job! Yes, go ahead with the 2nd command. After that, try and reboot. If that works, try and either install Disk Usage Analyzer, or reboot to the Ubuntu Live DVD/USB and use DUA there. Report back.

[arilwan]

OK, downloaded ubuntu. Will create a Bootable USB Stick now.

[Boris Hamanov]

@arilwan After completing the 2nd command, did you try and boot?

[arilwan]

Yes, I tried, the display goes black again, after manufacturer splash screen. Now I downloaded Ubuntu

[Boris Hamanov]

@arilwan also see Step #4. Also show me df -h and ls -al /boot from # again.

[arilwan]

@heynnema added df -hand ls -al /bootoutput in EDIT-7

[arilwan]

@heynnema Created the bootable USB stick now, how do I boot from it?

[Boris Hamanov]

@arilwan First reboot to Recovery Mode, and from the # prompt, type cd /boot and we need to manually delete some leftovers. Note the -76 files. Manually delete them with (example) sudo rm -i config-4.15.0-76-generic, and repeat for all remaining -76 files. Then a df -h command should show less than 402G used on /dev/sda1. If so, try and boot normally again. To boot the USB, power on your computer, and at the vendor splash screen, it should probably tell you to hit F12, or DEL, or ESC to bring up the BIOS boot menu, select the USB there.

[Boris Hamanov]

@arilwan ps: your last ls -al /boot was incomplete, yes?

[arilwan]

@yes, should I write it complete? Also, after deletiing -76files (about 3) df -his still 402 100%

[arilwan]

@heynnema Problem solved. I used the Bootable USB to start Ubuntu as Try Ubuntu, delete some folders containing data I can get later (about 15GB). Now it works. Thank you so much for your time.