rsyslogd - No space left on device

Question

I recently upgraded from ubuntu 16.04 to 18.04.1 and have been encountering a few problems. To try few things out I've started a brand new VM, installed all updates and mounted my media shares. I then tried to start plex media server but it failed and I get those errors:

Sep 26 15:05:44 plex rsyslogd[1053]: file '6' write error: No space left on device [v8.32.0 try http://www.rsyslog.com/e/2027 ]
Sep 26 15:05:44 plex rsyslogd[1053]: action 'action 3' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.32.0 try http://www.rsyslog.com/e/2027 ]
Sep 26 15:05:44 plex rsyslogd[1053]: file '6' write error: No space left on device [v8.32.0 try http://www.rsyslog.com/e/2027 ]
Sep 26 15:05:44 plex sudo[1895]: pam_unix(sudo:session): session closed for user root
Sep 26 15:05:49 plex sudo[1898]:     plex : TTY=pts/0 ; PWD=/dev/mapper ; USER=root ; COMMAND=/bin/journalctl -xe
Sep 26 15:05:49 plex sudo[1898]: pam_unix(sudo:session): session opened for user root by plex(uid=0)

I then checked my filesystem:

plex@plex:/dev/mapper$ df -h
Filesystem                         Size  Used Avail Use% Mounted on
udev                               5.9G     0  5.9G   0% /dev
tmpfs                              1.2G  1.2M  1.2G   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv  3.9G  3.7G     0 100% /
tmpfs                              5.9G  4.0K  5.9G   1% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
tmpfs                              5.9G     0  5.9G   0% /sys/fs/cgroup
/dev/loop1                          88M   88M     0 100% /snap/core/5328
/dev/loop0                          87M   87M     0 100% /snap/core/4917
/dev/sda2                          976M  142M  768M  16% /boot
//192.168.178.200/Coding            66T   52T   15T  79% /coding
//192.168.178.200/Music             66T   52T   15T  78% /music
//192.168.178.200/Series            66T   52T   15T  78% /series
//192.168.178.200/Movies            66T   52T   15T  79% /movies
//192.168.178.200/Audioooks         66T   52T   15T  79% /audiobooks
tmpfs                              1.2G     0  1.2G   0% /run/user/1000

As far as I know both snap mounts are supposed to use 100%. Is the mapper causing this?

Googling the problem I've found a possible reason that I dont have any inodes left so I checked:

plex@plex:/dev/mapper$ df -i
Filesystem                         Inodes  IUsed   IFree IUse% Mounted on
udev                              1529447    436 1529011    1% /dev
tmpfs                             1537097    693 1536404    1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv  262144 215596   46548   83% /
tmpfs                             1537097      2 1537095    1% /dev/shm
tmpfs                             1537097      3 1537094    1% /run/lock
tmpfs                             1537097     18 1537079    1% /sys/fs/cgroup
/dev/loop1                          12860  12860       0  100% /snap/core/5328
/dev/loop0                          12842  12842       0  100% /snap/core/4917
/dev/sda2                           65536    313   65223    1% /boot
//192.168.178.200/Coding                0      0       0     - /coding
//192.168.178.200/Music                 0      0       0     - /music
//192.168.178.200/Series                0      0       0     - /series
//192.168.178.200/Movies                0      0       0     - /movies
//192.168.178.200/Audioooks             0      0       0     - /audiobooks
tmpfs                             1537097     10 1537087    1% /run/user/1000

This is my /etc/fstab:

UUID=9dd0cc79-c15f-11e8-925a-000c291f8ddc / ext4 defaults 0 0
UUID=9dd0cc78-c15f-11e8-925a-000c291f8ddc /boot ext4 defaults 0 0

//192.168.178.200/Music /music cifs credentials=/home/plex/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/Movies /movies cifs credentials=/home/plex/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/Series /series cifs credentials=/home/plex/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/Audioooks /audiobooks cifs credentials=/home/plex/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/Coding /coding cifs credentials=/home/plex/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0

Am I missing anything?

Thanks for any help!

Solution

Your root file system is only 4GB and is full:

/dev/mapper/ubuntu--vg-ubuntu--lv  3.9G  3.7G     0 100% /

Comments

[Lewis M]

Based on your first df output, your root filesystem (/) is full. /dev/mapper/ubuntu--vg-ubuntu--lv shows to be 100% used.

[Jos]

Quick and easy ways to win back some space are: 1) remove obsolete kernels (sudo apt autoremove) and 2) removing or gzipping large log files in /var/log/.

[Kristopher Ives]

@Jos Good ideas, although I'm not sure how it was ever working with only 4GB of space. It's likely he accidentally picked that size during VM creation instead of 40GB for example.

[Jos]

True. You may be able to run Ubuntu within 4Gb, but you will need to carefully monitor it.

[In0cenT]

Funny enough 4GB is the default value, just created a new VM.