No space left on device

Question

I can no longer generate new columns in my database. My stack is below. I am new to managing my own server and am unfamiliar with Apache, Ubuntu and AWS. Currently I am receiving an error saying that I have no space left.

It looks like my folder /dev/xvda1 is at 100% capacity. How can I identify how space is being allocated and how can I open up more space?

I will eagerly share more information if you can help me with what to share.

My Error

irb(main):107:0> Survey.generate_for_appointment(a, b)
PG::DiskFull: ERROR:  could not extend file "base/16402/25415": No space left on device
HINT:  Check free disk space.
: INSERT INTO "questions" ("choices", "content", "created_at", "position", "question_type", "survey_group", "survey_group_question_id", "survey_id", "updated_at") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) RETURNING "id"
log writing failed. No space left on device @ io_write - /etc/projects/myapp/log/production.log
ActiveRecord::StatementInvalid: PG::DiskFull: ERROR:  could not extend file "base/16402/25415": No space left on device
HINT:  Check free disk space.

Storage

ubuntu@ip-172-31-33-140:/etc/projects/myapp$ sudo du -h --max-depth=1 /
4.0K    /media
752K    /tmp
0   /sys
9.6M    /bin
1.3G    /home
908M    /usr
4.0K    /mnt
20M /boot
24K /root
4.0K    /opt
3.7G    /etc
du: cannot access ‘/proc/7489/task/7489/fd/4’: No such file or directory
du: cannot access ‘/proc/7489/task/7489/fdinfo/4’: No such file or directory
du: cannot access ‘/proc/7489/fd/4’: No such file or directory
du: cannot access ‘/proc/7489/fdinfo/4’: No such file or directory
0   /proc
9.4M    /sbin
4.0K    /srv
16K /lost+found
60M /lib
12K /dev
649M    /var
4.0K    /lib64
212K    /run
7.5G    /

ubuntu@ip-172-31-33-140:/etc/projects/myapp$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  7.5G     0 100% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            284M   12K  284M   1% /dev
tmpfs            59M  212K   59M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            295M     0  295M   0% /run/shm
none            100M     0  100M   0% /run/user

Detailed Storage

ubuntu@ip-172-31-33-140:/etc/projects/myapp$ sudo du -x / | sort -n | tail -40
98880   /home/ubuntu/.rvm/rubies/ruby-2.1.2
98908   /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.4.1
98908   /home/ubuntu/.rvm/rubies/ruby-2.1.5/lib
99436   /home/ubuntu/.rvm/rubies/ruby-2.1.5
107920  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1/ext/nokogiri/tmp/x86_64-linux-gnu/ports
107924  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1/ext/nokogiri/tmp/x86_64-linux-gnu
107928  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1/ext/nokogiri/tmp
108520  /var/lib/apt/lists
108572  /var/lib/apt
111044  /usr/src
119420  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1/ext/nokogiri
119424  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1/ext
145780  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/nokogiri-1.6.1
146972  /var/cache
155528  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/passenger-4.0.45/buildout
155916  /usr/lib/x86_64-linux-gnu
158396  /etc/projects/myapp/public/assets
163068  /etc/projects/myapp/public
198320  /home/ubuntu/.rvm/rubies
200824  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems/passenger-4.0.45
207540  /home/ubuntu/.rvm/gems/ruby-2.1.5/gems
231732  /var/lib
264856  /home/ubuntu/.rvm/gems/ruby-2.1.5
271292  /var/log/apache2
284432  /var/log
292684  /usr/share
403300  /usr/lib
583604  /home/ubuntu/.rvm/gems/ruby-2.1.2/gems
650888  /home/ubuntu/.rvm/gems/ruby-2.1.2
664560  /var
915752  /home/ubuntu/.rvm/gems
929616  /usr
1174228 /home/ubuntu/.rvm
1262052 /home/ubuntu
1262056 /home
3470180 /etc/projects/myapp/log
3782520 /etc/projects/myapp
3782524 /etc/projects
3789104 /etc
7795236 /

Nodes in Use

ubuntu@ip-172-31-33-140:/etc/projects/myapp$ df -i
Filesystem     Inodes  IUsed  IFree IUse% Mounted on
/dev/xvda1     524288 153278 371010   30% /
none            75346      2  75344    1% /sys/fs/cgroup
udev            72620    381  72239    1% /dev
tmpfs           75346    266  75080    1% /run
none            75346      3  75343    1% /run/lock
none            75346      1  75345    1% /run/shm
none            75346      2  75344    1% /run/user

Passanger Version

Phusion Passenger version 4.0.55

Apache Version

Server version: Apache/2.4.7 (Ubuntu)

RVM Version

rvm 1.26.4 (latest) by Wayne E. Seguin <[email protected]>, Michal Papis <[email protected]> [https://rvm.io/]

Ruby Versions

rvm rubies

   ruby-2.1.2 [ x86_64 ]
=* ruby-2.1.5 [ x86_64 ]

# => - current
# =* - current && default
#  * - default

Solution

You only have 7.5G of total space. Your /etc folder is using up 3.7G, which seems like waaaay too much, and your home directory is 1.3G. Use du -sh * on each of these directories to find out what is using so much space in them and delete them if they are no longer needed. For example, items in your ~/Downloads folder, old logs in /var/log, or whatever is chewing up so much space in /etc.