Ubuntu No space left on device with plenty of space left
When attempting to use tab completion on my ubuntu 16.04 box running on an AWS EC2 instance I was greeted with the following error in the terminal.
cannot create temp file for here-document: No space left on device with plenty of space left
I ran the following command to see what my disk usage was like and was surprised to see that the box was only at 50% capacity.
df -h
$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 487M 0 487M 0% /dev
tmpfs 100M 4.3M 95M 5% /run
/dev/xvda1 7.8G 3.7G 3.7G 50% /
tmpfs 496M 0 496M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 496M 0 496M 0% /sys/fs/cgroup
tmpfs 100M 0 100M 0% /run/user/1000
Next, after a lot of research I was led to check the status of the inodes on the instance.
df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 122K 360 122K 1% /dev
tmpfs 124K 443 124K 1% /run
/dev/xvda1 512K 512K 0K 100% /
tmpfs 124K 1 124K 1% /dev/shm
tmpfs 124K 7 124K 1% /run/lock
tmpfs 124K 16 124K 1% /sys/fs/cgroup
tmpfs 124K 4 124K 1% /run/user/1000
Hey! look! Im all out of inodes for this instance. This is caused because there are too many file pointers.
The solution is to remove some files, but which ones? I cant remove the files that are serving my websites, so what can I do?
Chances are, there are tons of old linux header files from previous kernel versions. These can be removed to free up a good chunk of inodes.
First
lets find out what kernel we are using.
uname -a
Linux ip-172-31-55-89 4.4.0-93-generic #116-Ubuntu SMP Fri Aug 11 21:17:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Next
Lets see what headers are installed
dpkg -l | awk '/-headers-/ {print $2}'
linux-headers-4.4.0-64
linux-headers-4.4.0-64-generic
linux-headers-4.4.0-66
linux-headers-4.4.0-66-generic
linux-headers-4.4.0-70
linux-headers-4.4.0-70-generic
linux-headers-4.4.0-71
linux-headers-4.4.0-71-generic
linux-headers-4.4.0-72
linux-headers-4.4.0-72-generic
linux-headers-4.4.0-75
linux-headers-4.4.0-75-generic
linux-headers-4.4.0-78
linux-headers-4.4.0-78-generic
linux-headers-4.4.0-79
linux-headers-4.4.0-79-generic
linux-headers-4.4.0-81
linux-headers-4.4.0-81-generic
linux-headers-4.4.0-83
linux-headers-4.4.0-83-generic
linux-headers-4.4.0-87
linux-headers-4.4.0-87-generic
linux-headers-4.4.0-89
linux-headers-4.4.0-89-generic
linux-headers-4.4.0-91
linux-headers-4.4.0-91-generic
linux-headers-4.4.0-93
linux-headers-4.4.0-93-generic
linux-headers-4.4.0-96
linux-headers-4.4.0-96-generic
linux-headers-generic
linux-headers-virtual
Then
we probably need to make some space real quick so we can run the next command to purge the old kernel headers.
sudo dpkg -P linux-image-4.4.0-{64,66,70,71,72,75,78,79,81,83,87,89}-generic
Finally
we can purge the old kernel headers to free up a chunk of inodes and storage.
sudo apt autoremove --purge
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 122K 360 122K 1% /dev
tmpfs 124K 443 124K 1% /run
/dev/xvda1 512K 453K 60K 89% /
tmpfs 124K 1 124K 1% /dev/shm
tmpfs 124K 7 124K 1% /run/lock
tmpfs 124K 16 124K 1% /sys/fs/cgroup
tmpfs 124K 4 124K 1% /run/user/1000
That 11% should be enough to keep this instance happy