Ubuntu No space left on device with plenty of space left

When attempting to use tab completion on my ubuntu 16.04 box running on an AWS EC2 instance I was greeted with the following error in the terminal.

cannot create temp file for here-document: No space left on device with plenty of space left

I ran the following command to see what my disk usage was like and was surprised to see that the box was only at 50% capacity.

df -h

$ df -h
Filesystem      Size  Used Avail Use% Mounted on  
udev            487M     0  487M   0% /dev  
tmpfs           100M  4.3M   95M   5% /run  
/dev/xvda1      7.8G  3.7G  3.7G  50% /
tmpfs           496M     0  496M   0% /dev/shm  
tmpfs           5.0M     0  5.0M   0% /run/lock  
tmpfs           496M     0  496M   0% /sys/fs/cgroup  
tmpfs           100M     0  100M   0% /run/user/1000  

Next, after a lot of research I was led to check the status of the inodes on the instance.

df -ih

Filesystem     Inodes IUsed IFree IUse% Mounted on  
udev             122K   360  122K    1% /dev  
tmpfs            124K   443  124K    1% /run  
/dev/xvda1       512K  512K    0K  100% /
tmpfs            124K     1  124K    1% /dev/shm  
tmpfs            124K     7  124K    1% /run/lock  
tmpfs            124K    16  124K    1% /sys/fs/cgroup  
tmpfs            124K     4  124K    1% /run/user/1000  

Hey! look! Im all out of inodes for this instance. This is caused because there are too many file pointers.

The solution is to remove some files, but which ones? I cant remove the files that are serving my websites, so what can I do?

Chances are, there are tons of old linux header files from previous kernel versions. These can be removed to free up a good chunk of inodes.

First

lets find out what kernel we are using.
uname -a

Linux ip-172-31-55-89 4.4.0-93-generic #116-Ubuntu SMP Fri Aug 11 21:17:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux  

Next

Lets see what headers are installed
dpkg -l | awk '/-headers-/ {print $2}'

linux-headers-4.4.0-64  
linux-headers-4.4.0-64-generic  
linux-headers-4.4.0-66  
linux-headers-4.4.0-66-generic  
linux-headers-4.4.0-70  
linux-headers-4.4.0-70-generic  
linux-headers-4.4.0-71  
linux-headers-4.4.0-71-generic  
linux-headers-4.4.0-72  
linux-headers-4.4.0-72-generic  
linux-headers-4.4.0-75  
linux-headers-4.4.0-75-generic  
linux-headers-4.4.0-78  
linux-headers-4.4.0-78-generic  
linux-headers-4.4.0-79  
linux-headers-4.4.0-79-generic  
linux-headers-4.4.0-81  
linux-headers-4.4.0-81-generic  
linux-headers-4.4.0-83  
linux-headers-4.4.0-83-generic  
linux-headers-4.4.0-87  
linux-headers-4.4.0-87-generic  
linux-headers-4.4.0-89  
linux-headers-4.4.0-89-generic  
linux-headers-4.4.0-91  
linux-headers-4.4.0-91-generic  
linux-headers-4.4.0-93  
linux-headers-4.4.0-93-generic  
linux-headers-4.4.0-96  
linux-headers-4.4.0-96-generic  
linux-headers-generic  
linux-headers-virtual  

Then

we probably need to make some space real quick so we can run the next command to purge the old kernel headers.

sudo dpkg -P linux-image-4.4.0-{64,66,70,71,72,75,78,79,81,83,87,89}-generic  

Finally

we can purge the old kernel headers to free up a chunk of inodes and storage.

sudo apt autoremove --purge  
Filesystem     Inodes IUsed IFree IUse% Mounted on  
udev             122K   360  122K    1% /dev  
tmpfs            124K   443  124K    1% /run  
/dev/xvda1       512K  453K   60K   89% /
tmpfs            124K     1  124K    1% /dev/shm  
tmpfs            124K     7  124K    1% /run/lock  
tmpfs            124K    16  124K    1% /sys/fs/cgroup  
tmpfs            124K     4  124K    1% /run/user/1000  

That 11% should be enough to keep this instance happy