Ok, so LetsEncrypt is awesome but it has one MAJOR pitfall as of version 0.4.1-1 on Ubuntu, there is no delete option for the cli. There is however a workaround that will allow you to remove the domain from the cert renewal process and the chain. sudo rm