LetsEncrypt Certificates that Never Expire

Using DNSSEC and RFC2136 to delegate authority to LetsEncrypt

In this episode Devin learns how to install SSL certificates from LetsEncrypt, and how to keep them from ever expiring. Previously on Tangled Web Services, Devin learned to stop worrying and to embrace CORS.

LetsEncrypt certificates are popular because they are free. But they come with a hidden cost: they expire after three months. This makes them less appealing to organizations that don’t have dedicated support staff. But with a little effort, it doesn’t have to be that way.

Devin needed to create a certificate for his art supply store client, Rock Paper Scissors. He decided to tackle the problem of expiring certificates once and for all.

First, the basics. All certificates, whether they are from LetsEncrypt or another certificate authority, are only doled out to legitimate responsible parties of a domain. Responsible parties must be able to prove that they are not an impostor.

LetsEncrypt has more than one way for responsible parties to meet this burden of proof, but the gold standard is through DNS. The presumption is that if you have the capability to make changes to your DNS records, then you are a responsible party.

LetsEncrypt provides a command line tool certbot that automates the steps necessary to create a new certificate. Devin began by installing LetsEncrypt's certbot utility:

dnf install letsencrypt

There are two ways to use certbot, manual and fully automated. He started by trying out the manual process, issuing this command:

certbot certonly --manual --preferred-challenges dns-01 \
     -d rock-paper-scissors.com

After answering a few self-explanatory questions, the manual process generated a random single-use hash key and instructed Devin to add that key to the DNS records for rock-paper-scissors.com. The script waited for Devin to do his thing. He added a TXT record something like this:

_acme-challenge.rock-paper-scissors.com.  IN  TXT  \
     "yo2Fz_4BIQnJy73g8iDPtLwVgtDPna6W97Sb5NqMVew"

Then he went back to the terminal window and resumed the paused certbotscript. The LetsEncrypt server issued a query for the TXT record from Devin's DNS server, matched the response to the random hash key it previously generated, and issued the new certificate.

The certificate and private key were placed in /etc/letsencrypt/archive/ and symlinks to them were placed in /etc/letsencrypt/live/.

The TXT record had served its purpose and was no longer needed, so he removed it from his DNS records.

Devin modified his RWSERVE HTTP/2 Server configuration file rwserve.confto use the new certificate:

host {
    hostname rock-paper-scissors.com
    tls {
        private-key `/etc/letsencrypt/live/
                     rock-paper-scissors.com/privkey.pem` 
        certificate `/etc/letsencrypt/live/
                     rock-paper-scissors.com/fullchain.pem` 
    }
}

Devin had successfully created and installed his first LetsEncrypt certificate.

Delegating authority to change DNS

The next time he needed to create a new certificate, Devin decided to use certbot's fully-automated process. Its benefit was that he wouldn't need to manually create and delete the DNS TXT record required by LetsEncrypt. That meant that he could create the certificate with a single command line instruction.

The automated process worked by delegating authority to LetsEncrypt, to make the DNS changes. In order to allow LetsEncrypt to make those changes, but no one else, the automated process uses the protocol defined in RFC2136. That protocol describes a way to configure a DNS server with a secret key, and to grant holders of that key permission to make limited changes to DNS records.

Devin was savvy enough to make the necessary changes to his DNS server. It was a two step procedure. Here’s what he did.

The first step called for the creation of a new DNSSEC key, which he created using this command:

cd /etc/pki/named
dnssec-keygen -a HMAC-SHA512 -b 512 -n HOST -r /dev/urandom \ 
    letsencrypt-rfc2136.key.

The generated secret key was added to /etc/named.conf with these lines:

key "letsencrypt-rfc2136.key." {
  algorithm hmac-sha512;
  secret "4WFNjDX8Rg...AMpZs/bd1GqA==";
};

The second step was to change his DNS zone file to allow outside scripts having that key to add or remove TXT records beginning with _acme-challenge, which is the prefix used by LetsEncrypt. His new client was Rock 'n Roll Sushi, and he changed its zone file to allow that type of remote change:

zone "rock-n-roll-sushi.com"
{
    update-policy { grant letsencrypt-rfc2136.key. name \
        _acme-challenge.rock-n-roll-sushi.com. txt; };
};

With this in place Devin was prepared to set up the LetsEncrypt credentials file, which he would use for all future clients. He placed the credentials in the file /etc/pki/named/letsencrypt-rfc2136. It consisted of:

dns_rfc2136_server = 16.32.64.128
dns_rfc2136_port = 53
dns_rfc2136_name = letsencrypt-rfc2136.key.
dns_rfc2136_secret = 4WFNjDX8Rg...AMpZs/bd1GqA==
dns_rfc2136_algorithm = HMAC-SHA512

Finally, he was ready to issue the command line instruction to create a new certificate for his second client, Rock ‘n Roll Sushi, like this:

certbot certonly --dns-rfc2136
     --dns-rfc2136-credentials /etc/pki/named/letsencrypt-rfc2136
      -d rock-n-roll-shushi.com

The LetsEncrypt script worked beautifully. First it used the secret key to remotely create a new TXT record on Devin’s DNS server. Then it waited a minute for DNS to propagate. Then it created the new certificate and private key. Finally it cleaned up by removing the just created TXT record.

Devin had successfully automated the process of creating a new certificate.

Automated renewals

LetsEncrypt recommends that certificates be renewed rather than recreated. This is accomplished by adding the --keep-until-expiring option to the certbot command. When issuing certbot commands that way, a new certificate is generated and downloaded only when it is nearing its expiration date. Commands issued more than 30 days prior to expiration will gracefully exit without making any changes on the DNS servers, and without generating or downloading new certs and keys.

Setting up cron to automate the renewal was a one-liner. It looked like this:

30 02 * * 0 root certbot certonly --keep-until-expiring \
     --noninteractive \
     --quiet \
     --max-log-backups 0 \
     --dns-rfc2136 \
     --dns-rfc2136-credentials /etc/pki/named/letsencrypt-rfc2136 \
     -d rock-n-roll-shushi.com

This causes cron to quietly run the certbot command, as the root user, every Sunday at 2:30 a.m. The first eight weeks will return gracefully without doing anything. The ninth week, which falls within the 30 day window, is recognized as a legitimate request. A new certificate is generated and downloaded. The symlinks to the cert and the key, in /etc/letsencrypt/live/, are updated to point to the new cert and key in /etc/letsencrypt/archive/.

Voilà. The next time the HTTP/2 Server is restarted, it automatically picks up and uses the new certificate.

Wildcard certs

Wildcard certificates are a recent addition to LetsEncrypt. With a wildcard certificate, it is possible to use a single certificate for both the base domain and all its subdomains.

Devin’s third client, Rock City Vinyl, uses a subdomain for its API microservice. For it, he decided to create a wildcard certificate using LetsEncrypt. (This was still an experimental feature when he did this, so it required him to explicitly use the new LetsEncrypt server at https://acme-v02.api.letsencrypt.org/directory.)

He made the necessary change to the rock-city-vinyl.com DNS zone file in order to allow remote changes via the RFC2136 protocol. Then he issued this command:

certbot certonly --dns-rfc2136 \
     --dns-rfc2136-credentials /etc/pki/named/letsencrypt-rfc2136 \
     --server https://acme-v02.api.letsencrypt.org/directory \
     -d "rock-city-vinyl.com,*.rock-city-vinyl.com"

The only important note about this command, is that the base domain must be specified along with the wildcard subdomain. Without it, the base domain would not be covered by the certificate.

With this in place, he could now use HTTPS for the servers at www.rock-city-vinyl.com and api.rock-city-vinyl.com.

Devin was feeling pretty good about his DNS prowess. Still he thought it might be time to look at managed DNS providers, but which one, Amazon, Cloudflare, Google?

· · · EPISODE 4 · · · · · · · · TANGLED WEB SERVICES · · · · · · · EPISODE 6 · · ·

Devin’s #1 recommendation: got JavaScript? — download the HTTP/2 Server with modular JavaScript plugins.

Download RWSERVE HTTP/2 Server