AWS LetsEncrypt Lambda or Why I Wrote a Custom TLS Provider for AWS Using OpenTofu and Go

These days, it's challenging to imagine systems that have public API endpoints without TLS certificate protection. There are several ways to issue certificates:

• Paid wildcard certificates that can be bought from any big TLS provider
• Paid root certificates that sign all downstream certificates that are issued by corporate PKI systems
• Free certificates issued by TLS providers like LetsEncrypt or AWS Certificate Manager
• Self-signed certificates, issued by OpenSSL or another tool

Within the context of this post, I will mainly discuss free certificates that can be used inside of AWS, but not only by AWS services. Clearly, using anything other than AWS Certificate Manager makes no sense if you exclusively use managed AWS services and don't have strict security requirements. AWS Certificate Manager offers a very convenient and speedy method of issuing certificates via DNS or HTTP challenges; however, you face basic AWS limitations if you need to use these certificates outside of AWS services (API Gateway, ALB, NLB, etc.) , such as an EC2 instance running Nginx that needs a physical certificate file. Additionally, even if you request it, AWS Certificate Manager does not display the certificate content.

At this point it’s a good time to remind you about LetsEncrypt, a more widely used tool than Certificate Manager—at least because it doesn't depend on the cloud. Unfortunately, there are no built-in LetsEncrypt certificate issuance techniques available in AWS. It is possible to utilize the certbot tool for your EC2 or ECS services, but in that scenario, you will need to consider how to configure the renew process. I also don't want to combine different strategies since I think it's better to have a single procedure for everything since it reduces the whole system complexity.

Taking that into consideration, I created a Lambda function that automatically issues and renews LetsEncrypt certificates without requiring complex configuration. The certificate can be utilized at any AWS service using ARN along with AWS Certificate Manager certificates after the initial certificate issue. Additionally, you can use a physical certificate version that is kept in AWS Secrets Manager in whatever location you choose, whether it be an EC2 instance running Nginx or another place.

How does AWS LetsEncrypt Lambda works

The Lambda function that is described in this article is written on Go v1.22. All outcome resources such as DNS records, secrets, or certificates are controlled by Amazon IAM role, that is created via Terraform code by default. The sequence of Lambda actions is the following:

1. Get an event containing a certificate list. Typically, this event can be a result of manual execution, or execution by cron that is made via aws_cloudwatch_event_target. Event example:

{
   "domainName": "hackernoon.referrs.me",
   "acmeUrl": "prod",
   "acmeEmail": "alexander.sharov@cloudexpress.app",
   "reImportThreshold": 10,
   "issueType": "default",
   "storeCertInSecretsManager" : true
}

2. Verify whether the certificate exists in the AWS Certificate Manager. If yes, confirm the expiration date.
3. Start the LetsEncrypt DNS-01 challenge if the number of days until the expiration date is fewer than the reImportThreshold. This step involves Lambda creating a TXT record matching the domain name to the AWS Route53 zone and waiting for your certificate to be ready.
4. Lambda updates the certificate in the AWS Certificate Manager when it's ready.
5. Lambda will store certificate files inside the AWS Secrets Manager if storeCertInSecretsManager is true.

Lambda implementation details

The code

The Lambda is written on Go 1.22. Using as few libraries as possible helped me maintain my goal of keeping the code dry. The full list of required go libraries:

Docker image

I used gcr.io/distroless/static:nonroot as a basic docker image. For Go applications that don't require libc, this image is perfect. It is not completely empty as scratch and includes the following:

• CA certificates: no need to copy them from any other stage.
• /etc/passwd: contains users and groups such as nonroot.
• /tmp folder.
• tzdata: in case you want to set the timezone other than UTC.

Build process

In large software projects, overseeing the build process can turn into a laborious and time-consuming chore. Makefiles can help automate and streamline this process, ensuring that your project is built efficiently and consistently. For that reason, I prefer to use Makefile for all my Golang projects. The file is simple:

##@ General
help: ## Display this help.
	@awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n  make \033[36m<target>\033[0m\n"} /^[a-zA-Z_0-9-]+:.*?##/ { printf "  \033[36m%-15s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' $(MAKEFILE_LIST)

fmt: ## Run go fmt against code.
	go fmt ./...

vet: ## Run go vet against code.
	go vet ./...

##@ Build
build: fmt vet  ## Build service binary.
	go build -o bin/lambda main.go

run: vet ## Run service from your laptop.
	go run ./main.go

##@ Test
lint: ## Run Go linter
	golangci-lint run ./...

test: ## Run Go tests
	go test ./...

From the CICD side I used the typical setup for Go application

1. GitHub Actions as continuous integration.

2. ghcr.io as docker registry. Compared to DockerHub, this one offers two key features that make it my preference to use:

   1. The build, test, and deployment workflows can be more easily automated straight from the GitHub repository thanks to GHCR's smooth integration with GitHub Actions. This can increase productivity and simplify the development process.
   2. GHCR leverages GitHub’s permission model, allowing users to manage access to container images using the same teams and permissions they use for their code repositories. This simplifies user management and enhances security.

3. kvendingoldo/semver-action: my GitHub Actions plugin for automatic versioning. It’s A GitHub action that generates SemVer compatible tags for repository commits. The action can manage versions, generate GitHub releases, and control release branches inside of the repository. It works wonderfully with both single and monorepos.

4. Automatic changelog generation. I enjoy changelogs! In the context of other OpenSource projects that I manage (e.g., https://github.com/tofuutils/tenv, https://github.com/tofuutils/tofuenv, etc.), my team recognized the importance of informing users about changes.

5. golangci-lint. From my perspective, all code should be reviewed by a static code analyzer. SonarQube cannot be set up for all projects, however golangci, in my opinion, is sufficient for small to medium Go projects.

6. codespell.yml. In addition to code checking, it is a good idea to verify the grammar, especially if you have a large amount of documentation.

How to deploy Lambda to AWS via Terraform/OpenTofu

To work with AWS LetsEncrypt Lambda via OpenTofu you need to do the following steps:

1. Add module into to your OpenTofu / Terraform code

   module "letsencrypt_lambda" {
     source = "git@github.com:kvendingoldo/aws-letsencrypt-lambda.git//files/terraform/module?ref=0.31.4"
   
     blank_name = "kvendingoldo-letsencrypt-lambda"
     tags       = var.tags
   
     cron_schedule = var.letsencrypt_lambda_cron_schedule
     events        = var.letsencrypt_lambda_events
   
     ecr_proxy_username     = var.ecr_proxy_username
     ecr_proxy_access_token = var.ecr_proxy_access_token
   }
   

2. Specify variables

   variable "tags" {
     default = {
       hackernoon : "demo"
     }
   }
   
   variable "ecr_proxy_username" {
     default = "kvendingoldo"
   }
   
   variable "ecr_proxy_access_token" {
     default = "ghp_xxx"
   }
   
   variable "letsencrypt_lambda_cron_schedule" {
     default = "rate(168 hours)"
   }
   variable "letsencrypt_lambda_events" {
     default = [
       {
         "acmRegion" : "us-east-1",
         "route53Region" : "us-east-1",
         "domainName" : "hackernoon.referrs.me",
         "acmeUrl" : "stage",
         "acmeEmail" : "alexander.sharov@cloudexpress.app",
         "reImportThreshold" : 100,
         "issueType" : "default",
         "storeCertInSecretsManager" : false
       }
     ]
   }
   
   

3. Pay attention to variables ecr_proxy_username and ecr_proxy_access_token. By default, AWS Lambda cannot pull images from sources other than AWS ECR. Fortunately, the AWS team created ECR Proxy cache, which can fetch images from publicly available registries such as DockerHub or GHCR and store them inside ECR. Despite this possibility, AWS does not allow images to be pulled without a token, even from open public repositories, thus you must acquire a personal GitHub token to gain access to pre-built Docker images. Alternatively, you can pull my GitHub repository, build the image locally, then upload it to your pre-existing ECR repository. In this scenario, the example can be modified as follows:

   module "letsencrypt_lambda" {
     source = "../../"
   
     blank_name = "kvendingoldo-letsencrypt-lambda"
     tags       = var.tags
   
     cron_schedule = var.letsencrypt_lambda_cron_schedule
     events        = var.letsencrypt_lambda_events
   
     ecr_proxy_enabled = false
     ecr_image_uri     = "<YOUR_ACCOUNT_ID>.dkr.ecr.us-east-2.amazonaws.com/aws_letsencrypt_lambda:<VERSION>"
   }
   

4. When you complete changing the code, run the following command to install OpenTofu by OpenTofu version switcher tenv:

   $ tenv tofu install
   

5. And finally, execute the following commands to apply the produced code:

   $ tofu init
   $ tofu plan
   $ tofu apply
   

6. Wait until the code is deployed to AWS and events are triggered. After a few minutes, you will see ready certificates inside the certificate manager. Example:

   

7. Starting now, AWS can use the issued certificate at any services by ARN.

8. If you need to use the certificate outside of AWS services or have access to its content, set the storeCertInSecretsManager event option to true. In this situation, when Lambda completes the basic execution, the certificate will be saved in AWS Secrets Manager. It gives users more flexibility: they can inspect the certificate's content, work with it directly from EC2, etc. To learn more about AWS Secrets Manager, read the official guide.

   

Environment variables

How to check LetsEncrypt Lambda logs

In scope of work with aws-letsencrypt-lambda you may occasionally want to review the logs. It's quite easy to accomplish:

1. Go to AWS Cloudwatch, click to “Log groups”
2. Find log group with name, that you specified in OpenTofu code. For example, in my case it’s /aws/lambda/kvendingoldo-letsencrypt-lambda
3. Go to the group, select the desired stream from the list, and review the logs.

How to trigger Lambda manually via AWS UI

1. Go to Lambda function that has been created via OpenTofu. Click to “Tests” button.

   

2. Fill Test Event and click Test

   {
     "domainName": "<YOUR_VALID_DOMAIN>",
     "acmeUrl": <stage | prod>,
     "acmeEmail": "<ANY_VALID_EMAIL>",
     "reImportThreshold": 10,
     "issueType": "<default | force>",
     "storeCertInSecretsManager" : <true | false>
   }
   

   Example #1:

   {
      "domainName": "hackernoon.referrs.me",
      "acmeUrl": "prod",
      "acmeEmail": "alexander.sharov@cloudexpress.app",
      "reImportThreshold": 10,
      "issueType": "default"
   }
   

3. Wait until execution will be completed. You can the execution log is available in Cloudwatch. Usually the initial issue takes around 5 minutes.

How to test Lambda locally

1. Clone https://github.com/kvendingoldo/aws-letsencrypt-lambda repository to your laptop

2. Configure AWS Cli credentials via the official guide.

3. Examine the environment variables section and set the minimum number of variables needed. Since LetsEncrypt will limit the amount of retries per hour for ACME_URL="prod", I advise using ACME_URL="stage" for testing. Environment variables example:

   export AWS_REGION="us-east-2"
   export MODE=local
   export DOMAIN_NAME="hackernoon.referrs.me"
   export ACME_URL="stage"
   export ACME_EMAIL="alexander.sharov@cloudexpress.app"
   export REIMPORT_THRESHOLD=10
   export ISSUE_TYPE="default"
   export STORE_CERT_IN_SECRETSMANAGER="true"
   

4. Execute the lambda locally via the following command:

   go run main.go
   

5. Following Lambda's successful execution, the following log will appear.

   INFO[0000] Starting lambda execution ...                
   INFO[0000] Lambda will use STAGING ACME URL; If you need to use PROD URL specify it via 'ACME_URL' or pass in event body 
   INFO[0000] Certificate found, arn is arn:aws:acm:us-east-2:004867756392:certificate/72f872fd-e577-43f4-ae38-6833962630af. Trying to renew ... 
   INFO[0000] Checking certificate for domain 'hackernoon.referrs.me' with arn 'arn:aws:acm:us-east-2:004867756392:certificate/72f872fd-e577-43f4-ae38-6833962630af' 
   INFO[0000] Certificate status is 'ISSUED'               
   INFO[0000] Certificate in use by []                     
   INFO[0000] Certificate valid until 2024-08-31 13:50:49 +0000 UTC (89 days left) 
   INFO[0000] Try to get certificate for hackernoon.referrs.me domain 
   2024/06/02 17:56:23 [INFO] acme: Registering account for alex.sharov@referrs.me
   2024/06/02 17:56:24 [INFO] [hackernoon.referrs.me, www.hackernoon.referrs.me] acme: Obtaining bundled SAN certificate
   2024/06/02 17:56:25 [INFO] [hackernoon.referrs.me] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/12603809394
   2024/06/02 17:56:25 [INFO] [www.hackernoon.referrs.me] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/12603809404
   2024/06/02 17:56:25 [INFO] [hackernoon.referrs.me] acme: Could not find solver for: tls-alpn-01
   2024/06/02 17:56:25 [INFO] [hackernoon.referrs.me] acme: Could not find solver for: http-01
   2024/06/02 17:56:25 [INFO] [hackernoon.referrs.me] acme: use dns-01 solver
   2024/06/02 17:56:25 [INFO] [www.hackernoon.referrs.me] acme: Could not find solver for: tls-alpn-01
   2024/06/02 17:56:25 [INFO] [www.hackernoon.referrs.me] acme: Could not find solver for: http-01
   2024/06/02 17:56:25 [INFO] [www.hackernoon.referrs.me] acme: use dns-01 solver
   2024/06/02 17:56:25 [INFO] [hackernoon.referrs.me] acme: Preparing to solve DNS-01
   2024/06/02 17:56:26 [INFO] Wait for route53 [timeout: 5m0s, interval: 4s]
   2024/06/02 17:57:00 [INFO] [www.hackernoon.referrs.me] acme: Preparing to solve DNS-01
   2024/06/02 17:57:00 [INFO] Wait for route53 [timeout: 5m0s, interval: 4s]
   2024/06/02 17:57:30 [INFO] [hackernoon.referrs.me] acme: Trying to solve DNS-01
   2024/06/02 17:57:30 [INFO] [hackernoon.referrs.me] acme: Checking DNS record propagation. [nameservers=109.122.99.130:53,109.122.99.129:53]
   2024/06/02 17:57:34 [INFO] Wait for propagation [timeout: 5m0s, interval: 4s]
   2024/06/02 17:57:46 [INFO] [hackernoon.referrs.me] The server validated our request
   2024/06/02 17:57:46 [INFO] [www.hackernoon.referrs.me] acme: Trying to solve DNS-01
   2024/06/02 17:57:46 [INFO] [www.hackernoon.referrs.me] acme: Checking DNS record propagation. [nameservers=109.122.99.130:53,109.122.99.129:53]
   2024/06/02 17:57:50 [INFO] Wait for propagation [timeout: 5m0s, interval: 4s]
   2024/06/02 17:58:30 [INFO] [www.hackernoon.referrs.me] The server validated our request
   2024/06/02 17:58:30 [INFO] [hackernoon.referrs.me] acme: Cleaning DNS-01 challenge
   2024/06/02 17:58:30 [INFO] Wait for route53 [timeout: 5m0s, interval: 4s]
   2024/06/02 17:59:09 [INFO] [www.hackernoon.referrs.me] acme: Cleaning DNS-01 challenge
   2024/06/02 17:59:09 [INFO] Wait for route53 [timeout: 5m0s, interval: 4s]
   2024/06/02 17:59:43 [INFO] [hackernoon.referrs.me, www.hackernoon.referrs.me] acme: Validations succeeded; requesting certificates
   2024/06/02 17:59:43 [INFO] Wait for certificate [timeout: 30s, interval: 500ms]
   2024/06/02 17:59:45 [INFO] [hackernoon.referrs.me] Server responded with a certificate.
   INFO[0203] Certificate has been successfully imported. Arn is arn:aws:acm:us-east-2:004867756392:certificate/72f872fd-e577-43f4-ae38-6833962630af 
   INFO[0204] Secret updated successfully. SecretId: arn:aws:secretsmanager:us-east-2:004867756392:secret:hackernoon.referrs.me-NioT77 
   INFO[0204] Lambda has been completed
   

6. That is it. Starting now, AWS can use the issued certificate at any services by ARN or in other locations where it is physically necessary by obtaining it from the AWS Secrets Manager.

Hands-on experience using it over more than 4 years at AWS

I've been using the Lambda function in production for almost four years. Over the years, various aspects of the initial implementation have changed:

1. Previously, AWS prohibited the usage of any non-ECR registries as Lambda sources. It has not changed, however AWS has added ECR proxy for GitHub, DockerHub, and a few additional registries. Without this functionality, we had to manually push Lambda images to our personal ECR and replace URL to the image in Terraform code. Now the OpenTofu code does it automatically via ECR Proxy.

2. In the beginning, I considered introducing various challenges such as http-01 or tls-alpn-01, but no one questioned me about it for four years. It is still present on GitHub issues, and if this capability is required, we can work together to create it.

3. I didn't want to utilize LetsEncrypt certificates at pure EC2 instances when the project originally started, but these days it's standard practice. As I previously stated, in certain situations, a certificate can be retrieved from AWS Secrets Managed using the AWS cli.

4. I’ve written a lot of new Go code over the years, so I can tell that the original Lambda code in my repository isn't as fancy as it could be. There is a significant difference between it and my most recent Go project, tenv (OpenTofu, Terraform, Terragrunt, and Atmos version manager, written in Go), but in any case, the code is still generally supported, so making modifications to it won't be too problematic. Occasionally, I will undertake significant refactoring to make the code more elegant.

5. The same Lambda is being used for years in several different projects. Additionally, I'm co-founder of DevOps platform cloudexpress.app, where our team manages TLS certificates for all our clients using the AWS LetsEncrypt Lambda to simplify the automation processes.

Now let's talks about numbers. Over a period of 4 years, this project has helped many people and been used in numerous OpenSource and over 30 commercial projects. The Lambda issues more than 2000 certificates and don’t want to stop on that.

Conclusion

AWS LetsEncrypt Lambda is a suitable solution for you, if

• You must have a physical version of the certificate and will use it from non-AWS native services such as EC2 Nginx.
• You do not want to rely on AWS Certificate Manager to manage the TLS certificate issuance and renewal process (check logs, set renewal dates, etc.).
• You would like to get email notifications from LetsEncrypt when your certificate expires, or will be expired soon.
• You want to personalize the solution by changing the Golang code (for example, changing the LetsEncrypt challenge, storing the certificate in Hashicorp Vault, etc.).

If you discovered that at least one of these points applies to your situation, you are welcome to use AWS Lambda. Also, if you wish to participate in development, I am always open to new issues and Pull Requests at GitHub. Project URL: https://github.com/kvendingoldo/aws-letsencrypt-lambda.