TimHosting a private Docker Registry for your container-based application gives you the flexibility to control the storage and accessibility of your Docker images.
While hosting Docker images on 3rd party platforms is recommended for simple deployments, the process of downloading updated images can be time consuming. Removing Internet latency by hosting the images in your own environment can improve download performance dramatically.
However, this doesn’t come without a cost. Most 3rd party registries operate on redundant, highly available infrastructures which you would need to implement yourself to obtain a similar level of reliability. Your registry must be able to scale as your application scales to avoid overwhelming the registry at deployment time. It would be fairly easy to unintentionally create the symptoms of a DoS attack when lots of servers pull a new image during a deployment.
Fortunately, we can leverage some AWS services to build a scalable registry solution that’s load-balanced, secured by TLS, and supports authentication.
We’ll use ECS to run two containers: the official registry image and an NGiNX proxy. An ELB will pass SSL traffic over the TCP layer and allow NGiNX to terminate the SSL traffic and provide authentication. Finally, we’ll store our images in S3 to allow our load balanced containers to access the same storage facility. Do note, the registry itself will not require authentication this way, relying on NGiNX to authenticate each request. If the registry’s port is exposed to the outside, anyone would be able to push and pull images to and from your registry.
We’ll need to create an IAM role that will allow the agent application that runs on each EC2 instance to communicate with ECS. To do that, follow the instructions here.
A CloudFormation template provided at the end of this article can be used to easily provision the stack which includes all of the necessary resources with the exception of an IAM role described below.
The CloudFormation template requires several parameters:
NginxConfigBucketused to store your NGiNX configuration files, SSL certificates, and htpasswd authentication fileAwsRegionto launch the stack intoAwsAccessKeyAccess Key of an IAM user with appropriate permissions to launch entire CloudFormation stacks and have read access toNginxConfigBucketAwsSecretAccessKeySecret Access Key for the user aboveAwsRegistryBucketS3 bucket to store the Docker images inKeyPairKeyPair to use to launch the EC2 instances
Next, we’ll need to create an ECS cluster. You can do this manually by visiting the ECS console and clicking “Create Cluster”.
Once the cluster has been created we’ll launch an instance of AMI ami-b540eade with the IAM role previously created. For regions outside of us-east-1, see this link for alternate AMIs. In order for this instance to be aware of the ECS cluster, we’ll need to set an environment variable in the instance’s userdata before it launches. The following content added to the userdata section will configure the instance to use the cluster name provided:
#!/bin/bash
echo ECS_CLUSTER=CLUSTER_NAME_HERE >> /etc/ecs/ecs.config
The provided CloudFormation template automatically sets this up for you based on the name of the cluster.
Since we’ll be handling SSL termination and authentication using NGiNX, we need to configure NGiNX as a reverse proxy and provide it with an htpasswd file as well as our SSL key and certificate. To make this easier, I’ve created a public image that provides the following:
- NGiNX-1.9.4 compiled with nginx-headers-more (required for proxied authentication)
- NGiNX configuration that looks for nginx.htpasswd, domain.crt, and domain.key in
/usr/local/nginx/conf/conf.d - A run script that fetches those files from a configurable S3 bucket at runtime
- A health check URL for ELB at /ssl
- A connection to an upstream linked Docker container named
registryon port5000
The next step involves creating the files required by NGiNX.
Follow the instructions for creating an htpasswd file under the heading “Native basic auth” here
Upload the htpasswd file (name it nginx.htpasswd), your SSL key (named domain.key), and your SSL certificate (named domain.crt, ensure this also includes any intermediate certificates in the proper order) to your NginxConfigBucket.
Next, review the permissions on your AwsRegistyBucket. It’s important that the IAM user configured by the AwsAccessKey and AwsSecretAccessKey has the following permissions on the AwsRegistryBucket:
S3:ListBucketon the bucket itselfS3:GetObject,S3:PutObject,S3:DeleteObject,S3:ListObjectson all files in the bucket (i.e. [S3_BUCKET]/*)
For additional details on the setup of this stack including the configurations for VPC, ELB, and the ECS container definitions, please refer to the CloudFormation template below.
By default, the CloudFormation template creates an autoscaling group of 1 registry server. To enable true autoscaling, increase MaxSize above 1 and add any additional scaling properties such as load, bandwidth, etc. as needed.