Scaling Django Migrations on AWS Part 2

In our last post we described the steps necessary to build a Hubot-driven workflow for running Django’s migrate command in an ECS cluster.

Putting Hubot to work requires us to write some Javascript or CoffeeScript that processes commands we send it via Slack along with a response that Hubot sends back to the Slack channel.


Scaling Django Migrations on AWS

Django’s migration support provides developers with a fairly comprehensive toolset for managing and versioning schema and data changes. Writing migrations in code provides a helpful layer of abstraction and makes testing easier, while at the same time adding an extra layer of safety.

Running Django’s migrate command iterates through an application’s set of ordered migrations and runs each one, as long as it hasn’t already been executed. Fortunately, the migrate command is also idempotent – running it multiple times (but not simultaenously) has no ill effects. One particular area of contention however, is how to automate the running of the migrate command at deployment time without executing in parallel on all deployment targets in a cluster.


Email Testing in Development

Testing software that requires unique email addresses is a pain. Automated testing tools can lessen the burden by using counters or random strings to create unique email addresses, but that stops at localized testing – those email addresses can’t actually receive mail. A common method often used to resolve this is a “catch-all” email alias that accepts any email sent to it and forwards it to a specfic address. While this method can work well for the solo developer, it’s inadequate in a team environment where each member needs to see the message.


Build a Load Balanced Docker Registry on AWS

Hosting a private Docker Registry for your container-based application gives you the flexibility to control the storage and accessibility of your Docker images.

While hosting Docker images on 3rd party platforms is recommended for simple deployments, the process of downloading updated images can be time consuming. Removing Internet latency by hosting the images in your own environment can improve download performance dramatically.


Accessing Amazon ECS Environment Variables For Debugging

Amazon’s Elastic Beanstalk service supports both single and multi-container Docker environments through Amazon ECS. Despite being in its infancy, Amazon ECS does provide an adequate environment for scaling Docker containers, though debugging those containers can present a challenge.


Cronally vs Periodic Tasks

Amazon recently announced support for periodic tasks in Elastic Beanstalk. This can be an ideal solution for some users who:

  • are already using a worker tier
  • aren’t accessing data on the web server
  • can tolerate inaccuracies in job timing

If you’re not already using a worker tier for your Elastic Beanstalk environment, the costs involved with running a dedicated EC2 instance purely for a handful of scheduled tasks isn’t all that enticing. The other issue this creates has to do with how work is sent to the worker tier instances.


Why We Chose to Write Our CLI in Go

Currently, Cronally does not have a web UI. Instead, we chose to focus on a CLI for a number reasons:

  • Quicker development
  • More stable MVP
  • Easier to use
  • Fits into end-user CI/CD model

We considered a few different languages for development, including Python and Ruby, but ultimately chose Go.


Introducing pycronally

pycronally is the latest library created for those who desire a programmatic interface to Cronally without using our CLI. pycronally provides direct access to the underlying API without having to implement your own interface.


Single Server Cron Jobs on a Cluster

I often see questions on StackOverflow and other Q&A tech sites about running cron jobs on single servers within clusters of AWS instances. Running the same job on each machine tends to be simple – ensure the appropriate crontab exists on every machine (usually through your deployment process, Docker image, or AMI) and you’re good to go.


Hugo Shortcode for Embedded Gists

Hugo is a fantastic static site generator written in Go. One particularly nice feature of Hugo is how simple it is to create custom shortcodes for customizing content generation beyond what’s supported by Markdown.