Charms, Code, juju, Planet, Ubuntu

Deploying Discourse in production, lessons learned

This post is from several months ago. The Discourse charm has since evolved and we have merged our changes back in to trunk and use pure upstream + plugins

The Discourse charm has been updated to work as expected. There are still the lessons learned though!

For the past month I’ve been running the test Discourse site for Ubuntu, which has been a lot of fun. The upstream project is super responsive and working a break neck speeds to land not just new features but bug fixes alike. I wanted to document a bit on how I’ve been running this site using Juju, what I shouldn’t have done, and how I hope to break the mold with future deployments.

In the beginning there were clouds

For those interested, here’s the charm, as you read it you’ll notice it still needs a bit of work to become solid and charm store worthy, but if all you’re interested in is the charm, there you go!

Moving on, if you don’t already know about Juju, then here’s all you need to know. Juju is fucking amazing. So, to stand up the current production version of Discourse all I did was (after opening an account with HP Cloud)

juju bootstrap
juju deploy cs:~marcoceppi/discourse
juju deploy postgresql
juju add relation discourse postgresql:db-admin
juju expose discourse
juju set discourse admins=marcoceppi

Finally, I assigned a floating IP to the discourse unit via HP Cloud Console. That was it though, that’s all it took to get the latest version of discourse running in a production environment on a cloud that people could visit, register, and post.

So what? How did it get to how it looks today?

Writing charms is a pretty fun process. What I’ve learned from writing them is, if you want to write a good charm you need to be at least using that service in a production environment. When I started the Discourse charm I had simply translated the installation instructions to hooks and proclaimed that was that! Once I stood it up for the first time when running Ubuntu Discourse I realized I was missing quite a few key components.

Configuration is second only to relations

Juju really shines when you start building complex infrastructure using preexisting charms and just relating one service to another. This is really crazy awesome to just plug two services together using one command, it’s truly what lends a lot of power to Juju as an orchestration tool. However, in addition to being able to leverage other charms in the Charm Store, what made the Discourse charm something I could use in production, was adding a few simple configuration options

  • repository
  • release
  • thins
  • env

These may not seem like much, but I can now do versioned rollouts of Discourse from my own fork of the upstream code using the repository and release configuration options. This, being able to specify the source and version of the software, is considered a best practice of charming and it’s examples like this that really hit home why authors should take the time to invest in similar options. Repository is pretty straight forward, it’s the URL for a git repository (defaulting to Conversely, the release is a git ref (in this case a git ref can either be a branch or a git tag). It defaults to latest-release which is what upstream uses to always point at the latest tagged release of Discourse (in addition to also using numeric release tags).

In my fork of the project I have a master branch which I use to sync with upstream, then several “feature” branches which I’m constantly rebasing with master in order to have my patches always at the top of the revision history but also knowing when upstream changes something that “breaks” my patches. These branches have things like ubuntu-sso integration, an “Ubuntu” theme, enhancements to the markdown parser, onebox support for Ubuntu-centric blogs, and other misc changes that make sense in the context of an Ubuntu Discourse community.

When I get ready to do a new release, I create a “release” branch which involves branching the latest upstream commit and then squashing each of my feature branches on top of that. In doing so I’ve created a check point which has pure upstream with my modifications on top. When I need to roll out a change I can just juju set discourse release=20130716-2215 for example (as I name my branches using Ymd-hi format) and the charm will fetch that branch, make all changes necessary, perform all upgrades, the restart the discourse service. If I need to rollback to a previous release, I simply change the release configuration option and just like that everything is back to that snapshot.

At least, that’s what it’s supposed to do in theory.

Protoduction: When Prototyping becomes production.

My prototyping process clashes with production values, I like to think of it as my Dev bleeding too much in to Ops. When I write charms I usually build a base that should do something, deploy it, then rapidly iterate on top of it by shelling in to the deployed node, modifying the hook files, and using juju resolved --retry have the hook execute. Rinse, repeat, until I get a desired chunk of work done. Then rsync the changes back to my local machine, tear down the environment, stand it up again, rinse, repeat. The Ubuntu Discourse went from public prototype to “production” pretty quickly. As a result the charm that’s deployed is several version behind what’s in the repository and most of the prototyping has been done during maintenance windows on the machine. Effectively, I’ve been just hacking on things in production.

As I’ve noted time, after time, and time again in my maintenance notes; I don’t like doing this but because I’m doing this in my spare time it’s become a fact of life. To overcome this I’ve been slowly bringing the state of the charm to one where I can simply run juju upgrade-charm discourse and everything will just be OK. When this will be ready I can’t say for sure, could be next maintenance window could be three weeks from now. With local provider landing soon in Juju and testing become more awesome, I imagine it’ll be far easier to rev the charm than having to set up and tear down an environment every time I want to test a few things.

So, when it comes to update the source, I do – in essence – run juju set discourse release=.... However, due to the above I actually have to ssh in to the machine and run the commands manually, which looks something like this

cd ~/discourse
git fetch origin
git checkout -b 20130716-2215 origin/20130716-2215
bundle install
bundle exec rake db:migrate
bundle exec rake assets:clear
bundle exec rake assets:precompile
sudo restart discourse

As you can see, there isn’t much that’s complicated with this, it’s essentially copy and paste. However, because I wasn’t as diligent in the beginning with keeping the deployed charm sync’d I’m stuck in this loop where I must do this myself. Until I get the charm properly updated to reflect all the new items I’ve worked on it’ll remain this.

Where do we go from here?

It’s embarrassing to say the least that I haven’t been able to dedicate the time to finishing the charm, so hopefully shining a spotlight on my shameful practices will motivate me to finish the charm itself. If you’re interested in what it’s like to continually run an up-to-date Discourse setup I’ll continue updating what I do during maintenance windows on the site. The Discourse project is a very exciting one, one that can definitely benefit from a juju as an orchestration tool. Reading through the upstream forum there’s lots of people having trouble setting these up in their production environments. Having a tool like juju and an accompanying charm would go a long way in getting people setup and prepared to scale if their forum grows.

Once the charm is complete and the Ubuntu Discourse environment updated I’ll be able to not only submit the charm to the store but also save a bit of time during rollouts to the deployment!