Cloud, juju, Ubuntu, Uncategorized

Force upgrade, one of the best kept juju secret

This post has been updated to reflect the new commands for Juju 2.0, thanks babou

As I continue to work on building, reviewing, and troubleshooting charms there’s still a pain point that comes from when a charm enters an error state. In charm schools we’ve discussed various ways around this: debug-hooks, juju ssh, and finally juju resolved --retry. This works pretty well when you have a single unit of a service but comes with draw backs, mainly when you have multiple units.

To get around this, and to avoid modifying code on just one unit which I’ll forget to copy down and inevitably lose my patches, I’ve been employing the following:

juju deploy --repository=~/charms local:trusty/findon

Deploy the charm I’m creating, then get an error!

environment: local
machines:
  "0":
    agent-state: started
    agent-version: 1.23-alpha1.1
    dns-name: localhost
    instance-id: localhost
    series: trusty
    state-server-member-status: has-vote
  "1":
    agent-state: started
    agent-version: 1.23-alpha1.1
    dns-name: 10.0.3.226
    instance-id: marco-local-machine-2
    series: trusty
    hardware: arch=amd64
services:
  findon:
    charm: local:trusty/findon-12
    exposed: false
    units:
      findon/0:
        agent-state: error
        agent-version: 1.23-alpha1.1
        machine: "1"
        open-ports:
        - 2003/tcp
        - 9000/tcp
        public-address: 10.0.3.226

Perform typical investigation, view logs on the machine, piece together and solve the issue. Make my changes locally to my charm. Now for the interesting portion. I want to verify my changes but I need to publish them to the environment first. In order to do so, we would typically use juju upgrade-charm. However, this will not work since the unit is in an error state. As a result all future events are queued for execution and await the resolution of the error. I could run juju upgrade-chamr then juju resolved but that wouldn’t maintain the current errored event and I may not be able to replicate it (ie: it’s in the install hook). Typically this is when you would destroy the service or environment and try again but despite how quick juju is, this still is a time-consuming process.

Using the following you can upgrade the files for the charm in place without having to wait in the event queue:

juju upgrade-charm --force --repository=~/charms findon # Juju 1.X
juju upgrade-charm --force-units --path ~/charms/trusty/findon findon # Juju 2.X

This will replace the files without having to wait in the event queue. You can verify this by running juju status again and validating the charm version has incremented, despite still being in error:

environment: local
services:
  findon:
    charm: local:trusty/findon-13
    exposed: false
    units:
      findon/0:
        agent-state: error
        agent-version: 1.23-alpha1.1
        machine: "1"
        open-ports:
        - 2003/tcp
        - 9000/tcp
        public-address: 10.0.3.226

At this point you can correct the error to retry the hook with the new code:

juju resolved --retry findon/0 # Juju 1.x
juju resolved findon/0 # Juju 2.x

For me this has helped speed up my already quick paced development cycle for this project.