posted October 14, 2019 by dave

Using Github Actions For CI Of Lucky Apps

Recently, I tried out Github Actions for the first time (I know, I know, I am late to the game). It was a breeze to set up CI for my tiny identicon shard, mainly because the crystal template that Github offers more or less worked out-of-the-box.

But when I tried to use the same setup for fortunate, the Lucky based application powering this blog, things went a lot less smoothly.

The biggest hurdle was available software: The Github standard CI template for crystal uses the crystallang/crystal container image to run the specs. This image does not include NodeJS and yarn, which Lucky uses to manage JavaScript dependencies. Nor does it include Chrome, which is needed for LuckyFlow-based UI tests.

The Ubuntu-based standard environment for Github Actions does include node, yarn and Chrome, but of course it does not include crystal.

At this point I saw three possible ways out:

  1. Create my own container image including all dependencies, publish it to Docker Hub and keep maintaining it for the forseeable future. In many ways this would probably be the best solution and I still hope someone does this someday. But I am not (yet) so heavily invested in the whole Docker ecosystem that I felt up to that task.
  2. Install node, yarn and Chrome into the running crystal container. Totally possible, but I could not help but feel that downloading these on every CI run is too time-consuming and a waste of resources.
  3. Install Crystal into the standard Ubuntu environment provided by Github Actions. Of course this is only marginally better than the second option, but at least it saves some bandwidth.

So I went with option 3. Here is a step-by-step explanation of the working configuration I came up with:

name: Crystal CI

This is the default name taken from the original template. Not very creative but at the same time it is hard to come up with something as concise and fitting.

on:
  push:
    branches:
      - master
  pull_request:
    branches: 
      - master

Github Actions offers to run jobs when certain events occur. For CI purposes I found the push and pull_request events to be most suitable. In other projects I also used the schedule event that allows running time based jobs. Having a time based build, e.g. every night, can be great to catch flaky specs, but I thought it would be overkill for this small project.

jobs:
  build:

    runs-on: ubuntu-latest

This is the beginning of the actual job configuration. runs-on defines the virtual environment this job is supposed to run in. Github offers macos and windows environments as well. For my purposes using the latest Ubuntu LTS release is perfect.

    services:
      postgres:
        image: postgres:10.10
        env:
          POSTGRES_USER: fortunate
          POSTGRES_PASSWORD: fortunate
          POSTGRES_DB: fortunate_test
        ports:
        # will assign a random free host port
        - 5432/tcp
        # needed because the postgres container does not provide a healthcheck
        options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5

For some reason this took me a couple of days to figure out. While the services key is of course well-documented in Github's docs of the Configuration Syntax, I did not stumble upon one example where it was actually used until I googled for a very specific (but unrelated) problem and found an answer on SO.

Once you know you can use this to define services that your job needs, it is easy to find the Services Example Repository which thankfully uses PostgreSQL as one of the two examples (the other being Redis). I did not find any reference to this repository in the official Github Actions docs, but I might have simply missed it. Again, if you have no idea what to even look for, it is kind of hard to find.

This let me run a PostgreSQL server that I needed for my Lucky App without doing another costly installation into the virtual environment. Startup and setup are reasonably fast as well.

    steps:
    - uses: actions/checkout@v1

The steps sections describes an arbitrary number of commands that comprise a job. The steps will be listed individually when running the job and a failure of one step will lead to failure of the whole job.

The fist step for most (all?) CI jobs is to checkout the code. This is encapsulated within a ready-made "action" provided by Github. There are quite a few ready-made actions that can be used to compose jobs. Many can be found on Github's Marketplace.

    - name: Setup crystal repo
      run: curl -sSL https://dist.crystal-lang.org/apt/setup.sh | sudo bash
    - name: Install crystal and system dependencies
      run: sudo apt install crystal libyaml-dev

This is where I install crystal into the virtual environment. I use the steps from crystal's official documentation. This will always install the latest version of crystal, which may or may not be desired. crystal still regularly has breaking changes in new releases that might break your build. But this can also be a great reminder to upgrade, so it not necessarily a bad thing.

Note that libyaml-dev is another compile-time requirement that is not present in the stock virtual environment.

    - name: Install crystal dependencies
      run: shards install

This is standard crystal stuff again. Using the shards command to install dependencies mentioned in the shard.yml file.

    - name: Install JS dependencies
      run: yarn install
    - name: Compile Assets
      run: yarn dev

Besides crystal dependencies a Lucky app usually has JavaScript dependencies as well. These are managed with the yarn package manager.

After installing the dependencies the assets are being compiled. Lucky makes sure that the assets are available and will even raise an exception if they are missing. So it is important to generate them before running the specs.

    - name: Run tests
      run: crystal spec
      env:
        DATABASE_URL: "postgres://fortunate:fortunate@localhost:${{ job.services.postgres.ports['5432'] }}/fortunate_test"

And this is the final step: Running the actual specs. Note that I added an environment variable with the URL to the PostgreSQL database. This includes the credentials I set above and and also inserts a variable that Github provides with the port number. This is necessary because when a service is defined as noted above it will assign a random TCP port on each run. To get the actual port number from within a running job, one can query the job.services.postgres.ports array.

So that is it. It cost me some time to get this up and running but it was great to see the first build go green. You can find the finished version here.