Let's Improve Your and Co-workers' DX ~ CI Improvement Tips ~

そもそもの背景として、「自分や同僚のDX (Developer Experience) を上げる」というテーマで LT をしようとしていました。

そこで CI を設定、保守・運用するにあたって便利な Git、GitHub 周りの Tips と 具体的な CI として CircleCI の Tips を取り上げたんですが・・・

書いていたらスライドが30枚くらいになってしまい、これは15分でも終わるか分からないしそんな勉強会ないしなーということでブログ記事に起こします。Marpで書いていたスライドをそのまま貼っているので、見やすいかどうかは分からないです。例えばURLの生成はスライド幅の都合上、無理やり配列を使っていますが、通常であれば文字列宣言で終わります。

現職、前職でも参照できるように英語で資料を使っていましたが、まだ推敲前だった 1 のでかなり適当です。

スライド版は以下。

Let's improve your and co-workers' DX ~ CI improvement tips ~ - Speaker Deck

アジェンダ

  • Git Tips

    • Commit-aware
      • Know if HEAD is a merge-commit
      • Get the latest merge commit
      • Get a branch ref of a merge-commit
      • How can we know the last merged branch?
    • Change-aware
      • Know if a directory/file has been changed
      • What changes happned between merges?
  • Github Tips for CI

    • Get the base branch of your PR
    • Know if directories/files have been changed in the PR
    • Create a PR from CI
  • CircleCI Tips

    • General tips
      • md5 hash of multiple files
      • Download the latest artifact from the specific branch
    • Go with better caches!
      • Use version for cache keys
      • Use environment variables for cache key versions
    • For your good development
      • Don't need to source your utilities multiple time
      • Use {{ .Revision }} or {{ .BuildNum }} when spinning up CI environment

Let's Improve Your and Co-workers' DX

~ CI Improvement Tips ~


Who?


DX - Developer Experience

  • General speaking, there are several definitions:

    • User Experience for Developers
    • Quality of Your Developer Life etc...
  • What this presentation uses

    • Quality of Your Developer Life {on CI, when modifying CI conf}

Why should we improve DX?

OBVIOUS, right?

Why don't you improve your DX?

Enough? Your company is awesome!

Please recruit me to join


Git Tips for CI

It's better to know two approaches!

  • Commit-aware
    • Know if HEAD is a merge-commit
    • Get the latest merge commit
    • Get a branch ref of a merge-commit
    • How can we know the last merged branch?
  • Change-aware
    • Know if a directory/file has been changed
    • What changes happned between merges?

Commit-aware Git tips


Know if HEAD is a merge-commit

[[ -n "$(git show --merges HEAD -q)" ]]
  • --merges only merge commits will be shown
  • -q suppress diff part

So you can infer the job trigger! e.g. merge or direct push

allow_deploy() {
  # Never allow deployment if pushed directly!
  [[ ! -n "(git show --merges HEAD -q)" ]]
}

Know sha1 of the latest merge-commit

git show --merges -1 -q --format='%h'
  • --format='%h' only sha1 hash will be shown

Not useful if you will be using only this tip :)


Get a branch ref of a merge-commit

# origin/foo/bar
remote_slash_branch=$(
    git show $sha1 --merges --format='%s' -q | \
    awk '$0=$NF'
)

# origin/foo/bar => foo/bar, origin/foo => foo
echo $remote_slash_branch | sed 's/^[^\/]*//'
  • --format='%s' only subject will be shown
  • The message format must be like .... $REMOTE/$BRANCH_NAME
  • Omit the sed command if you wanan use this for local merge

How can we know the last merged branch?

  • Combine these tips!
    • Know sha1 of the latest merge-commit
    • Get a branch ref of a merge-commit
hash=$(git show --merges -1 -q --format='%h')

remote_slash_branch=$(
    git show $hash --merges --format='%s' -q | \
    awk '$0=$NF'
)

Change-aware Git tips


Was the file/directory changed?

# all workspace
[[ -n $(git diff ${ref}) ]]

# 'foo' file/directory
[[ -n $(git diff ${ref} -- 'foo') ]]

# .kt files under 'bar' directory
git diff ${ref} --name-only -- 'bar' | \
    grep "*.kt" >/dev/null 2>&1
    
# submodule change
[[ -n $(git diff ${ref} --diff-filter=T) ]]

What changes happned between merges?

l=$(git show --merges -q -1 --format=%h)
s=$(git show --merges -q -2 --format=%h)

# Show all files which have been changed between them
git diff $s...$l --name-only

# For example,
# If app's version has been changed,
if [[ -n "$(git diff $s...$l -- "app/version")" ]]; then

  # This should fail if REAMDE is not updated
  [[ -n "$(git diff $s...$l -- "README.md")" ]] || exit 1
fi

Not yet enough...

Let's see Github Tips and CircleCI Tips!


Github Tips for CI

  • Get the base branch of your PR
  • Know if directories/files have been changed in the PR
  • Create a PR from CI

Get the base branch of your PR

paths=(
  "https://api.github.com/repos"
  "/$CIRCLE_PROJECT_USERNAME"
  "/$CIRCLE_PROJECT_REPONAME/pulls"
  "/$(basename $CIRCLE_PULL_REQUEST)"
)
URL="$(echo ${paths[*]} | tr -d " ")"

curl -H "Authorization: token $GH_TOKEN" \
    "$URL" | jq '.base.ref'
# ruby -rjson -ne 'puts JSON.parse($_)["base"]["ref"]'
  • Don't fetch all remote refs because the cost is high.

Was the file/directory changed in the PR?

  • Combine these tips!
    • Was the file/directory changed? from Git Tips

    • Get the base branch of your PR

base_branch_name=$(...)

# make sure fetch the base branch. 
# Usually CI doesn't have it on local
git fetch origin $base_branch_name

# Use Git Tips! e.g.
git diff $base_branch_name...HEAD -- ...

Create a PR from CI

paths=(
  "https://api.github.com/repos"
  "/$CIRCLE_PROJECT_USERNAME"
  "/$CIRCLE_PROJECT_REPONAME/pulls"
)

URL="$(echo ${paths[*]} | tr -d " ")"
  
CURRENT_BRANCH="$(git rev-parse --abbrev-ref HEAD)"
fields=("\"head\": \"$CURRENT_BRANCH\"", \
      "\"base\": \"<base branch>\"", \
      "\"title\": \"<PR title>\"")
JSON_BODY="{${fields[*]}}"

curl -s -H "Authorization: token $GH_TOKEN" \
  -H "Content-Type: application/json" \
  -d "$JSON_BODY" "$URL"
  • Bash Script can create a PR! Don't need to use other languages!

CircleCI Tips

  • General tips
    • md5 hash of multiple files
    • Download the latest artifact from the specific branch
  • Go with better caches!
    • Use version for cache keys
    • Use environment variables for cache key versions
  • For your good development
    • Don't need to source your utilities multiple time
    • Use {{ .Revision }} or {{ .BuildNum }} when spinning up CI environment

md5 hash of multiple files

  • {{ checksum "a" }}-{{ checksum "b" }}-{{ checksum "c" }}-.... 😫

  • Get md5 hash of a file which contains the output of the following command

# For examaple, multi-module project
while read target_file; do
    md5sum $target_file
done < <(find . -name "build.gradle" | sort) 
# ↑ sort is important
# `find` might be more complicated in some use-cases

Download the latest artifact from the specific branch

EDITED cuz the previous implementation was wrong

latest_artifacts() {
    local -r build_num=$(curl -u "$CIRCLECI_TOKEN:" "https://circleci.com/api/v1.1/project/github/DeployGate/deploygate-android?filter=completed" | \
      ruby -rjson -e 'puts JSON.parse(STDIN.read).find { |j| j["branch"] == ENV.fetch['PRODUCTION_BRANCH'] && j["build_parameters"]["CIRCLE_JOB"] == "build" }["build_num"]')

    curl -u "$CIRCLECI_TOKEN:" "https://circleci.com/api/v1.1/project/github/DeployGate/deploygate-android/$build_num/artifacts"
}

get_url() {
    cat - | jq -r ".[].url" | grep "<use regexp>" | head -1
}

URL=$(latest_artifacts | get_url)

curl -o "<output>" "$URL?circle-token=$CIRCLECI_TOKEN"

Use version for cache keys

It would be your help when you want to refresh caches.

  • e.g. when configuring CI, when introducing new key parts
- v1-danger-{{ checksum "~/Gemfile.lock" }}
- v2-gradle-...

Cache key versions matter

  • You need to upgrade all cache versions of save/restore sections
  • Sometimes we forget to modify keys... 😩
  • If we can use environment variables for cache keys, this can be solved...
  • However, {{ .Environment.variableName }} doesn't mean we can use any arbitrary variables... So let's use workaround!

Use environment variables for cache key versions

cache_version_keys: &cache_version_keys
  CACHE_VERSION_OF_DANGER: v1
  
environment:
  <<: *cache_version_keys
 
run: |
  while read temp; do
    var_name="$(echo $temp | awk -F= '$0=$1')"
    echo "$temp" | sed "s/$var_name=//" > ~/$var_name
  done < <(env | grep "CACHE_VERSION_OF_")

# keys
- {{ checksum "~/CACHE_VERSION_OF_DANGER" }}-danger-...

You need to change only one variable when bumping up the version!


Don't need to source utils multiple times

BASH_ENV is a entry point to be loaded before running each steps.

run: echo 'source <your util file>' >> $BASH_ENV
run: echo 'export XXXX=YYYYY' >> $BASH_ENV
  • If you are using custom images which don't extend CircleCI's image, then this tips might not work for you.

Use {{ .Revision }} or {{ .BuildNum }} when spinning up CI environment

I guess some of you have written wrong caches by the correct key when configuring CI environment, right?

# Use this share caches between multiple jobs
- v1-foo-bar-{{ .Revision }}

# Use one-time cache key for the debug perpose 
- v1-foo-bar-{{ .BuildNum }}
# Use the key when restoring caches if cached 💯

After debugging, please remove {{ .Revision }} and {{ .BuildNum}}, and also bump up version of cache key, v1 to v2.


BTW


DroidKaigi 2019

Please join us and enjoy :)


  1. ただし推敲したからといってしっかりとした英語が書けるわけではないので大した話ではないです。