My Scripts

Ending the week, I figured I should have a showcase of the scripts that brought this site to life, no health checks, no logs, just lots of scaffolds

Before that I should talk about my “system”

Thecist’s git “philosophy” - heavy on the quotes

A bare repository is very limited in what it can do, it doesn’t even have a copy of the code, just the data to create one. So any real work you end up doing will more often than not be on a clone

Luckily git was made to be distributed, so a clone is as good as any other, and as much as I delegate my private git server to be the single source of truth, I can very easily create a new bare repository from my various clones or my synced repos on GitHub

So what does this mean? With the flexibility of git and root access to my git server, I was suddenly given the oppurtunity to make some decisions, so I banned PRs on the server. Every PR must happen off server with me being the final reviewer before it is merged with the main branch and then pushed

The same goes for conflicts or any other type of aggregation that updates the main branch

This led to some opinionated design decisions, and some interesting problems to solve - in the future. I am looking forward to it, but for now it should provide enough context for the current scripts holding my site together

post-receive hook

This is less of a script and more of an orchestrator, it calls other scripts passing whatever inputs it gets to respective scripts

Hook flow

Apart from being a really good orchestrator, an extra superpower is the ability of any of the scripts it runs to be able to take on the role of the post-receive hook since it accepts the same inputs

#!/usr/bin/env bash
set -e

HOOKS_DIR="/usr/local/lib/git-hooks"
INPUT=$(cat)

echo "$INPUT" | "$HOOKS_DIR/deploy-main" || echo "Deploying main branch failed"
echo "$INPUT" | "$HOOKS_DIR/mirror-to-github" || echo "Mirroring code to GitHub failed"
echo "$INPUT" | "$HOOKS_DIR/manage-archive" || echo "Archiving branch failed"

Jumpscare!

I’ll make GitHub gist for the longer scripts no worries

Background scripts

Unlike our frontliners from the above code, there are some soldiers in the backline providing support, they aren’t like the flashy scripts that are literally drop in replacement for hooks - show offs -, so to compensate I gave them little underscores to start their names

_deploy-core.sh

It’s the primary script for deploying - gasp, who would have thought

Currently, it’s inputs are REPO_DIR, TARGET_DIR, PROJECT_NAME and BRANCH but I plan on updating BRANCH to HASH - one of the really cool problems to solve

Let’s go through them, REPO_DIR is the directory of the REPO - bear with me, TARGET_DIR is the folder you’d like your production code to be in, PROJECT_NAME is a unique identifier used to create the Docker image and BRANCH - you will not believe this - is the branch you plan on deploying

For the most part, it goes through the motions

  • Ensures all variables are present
  • Exposes its inputs as environment variables - really useful for any script it calls, as they automatically have access to the inputs
  • Searches for and runs a pre-build script located in the repo’s scripts/production folder
  • Deploys code using docker compose
  • Switches HEAD ref back to main - incase we deployed a branch

And here’s the gist as promised!

_undeploy-core.sh

Brace yourself for the impending déjà vu

Currently it uno reverses whatever _deploy-core does, apart from messing with the HEAD ref - all must point back to main

It takes in the same input paramters: REPO_DIR, TARGET_DIR, PROJECT_NAME and BRANCH, although I really want to switch BRANCH to HASH but it’s a good segway so I’ll talk about it later

It also goes through the motions

  • Deletes the container assigned to PROJECT_NAME
  • Deletes the image created for the container
  • Deletes the directory that stores the production code

And that’s it!

Oh yeah it also spares the main branch. Don’t tell the other branches, but “undeploy” is a nice way of saying rm… -rf

And here’s the gist!

Archiving

Now you might be thinking, “other branches?”, did you… lie to me? And to that I say… yeo?

I didn’t! Anytime I push a branch I’m immediately stopped by this Black screen of death

And for me that’s more than enough to remind me we don’t push new branches around here, so instead of outright blocking other branches from being pushed, I entertained the idea, and came up with archiving!

That’s right, taking advantage of the post-receive hook I decided to enable deployments and undeployments for branches - but only when they’re created

And if I do end up with more contributors one day, I could easily create users for them and update the script to prevent pushes that aren’t to main from them

Or just straight up use GitHub

So technically there are branches but they’re all in this form: v[0-9]+. I mean come on, is that really a branch or something I’m going to have to change the moment I need sub versions of an archive?

Now that I’ve masterfully won your trust back, lets talk about my two cool scripts for archiving and unarchiving repos

They both exist locally, cos there’s no way I’m ssh’ing to archive a repo

archive_repo.ps1

One day I’m going to have original names… today isn’t the day

So this script is mostly used to call my python script archive_repo.py in the same directory, but I’ve given bash too much attention already, all things must be equal

python "$PSScriptRoot\archive_repo.py" $args

archive_repo.py

This script

  • Incrementally searches for a branch in the format v[0-9]+ that hasn’t been taken on the bare repo - repo on my git server
  • Creates said branch locally
  • Pushes the branch to my private git server - this triggers the post-receive hook
  • And then deletes the branch locally - this prevents me from mistakenly updating and pushing the branch

I know it’s a glorified branch creation script, but it prevents me from having a branch that doesn’t show that error from earlier - cos it’s the only thing stopping me from breaking my rule

Oh also, you remember those environment variables _deploy-core and _undeploy-core make available through export? I end up using it to generate dynamic .env files like HOST=v1.blog.thecist.dev - which may or may not exist ;)

Here’s the gist!

unarchive_repo.ps1

It’s the law!

python "$PSScriptRoot\unarchive_repo.py" $args

unarchive_repo.py

I’ll be honest with you, I was way too lost in the prompts and chats to realize this was a single command, but it kind of worked out, and my Scripts directory only has two global scripts

So what does it do?

It deletes the version branch on the git server…

And yes, it has a gist - it’s that many lines

Hook replacements

And now for the cool guys, the drop in hook replacements, the mini orchestrators

deploy_main.sh

This guy checks all inputs to see if the main branch was updated, if it was, it calls _deploy-core on the main branch

#!/usr/bin/env bash
set -e

REPO_DIR=$(pwd)                                # e.g., /opt/repos/lib.git
REPO_NAME=$(basename "$REPO_DIR" .git)         # e.g., lib
TARGET_DIR="/home/git/deploy/${REPO_NAME}"
HOOKS_DIR="/usr/local/lib/git-hooks"

# Read input and look for main branch
DEPLOY=false
while read oldrev newrev refname; do
  branch=$(basename "$refname")
  if [[ "$branch" == "main" ]]; then
    DEPLOY=true
    break
  fi
done

if [[ "$DEPLOY" != "true" ]]; then
  echo "No changes to main branch; skipping deployment."
  exit 0
fi

TARGET_DIR="/home/git/deploy/${REPO_NAME}"
"$HOOKS_DIR/_deploy-core" "$REPO_DIR" "$TARGET_DIR" "$REPO_NAME" "main"

mirror-to-github.sh

This superstar keeps GitHub in sync with my repo - free backup!

#!/usr/bin/env bash
set -e

REPO_DIR=$(pwd)
CHANGES=$(cat)
echo "Changes being pushed:"
echo "$CHANGES"

cd "$REPO_DIR"
unset GIT_WORK_TREE
git push --mirror github

manage-archive.sh

And the script responsible for my recent sleep schedule

It searches for the creation and/or deletion of branches, and based on that runs either _undeploy-core or _deploy-core

#!/usr/bin/env bash
set -e

REPO_DIR=$(pwd)
REPO_NAME=$(basename "$REPO_DIR" .git)
HOOKS_DIR="/usr/local/lib/git-hooks"

# Read from stdin
while read oldrev newrev refname; do
  branch=$(basename "$refname")

  if [[ "$branch" =~ ^v[0-9]+$ ]]; then
    TARGET_DIR="/home/git/deploy/${branch}.${REPO_NAME}"
    PROJECT_NAME="${branch}-${REPO_NAME}"

    if [[ "$oldrev" == "0000000000000000000000000000000000000000" ]]; then
      echo "Archiving $REPO_NAME at branch $branch..."
      "$HOOKS_DIR/_deploy-core" "$REPO_DIR" "$TARGET_DIR" "$PROJECT_NAME" "$branch"

    elif [[ "$newrev" == "0000000000000000000000000000000000000000" ]]; then
      echo "Unarchiving $REPO_NAME from branch $branch..."
      "$HOOKS_DIR/_undeploy-core" "$REPO_DIR" "$TARGET_DIR" "$PROJECT_NAME" "$branch"
    fi
  fi
done

But I’ll be honest, I had to pull myself out to get the blog out today. After successfully creating the archive logic with dynamic secret - thanks to redis!

I started thinking about rollbacks

  • What if I had a system that deploys hashes, not branches?
  • What if deploy-main gets the most recent hash on the main branch to deploy, but uses the same TARGET_DIR and PROJECT_NAME so there aren’t any orphan folders and images when a new deployment comes in
  • What if the same is done for branches?
  • What if moving HEAD from the top of main to a hash in main deploys main from that hash

What if we had a new pointer just for that? A pointer that determines what part of main is deployed with archiving still being supported.

If I nail that, and maybe create new inputs for _deploy-core and _undeploy-core, rolling back could be as simple as printing a statement

Plans for the future

Luckily I caught myself - I’m working on a time management system, and being TheCist is not helping haha

I’ll stock that up for next week, along with

  • A framework for testing - regression, unit, integration
  • A script to update discord invite links weekly