How does CI work in projects2?

My own code hosting platform, projects2, implements what I consider proper CI.

That is, it does something like this when you want to integrate changes from a branch in a Git repo:

def integrate(repo, branch):
    with lock(repo):
        sh("git clone {repo}")
        sh("git merge origin/{branch}")
        sh("<command to run test suite>")
        sh("git push")

In projects2, you actually don't need to create a branch. You just push to the main branch and a pre-receive hook intercepts that push and runs a CI script and rejects the commit if the script doesn't pass.

Let's see how it works by making two incompatible changes in two clones of the gittest repo:

gittest-a$ echo aaa >> README.md && git commit -am 'Add aaa'
[main 6bdffaa] Add aaa
 1 file changed, 1 insertion(+)

gittest-b$ echo bbb >> README.md && git commit -am 'Add bbb'
[main 60db681] Add bbb
 1 file changed, 1 insertion(+)

Then we push a first:

gittest-a$ git push
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 14 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 337 bytes | 337.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Hello from projects2 pre-receive hook!
remote: Building Dockerfile.ci...
remote: Running Dockerfile.ci...
remote: Sleep 0
remote: Sleep 1
...

And while that is running, we push b:

gittest-b$ git push
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 14 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 339 bytes | 339.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Hello from projects2 pre-receive hook!
remote: Building Dockerfile.ci...
remote: Running Dockerfile.ci...
remote: Sleep 0
remote: Sleep 1
...

We see that the CI script runs for both commits in parallel. Since we pushed a first, that finishes first:

...
remote: Sleep 28
remote: Sleep 29
remote: ('VCS_COMMITS_SINCE_1_0_0', '6')
remote: ('VCS_COMMITS_SINCE', '19')
remote: Ensuring /opt/rlprojects/web/gittest gone...
remote: Tagging not yet possible. Do manually:
remote: git tag 1.0.0 6bdffaa2b00f721c656f4a1fb3980d50e1020e39 && git push --tags
remote: Building /opt/rlprojects/web/index.html...
remote: Building /opt/rlprojects/web/gittest.html...
To projects.rickardlindberg.me:gittest.git
   db2c997..6bdffaa  main -> main

A little later, the CI script for b finishes as well, but with an error:

...
remote: Sleep 28
remote: Sleep 29
remote: ('VCS_COMMITS_SINCE', '19')
remote: ('VCS_COMMITS_SINCE_1_0_0', '6')
remote: Ensuring /opt/rlprojects/web/gittest gone...
remote: Tagging not yet possible. Do manually:
remote: git tag 1.0.0 60db681c900a82659c334139a72ebcc7d9e72a6f && git push --tags
remote: Building /opt/rlprojects/web/index.html...
remote: Building /opt/rlprojects/web/gittest.html...
remote: error: cannot lock ref 'refs/heads/main': is at 6bdffaa2b00f721c656f4a1fb3980d50e1020e39 but expected db2c997804a03d099a207b3c01ec5eeb54dba384
To projects.rickardlindberg.me:gittest.git
 ! [remote rejected] main -> main (failed to update ref)
error: failed to push some refs to 'projects.rickardlindberg.me:gittest.git'

That is, Git rejects the commit since changes have been made since the push was originally made.

If we try to push b again, we get another error:

gittest-b$ git push
To projects.rickardlindberg.me:gittest.git
 ! [rejected]        main -> main (fetch first)
error: failed to push some refs to 'projects.rickardlindberg.me:gittest.git'
hint: Updates were rejected because the remote contains work that you do not
hint: have locally. This is usually caused by another repository pushing to
hint: the same ref. If you want to integrate the remote changes, use
hint: 'git pull' before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

This time the error message makes a little more sense.

So projects2 ensures that only one commit can be pushed at a time and that that commit is also at the latest in the repository and passes the CI script.

It doesn't implement the exact same locking mechanism as in the pseudo example. This was actually the first time I tested this behavior. I expected Git to immediately reject the second push since there was already another push in progress. But it didn't.

One problem that projects2 has (even if Git had worked like I thought it would) is that it might produce artifacts that get published even if the commit is not accepted. That is not good. It should publish artifacts atomically based on if the commit is accepted by Git or not. I'll add it to the TODO.