As the first blog in the series let’s keep it short and informative.
If you work on a large scale project like compilers you will always come across the need to have multiple versions as a part of your development arsenal. One might be the latest one(in sync with upstream), one might be for a brand new feature you are working on and the other might be some crash issue that you are investigating.
When I started out I thought about 2 ways of managing it:
- Keep separate repositories for all, it was easy to use but very difficult to manage (eg moving around commits in between isn’t straightforward). When you have the source of size LLVM it will quickly eat up all your memory and also looks very redundant, surely we can do better.
- The other option was to create separate branches. This can work but has major cons:
- Working on multiple branches at once is a nightmare, to say the least, we all know what happens when you try to edit files on different branches at once in an IDE.
- Switching can be expensive, especially for large scale projects developing rapidly and branches diverge really quickly.
- It’s a hassle to make sure you build to a designated directory for a branch, a bit of carelessness and you’ll end up losing a lot of time.
To summarize we would like to:
- Have git branch like functionality.
- Easy to work on multiple features simultaneously.
- Clean folder structure, separate build folder associated with source.
This is exactly where git worktree shines.
Let’s understand it using LLVM source code as example.
You have the main source inside llvm-project. Let’s say you have to start working on a new compiler feature and very much like to keep both sources and build-directories/binaries separate.
Run the command
cd llvm-project git worktree add ../new-feature
The above command will create a new worktree named new-feature inside new-feature folder. This new-feaure is effectively a git branch but with its own copy of source code and hence can have its own build folders. Additionally providing all functionality you have between 2 git branches (cherry-pick, merge, rebase, diff, etc).
Below is how the folder structure will look like after adding new-feature worktree.
yassingh@pc ~/workstation $ ls llvm-project new-feature yassingh@pc ~/workstation $ cd llvm-project; git branch * main + new-feature
You can keep going, let’s say you create another worktree for some register-allocation issue you are working on.
yassingh@pc ~/workstation/llvm-project (main) $ git worktree add ../regalloc-issue Preparing worktree (new branch 'regalloc-issue') Updating files: 100% (127092/127092), done. HEAD is now at 003078b62d8d [Clang][Driver] Add -mcpu=help and -mtune=help to clang
Folder structure now:
yassingh@pc ~/workstation $ ls llvm-project new-feature regalloc-issue yassingh@pc ~/workstation $ cd llvm-project; git branch * main + new-feature + regalloc-issue
Once you are done, it only makes sense to get rid of them
Getting rid of regalloc worktree
yassingh@pc ~/workstation/llvm-project (main) $ git worktree remove ../regalloc-issue/ yassingh@pc ~/workstation/llvm-project (main) $ git branch -D regalloc-issue Deleted branch regalloc-issue (was 003078b62d8d). yassingh@pc ~/workstation/llvm-project (main) $ git branch * main + new-feature yassingh@pc ~/workstation/llvm-project (main) $ ls ../ llvm-project new-feature
To maintain multiple compiler binaries associated with each worktree you can follow below folder structure.
yassingh@pc ~/workstation $ tree -L 2 . . ├── llvm-project │ ├── bolt │ ├── build-dbg │ ├── build-rel │ ├── clang │ ├── clang-tools-extra │ ├── cmake │ ├── compiler-rt │ ├── CONTRIBUTING.md │ ├── cross-project-tests │ ├── flang │ ├── libc │ ├── libclc │ ├── libcxx │ ├── libcxxabi │ ├── libunwind │ ├── LICENSE.TXT │ ├── lld │ ├── lldb │ ├── llvm │ ├── llvm-libgcc │ ├── mlir │ ├── openmp │ ├── polly │ ├── pstl │ ├── README.md │ ├── runtimes │ ├── SECURITY.md │ ├── third-party │ └── utils └── new-feature ├── bolt ├── build-dbg ├── build-rel ├── clang ├── clang-tools-extra ├── cmake ├── compiler-rt ├── CONTRIBUTING.md ├── cross-project-tests ├── flang ├── libc ├── libclc ├── libcxx ├── libcxxabi ├── libunwind ├── LICENSE.TXT ├── lld ├── lldb ├── llvm ├── llvm-libgcc ├── mlir ├── openmp ├── polly ├── pstl ├── README.md ├── runtimes ├── SECURITY.md ├── third-party └── utils
As you can see we are maintaining separate release and debug builds(build-rel, build-dbg) for both the worktrees which will be a lot more difficult to maintain if we had followed branch-like development.
Thanks for reading!!