Guild Alpha Analysis: Recommendations for Open Source Development

Guild Alpha
#1

Wiki post open for editing

In the wiki post below we collect information for analysis of the entire open source “stack” from the perspective of the goals of Guild Alpha. Click ‘edit’ to add your own input.

:speech_balloon: → Discuss in: Guild Alpha Analysis: Discussion

1 Like
Guild Alpha Analysis: Discussion
Solidground 2024: Maintainers Log (wiki)
#2

Last Updated: 2024-02-10

Summary

An analysis of the entire open source “stack” from the perspective of the goals of Guild Alpha. Recommendations will be made alongside each analysis.

Creatively, I want each analysis to look (as opposed to actually being) official. As if it was written by some government body. Using Air Accident Investigation reports as reference.

Recommendation Targets

  • Lower the barrier to entry for contributing to open-source projects
  • Fast track the learning process of new contributors to equalise the skill set of all open-source contributors
  • Remove barriers for the formation of ad hoc teams
  • Minimise effort required to sustain development

Other Considerations

  • Avoid subscription to a specific ethos (i.e. “cloud-native”) or set of tools
  • Many contributors are not coders
  • Recommendations should be as close to their context as possible

This specific thread is for my research notes and general discussion around the idea. Each section will get its own thread after first draft.

Developing With Docker

Cloud-native but local

Development Containers - A specification that adds extra metadata to existing container formats such as Dockerfiles. This metadata can either be embedded in a container format or with a ‘devcontainer.json’ file. Using this metadata tools such as DevPod, Gitpod and Github Workspaces can spin up reproducible environments for development work. A good blog post explaining DevContainers.

DevPod - Open source tool used to create reproducible developer environments. Each developer environment runs in a separate container and is specified through metadata provided from a ‘devcontainer.json’ file (as per Development Containers spec).

Podman Pods - Pods make a lot of sense compared to “compose” in terms of structure for development environments. The services are essentially placed in one container together. Uses kubernetes yaml. Built into Podman instead of having to install another tool like docker-compose/podman-compose. Doesn’t work with Docker. Doesn’t work with Docker though or Dev Container spec. Does support custom metadata so could build it’s own dev container spec and integrate with podman/podman desktop.

All Require remote development from code editer/ide.
Why doesn’t Podman Desktop support dev containers?

Podman vs Docker daemon. Replacing docker with podman.
Notes for Podman compared to docker:

Developing Bare-metal

Tends to be NIX related.

Devbox - A command-line tool that lets you easily create isolated shells for development. You start by defining the list of packages required for your project, and Devbox creates an isolated, reproducible environment with those packages installed. Includes system packages and services like PostgreSQL. Uses a custom JSON file on top of NIX technologies. There’s a worry of lock-in.

devenv - Examples Allows the declaration, using Nix Language, of an isolated & reproducible development environment including system packages and services like PostgreSQL. Due to the versatility in configuration it avoids lock-in with downside of a harsher learning curve.

Devbox and devenv try to achieve the same goal. Biggest difference is that Devbox is a high level abstraction on Nix tech and a custom JSON format. devenv uses Nix lang, so harsher learning curve, and while you can use its custom format you can also base it off Nix’s flake system without much more effort. Doing it the second way means you aren’t locked-in to devenv.

I can identify 4 workflows:

  1. Devbox - Simple to use and doesn’t require special knowledge. Downside is the possibility of lock-in.
  2. devenv - Requires Nix language and is more powerful than Devbox. Still the possibility of lock-in.
  3. devenv + flakes - Requires Nix language and uses Nix native features. Only lock-in is with Nix itself. And I can’t work around that :stuck_out_tongue: Use docker I guess
  4. Flakes Only - If you don’t want to use devenv then you can use however the flake stuff works

Standard (std) - A DevOps framework for the SDLC with the power of Nix and Flakes. The std project provides a standardized way to structure nix code, based on flakes, in order to avoid a lot of the complexity that is inherent in nix projects as they grow. There’s good documentation, and an additional book.

Organist - Similar to Devbox and devenv, but organist is built to surface the nickel language instead of nix, while remaining compatible with Nix. The README has comparisons to similar tools.

Note: Flakes are experimental and have major gotcha’s + good practices. See: Flakes aren’t real.

Other relevant projects

direnv - Mentioned alongside devenv a lot. Seems to allow you to continue using your terminal rather than being forced to swtich to bash with devenv shell. It augments existing shells with a new feature that can load and unload environment variables depending on the current directory.
12 factor app -
FLEEK - A high level wrapper around Nix Home Manager that uses YAML instead of Nix language.
Nix Home Manager - OS independent way of decoratively configuring user environments. These user specific (non-global) configurations tend to include installing software or applications and creating dotfiles.
devshell - devenv is built on top of it. Confusing docs.
Starship - Referenced by devenv. This was also mentioned in relation to this Liquidprompt.
agenix Securely manage and deploy secrets in Nix environments using age encryption.
Josh - Just One Single History. A proxy for any git host, to combine the advantages of a monorepo with those of multirepo setups by leveraging a fast, incremental, and reversible implementation of git history filtering.
TestContainers: Provide throwaway, lightweight instances of databases, message brokers, web browsers, or just about anything that can run in a Docker container, then integration test against that. (HN discussion).

Developing Hybrid

Both Devbox and devenv can be used alongside the Dev Container spec and can generate a .devcontainer.json file.

Command Runners

Just - Similar to make but without the build functionality. Talk HN might be complementary to Nix, when invoking nix-shell in the justfile. Alts Interest
Make - Is a build tool but can function like Just as a Command Runner.
bash - Vanilla really. Readily available.

There’s a bit of discussion on this in the Earthly FAQ.

Maintaining consistent code style across editors

EditorConfig - Widely supported spec in the form of a single file for defining code styles without forcing a specific workflow or IDE onto devs.

If there is a requirement of extensions for code editors then they should be stated in a README.

Development Containers have a feature, though currently only for VS Code and IntelliJ IDEA, that allows “customizations” for specifying default settings and extensions. The downside to this is it forces an IDE and it’s configurations onto the developer. It may be useful for those unable to work on their local machine.

CI - Linting and Unit Testing (Excluding CD at the moment due to lack of knowledge)

Chosen due to “portability”. Decouples from a specific CI provider such as Github Actions and can be run locally on developer machine for ease of use:
Dagger
Earthly
Garden

Dagger and Earthly are the big ones. However, they all seem to do roughly the same thing with similar approaches.

The biggest difference between Dagger and Earthly seems to be regular code vs Earthfile. 1 Garden does similar and uses YAML.

Advantages:
??

Disadvantages:
??

https://continuousdelivery.com

On the Subject of Version Control

Add a nomenclature section here to help agree upon terms

Status Quo

Git is the de facto version control tool for open source development. In 2005, Linus Torvalds created git for the purpose of developing the Linux kernel. It’s a so-called “third-generation” version control system, otherwise known as a distributed version control system or DVCS.

Git’s main competitor at the time of its inception was Mercurial. Mercurial create just a couple months before Git it had many of the same benefits of Git being fast, distributed and scalable. Two Mercurial core developers, Raphaël Gomès and Pierre-Yves David, attribute Git’s popularity over mercurial to “The soaring rise of GitHub at that time, and to the natural endorsement of Git by the Linux community.”[^1]

DVCS’ “distributed” or “peer to peer” nature removed the need of connecting to centralised servers to make commits, as was common of the previous generations of VCSs, allowing multiple developers to work locally & concurrently. Putting the power into the hands of coders, cemented DVCSs place in open source development.

Centralised Version Control (CVCS) - Merge when talking about other industries

Otherwise known as the “second-generation” of version control systems. Prolific = Apache Subversion, Perforce. Built around access control. Better suited for tight-knit teams and enterprise. Central server. Empowered centralised organisations.

Next gen Version Control should go beyond the coder and seek to empower as many trades as possible.

Git in a team

It is typical of new developers who wish to contribute to open source to be self taught. They may have experience with Git but entirely on an individual basis. This has lead to problems with common operations that typically only occur when working with others.

Recommendation VC2024-01
It is recommended that the open source community shift away from the mindset of Git as a “means to an end” to an important prerequisite to open source contribution. As such, an informal consensus should be reached as to the subset of Git proficiency required. This should be done in the knowledge that many contributors are not proficient coders or indeed even contributing code.

Recommendation VC2024-02
It is recommended that open source projects stop assuming Git proficiency as a given. In conjunction with “VC2024-01”, a dedicated snippet or website with educational materials about Git should be placed in prominent locations. One such location could be at the top of Contribution Guidelines. This is to impress on new contributors the importance of Git while providing an avenue to reach the required proficiency.

Git as a development tool

Developers frequently struggle to understand the information they are presented bla bla

Recommendation VC2024-02
The proliferation of alternative Git workflows using various tools:

  • GUI workflows
  • CLI workflows using alternative Diff
  • Individual usage of Jujutsu or other git-compatible VCS

Recommendation VC2024-??
It is recommended GUI based workflows for Git are suggested to newcomers and non-technical users.

Diff

Git Alternatives

Put at bottom discussing if any of these hold up to the recommendations

Mercurial - Site

Sapling - Site

Forked from Mercurial 10 years ago. It’s differentiated itself with some opinionated changes. Made by Meta.

Pijul - Site or Theory of Patches

Piped - Mentions using Version Control in other industries.

Jujutsu - Site

Compatible with Git but attempts to improve upon it in several ways. Being a very young project it manages to take inspiration from the innovations by its peers in the version control space. It does add other benefits include committing of conflicts and automatic rebases (which it uses to offset the the drawbacks of snapshot vs patch theory). If this can reduce the pain points we’d identified before then it could be worth a look. See HN + article jj init (author announcement on fedi).

Perforce - Site

A proprietary CVCS, has carved out a niche for itself from the oversights of the current crop of DVCS. They mark the Game Development, Virtual Production, Semiconductor, and Automotive industries as major customers. Industries where code is forged alongside large assets, such as 3D models or PCB schematics, with different needs. Where coders work alongside the layperson, the designer, the artist, the musician.

The reasons why industries choose alternative forms of version control overlap heavily with the factors for why we find comparatively so few non-coder contributors within the open source community:

Usability - The command line interface is the home of Git. The CLI is also the home of the coder and the tech savvy. User friendliness takes a backseat alienating those who aren’t coders or tech savvy.

Code-First - Git treats everything as code, therefore it’s an imperfect solution for anything that isn’t code. This disempowers those who’s interactions are solely with non-code assets, placing control with the coder.

File Locking - Binary files aren’t able to be merged in the same way plain text files are. File locking prevents merge conflicts as well as wasted work.

Large File Support - Git does not support large files by default. Git LFS stuff here?

Unfamiliarity - The aforementioned technical factors are enough to require the usage of specialised version control in industries that aren’t built entirely on code. Non-coders therefore have more experience with these specialised solutions, perpetuating its usage, further separating their trade skills from open source contributions.

The open source community has built amazing tools to empower so many people, video editors, 3D modellers and office suites. The substrate of open source however, the version control, serves as an obstacle to the goodwill of those who wish to give back as contributors.

Prospective Recommendations
Normalise the use of version control in other industries and for non-code purposes
Version Control needs large asset support by default
Version control needs to be more user friendly/alternative workflows
Move away from version control as a code first/only solution

Embedded Version Control

Version control works perfectly well for the written word. So. why is it that we don’t see many writers using version control?
This PDF shows how version control of office docs are handled.

Discuss shows changes between post edits.

Google Docs & LibreOffice allows you to save named versions of files (Docs, Sheets, Slides). Each version acts as a commit and allows the document to be reverted.

Binary Files…blender version control. Blender people use Subversion

At the Blender Institute / Blender Animation Studio we use Subversion for our projects. It works fine for blend files, but you have to make sure they are not compressed. Compression can cause the entire file to be different when only a single byte changed, whereas in the uncompressed blend file only that one byte will differ. As a result, binary diffs will be much smaller, and your repository will be faster to work with.
Sunsetting Subversion — Developer Blog

Recommendation VC2024-??
It is recommended that tools, such as office suites, should be made Git-aware. Version control should be embedded into these tools, acting in the background for non-technical users.

Further Research

Replacing binary files with plain text - Video Post
Git LFS - File Locking

Relevant Tools

Git Town
Commitizen - Create committing rules for projects, auto bump versions and auto change-log generation.

Conventional Commits - A specification that is a lightweight convention on top of commit messages. It provides an easy set of rules for creating an explicit commit history; which makes it easier to write automated tools on top of.

External Git Diff

delta - Viewer for git and diff output.
Difftastic - Experimental diff tool that compares files based on their syntax.

Git GUIs

Gitnuro -Multi-platform open source Git client without any kind of constraint to how you can use it nor relying on web technologies.
gitg - GNOME GUI client to view git repositories.
qgit - Uses QT so it’s a KDE focused GUI for git repos

Terminal UIs
GitUI - Git GUI but right in your terminal
LazyGit - A simple terminal UI for git commands

https://rogerdudler.github.io/git-guide/

Sources

[^1]: Beyond Git: The other version control systems developers use - Stack Overflow
Do we think of git commits as diffs, snapshots, and/or histories?
https://www.perforce.com/blog/vcs/version-control-for-binary-files

Nomenclature

Forges -
Version Control System -

Curios

Coolify - An open-source & self-hostable Heroku / Netlify / Vercel alternative.
Bluefin - Based on Fedora Silverblue and can be rebased onto from silverblue. Tries to bring cloud-native patterns to your local computer.
Guix - Similar to Nix in a lot of ways with some key differences specifically replacing Nix Lang with Guile and only GNU software available. The big downside to Guix is that it doesn’t seem to have the same level of community support or tools built around the project.

Notes

circlebuilder

  1. Some things re:nix that may become additional considerations in the survey…
    • “reproducible builds” + “reproducible devenvs” + migrations thereof between environments.
    • Related to the above the trend that SBOM’s become way more relevant (avoid Log4J, supply-chain security, etc)
    • And that FOSS is sorta disadvantaged compared to OSS where companies give way more attention to doing the supply chain shebang, as their clients demand it.
    • Lastly also institutions / OSPO’s (think EC, and also fundraising to them) adore nix for these reasons
  2. This website proposes a good labeling system for issues in issue trackers.
3 Likes
[RFC] Solidground development environment (wiki)