- Presentation
- Technical challenges and barriers
- Idea and objectives
- Project outcomes
- Added value
- Relevance, socio-economic impact and benefits
- Free Software Licenses and/or open standards
- Commercial exploitation
- Project planning
- Executive summary
https://forgefriends.org/ is an online service to federate forges. The software projects hosted on one forge are synchronized in real time with their counterparts on other forges, via the W3C ActivityPub protocol. Developers can freely use the forge of their choosing while contributing to the same project. It operates independently from the forges and serves as an incubator with rapid prototyping to research the best user experience.
Presentation
Most Free Software projects are hosted on proprietary online services (called forges) that do not provide an export/import feature flexible enough to allow them to move easily (GitHub, SourceForge, etc.). Although the code repository can conveniently be moved from one service to another, other essential components such as pull or merge requests, issues, developer accounts, continuous integration, etc. cannot and the project, as a whole, is trapped.
When two forges are federated via forgefriends, the actions carried out by a developer on a forge are sent to the other, and vice versa. For instance:
- The forgefriends service runs with a proxy for GitLab and a proxy for GitHub
- The project Ceph exists on GitHub
- I run a self-hosted GitLab instance
- I ask the forgefriends service to federate the Ceph project from GitHub with my GitLab instance
- I browse the issues of the Ceph project on my GitLab instance
- I comment on an issue on my GitLab instance
- My comment is copied over to the GitHub Ceph project by forgefriends
- A GitHub user answers my comment which forgefriends copies over to the Ceph project that resides on my GitLab instance
Once all components of a software project are federated (issues, code, etc.) the project as a whole is effectively duplicated in real time on multiple forges, thus allowing the users to chose the one they prefer. Service portability is achieved because federated forges continuously maintain identical copies of the software project although they are operated by independent actors and running different servers and user interfaces.
Technical challenges and barriers
Most Free Software projects are hosted on proprietary online services (called forges) that do not provide an export/import feature flexible enough to allow them to move easily (GitHub, SourceForge, etc.). Although the code repository can easily be moved from one service to another, other essential components such as pull or merge requests, issues, developer accounts, continuous integration, etc. cannot and the project, as a whole, is trapped. This has many negative consequences:
- Fragility: When a service shutdown the development history and the community is lost, even when the code itself is preserved
- Censorship: A project may be unexpectedly blocked by the company because an embargo is imposed on a country where a contributor resides
- Lock-in: It may take years for some projects to manually migrate from one forge to another, as demonstrated by Gitea, a Free Software forge that started with code on GitHub and has not completed the migration after three years
- Exclusion: Users who do not agree (for ethical or practical reasons) to create an account on the forge are effectively excluded from participating in the development of the projects it hosts
- Vulnerability: Developers are required to trust the service provider with the integrity of the repository or the releases. For instance service providers could have used collision attacks to transparently modify the content of a commit.
- No interoperability: the API is not normalized and does not use open standards for most of its content. For instance it is only possible to export/import from GitHub to GitHub Enterprise.
Idea and objectives
Software forges became popular twenty years ago and the barriers appeared at the same time. The recommended solution at the time was for each forge to implement an export/import feature, which did not happen because proprietary services have a vested interest in locking their user in and exporting projects is contrary to this objective. Although it could have been implemented by Free Software forges, it did not happen either because it is an “all or nothing” solution: the project has to move all at once and the migration tool has to be perfect at the time it happens because there is no turning back. It may not be the only reason why it is still impossible to move a project as a whole from one forge to another but there is no denying, twenty years later, that it may be time for another approach.
There are two ways to move information regarding a project from one forge to another:
- import/export: an “all or nothing” approach where the project is moved to one forge and deleted from another
- federation: a two way communication between two forges where all actions happening on one forge are copied over to another
The number of forges publicly accessible was multiplied by at least two order of magnitude since 2001 and they would benefit from being federated. But no forge software supports it, although a data model was developed in 2019 a few years after the creation of the W3C ActivityPub protocol. Popular services such as mastodon or PeerTube appeared in the past five years: a source of inspiration for the implementation of federation in the context of forges. Although a full featured implementation is a very ambitious undertaking, it does not need to be complete to be usable. For instance, the federation of issue comments could be the only feature available while other are implemented. In addition, it is practical to experiment on federation with a proxy based on the forges API (or even web scrapping) before attempting a native integration.
The proposed incremental approach to forge federation is to:
- implement a federation proxy for each forge, which exposes the ActivityPub based protocol with a model similar to forgefed and translates it into the native forge API. Such a proxy needs credentials to interact with the forge API but can be run independently.
- implement a standalone forge federation service whose sole purpose is to connect the federation proxy of each forge together after obtaining the credentials required to carry out actions
An example use case could be:
- The forgefriends service runs with a proxy for GitLab and a proxy for GitHub
- The project Ceph exists on GitHub
- I run a self-hosted GitLab instance
- I ask the forgefriends service to federate the Ceph project from GitHub with my GitLab instance
- I browse the issues of the Ceph project on my GitLab instance
- I comment on an issue on my GitLab instance
- My comment is copied over to the GitHub Ceph project by forgefriends
- A GitHub user answers my comment which forgefriends copies over to the Ceph project that resides on my GitLab instance
The ultimate goal of forgefriends is to become obsolete, once all forges natively implement federation. It will become common place for a software project as a whole (i.e. code, issues, merge requests, CI, etc.) to exists simultaneously on multiple forges, with developers working together but using different forges. To get there, the forgefriends should be considered the first step of experimentation and incubation of federation features that are destined to land in the code base of each forge, eventually.
An example could be:
- The forgefriends service implements a proxy for GitLab that translates ActivityPub comments into the new issue note API call
- A native implementation is written as a contribution to the GitLab native federation effort and proposed as a merge request
- After the native GitLab implementation is released, the forgefriends code can be removed because GitLab natively supports the federation of issues comments
Project outcomes
- A software library is created, implementing federation proxy features for GitHub and another for GitLab:
- It communicates with the ActivityPub protocol using a common vocabulary and data model to:
- Create issues and add comments
- Create pull / merge requests and add comments
- A software for running a server is created, implementing a self-hostable service (named forgefriends) enabling all developers worldwide, including the 50 million GitHub users to:
- Authenticate on a self-hosted GitLab instance and on GitHub
- Request the federation of a project residing on both GitHub and GitLab
- One user research report based on at least 9 interviews in 3 different groups of potential users of the forge federation service
- A community of developers relying on forgefriends is created (either online or self-hosted): at least 2 of them use it on a daily basis, at least 10 of them use it occasionaly.
- At least 3 reports explaining the data model and vocabulary used by forgefriends and communicated to Gitea and forgefed with a request for comment
- At least 3 merge requests to GitLab are submitted to implement features relevant to forge federation
Added value
- The forgefriends server and libraries:
- bootstrap the work towards a full featured forge federation
- facilitate the implementation of additional features in the future
- provide a working implementation that allows developers to experiment
- The community of developers using the forgefriends service demonstrate the usage value of a partial implementation of forges federation and is therefore a practical method to incrementally make progress towards a full featured forge federation
- The user research report shows what current forge users consider important when it comes to their ability to move from one forge to another
- The reports explaining the data model and vocabulary
- provide real world data for forgefed to make progress toward a standard data model and vocabulary
- provide a foundation for Gitea to use when implementing forge federation natively
- The merge requests to GitLab:
- encourage the GitLab developers to make progress towards a native implementation of federation
- demonstrates how a feature implemented in forgefriends and be natively supported by a forge
Relevance, socio-economic impact and benefits
Enabler for developers to move between forges
The goal of forgefriends is not to attract a large audience but to serve as a demonstration and experimentation ground. It is designed to be an enabler towards the generalization of forge federation. Every software developers (and not just the 50 million GitHub users) should be able to freely move from one forge to another without friction. When forge federation becomes common place, all developers will use the forge they like instead of the forge on which the project resides.
Promotes the concept of federated development
Most developers work on Free Software in a centralized way and do not see the benefit for decentralization and federation. Implementing federation and using it for practical purposes is a way to discover use cases and imagine new ones. It can ultimately be a motivation to advocate for the generalisation of federation in software development. For instance, it is currently extremely difficult for a developer who does not have a GitHub account (because, for instance, they live in Iran) to participate in the development of the projects that are hosted there. This developer could use forgefriends and their own GitLab instance and become an active part of the community.
Improves the durability of software projects
Organizations are routinely impacted by the disapearance of forges which translates into a loss of value and money. For instance, if a software appliance contains software originating from a defunct forge, it will require an update to get it from another location, assuming the developers managed to migrate in time. Even if they did, past issues and merge requests would have to be migrated manually and the chances that they are permanently lost is high. By continuously duplicating issues and pull / merge request on GitLab and GitHub (redundancy is another way of looking at federation) on two forges, forgefriends improves the chances that they are recovered, thus saving value and money for all organizations depending on the impacted projects.
Scale out forge federation development
By providing a minimal infrastructure and a few features, forgefriends demonstrates that the development towards a full featured forge federation scales out, meaning that it can be conducted by many independent actors working on many independent features. By contrast, implementing an import/export feature working between forges is best developped by a single team because it needs to accurately represent all aspects of a given project at the time it is used. The scale out and incremental approach to forge federation is locally cheaper although it is much more expensive as a whole. For instance, a skilled developer could contribute a new feature within a week, a cost that even the most modest organization can afford. The sum of all those efforts is much higher than what would be needed to implement an export/import feature but it is distributed over a longer period of time and among many actors. It is therefore more likely to make progress and succeed.
Free Software Licenses and/or open standards
forgefriends will be released under the AGPLv3+ license. The code contributed to existing code bases will be released under licenses compatible with their own licenses. The data model and vocabulary developped by forgefriends will be contributed to the ongoing forgefed standardization effort. The ActivityPub W3C protocol will be used to communicate between forgefriends instances.
Commercial exploitation
forgefriends is a non-commercial project but with a relevant contribution to the internet community: it is an enabler for all software developers to use the forge of their choosing.
Project Planning
User research
Forge federation is a new idea and its implementation would benefit from a user centered design that requires user research.
- Prepare the research
- Find participants among software developers, organizations and forge maintainers
- Prepare the research sessions
- Create an intercept interview sript for software developers, organizations and forge maintainers
- Conduct interviews with the participants
- Organize an affinity mapping session to collect emerging themes
- Analyze the results and write a report with recommendations for the implementation of the user experience.
Infrastructure
While conducting user research, the building blocks can be implemented in parallel.
- Create a self-hosted development environment for forgefriends (with a GitLab forge and the associated CI)
- Create a website presenting the project and the roadmap
- Create forgefriends module a Django based server relying on the federation module for ActivityPub and
- Define a data model and vocabulary based on forgefed and GitLab import/export
- Create forgefriends GitLab, a module based on the forgefriends module using the GitLab API for creating and commenting on issues and merge requests
- Create forgefriends GitHub, a module based on the forgefriends module using the GitHub API for creating and commenting on issues and pull requests
- Create extensive end to end integration tests
- Create a reference documentation including instructions for self hosting
User experience
After the infrastructure is fully functional but can only be run with the integration tests, the roadmap for implementing the desired user experience is defined. Attempting to predict the outcome of the user research and how it will be implemented would be contrary to the primary objective of user research.
- Define the user experience roadmap for interacting with the forgefriends server based on the recommendations from the user research report
- Create a forgefriends server to enable users to connect the forgefriends GitLab together or with the forgefriends GitHub
Distributing and operating the service
- Setup a forgefriends server including both forgefriends GitLab and forgefriends GitHub
- Add monitoring and intrusion detection
- Publish forgefriends server, forgefriends GitLab and forgefriends GitHub
- As a container image usable with a oneliner
- As packages published on PyPI
- With self hosted documentation
Advocacy and feedback
- Reach out to users software developers, organizations and forge maintainers to foster a community of users
- Actively seek feedback from users
- Reach out to GitLab implementors and submit merge requests relevant to simplify the implementation of forgefriends GitLab
- Reach out to Gitea to implement a federation data model and vocabulary
Executive summary
Subdomain: Service Portability
Rationale: Forgefriends facilitates the portability of software projects and helps them move freely from a software development hosting service (also known as forge) to another.