DRAFT: State of the Forge Federation

Bonjour,

I would like to create a document describing the state of the Forge Federation, not limited to forgefriends, that is based on what people did in the past (sure thing) and tries to predict what they will do in the future based on that. It would span two years only, one year in the past, one year in the future.

This is something between a roadmap and a state of the art study. What is usually involved in a roadmap requires a team of paid staff. But none of the people involved in working on federation are paid at the moment, except for me, KN4CK3R. What it essentially means is that whatever I think would make sense to implement but I donā€™t intend to implement myself has a probability of apparition in the next year that entirely depends on other people.

I can give two examples to illustrate based on what I thought nine months ago:

  • I would have predicted that @KN4CK3R was going to work almost full time on Gitea federation. And I would have predicted that the chance of @Dodecahedron showing up and volunteering many weeks of his time on federation to be close to 0%. Turns out that I was wrong. But it also turns out that in the end the work was done, just not by a single person but by two. This does not diminish what @KN4CK3R did, thatā€™s not what I mean: his contributions to Gitea are astounding and Iā€™m very impressed by the quality of his work.
  • I would have predicted that @zeripath would take the opportunity of the NLnet grant to take a leave of absence and spend a significant amount of the grant to implement federated user (because it is kind of his area of expertise). But it turned out to no be the case.

I also find difficult to represent each element of the roadmap as dependent on each other. Representing federated users is likely to be implemented as a temporary hack to allow Gitea federation to move forward. Although it could be seen as a hard dependency without which nothing federated can happen.
Another example is what a forge is expected to do when receiving a ā€œCommitā€ message via ActivityPub. I think the best way is to sync the state of the remote repo via the git repository that contains the FFF state. But realistically thatā€™s not going to happen immediately and some other ā€œthingā€ will be implemented. So from my point of view FFF is a dependency without which forge federation cannot be implemented in Gitea. But from someone else point of view itā€™s not.

Regarding funding, what happened in the last years tends to demonstrate that most of the funding (50Kā‚¬, 25Kā‚¬, 5Kā‚¬ for diversity, over 10Kā‚¬ for the forgefed specifications, etc.) raised over the past two years was not spent because the beneficiaries did not claim it. And a number of opportunities to apply for grants (150Kā‚¬, 20Kā‚¬+ etc.) were missed or do not raise any interest. I therefore do not think necessary to prominently represent that in the roadmap, although my own contribution entirely depends on funding. At this point in time Iā€™m the only person who is both available and funded to work full time on federation. Making a roadmap based on that exception is unlikely to be useful.

So, in a nutshell, I will try to make a roadmap that:

  • Is a guess of what will happen in the next year
  • Shows what happened in the past year
  • Is based on who did what and who is likely (educated guess at best) to do what
  • Shows the difference between volunteer work and paid work

It will not include:

  • A long term vision of forge federation
  • Tasks that nobody publicly declared that they would commit to complete in the next 12 months
  • Dependencies between tasks unless there is no way around it (like someone not being available during a long period of time and unable to work on a given topic)
  • Considerations regarding funding work towards forge federation

At this point Iā€™m not entirely sure such a roadmap will be interesting or useful. The only risk I see is that it will not contain much because there are so few people committed to do work.

To be continued

1 Like

Draft is here.

@realaravinth @CSDUMMI @aschrijver @KN4CK3R @Dodecahedron @techknowlogick @zeripath Your review of the State of the Forge Federation would be very welcome. It covers the past year and tries to predict what will happen in the year to come. Feel free to edit to add anything youā€™d like or raise concerns on what it contains.

Cross-post :slight_smile: just gave some feedback in the chat.

1 Like

There has been extremely useful reviews and I feel like the document is ready to be published. Of course it can be improved but Iā€™m now convinced it is both useful and covers everything it should. Unless there is an objection, Iā€™ll publish it in the forgefriends blog tomorrow.

Here is the last version for the record:

DRAFT State of the Forge Federation.md (21.7 KB)

Published :tada:

i re-phrased the introductory paragraphs - i will explain the significant changes

Millions of Free Software developers forgot why it matters to own their tools.

i understand what this is trying to suggest; but clearly, it is a generalization of no individual persons, but an amorphous ā€œcollective consciousnessā€ - the people who wrote software before forges existed, did not forget the value of the unconstrained decentralized workflow - most of them still prefer it today - people who started programming in the era of forge popularity, never knew of it, so they had no such conceptions to forget - programmers especially, are predominantly ā€œlonersā€, or have at least a strong sense of individualism, and generally demand autonomy, unless there is a large paycheck involved - ā€œas a collective (on average), we forgotā€ is just too wishy-washy - it does not actually describe anyone that i know of

They know, better than anyone, how to fix and improve them.

this unfortunately is also not accurate - it probably is a fair estimate of people who wrote software before forges existed; but in recent years, the majority of people who use forges are web developers - they do not know the tools ā€œbetterā€ - they know absolutely nothing about how the build tools, run-times, and systems work, and do not want to know; because (the real problem) they do not need to know, as their predecessors did - they use only the tools which they are bestowed upon them from on high by their tech over-lords, they use them glibly as prescribed, and if the tools break, ā€œsome guruā€ (in the cloud) will fix it for them, someone who they do not know, and probably will not communicate with them directly

again, i understand the intention of that sentence; but i dont know how to word it sincerely - maybe they should know how (or at least want to), and maybe the people who came before them knew how, but this is probably not worth mentioning - some people do know how, but those people do not need convincing - the ones who dont know how, probably dont want to know, and never will - the majority of programmers will never be swayed by any arguments from ethics or practicality - a web developer is not going to be persuaded by arguments that suggest configuring apache, jenkins, or firewalls is a path to freedom, much less to suggest patching them and re-compiling - to many people, the convenience and laziness offered by freebie web services is the most beneficial form of freedom (ironically) - it frees people from the time, work, and expertise which self-sufficiency demands

NOTE: you can see? how difficult it is to express these ideas formally and concisely - it took me two paragraphs to unpack the subtleties lurking in that one sentence

Some Free Software developers chose to use different tools, such as email and DVCS without a web interface. Many others decided to run their own Free Software forge and work in a decentralized way, improving their own tools and independently deciding who they work with. But even then, they are isolated instead of being federated with each other.

as a whole, that is not true - email is federated and DVCS does not need to be federated, because it is fully-distributed, which is federationā€™s stronger, more robust cousin - email is the most popular and useful federated system in history, and it is still the most used federated system today - since the earliest days of the internet, no one was isolated from anyone else on the internet; because the internet itself is a federated system

after re-reading it though, i dont think that was the intention - it is those who ā€œdecided to run their own Free Software forgeā€ who were ā€œisolated instead of being federatedā€ - probably the wording is ambiguous, which gave me the first impression - if that was the intention, it would have been more clear like " But that latter group traded away federation and got isolation instead"

that paragraph is trying to make a point, that web forges have an indisputable natural advantage over patches via email; but they offer nothing new or unique - forges are only a collection of tools, which people had already been using in other forms, presented as a convenient unified web interface - some people consider that presentation to be more convenient; but convenience is subjective - it is only a different workflow - that workflow is perhaps unfamiliar to young people; but many major software projects today still prefer it - git itself is developed in that way, as well as GNU and linux, as examples - none of those are suffering from impedance or lack of contributors, due to that workflow preference

overall, that paragraph is really self-contradictory, for demonstrating the motivation of the forge-fed - the motivation is some (any) form of decentralization - if a system is decentralized, it does not need to be federated - a decentralized system is a more complete and robust extension of the weaker federation - the advantage of federation over full decentralization, is that each node may choose which portions to replicate, where fully-decentralized nodes must replicate everything - im pretty sure thats how git-ssb works - if someone wants to add a comment to a post, for example, one must first download the entire project (all tickets and historical edits, the forge itself, everything) - likewise with pagure - it is actually decentralized - to add a comment in that way, one must first clone the entire ā€˜ticketsā€™ VCS

federation is one form of decentralization; but self-hosting a typical forge such as git-tea, is not decentralization - it is definitely centralized - the only difference is where the ā€œcenterā€ is, and who controls the central machine - think of the ā€œhubā€ topology in networking terms - git-tea is literally a git ā€œhubā€ - that is the essence of centralization

FWIW, there also was not much of a choice to make - there were no other tools available; but it was not a problem, because those tools are perfectly adequate for unfettered global collaboration - no one who used them wanted to supplement them with a webby interface - corporations such as source-forge, google, and github wanted to displace them, but not to fill any demand for improvement - profit, was the only motivation for the invention of forges

the convenience of forges was so alluring to noobs, that now many people consider them to be so essential, that they too, would not ā€œchooseā€ any alternative to web forges - there are many of forges to choose from, but none are significantly different - the difference is that now, people can ā€œchooseā€ between the web and email workflows; but unfortunately they are not compatible - that is a key area where forge-fed comes to the rescue

to illustrate that point, forge-fed should allow people who prefer to use a forge (any forge) to collaborate/contribute to projects which only communicate and accept patches via email - and people who prefer to use only email should be able to collaborate/contribute to projects which only communicate and accept patches via a web forge - i do not yet know enough about the motivation or goals of the forge-friends group to presume anything; but if this is the first time that you have imagined that use-case, you are thinking of forge-fed in a severely impoverished way - it is not only about uniting web forges - it is about uniting people, regardless of their preferences of tools, workflows, service providers, or whatever

in short, if you write software, and your tools understand the concept of a patch, and you have some networking capability, whether that is a web browser or netcat, you should be able to collaborate with any one else, via the forge-fed protocol

likewise:

It must be possible for developers to work together on the same software project regardless of the user interface they use.

that is, regardless of all tools, workflows, machines, hosts, geographical locations, and so on - regardless of anything imaginable, related to networked computers - ā€œthe user interfaceā€ in that sentence is presumably: ā€œthe webby forge interfaceā€ - that is very much not regardless of oneā€™s choices - most forges do not function completely without javascript, for example - the differences between git-tea, gitlab, and pagure for example, are negligible - they are nearly identical ā€œinterfacesā€, all are essentially clones of the github model

that stated goal is too narrow - ā€œregardless of the user interface (which is a web forge)ā€ effectively implies: ā€œyou must use firefox or chromeā€ to communicate with this project in any way - i reject that notion whole-heartedly

This will help developers to move away from software forges they are forbidden to change and regain control of their tools.

this should also help developers to regain control of their tools, by moving away from web forges entirely; and instead using tools which are simpler, more efficient (native clients), and more robust than web forges, and can function completely and reliably, even when the entire team and/or the infrastructure is offline for 6 days each week

that is not a wild or fringe suggestion - git-ssb has that ā€œreliable offlineā€ feature baked in at the protocol level - it is actually impossible to remove that behavior - forge-fed could also work perfectly, even if a projectā€™s primary forge is running on a laptop, which is only online for one hour per week - but the protocol can not guarantee that level of robustness - forge developers would need to implement that robustness, per a strong recommendation - so, forge-fed must be sure to recommend it - i wanted to underline that - it is a highly desirable feature, and is one of the main benefits of federation and decentralizatoin; but fediverse fans seem to devalue or forget about it

I would oppose a statement that developers, as a collective, forgot why it matters to own their tools. But thatā€™s not what the phrase says. GitHub & GitLab claim they have users in the tenths of millions and thatā€™s a lot, it is worth mentioning. It is not a fringe of the population, it is a significant number.

Let say your assessment is true for most of the web developers, with a fair percentage of people curious enough to learn more than they are strictly required to (developers are like that sometimes :slight_smile: ). This leaves at least tenths of thousands of developer (Iā€™m being conservative in this estimate) who use GitHub/GitLab for their work daily and who are not web developers. And some of them are bound to have developed an excellent expertise in their respective field.

Letā€™s reverse the argument: Iā€™d be extremely surprised if it is possible to find a software development skill for which there does not exist a developer with a world class expertise, on GitHub/GitLab.

You are right and I overlooked that possible ambiguity. It was the last addition to the document and got less time and attention than the rest. Oh well :thinking:

That was not then intention. This paragraph is about how people are not subject to what is described in the previous paragraph. Some chose to not use forges and others did. This document is about forges which may give the impression that the people who did not chose forges are ignored. They are not and it is very important to have a phrase that is a reminder of that fact. But they are not in scope of the document and there is no working around that.

You are correct and there should be a better way to say this so it is clear for everyone. It definitely is an incorrect way to use ā€œdecentralizedā€. Here is how it is understood (and maybe Iā€™m wrong):

  • Centralized: an online service for which there is a single instance on the planet (Google, GitHub)
  • Decentralized: an online service for which there is an unlimited number of isolated instances (Gitea, GitLab CE, pagure, ā€¦)
  • Federated: an online service for which there is an unlimited number of instances that communicate with each other (Peertube, Mastodon, ā€¦)

And anyone working decentralized system would rightfully be outraged by this reuse of the term with a completely different definition. Iā€™ll think of a better word to use to avoid that. If you have an idea, that would be fantastic.

forgefriends focuses on forges, it is its focus, its scope. The ability for a forge to support an email based workflows is in scope. A boarder project exists, https://coding.social/ and also consider workflows that do not involve forges at all.

I think I understand where youā€™re coming from.

And Iā€™d like to clarify again: yes, the focus of forgefriends is on web based forges. Iā€™m sure thatā€™s what people who know what a ā€œforgeā€ means have in mind, reason why I agree that using ā€œforgeā€ without any further adjective such as ā€œweb forgeā€ is correct.

There is freedom in constraint: by choosing to focus on a limited scope (forges), the forgefriends project has the ability to move forward. Such a choice is debatable and it could be argued that it is impossible to sensibly move forward without embracing a broader scope. Or that moving forward without fully comprehending the global picture is bound to lead to dead end or exclusion. There are merits to these arguments but there also is a limit to what a human being is able to comprehend before experimenting and possibly making mistakes.

As a member of the forgefriends collective, Iā€™m happy with its current scope because it allows me to act, move forward, make mistakes, adjust etc. Iā€™m also very worried to miss something huge, reason why I appreciate your criticism very much. It forces me to rethink decisions made months ago: Iā€™d rather realize that Iā€™m going into a dead end now that later. However painful such a realization would be, ignoring it wonā€™t help, on the contrary.

we discussed this on IRC but i will add it here for posterity - Gitea and GitLab are ā€œcentralizedā€, in every sense of the word - the only difference between gitea and github is that gitea is libre

the only common factor between any two instances of gittea, is that they are running the same software - the only technical term to describe that is ā€œinstancesā€ - if more than one computer is running a copy of gittea, the best word for that relationship is ā€œcoincidenceā€

these would be more accurate definitions:

  • Centralized: an online service for which the only way data may enter or exit the system, is via a single ā€œgatekeeperā€ service - ie: data is generally not shared with any other service; and all users of the system must connect to the same central gatekeeper service (Google, GitHub, GitTea, GitLab)

  • Decentralized: an online service for which multiple instances replicate complete shared data-sets - decentralization is more about data storage than messaging - most often, the data is immutable (eg: append-only) (Git, SSB, IPFS, DHT, bittorrent, ā€¦)

  • Federated was basically accurate - a key feature of federation is that it propagates relatively small hunks of data - the data is usually time-sensitive and temporary - federation is more about server-to-server messaging and inter-operability than data storage (Email, IRC, The Internet itself)

the final point is significant - it makes activity-pub something of a strange bird, in that it looks more like federation; but it does assume long-term storage, where classic federation (email, IRC) can discard messages as soon as delivery is acknowledged by receiver(s)/subscribers, or if delivery has failed after repeated attempts - federation receivers are very likely to discard most messages also, after ā€œprocessingā€ them

1 Like

i also agree - i keep stressing that because forges are not the essential concept - a forge is a ā€œproject management systemā€ - it is a collection of various project management tools, all of which can be, and often are, completely separate services; and none of those components are inherently web services; but a forge is a web service by design

so i am stressing the point that forge-fed is not only about forges - it is about project management - forges are one way to manage projects - a bug tracker such as redmine, or an instance of mediawiki, or a forum or mailing list, could all be forge-fed peers, implementing only the features related to their respective component (issues, documentation, communication)

so if the goal only includes ā€œforgesā€, that excludes bug trackers, wikis, mailing lists, and anything else that is not a complete forge - it also excludes projects and contributors who do not want to use a forge - but those are artificial exclusions and unnecessary - the over-all goal is to give people more options, not to force them to use a single tool (eg: a web browser) or a single workflow (the workflow designed by github)

ā€œa forgeā€, regardless of which one, is one specific class of tool (the swiss army knife of project management), one which many projects do not need - though they probably use some of its components in other forms (an email-based bug tracker, code review via email) - forge-fed can accommodate everyoneā€™s preferences, by abstracting away the differences - it should not matter if a project uses email for patches, and a contributor wants to read them as a ā€œpull requestā€ or vice-versa - they are really the same thing - the difference is in the presentation, which is not the concern of any network protocol

for example, the semantics of a ā€œpull requestā€ is ā€œbob suggests some changes to some lines of code, and would like people to review themā€ - that is all the protocol needs to convey - a forge would handle that message by creating a pull request, a code-review tool would handle the same message by opening a review ticket, and a forum or mailing list would handle the same message by starting a discussion thread - it is very wrong for the protocol to assume anything of receivers, other than that they may be interested in the message somehow