Cheatlist for forgefriends development

Development
#1

forgefriends development maintenance guide

The upstream branch is updated from the Gitea main branch. Feature branches must be be rebased, base branches are reset to upstream and the feature branches it contains are merged into it. The order of the following list matters.

  • upstream → pulled from Gitea main
  • feature-gitlab-ci → rebase onto upstream
  • feature-development-environment → rebase onto upstream
  • base-development → reset to upstream
    • merge feature-gitlab-ci
    • merge feature-development-environment
  • feature-go-fed → rebase onto upstream
  • feature-projectbase → rebase onto base-development
  • base-activitypub → reset to upstream
    • merge feature-go-fed
    • merge feature-projectbase
  • feature-activitypub → rebase onto base-activitypub
  • main → reset to upstream
    • merge feature-activitypub

Gitea documentation

Local development tree checkout

In the following the remotes are setup as follows:

$ git remote -v
gitea https://github.com/go-gitea/gitea (fetch)
gitea https://github.com/go-gitea/gitea (push)
loic ssh://git@lab.forgefriends.org:2222/dachary/forgefriends.git (fetch)
loic ssh://git@lab.forgefriends.org:2222/dachary/forgefriends.git (push)
origin https://lab.forgefriends.org/forgefriends/forgefriends.git (fetch)
origin https://lab.forgefriends.org/forgefriends/forgefriends.git (push)

Show forgefriends commits only (ignore commits originating from Gitea)

  • git log --oneline --no-merges gitea/main..origin/main

Every week sync with gitea

  • git fetch gitea
  • git checkout -b wip-gitea-merge origin/main
  • git merge gitea/main
  • git push loic/wip-gitea-merge
  • create merge request with subject “Merge remote-tracking branch ‘gitea/main’ into main”
  • merge when tests pass

Running tests

  • All tests
    time drone exec --pipeline testing-amd64 .gitlab-ci-drone.yml
    firefox coverage.html
  • Only integration tests
    docker run --rm -ti -e TEST_TAGS='bindata sqlite sqlite_unlock_notify' -e GOPATH=/drone/src/go-path -e GOCACHE=/drone/src/go-build --volume $(pwd):/drone/src --workdir /drone/src --user gitea gitea/test_env:linux-amd64 bash -c 'make integrations.sqlite.test generate-ini-sqlite && GITEA_CONF=integrations/sqlite.ini ./integrations.sqlite.test'
  • Only unit tests with SQL statements displayed
    docker run --rm -ti -e GOPATH=/drone/src/go-path -e GOCACHE=/drone/src/go-build -e GITEA_UNIT_TESTS_LOG_SQL=1 -e TAGS="bindata sqlite sqlite_unlock_notify" -e GOPROXY=off --user gitea -v $(pwd):/drone/src --workdir=/drone/src gitea/test_env:linux-amd64 make unit-test-coverage
  • Only unit tests with coverage
    docker run --rm -ti -e GOPATH=/drone/src/go-path -e GOCACHE=/drone/src/go-build -e TAGS="bindata sqlite sqlite_unlock_notify" -e GOPROXY=off --user gitea -v $(pwd):/drone/src --workdir=/drone/src gitea/test_env:linux-amd64 make unit-test-coverage
    docker run --rm --volume $(pwd):/drone/src --workdir /drone/src --user gitea gitea/test_env:linux-amd64 go tool cover -html=coverage.out -o coverage.html
    firefox coverage.html
  • A specific test only
    docker run --rm -ti -e GOPATH=/drone/src/go-path -e GOCACHE=/drone/src/go-build -e TAGS="bindata sqlite sqlite_unlock_notify" -e GOPROXY=off --user gitea -v $(pwd):/drone/src --workdir=/drone/src gitea/test_env:linux-amd64 make GO_PACKAGES=code.gitea.io/gitea/modules/secret unit-test-coverage
  • A specific unit test in verbose mode
    docker run -ti --rm -e GOTESTFLAGS=-v -e GO_PACKAGES=code.gitea.io/gitea/modules/foobar -e GOPATH=/drone/src/go-path -e GOCACHE=/drone/src/go-build --volume $(pwd):/drone/src --workdir /drone/src --user gitea gitea/test_env:linux-amd64 make 'test#Test'
  • A specific integration test in verbose mode to see log.Error, fmt.Printnl…
    docker run --rm -ti -e TEST_TAGS='bindata sqlite sqlite_unlock_notify' -e GOPATH=/drone/src/go-path -e GOCACHE=/drone/src/go-build --volume $(pwd):/drone/src --workdir /drone/src --user gitea gitea/test_env:linux-amd64 bash -c 'make integrations.sqlite.test generate-ini-sqlite && GITEA_CONF=integrations/sqlite.ini ./integrations.sqlite.test -test.v -test.run TestActivityPubPerson'
  • A specific unit test without caching (note the -count=1, that’s what disables caching)
    docker run -ti --rm -e GOPATH=/drone/src/go-path -e GOCACHE=/drone/src/go-build --volume $(pwd):/drone/src --workdir /drone/src --user gitea gitea/test_env:linux-amd64 env GOTESTFLAGS=-count=1 GITEA_UNIT_TESTS_VERBOSE=true GO_PACKAGES=code.gitea.io/gitea/models make 'test#TestRepository_HasWiki'
  • Rebuild the bindata after migration/schemas is modified
    docker run -ti --rm -e TAGS='bindata sqlite sqlite_unlock_notify' -e GOPATH=/drone/src/go-path -e GOCACHE=/drone/src/go-build --volume $(pwd):/drone/src --workdir /drone/src --user gitea gitea/test_env:linux-amd64 make backend

Getting code from a given PR

Creating a branch

PR=16834 ; git fetch gitea pull/$PR/head:pr-$PR

Creating a diff with the content of the PR

PR=16834 ; git diff gitea/main…pr-$PR > /tmp/$PR.patch

GitLab runner pitfalls

Because of this bug it is not possible to cancel a running job and expect it to be terminated. It will run to the end.

Working with Emacs and Magit

See Cheatlist for forgefriends development - #6 by dachary

3 Likes
DRAFT: monthly report, December 2021
DRAFT: monthly report, November 2021
#2

As of today:

$ git log --no-decorate --oneline --no-merges gitea/main..origin/main
314ab214d activitypub: signing http client
e72d8d58c activitypub: go-fed conformant Clock instance
4d09135ae go.mod: add go-fed/httpsig dependency (#14186)
6bf561516 https://github.com/go-gitea/gitea/pull/16834 (part 2)
e1254e06f activitypub: add the public key to Person (#14186)
1cdc1596d go.mod: add go-fed/activty/pub dependency (#14186)
eb28575bb https://github.com/go-gitea/gitea/pull/16834
41574c3dd activitypub: implement /api/v1/activitypub/user/{username} (#14186)
eabe93a50 go.mod: add go-fed dependency (#14186)
8e43d2e66 gitignore: emacs backups
de98d1860 ci: setup GitLab CI
#3

Notes on this week merge with gitea.

  • User settings were merged and the forgefriends backport is not longer needed, it was reverted. The samantic is slightly different and the forgefriends code was adapted.
  • Conflicts had to be resolved as a result of three refactors impacting the forgefriends code (1, 2, 3)
#4

It turns out that there are rather frequent and significant refactoring in the gitea codebase, which makes the merge strategy inconvenient (or difficult to read). It will be replaced with a feature branches, each carrying a self contained set of changes, that can be rebased onto the main gitea branch. These branches will then be merged into the main forgefriends branch.

The latest forgefriends branch is kept around for archive under the name main-2021-11.

The conventions for maintaining the feature branches are described in a guide.

DRAFT: monthly report, December 2021
#5

As of today:

$ git log --oneline --no-merges gitea/main..origin/main
dca7af25c (HEAD -> main, origin/main, origin/feature-activitypub, feature-activitypub) activitypub: implement Repository
e8677611f activitypub: hack_16834
970ae9403 activitypub: implement the ReqSignature middleware
15f45756f activitypub: signing http client
07928ac20 activitypub: go-fed conformant Clock instance
5ba5382b1 activitypub: add the public key to Person (#14186)
911138d8c activitypub: implement /api/v1/activitypub/user/{username} (#14186)
e31f60cc8 (origin/feature-projectbase, feature-projectbase) repo: add a .projectbase repository to each project
216b788ea repository: cloneLink(suffix) instead of cloneLink(isWiki)
f164a65a6 (origin/feature-development-environment, feature-development-environment) gitignore: emacs backups
e3490af21 (origin/feature-gitlab-ci, feature-gitlab-ci) tests: do not test against the GitHub API
a38b3bea9 CI: unit tests for models race with each other and sometime fail
e89134b18 ci: setup GitLab CI
84f8b9192 (origin/feature-go-fed, feature-go-fed) go.mod: add go-fed/httpsig dependency (#14186)
88b71226c go.mod: add go-fed/activty/pub dependency (#14186)
c716a14e1 go.mod: add go-fed dependency (#14186)
DRAFT: monthly report, December 2021
#6

Rebasing forgefriends on today’s Gitea main branch, there were no conflicts. Using the list of branches as a checklist worked well. Using magit it goes like this and takes around five minutes:

image

  • For rebasing:
    • r (rebase) u (upstream): to rebase onto the base branch
    • P (push) -f (force) p (push): to force push
  • For reset+merge:
    • X (reset) h (hard) name-of-the-base-branch: to reset to the newer base branch and discard previous merge
    • m (merge) m (merge) name-of-the-branch-to-merge: to merge the newer version of the branch into the newer version of the base branch
    • P (push) -f (force) p (push): to force push

To improve readability, the pushRemote of each branch is set to origin using b (branch) p (pushRemote):

image

The CI the schedule tests in order for each branch:

image

There currently are nine branches for which the CI runs and each run takes around 20 minutes to complete. That’s a total of three hours altogether.

#7

As of today:

$ git log --oneline --no-merges gitea/main..origin/main
bbec2f378 (origin/main, origin/feature-activitypub, main, feature-activitypub) activitypub: implement Repository
b0e62c848 activitypub: hack_16834
90e7d7f7e activitypub: implement the ReqSignature middleware
5f33a3302 activitypub: signing http client
f0e2cc174 activitypub: go-fed conformant Clock instance
35cfcf3a3 activitypub: add the public key to Person (#14186)
7601dbc8c activitypub: implement /api/v1/activitypub/user/{username} (#14186)
a88a3df66 (origin/feature-projectbase, feature-projectbase) projectbase: implement tests for UpdateProjectBase
bbc588200 repo: add a .projectbase repository to each project
cff7128d0 repository: cloneLink(suffix) instead of cloneLink(isWiki)
d949fcb05 (origin/feature-development-environment, feature-development-environment) gitignore: emacs backups
1d030bca2 (origin/feature-gitlab-ci, feature-gitlab-ci) tests: do not test against the GitHub API
153f7b214 CI: unit tests for models race with each other and sometime fail
5e945051d CI: setup GitLab CI
fdc01ebda (origin/feature-go-fed, feature-go-fed) go.mod: add go-fed/httpsig dependency (#14186)
bbe81f309 go.mod: add go-fed/activty/pub dependency (#14186)
f03baf29e go.mod: add go-fed dependency (#14186)
#8

As of today:

$ git log --oneline --no-merges gitea/main..origin/main
ba8f45795 (HEAD -> main, origin/main, origin/feature-activitypub, feature-activitypub) activitypub: implement Repository
7dab6c39d activitypub: hack_16834
9e1c1ad1f activitypub: implement the ReqSignature middleware
d88740ce6 activitypub: signing http client
459b21973 activitypub: go-fed conformant Clock instance
20f6688ca activitypub: add the public key to Person (#14186)
c996d4fff activitypub: implement /api/v1/activitypub/user/{username} (#14186)
2aed47be9 (origin/feature-projectbase, feature-projectbase) projectbase: implement tests for UpdateProjectBase
eef302911 repo: add a .projectbase repository to each project
daf2f859e repository: cloneLink(suffix) instead of cloneLink(isWiki)
3048430dc (origin/feature-gitlab-ci, feature-gitlab-ci) tests: do not test against the GitHub API
8421f083f CI: unit tests for models race with each other and sometime fail
4387636b8 CI: setup GitLab CI
ee323b1c1 (origin/feature-development-environment, feature-development-environment) gitignore: emacs backups
eca651baa (origin/feature-go-fed, feature-go-fed) go.mod: add go-fed/httpsig dependency (#14186)
5b2626aa0 go.mod: add go-fed/activty/pub dependency (#14186)
22ef32733 go.mod: add go-fed dependency (#14186)
#9

As of today (there were conflicts due to the removal of the vendor directory), it took me about one hour to resolve them for all branches:

$ git log --oneline --no-merges gitea/main..origin/main
3ba2b3bbb (origin/main, origin/feature-activitypub, main, feature-activitypub) activitypub: implement Repository
24e1c3dcc activitypub: hack_16834
8a3c86573 activitypub: implement the ReqSignature middleware
323cdf840 activitypub: signing http client
7a3ac62f2 activitypub: go-fed conformant Clock instance
c023aee5f activitypub: add the public key to Person (#14186)
530967a97 activitypub: implement /api/v1/activitypub/user/{username} (#14186)
3008323ed (origin/feature-projectbase, feature-projectbase) projectbase: implement tests for UpdateProjectBase
f7c1894e7 repo: add a .projectbase repository to each project
8c1a9a51d repository: cloneLink(suffix) instead of cloneLink(isWiki)
de3d427c8 (origin/feature-gitlab-ci, feature-gitlab-ci) tests: do not test against the GitHub API
ff39ee4a5 CI: unit tests for models race with each other and sometime fail
f523580fe CI: setup GitLab CI
d5e6289aa (origin/feature-development-environment, feature-development-environment) gitignore: emacs backups
b73a5a0d4 (origin/feature-go-fed, feature-go-fed) go.mod: add go-fed/{httpsig,activity/pub,activity/streams} dependency (#14186)
#10

Today’s rebasing took three hours because there were conflicts due to:

  • changes made to this merge request
  • widespread refactoring adding a Context argument to many methods/functions
  • renaming of SetSettings and GetSettings to SetUserSettings and GetUserSettings
  • numerous syntactic changes introduced by the switch from gofmt to gofumpt
$ git log --oneline --no-merges gitea/main..origin/main
a9ed92a27 (origin/main, origin/feature-activitypub, main, feature-activitypub) activitypub: implement Repository
00179d9d4 activitypub: hack_16834
4ba7fccbd activitypub: implement the ReqSignature middleware
a973a7bfe activitypub: signing http client
f6772c76d activitypub: go-fed conformant Clock instance
22e278f37 activitypub: add the public key to Person (#14186)
994a0727c activitypub: implement /api/v1/activitypub/user/{username} (#14186)
fc49dec50 (origin/feature-projectbase, feature-projectbase) projectbase: implement tests for UpdateProjectBase
c8022434c repo: add a .projectbase repository to each project
16a4b9c12 repository: cloneLink(suffix) instead of cloneLink(isWiki)
2a14ba308 (origin/feature-go-fed, feature-go-fed) go.mod: add go-fed/{httpsig,activity/pub,activity/streams} dependency (#14186)
95d7b9881 (origin/feature-gitlab-ci, feature-gitlab-ci) tests: do not test against the GitHub API
93d915dc2 CI: unit tests for models race with each other and sometime fail
d12f061ce CI: setup GitLab CI
7608c5650 (origin/feature-development-environment, feature-development-environment) gitignore: emacs backups