chore: merge chatmail/filtermail into chatmail/relay to have a monorepo

[missytake]

This is a proposal to merge chatmail/filtermail and chatmail/relay into one monorepo, so we can change filtermail with only one PR in the future.

The paths in the filtermail commits have been rewritten so that they were always in the filtermail/ subdirectory, so the commit history of each file is preserved. What I did to merge the repositories, in case it needs to be repeated slightly differently:

cd $USER/code/relay/..
git clone https://github.com/chatmail/filtermail
cd filtermail
git filter-repo --to-subdirectory filtermail --tag-rename v:filtermail-
cd ../relay
git checkout main
git pull --rebase
git checkout -b merge-filtermail
git remote add filtermail-merge ../filtermail
git fetch filtermail-merge
git merge --allow-unrelated-histories filtermail-merge/main

Then I did 3cf3937 on top of it to integrate filtermail's .github/ and .gitignore into chatmail/relay.

open questions:

• how should we distinguish tags? does filtermail still need its own software versions? Other projects import filtermail, afaik. I renamed the filtermail tags from v1.2.3 to filtermail-1.2.3 for now, but didn't push them so far.
• should we rename the relay tags as well, from 1.2.3 to relay-1.2.3?

[j-g00da]

does filtermail still need its own software versions?

In theory it does not, but then we probably want to tag relay more often, as it won't be possible to deploy relay from main if filtermail changed without compiling it locally.

[j4n]

Thanks for tackling this, two non-inline comments:

• ci.yaml still polls the old filtermail release, adapt that to the build pattern from filtermail-ci.yaml.
• Changelog: There are now two and two cliff.toml; needs some --include-path work so this won't clobber.

Re versioning: prefixed filtermail-X.Y.Z tags, with relay staying unprefixed, seems the conventional shape; I would not retroactively rename relay's tags to relay-X.Y.Z as it breaks existing release-download URLs.

[missytake]

CI requires chatmail/cmlxc#31 I think - edit: apparently not, it's green :)

[j4n]

Changes looks good. Some more comments inline.

One thing for a followup-PR: RELEASE.md is currently relay-only and should have a filtermail section, something like:

# Releasing chatmail relay
...

# Releasing filtermail

filtermail is versioned independently and released from the same repo,
using `filtermail-` prefixed tags. To release filtermail 0.7.5:

1. Update the changelog:
   `git cliff --config filtermail/cliff.toml -u -t filtermail-0.7.5 -p filtermail/CHANGELOG.md`
2. Bump `version` in `filtermail/Cargo.toml` and commit `filtermail/Cargo.lock`.
3. Commit with the message `feat(release): prepare filtermail for 0.7.5`;  or chore() if it should be excluded from relay changelog.
4. Open a PR with the new commit, merge it to main after review.
5. In the web interface, create a GitHub release, tell it to create a new tag `filtermail-0.7.5`. Do not push the tag by hand: the release must exist before the upload job runs.
6. Bump the pinned version and sha256sums in`cmdeploy/src/cmdeploy/filtermail/deployer.py`.

[j4n]

Suggested change

	cmlxc -v deploy-cmdeploy --source ./repo cm0
	cmlxc -v deploy-cmdeploy --source ./repo --filtermail repo/filtermail/target/x86_64-unknown-linux-musl/release/filtermail cm0

if we do build filtermail at all times, we should probably also use it in the test, otherwise we dont need to build it; though we will have a path clash here that will need fixing in cmlxc.

[missytake]

I assume this is the path clash you mean? https://github.com/chatmail/relay/actions/runs/32125762857/job/95675867660

[j4n]

This needs to be updated.

[missytake]

Hm, how do we want to distribute binaries? Maybe https://download.delta.chat/filtermail? I couldn't see a way to filter github.com/chatmail/relay/releases by tag prefix :/ searching for ?q=tag:1.12.0 matches one release, but ?q=tag:1.1 matched none

[missytake]

6. Bump the pinned version and sha256sums incmdeploy/src/cmdeploy/filtermail/deployer.py.

Hm, I added it - but this means we again need 2 PRs until filtermail changes arrive in cmdeploy? Wasn't this the purpose of this PR, that this is simpler now, e.g. by building filtermail in cmdeploy if there are changes?

[j-g00da]

but this means we again need 2 PRs until filtermail changes arrive in cmdeploy? Wasn't this the purpose of this PR, that this is simpler now, e.g. by building filtermail in cmdeploy if there are changes?

Yes I brought up this topic on a group chat some time ago. If we don't come up with a way to do this in one PR, then I don't really see any benefit of merging the repos. In fact it IMO only makes things more confusing.

[missytake]

but this means we again need 2 PRs until filtermail changes arrive in cmdeploy? Wasn't this the purpose of this PR, that this is simpler now, e.g. by building filtermail in cmdeploy if there are changes?

Yes I brought up this topic on a group chat some time ago. If we don't come up with a way to do this in one PR, then I don't really see any benefit of merging the repos. In fact it IMO only makes things more confusing.

Hm, the options I see:

• stop having filtermail releases, and using the main branch instead:
  • a) We can build filtermail from source during cmdeploy run. (this is basically what we do with chatmaild.)
  • b) We could have a github action which builds filtermail and uploads it to https://download.delta.chat/filtermail on pushes to main which change the filtermail/ directory.
    • Any PR would need to update the sha256sum in cmdeploy, a github action or git hook could take care we don't forget it.
• c) keep it as it is.

What do you think? I think I prefer option b).

[j-g00da]

I think a) won't work as IIRC we don't want to force users to build it locally. I like option b), but then what do we want to do with version string, ideally we would auto-bump it on each change.

Alternative, (let's call it "d)") I had in mind was to use the latest available binary or build from source if there were any changes in filtermail subdir. This way we can still build filtermail only when tagging relay, but if someone doesn't mind building from source - they can use main branch.

I'm fine with either b) or d) whichever seems simpler.

[hpk42]

this filtermail/relay merging all turned into pretty much of a quagmire 😅 Considering that we actually want to go for a new single-file installer i don't think it's worth to spend our collective time on this PR any further, i am afraid. I don't have easy answers to some of the questions above, and also think that if "single-PR" changes of both relay and filtermail don't become reliably and somewhat easily possible, we don't win much.

So i suggest for now to close this PR, and stay with the two-repo solution until we get more clarity on the single-file installer layout/changes. Sorry for triggering this two-repo merge early on without thinking it through, and thanks Nami and all for engaging.

[missytake]

Sounds good to me :)