Skip to content
This repository has been archived by the owner on Jul 1, 2021. It is now read-only.

Status of this repo / docker image #418

Closed
2 tasks done
prometheanfire opened this issue Nov 4, 2019 · 62 comments
Closed
2 tasks done

Status of this repo / docker image #418

prometheanfire opened this issue Nov 4, 2019 · 62 comments

Comments

@prometheanfire
Copy link

Classification

  • Question

Reproducibility

  • Always

Description

I see that you marked this repo as Unmaintained repo, are there plans to hand this off to someone. I'd be a shame to loose this (especially given the 1 million+ pulls). If you are looking for manpower I could probably help out a bit. I ran a very similiar setup to the docker image before migrating to it about a month ago...

@hardware
Copy link
Owner

hardware commented Nov 4, 2019

Unmaintained status is not final if someone wants to take care of it. For now I have not found anyone to maintain this repo in an active and regular way, most people just want to make pull request (and that's fine), but that's not enough to maintain a project properly. And I do not want to leave any status, so I preferred to be clear about my position about the images set to find someone asap.

I have probably spent more than 1500+ hours on this docker image since 2016 and I need a maintainer who can take care of it on a monthly basis, because security and auditing are important, not a bit from time to time. But if several people can take care of it from time to time, it's good too.

This image was downloaded 1 million times last 6 months (jeez it's crazy), so that means a lot of people are using it, even small companies to my knowledge, so it's a responsibility.

@hardware hardware pinned this issue Nov 4, 2019
@prometheanfire
Copy link
Author

I'd be willing to help out, I already maintain stuff for a distro downstream, though not mail (mainly ldap as related to this repo). I think the problem may be that it is more than one person can handle. What about this is taking the most time? Where would help be best applied?

@shawniverson
Copy link

Following...

@hardware
Copy link
Owner

hardware commented Nov 5, 2019

What about this is taking the most time ?

Testing / building / upgrade / project monitoring.

When you maintain an image for yourself it's easy, you can try and fail, then retry. But when people are using it with different purpose and usage, a good testing process, stability and backward compatibility are important, but that takes time.

Where would help be best applied ?

The next steps for this image are :

  • Upgrade to Rspamd 2
  • Upgrade to Treafik 2
  • Upgrade to Debian 10 in stable branch (already working on the latest branch)

When you upgrade something, you need to test each piece of software, unit tests are very useful in this case, but it is not enough. After each change, you need to check the building process, the logs and behavior of all 6 images (mailserver_default, mailserver_reverse, mailserver_ecdsa, mailserver_ldap, mailserver_ldap2, mailserver_traefik_acmev1, mailserver_traefik_acmev2), then test in real conditions for weeks before pushing in stable branch.

@sknight80
Copy link
Contributor

@hardware if we start adding testing infrastructure to this:

When you upgrade something, you need to test each piece of software, unit tests are very useful in this case, but it is not enough. After each change, you need to check the building process, the logs and behavior of all 6 images (mailserver_default, mailserver_reverse, mailserver_ecdsa, mailserver_ldap, mailserver_ldap2, mailserver_traefik_acmev1, mailserver_traefik_acmev2), then test in real conditions for weeks before pushing in stable branch.

Would be easier for maintaining the images? I am using this image and others from you, and I am willing to allocate my time to review, improve the repo.

@prometheanfire
Copy link
Author

I believe the traefik-2 change was merged a week or two ago.
I could look at the rspamd-1 -> rspamd-2 upgrade. Though that'd fall behind the traefik-2 and debian-10 changes as they are already merged (just time wise).
For Debian 10, this means the testing needs to hit the scenarios to be approved for stable.

I wonder if some CI testing like we've done in openstack would help (we have a test unit called 'grenade' that tests upgrades, along with various scenarios).

After each change, you need to check the building process, the logs
and behavior of all 6 images (mailserver_default, mailserver_reverse,
mailserver_ecdsa, mailserver_ldap, mailserver_ldap2,
mailserver_traefik_acmev1, mailserver_traefik_acmev2)

I personally do not know what goes into each of these configurations and tests, but it can likely be automated.

acmev1 and v2 can likely be merged as they are hitting the same code path (just verify that it gets the right data from the datastructure and puts it in the right place for both acmev1 and acmev2). Once the datastructure munging part is tested, whatever goes for standard testing would occur.

The ecdsa testing sounds like it's testing if random_curve can be made to work with postfix? The readme looks like it handles that already documentation wise. It may also be easier to state what is supported / tested and leave going beyond that to the end user. Need to know when and what to test and all that. (hopefully this would drop that test).

I do not know why there are two ldap tests either, nor do I know the reason for the default and reverse tests, but perhaps they could be pruned down or focused.

Should be able to drop to 5 tests in my view.

  1. mailserver_default
  2. mailserver_reverse
  3. mailserver_ecdsa
    a. dropped
  4. mailserver_ldap
  5. mailserver_ldap2
  6. mailserver_traefik_acmev1
  7. mailserver_traefik_acmev2
    a. merged with other acme test

I'm not sure why the images need to be deployed in the real world for so long and so many configurations. Are you looking to test something specific or has it caught something unexpected in the past? If not I'd reduce this test to be what's needed, not what feels good (sounds like over-caution may be a thing here).

@marcdeop
Copy link

I would be willing to help maintaining this software :-)

I have created a room at matrix.org: #mailserver:matrix.org in case anybody wants to join and have a more interactive talk/discussion.

Best,

Marc

@sknight80
Copy link
Contributor

Hi Marcdeop! Sounds good. I joined to the chat.

@prometheanfire
Copy link
Author

joined as well

@chuckmckinnon
Copy link

I'll join the chat. This is a great project, the best of its kind I've found, and it would be a shame for it to go unmaintained.

@shawniverson
Copy link

Count me in. I'll join.

@albjeremias
Copy link

is there a donation button somewhere? thank you so much for this project..

@albjeremias
Copy link

albjeremias commented Dec 18, 2019

I'm not sure why the images need to be deployed in the real world for so long and so many configurations. Are you looking to test something specific or has it caught something unexpected in the past? If not I'd reduce this test to be what's needed, not what feels good (sounds like over-caution may be a thing here).

because email is complicated.. and if you want to test acme and SSL certificates you will need a real server for the tests!

mailserver_default
mailserver_reverse
mailserver_ecdsa
a. dropped
mailserver_ldap
mailserver_ldap2
mailserver_traefik_acmev1
mailserver_traefik_acmev2
a. merged with other acme test

I would suggest some other tests:

  • test LDAP authentication
  • test send an email
  • test receive an email
    • test imap if email was received
    • test pop3 if email was received
  • test spam filters somehow (SPF, DKIM, DMARC, ARC, ratelimit and greylisting capabilities)
  • test automatic GPG
  • test auto-responder, auto-forward
  • test fetchmail
  • test the webmail if logging in works
  • test that you can login to postfixadmin
  • find a test for unbound/NSD/traefik/SSL..?
  • test LDAP (if login works)

@prometheanfire
Copy link
Author

You can get certificates not via public IPs and not exposed to the internet. I'm using traefik and rfc2136 (send updates to bind), there are other plugins that can be used as well (cloudflare, etc). I'd say for testing self signed could even work, though I see no reason why that's needed.

@padrino121
Copy link

Time is a fleeting resource however given the popularity would you support some fundraising to help fund your continued time to maintain or fund maintenance utilizing other resources?

@alexanderadam
Copy link

alexanderadam commented Feb 7, 2020

This is a great project, the best of its kind I've found

May I ask what exactly makes it better than projects like Mailu, mailcow, tomav's mailserver, jeboehm's mailserver, kopano and other projects?

I mean, the only thing I'm personally missing on the other projects is the Zeyple integration (1, 2, 3) and security features like MailCrypt aren't even in this project (i.e. Mailcow has at least MailCrypt support in global mode and tomav's solution and Mailu might get it someday).

Also LDAP is already build in in tomavs mailserver, somehow working in mailcow (not yet in Mailu).

Mailserver administration is not an easy task and maybe it would be a good thing to merge efforts?
Or are those projects too different on their own? 🤔

@prometheanfire
Copy link
Author

For me it was basically identical to the setup I was migrating from. There's also a cost to migration to consider. This project did a lot of things and was reasonably popular.

@albjeremias
Copy link

albjeremias commented Feb 7, 2020

thank you @alexanderadam awesome post! :)

@prometheanfire let's think about that together? I also need to do it...

@navossoc
Copy link
Contributor

navossoc commented Feb 7, 2020

At the time, this was one of the few projects that use rspamd.

@msander
Copy link

msander commented Mar 9, 2020

Is there a convenient way to migrate to another project?

@KochFolie
Copy link

I would also like to thank you for this great project, which I've been using for some years.

That said I migrated to tomav's mailserver and, after realizing it doesn't support per-mailbox-quota, moved on to mailcow a few days ago. I basically just created my domains, mailboxes and aliases, sent test mails to them, so the folders would get created, copied over all mail-files and set correct owner/permissions. I would guess that this process is similar for most other projects.

@denji
Copy link
Contributor

denji commented Mar 14, 2020

@KochFolie #48 (comment)

@AndrewSav
Copy link

@alexanderadam there are little differences everywhere. For example, I'm fond of Rainloop and in particular about it's feature that allows you to specify raw Sieve script for filtering. I'm using this feature extensively. From the list you've given, it appears that only mailu comes with Rainloop out of the box and then it has Sieve scripts disabled. It also has Rainloop admin panel disabled so you cannot re-enable it easily. Some people was able to do this via a config change, but that gets regenerated on every restart of the container.

Of course one could install rainloop separately, but the point of a project like this, is that you do not have to.

In conclusion, I do not believe that there is one size fits all.

@alexanderadam
Copy link

alexanderadam commented Apr 17, 2020

there are little differences everywhere.

Of course there are. It's also a good thing obviously. 😉

In conclusion, I do not believe that there is one size fits all.

I didn't claim that it have to. But a project should at least be maintained. And maybe even maintained by more than one person.
IMHO it would even better if there would be less popular projects at all. So that there are more maintainers/contributors per project.

This could increase development speed and increase the probability of implementing even "less popular" features (i.e. specifying Sieve scripts from Rainloop).

Having said that, the maintainers of mailu (or any other project) would probably even be happy if you would add a PR that adds this feature.

@neuffer
Copy link

neuffer commented Apr 17, 2020

The questions do we have a chance to get the project back into the status maintained?

@AndrewSav
Copy link

@neuffer from the state of this thread it does not seem likely

@neuffer
Copy link

neuffer commented Apr 17, 2020

Which is a shame
The one alternative that I have found is sovereign, but it refused postfixadmin when Adrian Goins contributed the necessary patches oscapt/sovereign and has some very strange notions about users and their data.
An upgrade/install always includes the deletion of all accounts & user data. The users are predefined in the Ansible config files.

@AndrewSav
Copy link

Having said that, the maintainers of mailu (or any other project) would probably even be happy if you would add a PR that adds this feature.

Having a bit of experience with open source, it's not that simple. There are 30 open PR's at mailu right now, some of which months old. Some open project maintainers (I'm generalising I have no information about mailu in particular) are very strict with what they would merge. At the same, they are usually busy, and not available for a proposed PR discussion.

Not everyone has skills to create good PR too. Assuming I do, I would not waste my time working on a PR, that does not have a chance to get merged, so to me an approval in principal form the maintainers is a prerequisite for working on PR.

From my analysis of previous similar issues on mailu GitHub, the maintainer has a philosophy of "keeping it simple", this was the reason, why exposing Rainloop admin UI was rejected. I did open an issue re:sieve, just to find out how they go about such things, but I'm not very optimistic. Unfortunately, the way containers are stand up in mailu affect configuration management in Rainloop, in particular, main configuration files are assembled from scratch from minimalist templates, overwriting any changes that an admin could do should be thre Rainloop admin UI enabled. I'm not clear about the reason, why it's done this way, but it seems to have to do how discovery works for "neighbouring" containers which can differ, say between docker and kubernetes.

In areas of that complexity, from my experience maintainers sometimes are very reluctant to accept changes, since a person outside of the project has no chance to guess what is "the right" way from the maintainer perspective in such complex situations. At the same time they are understandably busy to be able to explain those to everyone who asks.

So in my opinion, it is possible but not that probable ;)

@AndrewSav
Copy link

AndrewSav commented Apr 28, 2020

@sknight80 so do you prefer to use your fork going forward or @SaraSmiseth 's?

I would like to also include rainloop updates (which I've done), under which repo do you guys think it's best to do?

It's kind of funny, three people duplicated the same work independently: same as you guys I upgraded postfixadmin, debian-mail-overlay, rspamd, etc to the latest. Did not go as far as including pull requests though and did not try to run the test.

Oh and I also updated traefik config to v2. I'll try and publish rainloop and traefik changes shortly, but it would be nice to decide in which repo the new home will be.

@AndrewSav
Copy link

AndrewSav commented Jun 2, 2020

In case anyone is interested I published my traefik v2 configs here: https://github.com/AndrewSav/mailserver/tree/master/docker-compose

Rainloop update is here: https://github.com/AndrewSav/rainloop

@neuffer
Copy link

neuffer commented Jun 2, 2020

@hardware @AndrewSav @SaraSmiseth @sknight80 If the four of you would come to some kind of understanding of working together, maybe this project can be resurrected and people have once again a central converging point?
That would be awsome. :-)

@navossoc
Copy link
Contributor

navossoc commented Jun 2, 2020

@AndrewSav going to traefik v2 is on my TODO list ;p

Have you tested the renew? Is it working properly?
Last time I checked there was some work being done to auto reload the certificates when they are renewed, not sure how it is right now...

@AndrewSav
Copy link

@navossoc I've been running traefik v2 elsewhere for months with no problems with renewals.

@sknight80
Copy link
Contributor

Wow! Great @AndrewSav ! Thanks for sharing! If you don't mind, I would love to merge into my branch to test out in my setup.

@neuffer, I would be more than happy to start a collaboration in this project with the others. @AndrewSav , @SaraSmiseth what do you think?

@neuffer
Copy link

neuffer commented Jun 2, 2020

@ksylvan How up-to-date are your mailserver Ansible configs/scripts? Maybe this could be merged in as well?

@AndrewSav
Copy link

@sknight80 I tried to reach out to you above, basically asking the same thing.

I did NOT do the tests and did NOT do the PRs. However I did traefik and rainloop.

The new organisation created by @SaraSmiseth looks like a perfect place to accumulate all the changes but they have not responded so far. Ideally we would need rights to merge there or have PRs merged relatively quickly, which is a big ask.

I am personally not volunteering to do that (lead maintaining a fork), is because there are a few areas that require some time investment. For example I think the tests are crucial, yet I'm not familiar with the testing framework, and I don't feel I can familiarise myself with it in reasonable time.

Similarly I have no experience with Clamav, Zeyple, Unbound, NSD, or OpenLDAP, and I'm personally not using these.

If you are happy to lead and maintain a fork, then you are of course welcome, I will help within constraints of abilities and time available. If not, then it will remain as it is now.

If you have any issues with integrating traefik 2, I'll do my best to work with you to resolve them, I'm quite experienced with traefik, and I do not anticipate any hard obstacles there.

@sknight80
Copy link
Contributor

@neuffer : Currently, my fork is up-to-date. However, I need to test if the image is actually is capable to start. Last time when I checked there was some problem so, I have to check it.

@AndrewSav
Copy link

Oh and clamav now complains that it's out of date, even though it's the latest version available in debian apt

@sknight80
Copy link
Contributor

@AndrewSav same here, however, I checked a couple of existing test but I need time to get familiar with it.

I added a little roadmap into my fork what I planned to do with the project, but I am more than happy to start working in a centralized place.

@SaraSmiseth
Copy link

@sknight80 I tried to reach out to you above, basically asking the same thing.

The new organisation created by @SaraSmiseth looks like a perfect place to accumulate all the changes but they have not responded so far. Ideally we would need rights to merge there or have PRs merged relatively quickly, which is a big ask.

I will invite you and the others and give you rights to merge PRs.

I am personally not volunteering to do that (lead maintaining a fork), is because there are a few areas that require some time investment. For example I think the tests are crucial, yet I'm not familiar with the testing framework, and I don't feel I can familiarise myself with it in reasonable time.

Well I think thats the main problem. I don't really want to be the lead maintainer either. Maybe we can split this if we have enough maintainers and whoever has time can review and merge PRs but I'm not really sure if thats a good idea.

I have experience with the tests as I have updated them. I have protected the master branch and made it so that changes are only possible in form of PRs which pass all tests. We should also require new tests for new features.

We should probably also require that at least 2 maintainers review PRs before they get merged.

@AndrewSav
Copy link

AndrewSav commented Jun 2, 2020

I don't think there will be a lot of PRs to be honest so it could be workable. We'll see. Thank you for the invites.

I added rainloop here: https://github.com/mailserver2/rainloop

@AndrewSav
Copy link

@sknight80 can you confirm that you got invites for the new org? Is there anything in your fork that is worth porting to that repo, now when you can merge there?

I probably need to make a PR for traefik and merge it. I put the new configs separately in my repo, but I think they need to replace the v1 in the common repo.

Also @SaraSmiseth @sknight80 what do we do with the docker images? We cannot update the old ones in place for obvious reasons, should we also create a new project on DockerHub?

@pollux
Copy link

pollux commented Jun 2, 2020

@AndrewSav there already is a new project on dockerhub, its linked in the mailserver2/mailserver repo.

@AndrewSav
Copy link

Sure, but how do I, say, get the rainloop image pushed to it?

I just created a brand new discord server here https://discord.gg/nb8tRGs in case people want to discuss this in a chat rather than a thread.

@SaraSmiseth
Copy link

I probably need to make a PR for traefik and merge it. I put the new configs separately in my repo, but I think they need to replace the v1 in the common repo.

Yes please create a PR. Yeah I think its better for the average user to put it all in one docker-compose.yml file. Makes it easier to get started if you only run mailserver.

Sure, but how do I, say, get the rainloop image pushed to it?

I just created a brand new discord server here https://discord.gg/nb8tRGs in case people want to discuss this in a chat rather than a thread.

I have added you to the dockerhub organization and created a new repository for rainloop and setup automated builds.

@sknight80 Tell me your dockerhub account and I'll add you as well.

@ksylvan
Copy link
Contributor

ksylvan commented Jun 3, 2020

My docker hub user is kayvan

I’d love to check this out as it cones back alive and update my ansible playbooks and contribute them to the repo.

@sknight80
Copy link
Contributor

@sknight80 can you confirm that you got invites for the new org? Is there anything in your fork that is worth porting to that repo, now when you can merge there?

I probably need to make a PR for traefik and merge it. I put the new configs separately in my repo, but I think they need to replace the v1 in the common repo.

Also @SaraSmiseth @sknight80 what do we do with the docker images? We cannot update the old ones in place for obvious reasons, should we also create a new project on DockerHub?

I got it! Will check it today.

@sknight80
Copy link
Contributor

I probably need to make a PR for traefik and merge it. I put the new configs separately in my repo, but I think they need to replace the v1 in the common repo.

Yes please create a PR. Yeah I think its better for the average user to put it all in one docker-compose.yml file. Makes it easier to get started if you only run mailserver.

Sure, but how do I, say, get the rainloop image pushed to it?

I just created a brand new discord server here https://discord.gg/nb8tRGs in case people want to discuss this in a chat rather than a thread.

I have added you to the dockerhub organization and created a new repository for rainloop and setup automated builds.

@sknight80 Tell me your dockerhub account and I'll add you as well.

My dockerhub account: knightsecret

@AndrewSav
Copy link

AndrewSav commented Jun 3, 2020

I got it! Will check it today.

@sknight80 no rush ;)

@AndrewSav
Copy link

@SaraSmiseth do you happen to know if one can run tests locally?

@AndrewSav
Copy link

@SaraSmiseth sorry I recreated rainloop repo to fork from the origin, not from my fork, I saw you accessed it, and your access key may be lost now. I invite you to discord (link above) because I feel that discussing these details here is no longer appropriate.

@SaraSmiseth
Copy link

SaraSmiseth commented Jun 4, 2020

@SaraSmiseth do you happen to know if one can run tests locally?

Yes. Make sure you also initialize and clone the submodules. Just run make. This will build the image and run the tests.

@SaraSmiseth sorry I recreated rainloop repo to fork from the origin, not from my fork, I saw you accessed it, and your access key may be lost now. I invite you to discord (link above) because I feel that discussing these details here is no longer appropriate.

I'll join later.

EDIT: Well I joined #mailserver:matrix.org. Discord wants too much personal information. I'm not gonna give them my phone number.

@AndrewSav
Copy link

AndrewSav commented Jun 4, 2020

@SaraSmiseth Understood. I'm not going to use matrix org as it's most clunky and slow piece of software IMO, I've ever used. It boggles my mind why people put up with using it. I just created another account on discord and joined the server all I was asked is my email and a new password - no other information, personal or not. Where did you get the phone number thing from? https://discord.com/register

@AndrewSav
Copy link

I opened a new thread here: mailserver2#9 since it could be more appropriate.

@stale
Copy link

stale bot commented Dec 4, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the Stale label Dec 4, 2020
@stale stale bot closed this as completed Dec 12, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests