frequency of breaking / major releases and release schedule

[decathorpe]

gtk-rs projects are used by more and more projects. Linux distributions like Fedora and Debian are not building applications that use these bindings with vendored dependencies, resulting in a high workload for package maintainers: Not all applications update their gtk-rs dependencies on the same schedule.

This affects all applications that depend on crates from gtk-rs-core, gtk3-rs, gtk4-rs, gstreamer-rs, and various other Rust bindings that are tied to them, like bindings for libhandy, libadwaita, libpanel, vte4, etc.

GNOME applications typically update somewhere between their alpha and final releases, but not all applications bump their dependencies at the same point in time, and it also depends on how the major release of gtk-rs lines up with the GNOME release cycle.

Some applications like system76-keyboard-configurator or squeekboard are using the GTK3 bindings which were deprecated with the previous gtk-rs release, and hence pull in increasingly old versions of the gtk-rs stack and don't have a way forward without also porting to GTK4.

Other components like gstreamer-plugins-rs would ideally be "updatable" to the latest version, but those are also tied to the latest gstreamer-rs bindings, and that doesn't line up with GNOME releases (i.e. snapshot).

This situation currently results in two different outcomes:

• The gtk-rs stack is outdated, and is only updated when all dependent applications and libraries are ready for the next major version. Only one version of the gtk-rs stack is available at a time, and it's likely not the latest one. This is often blocking updates for core GNOME applications like loupe, snapshot, or gnome-tour. This is what is currently happening in debian (to the best of my knowledge).

• Multiple versions of the entire gtk-rs stack (i.e. packages for all the Rust crates that make up gtk-rs-core, gtk4-rs, gstreamer-rs, gstreamer-plugins-rs, etc.) need to be provided and maintained in parallel to allow for different adoption speeds of major gtk-rs versions in GNOME and other applications, and to be able to ship application updates independent of the GNOME release cycle. Depending on how the releases of gtk-rs and GNOME align, this places the burden of having to update dozens of Rust crate packages (and creating dozens of new ones for the previous version), often under time pressure, i.e. in the short window of time between the gtk-rs major release and GNOME applications starting to depend on the new version. This is what is currently happening in Fedora (I am the maintainer of the gtk-rs stack in Fedora).

I don't think either situation is good or sustainable. Given that neither debian nor Fedora package maintainers consider building applications with vendored Rust dependencies acceptable, that is also not a solution.

---

I have briefly talked with @sdroege about the possibility of reducing the frequency of breaking / major releases. This could make lives of package maintainers much easier and less stressful.

It would likely require discussing an actual release schedule, and a discussion of the target release frequency. Currently gtk-rs major releases seem to be done every 5-7 months, ~mostly lining up with being shortly before GNOME alpha/beta releases, but not always. Depending on how things line up, it's much more (or much less) stressful to handle.

For example, currently Fedora ships gtk-rs-core v0.19 / gtk4-rs v0.8 / gstreamer-rs v0.22, but GNOME applications will very soon start to depend on the next major version of these three stacks of crates. I'll try to proactively make the latest / next major versions of all crates available soon, but I'm not sure if it will be done before GNOME maintainers start breaking down my door again.

On the other hand, Fedora also currently ships gtk-rs-core v0.15, v0.17, and v0.18, and gtk4-rs v0.7, because some applications still haven't been ported to gtk-rs-core v0.19 / gtk4-rs v0.8 yet. As things stand, this is not sustainable - the number of versions that I would need to provide in parallel cannot grow forever.

Adapting the release schedule (for example, one breaking / major release per year instead of ~two) would help a lot. In Fedora, we might need to keep old versions around for a little longer, but hopefully not as many old versions as right now. It would also help reduce the debian maintainers' workload, making it necessary to port everything to the latest versions only ~once per year instead of ~twice.

---

An alternative solution in Fedora might be to only provide major new versions of the gtk-rs stack in new release of Fedora, but not on stable releases, and not to provide both old and new versions in parallel. This is how we used to handle gtk-rs updates up until a few years ago when there were much fewer applications that used the gtk-rs bindings. This approach has two big downsides though - it would prevent us from shipping some application updates on stable branches since not everything follows the GNOME release cycle, and it would result in some applications being broken for indeterminate amounts of time because they're not ready for the new major version. Judging by the speed at which non-GNOME applications update their gtk-rs dependencies, this would be a major problem, and would likely result in some applications being removed from Fedora. Which is the reason why we ultimately decided to go with the current multi-version-availability approach over the old "update or die" approach.

[bilelmoussaoui]

Tbh, one of the major reasons why someone would use Rust instead of another language is to statically link dependencies, what distros are doing is wrong and I won't change my mind about it nor do more work on the bindings I maintain (gtk4-rs & most of the stuff in World/Rust), just for the sake of making it easier for distros.

[decathorpe]

There seems to be a misunderstanding here. We do statically link dependencies. We just don't vendor them with applications.

[bilelmoussaoui]

There was no misunderstanding, I am familiar with Rust packaging in fedora but I didn't express my thoughts properly.

• I agree, we are doing releases too often, I am quite burned out because of that. Too much work. So reducing the amount of major releases would be super appreciated.
• I don't agree that we should avoid doing breaking releases if we have some breaking API changes to make, although, those have reduced drastically as the core bindings are being "stabilized"

[decathorpe]

Sure, if you need to make breaking API changes to fix bugs or something like that, that's a good reason to do them.

Though if it's a safety fix, you might even get away with doing those "technically breaking API" changes without bumping the major version. :)

[werdahias]

gtk-rs projects are used by more and more projects. Linux distributions like Fedora and Debian are not building applications that use these bindings with vendored dependencies, resulting in a high workload for package maintainers: Not all applications update their gtk-rs dependencies on the same schedule.

This affects all applications that depend on crates from gtk-rs-core, gtk3-rs, gtk4-rs, gstreamer-rs, and various other Rust bindings that are tied to them, like bindings for libhandy, libadwaita, libpanel, vte4, etc.

FWIW, gtk3-rs in Debian has been removed (except for two source packages where it's vendored in).

GNOME applications typically update somewhere between their alpha and final releases, but not all applications bump their dependencies at the same point in time, and it also depends on how the major release of gtk-rs lines up with the GNOME release cycle.

Some applications like system76-keyboard-configurator or squeekboard are using the GTK3 bindings which were deprecated with the previous gtk-rs release, and hence pull in increasingly old versions of the gtk-rs stack and don't have a way forward without also porting to GTK4.

Other components like gstreamer-plugins-rs would ideally be "updatable" to the latest version, but those are also tied to the latest gstreamer-rs bindings, and that doesn't line up with GNOME releases (i.e. snapshot).

This situation currently results in two different outcomes:

* The gtk-rs stack is outdated, and is only updated when all dependent applications and libraries are ready for the next major version. Only one version of the gtk-rs stack is available at a time, and it's likely not the latest one. This is often blocking updates for core GNOME applications like loupe, snapshot, or gnome-tour. This is what is currently happening in debian (to the best of my knowledge).

Yes. We only can have one version of gtk-rs at a time and all applications consuming those will need to be updated all at the same time. This is frustrating at times since we need to manually patch some apps for the new release when they haven't switched yet.
Just yesterday I caught up with updating gtk-rs to 0.8. The main reason were other crates, but having to update 50+ source packages every ~6 months is a huge drain on my maintainer time. As the sole maintainer of gtk-rs, gstreamer and most associated apps longer release cycles / less ABI breaking releases would be much appreciated.

I don't think either situation is good or sustainable. Given that neither debian nor Fedora package maintainers consider building applications with vendored Rust dependencies acceptable, that is also not a solution.
FWIW, partially vendoring is somewhat acceptable, but in no way the norm. I hope I don't have to argue here why we build everything from source.

I have briefly talked with @sdroege about the possibility of reducing the frequency of breaking / major releases. This could make lives of package maintainers much easier and less stressful.

It would likely require discussing an actual release schedule, and a discussion of the target release frequency. Currently gtk-rs major releases seem to be done every 5-7 months, ~mostly lining up with being shortly before GNOME alpha/beta releases, but not always. Depending on how things line up, it's much more (or much less) stressful to handle.

Full ack. CC @jbicha

Adapting the release schedule (for example, one breaking / major release per year instead of ~two) would help a lot. In Fedora, we might need to keep old versions around for a little longer, but hopefully not as many old versions as right now. It would also help reduce the debian maintainers' workload, making it necessary to port everything to the latest versions only ~once per year instead of ~twice.

An alternative solution in Fedora might be to only provide major new versions of the gtk-rs stack in new release of Fedora, but not on stable releases, and not to provide both old and new versions in parallel. This is how we used to handle gtk-rs updates up until a few years ago when there were much fewer applications that used the gtk-rs bindings. This approach has two big downsides though - it would prevent us from shipping some application updates on stable branches since not everything follows the GNOME release cycle, and it would result in some applications being broken for indeterminate amounts of time because they're not ready for the new major version. Judging by the speed at which non-GNOME applications update their gtk-rs dependencies, this would be a major problem, and would likely result in some applications being removed from Fedora. Which is the reason why we ultimately decided to go with the current multi-version-availability approach over the old "update or die" approach.

[sdroege]

I think it would make sense to reduce the number of breaking releases at least.

• It's causing a lot of work for Linux distributions. Let's ignore that it's silly to package Rust crates this way: things are what they are, and exactly the same problem happens with curated, non-crates.io crate registries (which arguably would make things a lot easier for Linux distributions in other regards)
• Problems for curated, non-crates.io crate registries. While private ones by companies are outside our scope, in discussions with @mcatanzaro we came to that as a possible solution for reducing the risk of supply-chain attacks in the context of GNOME (unlike C, we actually have tools for this). This would be in combination with something like Building a cargo vet infrastructure release#196 .
• It's a lot of work for application developers to update everything every 6 months, and even GNOME core software like librsvg is apparently struggling with that (cc @federicomenaquintero). It causes a lot of churn, requires apps to wait until all their dependencies are finally updated (which might also take a while), and doesn't really bring any great improvements anymore at this point so people are questioning the point of 6 monthly breaking changes.
• It's a lot of work for us and other bindings maintainers for the very same reasons. Doing a micro release is usually a matter of a few minutes for me, doing a big breaking release a matter of 2 days.

Currently the main reasons for the breaking releases every 6 months are

• New versions of the libraries with new APIs. There's usually no reason why we couldn't backport these to the previous release as this doesn't cause any breakages, and this already happened in the past. We also regularly backport other new API.
• Breaking API changes where the bindings were previously incorrect. The most common case of this are changes between Option<T> and T in either direction. These are arguably bugfixes and while they potentially break compilation of code that currently might happen to work fine, at this point I think these only really affect very rarely used APIs. Chances are very good that the only code that gets broken by such a change is whoever reported that the API is wrong. I personally would be fine with backporting such things too on a case by case basis after considering the impact.
• Breaking API changes that are just cleanup or usability improvements.

My suggestion would be that we do a breaking release every 12 months (2 GNOME releases), or maybe even every 18 months, depending on how much has accumulated in the meantime and if it's really worth making a breaking release. And different to now we would make sure that every 6 months (every GNOME release) we have support for all the new APIs backported, and we would also more aggressively backport breaking changes that are binding/API fixes.

I think this would overall reduce everybody's workload, including our own.

[sdroege]

Ok, so let's try to not make a breaking release in February and then let's reconsider for July/August next year. In the meantime we're going to backport more things than usual.

[gdesmott]

Ok, so let's try to not make a breaking release in February and then let's reconsider for July/August next year

One problem I didn't foresee if the MRSV. We currently don't change it in stable branches which may prevent us from merging important fixes. We have this problem in the gst spotify plugin which is currently unusable and we can't backport the branch to stable beacause of MSRV bump.

Having to wait one year to have a released working version is not great. So maybe we should either bump MSRV in stable or do it in a newer stable forked from it?

[decathorpe]

I don't think raising the MSRV for leaf crates like the stuff in gst-plugins-rs is a problem at all - and if you really need to update a dependency that raises MSRV there's no way around that either way.

Raising MSRV for the library crates in gtk-rs-core / gtk4-rs / gstreamer-rs would a different thing, but there I assume you can backport fixes without needing to update dependencies and / or raising MSRV yourself.

Still, for packaging purposes, raising MSRV would not be a problem for Fedora - we get current stable Rust within a few days of release, usually. Though if you bump MSRV too far we might get into trouble wrt/ stable RHEL releases, which are (depending on the timing) always ~3-7 Rust releases behind current stable.

[sdroege]

I just heard the other day concerns about updating the MSRV of gst-plugins-rs too, in the context of enterprise distros and yocto. The usual causes of preventing progress.

@ocrete can you write your concerns here in more detail?

[ocrete]

My concerns are more around embedded use-case, updating compilers on Yocto is such a pain, if we depend on latest versions, it won't be packaged, and that becomes a problem as GStreamer has more important components in Rust. I'd rather have the bindings stay 2-3 years behind the latest Rust.

For context, here are the yocto versions

• Kirkstone (LTS), May 2022: 1.59
• Mickledore, May 2023: 1.68.2
• Scarthgap (current), April 2024: 1.75
• git master: 1.79

So requiring Rust 1.80 means that the GStreamer rust plugins won't get packaged at all until someone does all the painful work of updating Yocto

[decathorpe]

FWIW, that looks identical to RHEL - current RHEL 9.4 is at Rust 1.75, CentOS Stream is at Rust 1.79, so RHEL 9.5 (released in a few weeks?) will have 1.79 too.

I still don't think requiring a newer MSRV for one non-default GStreamer plugin would be a big deal.

EDIT: Looking at the upstream discussion for the Spotify plugin, the changes needed to "fix it" look like breaking changes regardless of MSRV, so it's not even really that relevant here.

[werdahias]

Ok, so let's try to not make a breaking release in February and then let's reconsider for July/August next year One problem I didn't foresee if the MRSV. We currently don't change it in stable branches which may prevent us from merging important fixes. We have this problem in the gst spotify plugin[https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1801#note_2612557] which is currently unusable and we can't backport the branch to stable beacause of MSRV bump. Having to wait one year to have a released working version is not great. So maybe we should either bump MSRV in stable or do it in a newer stable forked from it?

On the Debian side we are pretty much current with the stable rustc (we have 1.81 right now). Ack from my side to bump the MSRV.