-
Notifications
You must be signed in to change notification settings - Fork 897
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix qpid proton require under zeitwerk - resolves rake locale:update_all failure in CI #22752
Fix qpid proton require under zeitwerk - resolves rake locale:update_all failure in CI #22752
Conversation
I stumbled upon this when I noticed we haven't been building the updated translations. https://github.com/ManageIQ/manageiq/actions/runs/5861960194/job/15892980355 (last good) This is because zeitwerk was enabled on August 23rd. By eager requiring qpid_proton on linux, we should be able to satisfy zeitwerk. |
FWIW, I think we are "close" to getting this building on a Mac just fine. Separately, @agrare has been trying to get us to at least 0.37.0 such as in #22271 . It's strange that rubygems doesn't have a 0.39.0, but maybe we can help make that happen once we get futher on supporting the newer qpid-protons |
I actually got qpid-proton 0.37.0 installed locally on mac just fine with some brew tweaking to pull in the old version. cc @agrare brew tap homebrew/core
cd /opt/homebrew/Library/Taps/homebrew/homebrew-core/Formula/q/
git show ab90ea1d3356382d55bd6aba6e87fc29d5afdb33:Formula/qpid-proton.rb > qpid-proton.rb
HOMEBREW_NO_INSTALL_FROM_API=1 brew install qpid-proton
cd ~/dev/manageiq
gem install qpid_proton -- --with-qpid-proton-dir=$(brew --prefix qpid-proton) |
That's awesome, yeah I think we're ready to upgrade to that once we get the appliance rpm side updated ManageIQ/manageiq-rpm_build#326 (CC @bdunne ) |
I do find it interesting that qpid-proton is "optional" from our Gemfile, but not actually optional because Nuage needs it? Does Zeitwerk not respect optional gem groups? |
@agrare Is it possible to go to 0.39.0, since that's the latest (at least the latest in brew)...that would help avoid the annoying homebrew games to install an old version. I'm not sure what to do about rubygems though since that's only at 0.37.0 |
19321e2
to
9f3f0e2
Compare
I believe you can use version 0.39 for the library and 0.37 for the gem (especially for development). |
It's only "needed" so far for:
Yeah, it's messy because of mac. We moved the requires down into methods to avoid hitting |
That failed to compile the gem for me with lots of errors :( |
Okay that's unfortunate, I've used mixed versions in the past but maybe something breaking about 0.39 |
This is what I get trying to build 0.37.0 gem against the 0.39.0_1 qpid-proton library:
|
I'll manually generate the message catalogs while we try to get this merged so the bot can generate them from the update_all github workflow. |
ok, manually opened a PR to do the updated english messages: #22754 |
Do we have any problems with merging this for now until we can build qpid_proton on mac? I'd like to see if this allows me to run the github workflow for translations so we can get the bot posting updates to translations: https://github.com/ManageIQ/manageiq/actions/workflows/locale_update_all.yaml |
lib/qpid/proton/container.rb
Outdated
@@ -0,0 +1,9 @@ | |||
# Fake qpid_proton stub so zeitwerk can autoload on platforms that can't build the gem, such as macos. | |||
if RUBY_PLATFORM.include?("darwin") | |||
module Qpid |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Surprised this doesn't thrown an error like "expected this file to define a class".
Can't remember why that gets thrown.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doesn't blow up just seeing
D, [2023-10-24T16:55:14.848893 #69545] DEBUG -- : [email protected]: expected file /home/grare/adam/src/manageiq/manageiq/lib/qpid/proton/messaging_handler.rb to define constant Qpid::Proton::MessagingHandler, but didn't
expected file lib/qpid/proton/messaging_handler.rb to define constant Qpid::Proton::MessagingHandler
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, I gotta fix that then. 😢
Yea, https://github.com/apache/qpid-proton is only for pull requests and not for issues. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good to me.
Feel like getting it to something that works for now keeps us moving forward and then we can upgrade qpid-proton when the gem is updated.
9f3f0e2
to
3b3f192
Compare
config/initializers/zeitwerk.rb
Outdated
# These stubs won't define constants elsewhere, so we need to tell zeitwerk to ignore them. | ||
unless RUBY_PLATFORM.include?("darwin") | ||
Rails.autoloaders.main.ignore(Rails.root.join("lib/qpid/proton/container.rb")) | ||
Rails.autoloaders.main.ignore(Rails.root.join("lib/qpid/proton/messaging_handler.rb")) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't like it but if we're conditionally defining classes in lib/qpid/proton/* below, we need to tell the autoloaders to ignore them on platforms that will not define them as zeiwerk will complain about them.
This should fix DEBUG_MANAGEIQ_ZEITWERK=true bin/rails zeitwerk:check --trace
on linux @agrare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
NameError: uninitialized constant Qpid
/home/grare/adam/.gem/ruby/3.0.0/bundler/gems/manageiq-providers-nuage-541494cded64/app/models/manageiq/providers/nuage/network_manager/event_catcher/messaging_handler.rb:1:in `<main>'
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@agrare Thanks for testing. BUT WAT. Is it possible you're excluding qpid_proton
group or you're not on this branch with require: false removed?
If not, wow, weird. I will have to try to recreate it on an appliance.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I don't know, I have BUNDLE_WITH=qpid_proton
and it is in my Gemfile.lock
$ grep BUNDLE_WITH .bundle/config
BUNDLE_WITH: "qpid_proton,systemd"
$ grep qpid_proton Gemfile.lock
qpid_proton (0.30.0)
qpid_proton (~> 0.30.0)
$ DEBUG_MANAGEIQ_ZEITWERK=true bin/rails zeitwerk:check --trace
NameError: uninitialized constant Qpid
/home/grare/adam/.gem/ruby/3.0.0/bundler/gems/manageiq-providers-nuage-541494cded64/app/models/manageiq/providers/nuage/network_manager/event_catcher/messaging_handler.rb:1:in `<main>'
<internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:38:in `require'
/home/grare/adam/.gem/ruby/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require
and I merged this PR locally
commit 5803d9e827854b4662cec6569185b04d610faff7 (HEAD -> master)
Merge: 883525adba 3b3f192068
Author: Adam Grare <[email protected]>
Date: Wed Oct 25 11:10:27 2023 -0400
Merge pull request #22752 from jrafanie/fix_qpid_proton_require_under_zeitwerk
Fix qpid proton require under zeitwerk
commit 3b3f192068fd41c3f235912f4a4bfd3bfdcdaab0
Author: Joe Rafaniello <[email protected]>
Date: Mon Oct 23 16:11:15 2023 -0400
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Making this up:
The error is due to zeitwerk wanting to define the constant qpid, finding the file qpid.rb
and requiring it.
I bet they expect it to work just fine
Speculation:
If @agrare were to delete those 2 files, it would work just fine
Maybe we could zeitwerk ignore those files on all but mac? (ugh, you do this)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sorry ^ is useless and obvious. ignore
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok. so these files (aka defining these constants) are only required for development on a mac.
Wonder if we want to add an initializer that defines these 2 constants on the mac?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@agrare can you try this:
BUNDLER_GROUPS="manageiq_default,ui_dependencies,qpid_proton" DEBUG_MANAGEIQ_ZEITWERK=true bin/rails zeitwerk:check --trace
We weren't auto-requiring qpid_proton during rails boot because qpid_proton wasn't in the default bundler groups.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With rails c
in core:
>> require 'qpid_proton'
=> true
$ BUNDLER_GROUPS="manageiq_default,ui_dependencies,qpid_proton" DEBUG_MANAGEIQ_ZEITWERK=true bin/rails zeitwerk:check --trace
...
NameError: uninitialized constant Qpid
/home/grare/adam/.gem/ruby/3.0.0/bundler/gems/manageiq-providers-nuage-541494cded64/app/models/manageiq/providers/nuage/network_manager/event_catcher/messaging_handler.rb:1:in `<main>'
<internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:38:in `require'
/home/grare/adam/.gem/ruby/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
- name: Set up Ruby | ||
uses: ruby/setup-ruby@v1 | ||
with: | ||
ruby-version: "3.0" | ||
bundler-cache: true | ||
timeout-minutes: 30 | ||
- name: Set bundle config with qpid_proton | ||
run: bundle config with qpid_proton |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was able to fix the locale_update_all workflow by installing qpid_proton, adding it to the BUNDLER_GROUPS so it's auto-required, and setting the bundle
--with qpid_proton
.
https://github.com/jrafanie/manageiq/actions/runs/6659788595/job/18099615071
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
setup-ruby installs all the gems, so I think this is too late. I think BUNDLE_WITH env var can be set
- name: Install qpid dependency - libcurl4-openssl-dev | ||
run: sudo apt-get update && sudo apt-get install -y libcurl4-openssl-dev | ||
- name: Install qpid dependency - libqpid-proton11-dev | ||
run: sudo apt-add-repository --yes 'deb http://us.archive.ubuntu.com/ubuntu lunar universe' && sudo apt-get install -t lunar -y libqpid-proton11-dev |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should already be in the bin/before_install
- if not, that s where these belong - There were some recent changes for Ubuntu Lunar in other plugins, so maybe that same change needs to happen here? cc @agrare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We didn't need to install qpid_proton in core before, only in nuage/openstack. @jrafanie are we going to need to do this for every plugin?
And this is the correct repo, https://github.com/ManageIQ/manageiq-providers-nuage/pull/297/files#diff-14d541c92b03a497ca2efd069b0a5a0fbc47487e0d8a2d366933d6ee44103a4cR7-R8
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We only need this because locale:update_all
ends up calling eager_load! from the call to gettext:store_model_attributes. Providers don't need qpid_proton. Neither does core.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just had a thought, are we going to have to do the same with the systemd gems that are based on ruby-dbus? Those are in an optional group because of Mac as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can continue to ignore the file that can't be required without qpid_proton gem installed and loaded.
message_handler_path = Pathname.new(Vmdb::Plugins.paths[ManageIQ::Providers::Nuage::Engine]).join("app/models/manageiq/providers/nuage/network_manager/event_catcher/messaging_handler.rb")
Rails.autoloaders.main.ignore(message_handler_path)
I was trying to remove it but I guess we can keep it to avoid this shotgun surgery.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For now, I'll drop the qpid dependency and unconditionally tell zeitwerk to ignore eager loading the file that depends on the library at load time.
I'm starting to think we should do this another way. I think the fundamental problem is that the nuage code is requiring qpid-proton too early, because it is subclassing a Qpid::Proton class directly and that subclassing is suspicious to begin with. If you look closely, the initialize does Ideally no provider plugin should require its client libraries until we actually need them (I do understand this case is slightly different because it's about i18n which effectively loads everything). If we changed the nuage plugin to not subclass the Qpid::Proton library but instead used composition, would that "avoid" the problem at i18n time? |
👍 i18n time really means load time, so yes, if the instances were composed of proton classes, that would move the dependence on the library until instantiation time. |
fb07da1
to
4b3e8e6
Compare
Ok, dropped qpid installation and kept the part where we tell zeitwerk to explicitly ignore eager loading the file that cannot be loaded without requiring the qpid_proton gem. The github action for update_all was tested and worked: https://github.com/jrafanie/manageiq/actions/runs/6669465925/job/18127299126 |
d852b20
to
dd2a199
Compare
restarting after merge of #22758 |
This class can't be eager loaded because not all platforms install qpid_proton and we need qpid_proton's class at load time. We can't install qpid_proton on mac but also don't want to install qpid's dependencies in CI in various providers, core, etc.
dd2a199
to
5dc641f
Compare
I'm good with this approach for now. Slight alternative, is it possible to define that "exception" in the nuage plugin itself? That way these exceptions can be made pluggable (and thus a future fix could remove the exception in the same PR). |
I think I tried this in the past
I assume you'd define that initializer in the engine and update it to point to the correct path. I think I chose to fix what I could to avoid that or define them here because engines/gems don't necessarily know if they're using zeitwerk. Engines could do above. Gems, no. They'd need to check if the |
Yeah in this case it's explicitly an engine (and it's an miq engine at that) |
I've moved the nuage loader ignore to ManageIQ/manageiq-providers-nuage#299. |
Follow, this enables us to do an explicit require in the troublesome file before it tries to access a constant that is not auto-loadable as it's from a library: ManageIQ/manageiq-providers-nuage#299