Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix qpid proton require under zeitwerk - resolves rake locale:update_all failure in CI #22752

Conversation

jrafanie
Copy link
Member

@jrafanie jrafanie commented Oct 23, 2023

  • Don't eager load a class needing qpid_proton loaded at load time
This class can't be eager loaded because not all platforms install qpid_proton
and we need qpid_proton's class at load time.  We can't install qpid_proton on mac
but also don't want to install qpid's dependencies in CI in various providers, core,
etc.
  • Ensure store_model_attributes loads rails environment first

Follow, this enables us to do an explicit require in the troublesome file before it tries to access a constant that is not auto-loadable as it's from a library: ManageIQ/manageiq-providers-nuage#299

@jrafanie
Copy link
Member Author

I stumbled upon this when I noticed we haven't been building the updated translations.

https://github.com/ManageIQ/manageiq/actions/runs/5861960194/job/15892980355 (last good)
https://github.com/ManageIQ/manageiq/actions/runs/6043713331/job/16401194211 (first bad)

This is because zeitwerk was enabled on August 23rd. By eager requiring qpid_proton on linux, we should be able to satisfy zeitwerk.

Gemfile Outdated Show resolved Hide resolved
@Fryguy
Copy link
Member

Fryguy commented Oct 23, 2023

FWIW, I think we are "close" to getting this building on a Mac just fine. brew install qpid-proton currently installs 0.39.0, but the latest gem is 0.37.0, which is where the mismatch occurs.

Separately, @agrare has been trying to get us to at least 0.37.0 such as in #22271 . It's strange that rubygems doesn't have a 0.39.0, but maybe we can help make that happen once we get futher on supporting the newer qpid-protons

@Fryguy
Copy link
Member

Fryguy commented Oct 23, 2023

I actually got qpid-proton 0.37.0 installed locally on mac just fine with some brew tweaking to pull in the old version. cc @agrare

brew tap homebrew/core
cd /opt/homebrew/Library/Taps/homebrew/homebrew-core/Formula/q/
git show ab90ea1d3356382d55bd6aba6e87fc29d5afdb33:Formula/qpid-proton.rb > qpid-proton.rb
HOMEBREW_NO_INSTALL_FROM_API=1 brew install qpid-proton

cd ~/dev/manageiq
gem install qpid_proton -- --with-qpid-proton-dir=$(brew --prefix qpid-proton)

@agrare
Copy link
Member

agrare commented Oct 23, 2023

That's awesome, yeah I think we're ready to upgrade to that once we get the appliance rpm side updated ManageIQ/manageiq-rpm_build#326 (CC @bdunne )

@Fryguy
Copy link
Member

Fryguy commented Oct 23, 2023

I do find it interesting that qpid-proton is "optional" from our Gemfile, but not actually optional because Nuage needs it? Does Zeitwerk not respect optional gem groups?

@Fryguy
Copy link
Member

Fryguy commented Oct 23, 2023

@agrare Is it possible to go to 0.39.0, since that's the latest (at least the latest in brew)...that would help avoid the annoying homebrew games to install an old version. I'm not sure what to do about rubygems though since that's only at 0.37.0

@jrafanie jrafanie force-pushed the fix_qpid_proton_require_under_zeitwerk branch from 19321e2 to 9f3f0e2 Compare October 23, 2023 21:30
@agrare
Copy link
Member

agrare commented Oct 23, 2023

I believe you can use version 0.39 for the library and 0.37 for the gem (especially for development).
As for going to 0.39 for the gem we are at the mercy of the qpid_proton developers and whenever they push to rubygems. We could probably ask the mailing list for a release but there is no GH repo as far as I know to open an issue on.

@jrafanie
Copy link
Member Author

jrafanie commented Oct 23, 2023

I do find it interesting that qpid-proton is "optional" from our Gemfile, but not actually optional because Nuage needs it? Does Zeitwerk not respect optional gem groups?

It's only "needed" so far for:

  • zeitwerk:check since that tries to eager load the app and engines
  • locale:update_all because it eager loads models to be able to capture the messages for i18n for each model.

It blows up on https://github.com/ManageIQ/manageiq-providers-nuage/blob/541494cded64b1ef281c71767d7d5f88cf4bb08d/app/models/manageiq/providers/nuage/network_manager/event_catcher/messaging_handler.rb#L1

Yeah, it's messy because of mac. We moved the requires down into methods to avoid hitting require 'qpid_proton' at require time. Unfortunately, the toplevel constant above subclasses from a qpid constant so at require/load time, qpid is needed.

@Fryguy
Copy link
Member

Fryguy commented Oct 23, 2023

I believe you can use version 0.39 for the library and 0.37 for the gem (especially for development).

That failed to compile the gem for me with lots of errors :(

@agrare
Copy link
Member

agrare commented Oct 23, 2023

That failed to compile the gem for me with lots of errors :(

Okay that's unfortunate, I've used mixed versions in the past but maybe something breaking about 0.39

@Fryguy
Copy link
Member

Fryguy commented Oct 23, 2023

This is what I get trying to build 0.37.0 gem against the 0.39.0_1 qpid-proton library:

$ gem install qpid_proton -- --with-qpid-proton-dir=$(brew --prefix qpid-proton)
Fetching qpid_proton-0.37.0.gem
Building native extensions with: '--with-qpid-proton-dir=/opt/homebrew/opt/qpid-proton'
This could take a while...
ERROR:  Error installing qpid_proton:
	ERROR: Failed to build gem native extension.

    current directory: /Users/jfrey/.gem/ruby/3.0.6/gems/qpid_proton-0.37.0/ext/cproton
/Users/jfrey/.rubies/ruby-3.0.6/bin/ruby -I /Users/jfrey/.rubies/ruby-3.0.6/lib/ruby/3.0.0 -r ./siteconf20231023-29924-hp6l41.rb extconf.rb --with-qpid-proton-dir\=/opt/homebrew/opt/qpid-proton
checking for -lqpid-proton... yes
checking for proton/engine.h... yes
checking for proton/message.h... yes
checking for proton/sasl.h... yes
checking for proton/messenger.h... yes
creating Makefile

current directory: /Users/jfrey/.gem/ruby/3.0.6/gems/qpid_proton-0.37.0/ext/cproton
make DESTDIR\= clean

current directory: /Users/jfrey/.gem/ruby/3.0.6/gems/qpid_proton-0.37.0/ext/cproton
make DESTDIR\=
compiling cproton.c
cproton.c:1172:1: warning: function 'Ruby_Format_OverloadedError' could be declared with attribute 'noreturn' [-Wmissing-noreturn]
{
^
cproton.c:2126:12: warning: implicit conversion loses integer precision: 'ssize_t' (aka 'long') to 'int' [-Wshorten-64-to-32]
    return sz;
    ~~~~~~ ^~
cproton.c:2137:12: warning: implicit conversion loses integer precision: 'ssize_t' (aka 'long') to 'int' [-Wshorten-64-to-32]
    return sz;
    ~~~~~~ ^~
cproton.c:2148:12: warning: implicit conversion loses integer precision: 'ssize_t' (aka 'long') to 'int' [-Wshorten-64-to-32]
    return sz;
    ~~~~~~ ^~
cproton.c:2399:31: error: call to undeclared function 'PN_CLASS'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
    static pn_class_t clazz = PN_CLASS(Pn_rbkey);
                              ^
cproton.c:2399:23: error: variable has incomplete type 'pn_class_t' (aka 'struct pn_class_t')
    static pn_class_t clazz = PN_CLASS(Pn_rbkey);
                      ^
/opt/homebrew/opt/qpid-proton/include/proton/types.h:472:16: note: forward declaration of 'struct pn_class_t'
typedef struct pn_class_t pn_class_t;
               ^
cproton.c:2467:11: warning: unused variable 'result' [-Wunused-variable]
    VALUE result = rb_funcall(pni_ruby_get_proton_module(), rb_intern("add_to_registry"), 2, key, value);
          ^
cproton.c:4175:45: error: use of undeclared identifier 'PN_WEAKREF'
  _val = SWIG_NewPointerObj(SWIG_as_voidptr(PN_WEAKREF), SWIGTYPE_p_pn_class_t,  0 );
                                            ^
cproton.c:4282:20: error: call to undeclared function 'pn_class_incref'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  result = (void *)pn_class_incref((struct pn_class_t const *)arg1,arg2);
                   ^
cproton.c:4282:12: warning: cast to 'void *' from smaller integer type 'int' [-Wint-to-void-pointer-cast]
  result = (void *)pn_class_incref((struct pn_class_t const *)arg1,arg2);
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cproton.c:4312:17: error: call to undeclared function 'pn_class_refcount'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  result = (int)pn_class_refcount((struct pn_class_t const *)arg1,arg2);
                ^
cproton.c:4312:17: note: did you mean 'pn_class_reactor'?
/opt/homebrew/opt/qpid-proton/include/proton/reactor.h:171:26: note: 'pn_class_reactor' declared here
PNX_EXTERN pn_reactor_t *pn_class_reactor(const pn_class_t *clazz, void *object);
                         ^
cproton.c:4342:17: error: call to undeclared function 'pn_class_decref'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  result = (int)pn_class_decref((struct pn_class_t const *)arg1,arg2);
                ^
cproton.c:4370:3: error: call to undeclared function 'pn_class_free'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  pn_class_free((struct pn_class_t const *)arg1,arg2);
  ^
cproton.c:4399:26: error: call to undeclared function 'pn_class_reify'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  result = (pn_class_t *)pn_class_reify((struct pn_class_t const *)arg1,arg2);
                         ^
cproton.c:4399:12: warning: cast to 'pn_class_t *' (aka 'struct pn_class_t *') from smaller integer type 'int' [-Wint-to-pointer-cast]
  result = (pn_class_t *)pn_class_reify((struct pn_class_t const *)arg1,arg2);
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cproton.c:4429:23: error: call to undeclared function 'pn_class_hashcode'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  result = (uintptr_t)pn_class_hashcode((struct pn_class_t const *)arg1,arg2);
                      ^
cproton.c:4465:12: error: call to undeclared function 'pn_class_compare'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  result = pn_class_compare((struct pn_class_t const *)arg1,arg2,arg3);
           ^
cproton.c:4465:12: note: did you mean 'pn_class_create'?
/opt/homebrew/opt/qpid-proton/include/proton/object.h:36:23: note: 'pn_class_create' declared here
PN_EXTERN pn_class_t *pn_class_create(const char *name,
                      ^
cproton.c:4501:18: error: call to undeclared function 'pn_class_equals'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  result = (bool)pn_class_equals((struct pn_class_t const *)arg1,arg2,arg3);
                 ^
cproton.c:4513:3: error: use of undeclared identifier 'pn_string_t'
  pn_string_t *arg3 = (pn_string_t *) 0 ;
  ^
cproton.c:4513:16: error: use of undeclared identifier 'arg3'
  pn_string_t *arg3 = (pn_string_t *) 0 ;
               ^
cproton.c:4513:37: error: expected expression
  pn_string_t *arg3 = (pn_string_t *) 0 ;
                                    ^
cproton.c:4513:24: error: use of undeclared identifier 'pn_string_t'
  pn_string_t *arg3 = (pn_string_t *) 0 ;
                       ^
cproton.c:4538:24: error: expected expression
  arg3 = (pn_string_t *)(argp3);
                       ^
cproton.c:4538:11: error: use of undeclared identifier 'pn_string_t'
  arg3 = (pn_string_t *)(argp3);
          ^
cproton.c:4538:3: error: use of undeclared identifier 'arg3'
  arg3 = (pn_string_t *)(argp3);
  ^
cproton.c:4539:17: error: call to undeclared function 'pn_class_inspect'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  result = (int)pn_class_inspect((struct pn_class_t const *)arg1,arg2,arg3);
                ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
7 warnings and 20 errors generated.
make: *** [cproton.o] Error 1

make failed, exit code 2

Gem files will remain installed in /Users/jfrey/.gem/ruby/3.0.6/gems/qpid_proton-0.37.0 for inspection.
Results logged to /Users/jfrey/.gem/ruby/3.0.6/extensions/arm64-darwin-22/3.0.0/qpid_proton-0.37.0/gem_make.out

@jrafanie
Copy link
Member Author

I'll manually generate the message catalogs while we try to get this merged so the bot can generate them from the update_all github workflow.

@jrafanie
Copy link
Member Author

ok, manually opened a PR to do the updated english messages: #22754

@jrafanie
Copy link
Member Author

Do we have any problems with merging this for now until we can build qpid_proton on mac? I'd like to see if this allows me to run the github workflow for translations so we can get the bot posting updates to translations: https://github.com/ManageIQ/manageiq/actions/workflows/locale_update_all.yaml

@@ -0,0 +1,9 @@
# Fake qpid_proton stub so zeitwerk can autoload on platforms that can't build the gem, such as macos.
if RUBY_PLATFORM.include?("darwin")
module Qpid
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Surprised this doesn't thrown an error like "expected this file to define a class".
Can't remember why that gets thrown.

Copy link
Member Author

@jrafanie jrafanie Oct 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kbrock you might be right if it's non-darwin.

@agrare can you verify the PR doesn't blow up for you when you let zeitwerk try to eager load things?

DEBUG_MANAGEIQ_ZEITWERK=true bin/rails zeitwerk:check --trace

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't blow up just seeing

D, [2023-10-24T16:55:14.848893 #69545] DEBUG -- : [email protected]: expected file /home/grare/adam/src/manageiq/manageiq/lib/qpid/proton/messaging_handler.rb to define constant Qpid::Proton::MessagingHandler, but didn't
expected file lib/qpid/proton/messaging_handler.rb to define constant Qpid::Proton::MessagingHandler

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, I gotta fix that then. 😢

@kbrock
Copy link
Member

kbrock commented Oct 24, 2023

Yea, https://github.com/apache/qpid-proton is only for pull requests and not for issues.

Copy link
Member

@kbrock kbrock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me. :shipit:

Feel like getting it to something that works for now keeps us moving forward and then we can upgrade qpid-proton when the gem is updated.

@jrafanie jrafanie force-pushed the fix_qpid_proton_require_under_zeitwerk branch from 9f3f0e2 to 3b3f192 Compare October 25, 2023 14:09
# These stubs won't define constants elsewhere, so we need to tell zeitwerk to ignore them.
unless RUBY_PLATFORM.include?("darwin")
Rails.autoloaders.main.ignore(Rails.root.join("lib/qpid/proton/container.rb"))
Rails.autoloaders.main.ignore(Rails.root.join("lib/qpid/proton/messaging_handler.rb"))
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like it but if we're conditionally defining classes in lib/qpid/proton/* below, we need to tell the autoloaders to ignore them on platforms that will not define them as zeiwerk will complain about them.

This should fix DEBUG_MANAGEIQ_ZEITWERK=true bin/rails zeitwerk:check --trace on linux @agrare

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NameError: uninitialized constant Qpid
/home/grare/adam/.gem/ruby/3.0.0/bundler/gems/manageiq-providers-nuage-541494cded64/app/models/manageiq/providers/nuage/network_manager/event_catcher/messaging_handler.rb:1:in `<main>'

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@agrare Thanks for testing. BUT WAT. Is it possible you're excluding qpid_proton group or you're not on this branch with require: false removed?

If not, wow, weird. I will have to try to recreate it on an appliance.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I don't know, I have BUNDLE_WITH=qpid_proton and it is in my Gemfile.lock

$ grep BUNDLE_WITH .bundle/config 
BUNDLE_WITH: "qpid_proton,systemd"
$ grep qpid_proton Gemfile.lock 
    qpid_proton (0.30.0)
  qpid_proton (~> 0.30.0)
$ DEBUG_MANAGEIQ_ZEITWERK=true bin/rails zeitwerk:check --trace
NameError: uninitialized constant Qpid
/home/grare/adam/.gem/ruby/3.0.0/bundler/gems/manageiq-providers-nuage-541494cded64/app/models/manageiq/providers/nuage/network_manager/event_catcher/messaging_handler.rb:1:in `<main>'
<internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:38:in `require'
/home/grare/adam/.gem/ruby/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require

and I merged this PR locally

commit 5803d9e827854b4662cec6569185b04d610faff7 (HEAD -> master)
Merge: 883525adba 3b3f192068
Author: Adam Grare <[email protected]>
Date:   Wed Oct 25 11:10:27 2023 -0400

    Merge pull request #22752 from jrafanie/fix_qpid_proton_require_under_zeitwerk
    
    Fix qpid proton require under zeitwerk

commit 3b3f192068fd41c3f235912f4a4bfd3bfdcdaab0
Author: Joe Rafaniello <[email protected]>
Date:   Mon Oct 23 16:11:15 2023 -0400

Copy link
Member

@kbrock kbrock Oct 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Making this up:

The error is due to zeitwerk wanting to define the constant qpid, finding the file qpid.rb and requiring it.
I bet they expect it to work just fine

Speculation:

If @agrare were to delete those 2 files, it would work just fine

Maybe we could zeitwerk ignore those files on all but mac? (ugh, you do this)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry ^ is useless and obvious. ignore

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok. so these files (aka defining these constants) are only required for development on a mac.
Wonder if we want to add an initializer that defines these 2 constants on the mac?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@agrare can you try this:

BUNDLER_GROUPS="manageiq_default,ui_dependencies,qpid_proton" DEBUG_MANAGEIQ_ZEITWERK=true bin/rails zeitwerk:check --trace

We weren't auto-requiring qpid_proton during rails boot because qpid_proton wasn't in the default bundler groups.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With rails c in core:

>> require 'qpid_proton'
=> true
$ BUNDLER_GROUPS="manageiq_default,ui_dependencies,qpid_proton" DEBUG_MANAGEIQ_ZEITWERK=true bin/rails zeitwerk:check --trace
...
NameError: uninitialized constant Qpid
/home/grare/adam/.gem/ruby/3.0.0/bundler/gems/manageiq-providers-nuage-541494cded64/app/models/manageiq/providers/nuage/network_manager/event_catcher/messaging_handler.rb:1:in `<main>'
<internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:38:in `require'
<internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:38:in `require'
/home/grare/adam/.gem/ruby/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'

@jrafanie jrafanie changed the title Fix qpid proton require under zeitwerk [WIP] Fix qpid proton require under zeitwerk Oct 26, 2023
@jrafanie jrafanie changed the title [WIP] Fix qpid proton require under zeitwerk Fix qpid proton require under zeitwerk Oct 26, 2023
- name: Set up Ruby
uses: ruby/setup-ruby@v1
with:
ruby-version: "3.0"
bundler-cache: true
timeout-minutes: 30
- name: Set bundle config with qpid_proton
run: bundle config with qpid_proton
Copy link
Member Author

@jrafanie jrafanie Oct 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was able to fix the locale_update_all workflow by installing qpid_proton, adding it to the BUNDLER_GROUPS so it's auto-required, and setting the bundle --with qpid_proton.

https://github.com/jrafanie/manageiq/actions/runs/6659788595/job/18099615071

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

setup-ruby installs all the gems, so I think this is too late. I think BUNDLE_WITH env var can be set

Comment on lines 34 to 37
- name: Install qpid dependency - libcurl4-openssl-dev
run: sudo apt-get update && sudo apt-get install -y libcurl4-openssl-dev
- name: Install qpid dependency - libqpid-proton11-dev
run: sudo apt-add-repository --yes 'deb http://us.archive.ubuntu.com/ubuntu lunar universe' && sudo apt-get install -t lunar -y libqpid-proton11-dev
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should already be in the bin/before_install - if not, that s where these belong - There were some recent changes for Ubuntu Lunar in other plugins, so maybe that same change needs to happen here? cc @agrare

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We didn't need to install qpid_proton in core before, only in nuage/openstack. @jrafanie are we going to need to do this for every plugin?

And this is the correct repo, https://github.com/ManageIQ/manageiq-providers-nuage/pull/297/files#diff-14d541c92b03a497ca2efd069b0a5a0fbc47487e0d8a2d366933d6ee44103a4cR7-R8

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We only need this because locale:update_all ends up calling eager_load! from the call to gettext:store_model_attributes. Providers don't need qpid_proton. Neither does core.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just had a thought, are we going to have to do the same with the systemd gems that are based on ruby-dbus? Those are in an optional group because of Mac as well.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can continue to ignore the file that can't be required without qpid_proton gem installed and loaded.

    message_handler_path = Pathname.new(Vmdb::Plugins.paths[ManageIQ::Providers::Nuage::Engine]).join("app/models/manageiq/providers/nuage/network_manager/event_catcher/messaging_handler.rb")
    Rails.autoloaders.main.ignore(message_handler_path)

I was trying to remove it but I guess we can keep it to avoid this shotgun surgery.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For now, I'll drop the qpid dependency and unconditionally tell zeitwerk to ignore eager loading the file that depends on the library at load time.

@Fryguy
Copy link
Member

Fryguy commented Oct 27, 2023

I'm starting to think we should do this another way. I think the fundamental problem is that the nuage code is requiring qpid-proton too early, because it is subclassing a Qpid::Proton class directly and that subclassing is suspicious to begin with. If you look closely, the initialize does require 'qpid-proton', which doesn't make sense if we're subclassing qpid-proton.

Ideally no provider plugin should require its client libraries until we actually need them (I do understand this case is slightly different because it's about i18n which effectively loads everything).

If we changed the nuage plugin to not subclass the Qpid::Proton library but instead used composition, would that "avoid" the problem at i18n time?

@jrafanie
Copy link
Member Author

Ideally no provider plugin should require its client libraries until we actually need them (I do understand this case is slightly different because it's about i18n which effectively loads everything).

If we changed the nuage plugin to not subclass the Qpid::Proton library but instead used composition, would that "avoid" the problem at i18n time?

👍 i18n time really means load time, so yes, if the instances were composed of proton classes, that would move the dependence on the library until instantiation time.

@jrafanie jrafanie force-pushed the fix_qpid_proton_require_under_zeitwerk branch from fb07da1 to 4b3e8e6 Compare October 27, 2023 16:08
@jrafanie
Copy link
Member Author

Ok, dropped qpid installation and kept the part where we tell zeitwerk to explicitly ignore eager loading the file that cannot be loaded without requiring the qpid_proton gem.

The github action for update_all was tested and worked: https://github.com/jrafanie/manageiq/actions/runs/6669465925/job/18127299126

@jrafanie jrafanie force-pushed the fix_qpid_proton_require_under_zeitwerk branch 2 times, most recently from d852b20 to dd2a199 Compare October 27, 2023 16:11
@jrafanie jrafanie changed the title Fix qpid proton require under zeitwerk Fix qpid proton require under zeitwerk - resolves rake locale:update_all failure in CI Oct 27, 2023
@jrafanie
Copy link
Member Author

restarting after merge of #22758

@jrafanie jrafanie closed this Oct 27, 2023
@jrafanie jrafanie reopened this Oct 27, 2023
This class can't be eager loaded because not all platforms install qpid_proton
and we need qpid_proton's class at load time.  We can't install qpid_proton on mac
but also don't want to install qpid's dependencies in CI in various providers, core,
etc.
@jrafanie jrafanie force-pushed the fix_qpid_proton_require_under_zeitwerk branch from dd2a199 to 5dc641f Compare October 27, 2023 20:09
@Fryguy
Copy link
Member

Fryguy commented Oct 27, 2023

I'm good with this approach for now. Slight alternative, is it possible to define that "exception" in the nuage plugin itself? That way these exceptions can be made pluggable (and thus a future fix could remove the exception in the same PR).

@jrafanie
Copy link
Member Author

Slight alternative, is it possible to define that "exception" in the nuage plugin itself? That way these exceptions can be made pluggable (and thus a future fix could remove the exception in the same PR).

I think I tried this in the past

initializer 'manageiq-decorators.configure_autoloader' do
  Rails.autoloaders.main.ignore("#{root}/lib/manageiq-decorators.rb")
end

I assume you'd define that initializer in the engine and update it to point to the correct path. I think I chose to fix what I could to avoid that or define them here because engines/gems don't necessarily know if they're using zeitwerk. Engines could do above. Gems, no. They'd need to check if the Rails.autoloaders exists or that it's being used with zeitwerk.

@Fryguy
Copy link
Member

Fryguy commented Oct 27, 2023

Yeah in this case it's explicitly an engine (and it's an miq engine at that)

@jrafanie
Copy link
Member Author

jrafanie commented Oct 30, 2023

I've moved the nuage loader ignore to ManageIQ/manageiq-providers-nuage#299.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants