Puppetserver returns "Function load error eyaml_lookup_key" on puppet agent 8.7.0+

[mdechiaro]

Describe the Bug

Puppetserver fails with Function Load Error for function 'eyaml_lookup_key' and produces error messages in the logs if puppet agent is upgraded to v8.7.0+. The latest version of puppet agent, v8.8.1, also produces this error.

2024-09-05T09:53:20.732-05:00 ERROR [qtp944291823-53] [puppetserver] Puppet Evaluation Error: Error while evaluating a Function Call, Function Load Error for function 'eyaml_lookup_key': Look
up using eyaml lookup_key function is only supported when the hiera_eyaml library is present (file: /etc/puppetlabs/code/environments/development/modules/manifests/00site.pp, line: 18, column
: 16) on node hostname.domain.com

Downgrading puppet agent to v8.6.0 on system running puppetserver fixes the issue.

Expected Behavior

Hiera-eyaml shouldn't break when puppet agent is upgraded to 8.7.0+ on a system running puppetserver.

Steps to Reproduce

Steps to reproduce the behavior:

1. Install puppetserver v8.6.2 and setup hiera-eyaml.
2. Install puppet agent on v8.6.0 on system running puppetserver and ensure the config works.
3. Upgrade to puppet agent v8.7.0 on system running puppetserver.
4. restart puppetserver.
5. puppetserver fails with Function Load Error for function 'eyaml_lookup_key' errors.

Environment

• Version (OSS) 8.7.0
• Platform Ubuntu 22.04
• puppetserver (OSS) v8.6.2

Additional Context

Full error in puppetserver.log is below

2024-09-05T09:53:20.732-05:00 ERROR [qtp944291823-53] [puppetserver] Puppet Evaluation Error: Error while evaluating a Function Call, Function Load Error for function 'eyaml_lookup_key': Look
up using eyaml lookup_key function is only supported when the hiera_eyaml library is present (file: /etc/puppetlabs/code/environments/development/modules/manifests/00site.pp, line: 18, column
: 16) on node hostname.domain.com                                                              
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/functions.rb:195:in `create_function'                                                                                                       
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/functions/eyaml_lookup_key.rb:9:in `create'                                                                                                 
org/jruby/RubyKernel.java:1274:in `eval'                                                                                                                                                       
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/ruby_function_instantiator.rb:25:in `create'        
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/module_loaders.rb:281:in `instantiate'      
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/module_loaders.rb:255:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/base_loader.rb:174:in `internal_load'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/base_loader.rb:44:in `block in load_typed'                 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/concurrent/lock.rb:11:in `synchronize'                                                                                                      
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/loader.rb:154:in `synchronize'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/base_loader.rb:42:in `load_typed'                 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/base_loader.rb:166:in `internal_load'        
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/base_loader.rb:44:in `block in load_typed'                 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/concurrent/lock.rb:11:in `synchronize' 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/loader.rb:154:in `synchronize'                                                                                                  
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/base_loader.rb:42:in `load_typed'                                                                                               
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/base_loader.rb:166:in `internal_load'             
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/base_loader.rb:44:in `block in load_typed'      
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/concurrent/lock.rb:11:in `synchronize'                                                                                                      
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/loader.rb:154:in `synchronize'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/base_loader.rb:42:in `load_typed'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/base_loader.rb:166:in `internal_load'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/base_loader.rb:44:in `block in load_typed'                                                                                      
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/concurrent/lock.rb:11:in `synchronize'                                                                                                      
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/loader.rb:154:in `synchronize' 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/base_loader.rb:42:in `load_typed'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/function_provider.rb:93:in `load_function'                                                                                      
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/function_provider.rb:81:in `function'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/function_provider.rb:39:in `create_function_context'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/function_provider.rb:35:in `function_context'                                                                                   
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/lookup_key_function_provider.rb:50:in `lookup_key'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/lookup_key_function_provider.rb:33:in `block in invoke_with_location'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/invocation.rb:140:in `with'                                                                                                     
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/lookup_key_function_provider.rb:32:in `invoke_with_location'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/lookup_key_function_provider.rb:22:in `block in unchecked_key_lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/merge_strategy.rb:137:in `block in lookup'
org/jruby/RubyKernel.java:1423:in `catch'                                                      
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/merge_strategy.rb:136:in `block in lookup'                                                                                             
org/jruby/RubyArray.java:1981:in `each'                                                                                                                                                        
org/jruby/RubyEnumerable.java:1084:in `inject'                                                                                                                                                 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/merge_strategy.rb:134:in `block in lookup'                                                                                             
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/invocation.rb:140:in `with'                                                                                                     
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/merge_strategy.rb:133:in `lookup'                                                                                                      
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/lookup_key_function_provider.rb:21:in `block in unchecked_key_lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/invocation.rb:140:in `with'                                                                                              [46/98]
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/lookup_key_function_provider.rb:20:in `unchecked_key_lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/configured_data_provider.rb:58:in `block in unchecked_key_lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/merge_strategy.rb:131:in `lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/configured_data_provider.rb:57:in `block in unchecked_key_lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/invocation.rb:140:in `with'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/configured_data_provider.rb:50:in `unchecked_key_lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/global_data_provider.rb:52:in `unchecked_key_lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/data_provider.rb:28:in `block in key_lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/invocation.rb:99:in `check'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/data_provider.rb:28:in `key_lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/lookup_adapter.rb:133:in `lookup_global'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/lookup_adapter.rb:378:in `block in global_lookup_options'
org/jruby/RubyKernel.java:1423:in `catch'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/lookup_adapter.rb:378:in `global_lookup_options'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/lookup_adapter.rb:387:in `env_lookup_options'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/lookup_adapter.rb:353:in `block in retrieve_lookup_options'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/invocation.rb:140:in `with'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/lookup_adapter.rb:349:in `block in retrieve_lookup_options'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/invocation.rb:84:in `lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/lookup_adapter.rb:348:in `retrieve_lookup_options'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/lookup_adapter.rb:241:in `lookup_lookup_options'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/lookup_adapter.rb:65:in `block in lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/invocation.rb:84:in `lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup/lookup_adapter.rb:60:in `lookup' 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup.rb:82:in `search_and_merge'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup.rb:34:in `block in lookup'
org/jruby/RubyKernel.java:1423:in `catch'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup.rb:34:in `block in lookup'
org/jruby/RubyArray.java:1981:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/lookup.rb:32:in `lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/functions/lookup.rb:210:in `do_lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/functions/lookup.rb:190:in `lookup_1'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatch.rb:62:in `invoke'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:48:in `block in dispatch'
org/jruby/RubyKernel.java:1423:in `catch'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:47:in `dispatch'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/function.rb:44:in `block in call'
org/jruby/RubyKernel.java:1423:in `catch'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/function.rb:43:in `call'
/etc/puppetlabs/code/environments/development/modules/manifests/00site.pp:0
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/puppet_stack.rb:44:in `stack'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:309:in `block in call_function'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:59:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:53:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:307:in `call_function'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:994:in `call_function_with_block'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:961:in `eval_CallNamedFunctionExpression'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:55:in `block in visit_this'
org/jruby/RubyArray.java:1981:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:47:in `visit_this'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:109:in `visit_this_1'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:77:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:363:in `eval_AssignmentExpression'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:55:in `block in visit_this'
org/jruby/RubyArray.java:1981:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:47:in `visit_this'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:109:in `visit_this_1'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:77:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:674:in `block in eval_BlockExpression'
org/jruby/RubyArray.java:1981:in `each'
org/jruby/RubyEnumerable.java:1084:in `inject'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:674:in `eval_BlockExpression'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:55:in `block in visit_this'
org/jruby/RubyArray.java:1981:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:47:in `visit_this'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:109:in `visit_this_1'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:77:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/puppet_stack.rb:44:in `stack'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:750:in `eval_Program'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:55:in `block in visit_this'
org/jruby/RubyArray.java:1981:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:47:in `visit_this'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:109:in `visit_this_1'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:77:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/parser/evaluating_parser.rb:62:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:110:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast.rb:32:in `safeevaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:7:in `block in evaluate'
org/jruby/RubyArray.java:1981:in `each'
org/jruby/RubyEnumerable.java:1084:in `inject'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:7:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast.rb:32:in `safeevaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/type.rb:81:in `evaluate_code'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:83:in `block in evaluate' 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:59:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:53:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:75:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:399:in `evaluate_main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:134:in `block in compile' 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:59:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:53:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:134:in `block in compile' 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'

[joshcooper]

That error is coming from this line

puppet/lib/puppet/functions/eyaml_lookup_key.rb

Line 10 in 32499bc

unless Puppet.features.hiera_eyaml?

For some reason Puppet.features.hiera_eyaml? is returning false. I can't reproduce that using puppetserver 8.6.0 and puppet-agent 8.8.1:

# /opt/puppetlabs/puppet/bin/facter os
{
  architecture => "amd64",
  distro => {
    codename => "jammy",
    description => "Ubuntu 22.04.3 LTS",
    id => "Ubuntu",
    release => {
      full => "22.04",
      major => "22.04"
    }
  },
  family => "Debian",
  hardware => "x86_64",
  name => "Ubuntu",
  release => {
    full => "22.04",
    major => "22.04"
  },
  selinux => {
    enabled => false
  }
}
# dpkg -l puppet-agent puppetserver
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-==========================================================================================>
ii  puppet-agent   8.8.1-1jammy amd64        The Puppet Agent package contains all of the elements needed to run puppet, including ruby>
ii  puppetserver   8.6.2-1jammy all          Puppet Labs puppetserver

# /opt/puppetlabs/server/bin/puppetserver ruby -rpuppet -e 'puts Puppet.features.hiera_eyaml?'
true
# /opt/puppetlabs/server/bin/puppetserver gem which hiera/backend/eyaml/parser/parser
/opt/puppetlabs/puppet/lib/ruby/vendor_gems/gems/hiera-eyaml-3.4.0/lib/hiera/backend/eyaml/parser/parser.rb

Additionally the version of the hiera-eyaml gem 3.4.0 is unchanged between puppet-agent 8.6.0 and 8.8.1, so seems unlikely this is caused by our agent.

Are you sure you did a full restart of puppetserver (as opposed to reload)?

Are you sure you installed puppet-agent from our repos? As opposed to Ubuntu/Debian repos?

[mdechiaro]

@joshcooper Thank for you the fast response! Also, thanks for those extra troubleshooting commands. I think I found the problem, and it seems there's an issue with external hiera-eyaml gem, v4.1.0, with puppet versions 8.7.0+.

v8.6.0 (works with hiera-eyaml v4.1.0)

# dpkg -l puppet-agent puppetserver
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-======================================================================================================
ii  puppet-agent   8.6.0-1jammy amd64        The Puppet Agent package contains all of the elements needed to run puppet, including ruby and facter.
ii  puppetserver   8.6.2-1jammy all          Puppet Labs puppetserver
# sudo /opt/puppetlabs/server/bin/puppetserver ruby -rpuppet -e 'puts Puppet.features.hiera_eyaml?'
true
# sudo /opt/puppetlabs/server/bin/puppetserver gem which hiera/backend/eyaml/parser/parser
/opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-4.1.0/lib/hiera/backend/eyaml/parser/parser.rb

Upgrade to v8.7.0 (fails with hiera-eyaml v4.1.0)

# sudo apt install puppet-agent=8.7.0-1jammy
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be upgraded:
  puppet-agent
1 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
Need to get 35.9 MB of archives.
After this operation, 492 kB of additional disk space will be used.
Get:1 https://apt.puppetlabs.com jammy/puppet8 amd64 puppet-agent amd64 8.7.0-1jammy [35.9 MB]
Fetched 35.9 MB in 1s (71.7 MB/s)
(Reading database ... 152735 files and directories currently installed.)
Preparing to unpack .../puppet-agent_8.7.0-1jammy_amd64.deb ...
Unpacking puppet-agent (8.7.0-1jammy) over (8.6.0-1jammy) ...
Setting up puppet-agent (8.7.0-1jammy) ...
Processing triggers for libc-bin (2.35-0ubuntu3.8) ...
Scanning processes...
Scanning linux images...

Running kernel seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.
# sudo /opt/puppetlabs/server/bin/puppetserver ruby -rpuppet -e 'puts Puppet.features.hiera_eyaml?'
false
# sudo /opt/puppetlabs/server/bin/puppetserver gem which hiera/backend/eyaml/parser/parser
/opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-4.1.0/lib/hiera/backend/eyaml/parser/parser.rb

Uninstall external hiera-eyaml v4.1.0.

This fixes the issue.

# sudo /opt/puppetlabs/server/bin/puppetserver gem uninstall hiera-eyaml

Select gem to uninstall:
 1. hiera-eyaml-3.4.0
 2. hiera-eyaml-4.1.0
 3. All versions
> 2
Successfully uninstalled hiera-eyaml-4.1.0
# sudo /opt/puppetlabs/server/bin/puppetserver gem which hiera/backend/eyaml/parser/parser
/opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-3.4.0/lib/hiera/backend/eyaml/parser/parser.rb
# sudo /opt/puppetlabs/server/bin/puppetserver ruby -rpuppet -e 'puts Puppet.features.hiera_eyaml?'
true

Uninstall external hiera-eyaml v3.4.0

It still works.

:~$ sudo /opt/puppetlabs/server/bin/puppetserver gem uninstall hiera-eyaml
Remove executables:
        eyaml

in addition to the gem? [Yn]  y
Removing eyaml
Successfully uninstalled hiera-eyaml-3.4.0
:~$ sudo /opt/puppetlabs/server/bin/puppetserver gem which hiera/backend/eyaml/parser/parser
/opt/puppetlabs/puppet/lib/ruby/vendor_gems/gems/hiera-eyaml-3.4.0/lib/hiera/backend/eyaml/parser/parser.rb
:~$ sudo /opt/puppetlabs/server/bin/puppetserver ruby -rpuppet -e 'puts Puppet.features.hiera_eyaml?'
true

After removing all external installs of hiera-eyaml, we are now using the vendor_gem. That fixes the issue. It seems hiera-eyaml v4.1.0 was working until puppet agent 8.6.0, but fails on 8.7.0. We removed all external installs to use the vendor supplied gem to fix the issue.

[bastelfreak]

Hey,
do I read it correctly that puppetserver vendors hiera-eyaml 3.4.0 but you also install hiera-eyaml 3.4.0 and 4.1.0? Did you try uninstalling 3.4.0 and only have 4.1.0? I want to figure out if this is a conflict of between differeny hiera-eyaml versions or hiera-yaml and puppet.

[bastelfreak]

I cannot reproduce it:

My test system is Ubuntu 22.04

root@puppetserver:~/# puppet --version
8.8.1
root@puppetserver:~/# /opt/puppetlabs/server/bin/puppetserver --version
puppetserver version: 8.6.2
root@puppetserver:~/#

The vanilla installation works:

root@puppetserver:~/# /opt/puppetlabs/server/bin/puppetserver ruby -rpuppet -e 'puts Puppet.features.hiera_eyaml?'
2024-09-06 16:07:56,683 INFO  [p.s.j.jruby-puppet-core] Disabling i18n for puppet because using multithreaded jruby
true
root@puppetserver:~/#

But it still works after installing hiera-eyaml 4.1.0

root@puppetserver:~/controlrepo# /opt/puppetlabs/server/bin/puppetserver gem install hiera-eyaml
2024-09-06 16:17:59,485 INFO  [p.s.j.jruby-puppet-core] Disabling i18n for puppet because using multithreaded jruby
Fetching highline-2.1.0.gem
Fetching hiera-eyaml-4.1.0.gem
Successfully installed highline-2.1.0
Successfully installed hiera-eyaml-4.1.0
Parsing documentation for highline-2.1.0
Installing ri documentation for highline-2.1.0
Parsing documentation for hiera-eyaml-4.1.0
Installing ri documentation for hiera-eyaml-4.1.0
Done installing documentation for highline, hiera-eyaml after 12 seconds
2 gems installed
root@puppetserver:~/controlrepo# /opt/puppetlabs/server/bin/puppetserver ruby -rpuppet -e 'puts Puppet.features.hiera_eyaml?'
2024-09-06 16:18:57,221 INFO  [p.s.j.jruby-puppet-core] Disabling i18n for puppet because using multithreaded jruby
true
root@puppetserver:~/# /opt/puppetlabs/server/bin/puppetserver gem which hiera/backend/eyaml/parser/parser
2024-09-06 16:21:40,126 INFO  [p.s.j.jruby-puppet-core] Disabling i18n for puppet because using multithreaded jruby
/opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-4.1.0/lib/hiera/backend/eyaml/parser/parser.rb
root@puppetserver:~/#

it even works after restarting puppetserver:

root@puppetserver:~/# systemctl restart puppetserver
root@puppetserver:~/# /opt/puppetlabs/server/bin/puppetserver gem which hiera/backend/eyaml/parser/parser
2024-09-06 16:22:34,480 INFO  [p.s.j.jruby-puppet-core] Disabling i18n for puppet because using multithreaded jruby
/opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-4.1.0/lib/hiera/backend/eyaml/parser/parser.rb
root@puppetserver:~/controlrepo# /opt/puppetlabs/server/bin/puppetserver ruby -rpuppet -e 'puts Puppet.features.hiera_eyaml?'
2024-09-06 16:22:46,146 INFO  [p.s.j.jruby-puppet-core] Disabling i18n for puppet because using multithreaded jruby
true
root@puppetserver:~/#

any chance your umask isn't 0022 and the files, for example /opt/puppetlabs/server/data/puppetserver/jruby-gems/gems/hiera-eyaml-4.1.0/lib/hiera/backend/eyaml/parser/parser.rb, don't have 0644 permissions?

[joshcooper]

Another possibility is hiera-eyaml depends on strscan which I believe is a gem with native extensions. So you'd need a JRuby compatible version of strscan in order to load hiera-eyaml.

[bastelfreak]

indeed there's a require 'strscan'
https://github.com/voxpupuli/hiera-eyaml/blob/dd86f485a35f4dd75d09334246ad097214d5b48a/lib/hiera/backend/eyaml/parser/parser.rb#L1

[joshcooper]

@mdechiaro you'll need to look closely at which gems are installed in JRuby. Recommend running these commands to try to narrow down the problem:

/opt/puppetlabs/server/bin/puppetserver gem list
/opt/puppetlabs/server/bin/puppetserver gem env

[mdechiaro]

@joshcooper unfortunately I don't have time to troubleshoot this issue further. The fix for us was to use the vendor_gem version, and I am OK with closing this issue. Thanks for your help.

[joshcooper]

Thanks @mdechiaro, I'll close this