Puppet agent throws random errors

[lordwebbie]

Describe the Bug

For a project, we operate approx. 40 servers of exact same hardware specifications. On most of them, puppet runs stable. But on several others, puppet seems to be instable and throws random errors. We do not have any clue, what the problem could be.

Expected Behavior

Puppet agent should not throw random errors.

Steps to Reproduce

To reproduce, we can just use any simple puppet agent command and execute it multiple times after one another.

e.g. configprint:

root@host ~ # puppet agent --configprint ssldir
Error: Could not initialize global default settings: can't modify frozen File: #<File:/opt/puppetlabs/puppet/lib/ruby/vendor_gems/gems/hocon-1.3.1/lib/hocon/impl/config_node_simple_value.rb>
root@host ~ # puppet agent --configprint ssldir
/etc/puppetlabs/puppet/ssl
root@host ~ # puppet agent --configprint ssldir
/etc/puppetlabs/puppet/ssl
root@host ~ # puppet agent --configprint ssldir
/etc/puppetlabs/puppet/ssl
root@host ~ #
root@host ~ # puppet agent --configprint ssldir
Traceback (most recent call last):
        21: from /opt/puppetlabs/puppet/bin/puppet:4:in `<main>'
        20: from /opt/puppetlabs/puppet/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        19: from /opt/puppetlabs/puppet/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        18: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:12:in `<top (required)>'
        17: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:12:in `require_relative'
        16: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:335:in `<top (required)>'
        15: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:335:in `require_relative'
        14: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser.rb:6:in `<top (required)>'
        13: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser.rb:6:in `require_relative'
        12: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:8:in `<top (required)>'
        11: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:8:in `require_relative'
        10: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops.rb:1:in `<top (required)>'
         9: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops.rb:120:in `<module:Puppet>'
         8: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/model/ast.rb:4669:in `register_pcore_types'
         7: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loaders.rb:111:in `static_loader'
         6: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/pcore.rb:65:in `init'
         5: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/types.rb:61:in `register_ptypes'
         4: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/types.rb:61:in `each'
         3: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/types.rb:65:in `block in register_ptypes'
         2: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/types.rb:760:in `register_ptype'
         1: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/types.rb:53:in `create_ptype'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/pcore.rb:103:in `create_object_type': can't modify frozen Hash: {} (FrozenError)

Example: confdir

root@host ~ # puppet agent --configprint confdir
/etc/puppetlabs/puppet
root@host ~ # puppet agent --configprint confdir
/etc/puppetlabs/puppet
root@host ~ # puppet agent --configprint confdir
Traceback (most recent call last):
        27: from /opt/puppetlabs/puppet/bin/puppet:4:in `<main>'
        26: from /opt/puppetlabs/puppet/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        25: from /opt/puppetlabs/puppet/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        24: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:12:in `<top (required)>'
        23: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:12:in `require_relative'
        22: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:335:in `<top (required)>'
        21: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:335:in `require_relative'
        20: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser.rb:6:in `<top (required)>'
        19: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser.rb:6:in `require_relative'
        18: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:8:in `<top (required)>'
        17: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:8:in `require_relative'
        16: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops.rb:1:in `<top (required)>'
        15: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops.rb:120:in `<module:Puppet>'
        14: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/model/ast.rb:4669:in `register_pcore_types'
        13: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loaders.rb:111:in `static_loader'
        12: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/pcore.rb:65:in `init'
        11: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/types.rb:68:in `register_ptypes'
        10: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/types.rb:68:in `each'
         9: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/types.rb:68:in `block in register_ptypes'
         8: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/p_meta_type.rb:48:in `resolve'
         7: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/p_object_type.rb:738:in `_pcore_init_from_hash'
         6: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/p_object_type.rb:738:in `map'
         5: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/p_object_type.rb:738:in `each'
         4: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/p_object_type.rb:744:in `block in _pcore_init_from_hash'
         3: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/p_object_type.rb:744:in `new'
         2: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/p_object_type.rb:286:in `initialize'
         1: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/type_asserter.rb:33:in `assert_instance_of'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/types/type_asserter.rb:41:in `report_type_mismatch': Puppet::Pops::Types::TypeAssertionError
root@host ~ # puppet agent --configprint confdir
/etc/puppetlabs/puppet

or even only version output:

root@host ~ # puppet agent --version
7.30.0
root@host ~ # puppet agent --version
Traceback (most recent call last):
        11: from /opt/puppetlabs/puppet/bin/puppet:4:in `<main>'
        10: from /opt/puppetlabs/puppet/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
         9: from /opt/puppetlabs/puppet/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
         8: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:12:in `<top (required)>'
         7: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:12:in `require_relative'
         6: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:41:in `<top (required)>'
         5: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:129:in `<module:Puppet>'
         4: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/defaults.rb:231:in `initialize_default_settings!'
         3: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/settings.rb:1019:in `define_settings'
         2: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/settings.rb:1019:in `each'
         1: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/settings.rb:1026:in `block in define_settings'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/settings.rb:763:in `newsetting': can't modify frozen Hash: {:default=>"", :desc=>"A command to run before every agent run.  If this command returns a non-zero\\n      return code, the entire Puppet run will fail.", :name=>:prerun_command, :section=>:main} (FrozenError)

Environment

• e.g. with Version 7.30.0 and Version 7.23.0
• We are seeing this on some servers running on Debian Bookworm and also on some servers running on Debian Bullseye.
• uname -a:
  • Linux host 6.5.13-5-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-5 (2024-04-05T11:03Z) x86_64 GNU/Linux
  • Linux host 6.5.11-8-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.11-8 (2024-01-30T12:27Z) x86_64 GNU/Linux

Additional Context

Nothing.

[joshcooper]

@lordwebbie I can't reproduce this. There's something going on in your environment. If I had to guess you have a mix of ruby and/or puppet versions. You're better off reaching out in community slack to help track down the issue:

root@red-empathy:~# dpkg-query -W puppet-agent
puppet-agent	7.30.0-1bookworm
root@red-empathy:~# facter os
{
  architecture => "amd64",
  distro => {
    codename => "bookworm",
    description => "Debian GNU/Linux 12 (bookworm)",
    id => "Debian",
    release => {
      full => "12.5",
      major => "12",
      minor => "5"
    }
  },
  family => "Debian",
  hardware => "x86_64",
  name => "Debian",
  release => {
    full => "12.5",
    major => "12",
    minor => "5"
  },
  selinux => {
    enabled => false
  }
}
root@red-empathy:~# puppet agent --configprint confdir
/etc/puppetlabs/puppet
root@red-empathy:~# puppet agent --configprint confdir
/etc/puppetlabs/puppet
root@red-empathy:~# puppet agent --configprint confdir
/etc/puppetlabs/puppet
root@red-empathy:~# puppet agent --configprint confdir
/etc/puppetlabs/puppet
root@red-empathy:~# puppet agent --configprint confdir
/etc/puppetlabs/puppet
root@red-empathy:~# puppet agent --configprint confdir
/etc/puppetlabs/puppet
root@red-empathy:~# puppet agent --configprint confdir
/etc/puppetlabs/puppet