feat: support other js package managers (#349)

* feat: use `package_json` to manage `package.json`

* fix: add `require` back maybe?

* fix: use `read` and leave the package manager to `package_json` (as it now has an env variable)

* fix: update `PackageJson` call

* fix: don't check yarn anymore

* fix: use bang method & exit early on error

* fix: use `package_json` rather than `yarn bin` to exec webpack

* fix: replace `check_yarn` with `check_manager` task

* fix: show general package manager name & version in info task

* refactor: make runners use of `PackageJson` opt-in

* refactor: make rake tasks use of `PackageJson` opt-in

* refactor: restore `check_yarn` rake task, based on opt-in flag for `PackageJson`

* refactor: make use of `PackageJson` in template tasks opt-in

* fix: only use `npm` for getting node package version if not using `package_json` gem

* fix: preserve old load paths when requiring `package-json` gem

It seems that requires which happen after this are mucked up due to the load paths having been
changed - this really only seems to impact RSpec which requires the diff library lazily

* fix: read `package.json` within app path

* fix: address minor bugs discovered from adding specs for when `package_json` is enabled

* test: update generator specs

* ci: run tests both with and without using `package_json` gem to ensure everything is valid

* docs: include a section about `SHAKAPACKER_USE_PACKAGE_JSON_GEM`

* fix: properly set babel preset for react

* feat: support bun

* feat: properly include `package_json` as a dependency

* ci: run generator specs with and without `package_json` gem in parallel

* docs: add changelog entry

.github/workflows/generator.yml

@@ -10,6 +10,7 @@ jobs:
       matrix:
         os: [ubuntu-latest]
         ruby: ['2.6', '2.7', '3.0']
+        use_package_json_gem: ['true', 'false']
         gemfile:
           - gemfiles/Gemfile-rails.6.0.x
           - gemfiles/Gemfile-rails.6.1.x
@@ -37,3 +38,5 @@ jobs:
       - name: Install dependencies
         run: bundle install
       - run: bundle exec rake run_spec:generator
+        env:
+          SHAKAPACKER_USE_PACKAGE_JSON_GEM: ${{ matrix.use_package_json_gem }}

.github/workflows/ruby-backward-compatibility.yml

@@ -39,3 +39,9 @@ jobs:
 
       - name: Ruby specs - Backward compatibility
         run: bundle exec rake run_spec:gem_bc
+        env:
+          SHAKAPACKER_USE_PACKAGE_JSON_GEM: "false"
+      - name: Ruby specs - Backward compatibility
+        run: bundle exec rake run_spec:gem_bc
+        env:
+          SHAKAPACKER_USE_PACKAGE_JSON_GEM: "true"

.github/workflows/ruby.yml

@@ -39,3 +39,10 @@ jobs:
 
       - name: Ruby specs
         run: bundle exec rake run_spec:gem
+        env:
+          SHAKAPACKER_USE_PACKAGE_JSON_GEM: "false"
+
+      - name: Ruby specs
+        run: bundle exec rake run_spec:gem
+        env:
+          SHAKAPACKER_USE_PACKAGE_JSON_GEM: "true"

CHANGELOG.md

@@ -9,6 +9,9 @@ Changes since the last non-beta release.
 
 _Please add entries here for your pull requests that are not yet released._
 
+### Added
+- Experimental support for other JS package managers using `package_json` gem [PR 349](https://github.com/shakacode/shakapacker/pull/349) by [G-Rath](https://github.com/g-rath).
+
 ## [v7.1.0] - September 30, 2023
 
 ### Added

README.md

@@ -51,6 +51,7 @@ Read the [full review here](https://clutch.co/profile/shakacode#reviews?sort_by=
     - [Optional support](#optional-support)
   - [Installation](#installation)
     - [Rails v6+](#rails-v6)
+    - [Using alternative package managers](#using-alternative-package-managers)
     - [Note for Yarn v2 usage](#note-for-yarn-v2-usage)
   - [Concepts](#concepts)
   - [Usage](#usage)
@@ -163,9 +164,21 @@ yarn add @babel/core @babel/plugin-transform-runtime @babel/preset-env @babel/ru
 Previously, these "webpack" and "babel" packages were direct dependencies for `shakapacker`. By
 making these peer dependencies, you have control over the versions used in your webpack and babel configs.
 
+### Using alternative package managers
+
+There is experimental support for using package managers besides Yarn classic for managing JavaScript dependencies using the [`package_json`](https://github.com/G-Rath/package_json) gem.
+
+This can be enabled by setting the environment variable `SHAKAPACKER_USE_PACKAGE_JSON_GEM` to `true`; Shakapacker will then use the `package_json` gem which in turn will look for the [`packageManager`](https://nodejs.org/api/packages.html#packagemanager) property in the `package.json` or otherwise the `PACKAGE_JSON_FALLBACK_MANAGER` environment variable to determine which manager to use, defaulting to `npm` if neither are found.
+
+See [here](https://github.com/G-Rath/package_json#specifying-a-package-manager) for a list of the supported package managers and more information; note that `package_json` does not handle ensuring the manager is installed.
+
+> **Note**
+> 
+> The rest of the documentation assumes that `package_json` is not being used, and so always references `yarn` - you should instead use the package manager of your choice for these commands.  
+
 ### Note for Yarn v2 usage
 
-If you are using Yarn v2 (berry), please note that PnP modules are not supported.
+If you are using Yarn v2 (berry), please note that PnP modules are not supported unless you're using `SHAKAPACKER_USE_PACKAGE_JSON_GEM`.
 
 To use Shakapacker with Yarn v2, make sure you set `nodeLinker: node-modules` in your `.yarnrc.yml` file as per the [Yarn docs](https://yarnpkg.com/getting-started/migration#step-by-step) to opt out of Plug'n'Play behavior.
 

lib/install/template.rb

@@ -1,3 +1,4 @@
+require "shakapacker/utils/misc"
 require "shakapacker/utils/version_syntax_converter"
 
 # Install Shakapacker
@@ -41,12 +42,28 @@
   say %(        Add <%= javascript_pack_tag "application" %> within the <head> tag in your custom layout.)
 end
 
+def package_json
+  if @package_json.nil?
+    Shakapacker::Utils::Misc.require_package_json_gem
+
+    @package_json = PackageJson.read
+  end
+
+  @package_json
+end
+
 # Ensure there is `system!("bin/yarn")` command in `./bin/setup` file
 if (setup_path = Rails.root.join("bin/setup")).exist?
-  say "Run bin/yarn during bin/setup"
+  def native_install_command
+    return "bin/yarn" unless Shakapacker::Utils::Misc.use_package_json_gem
+
+    package_json.manager.native_install_command.join(" ")
+  end
+
+  say "Run #{native_install_command} during bin/setup"
 
   if File.read(setup_path).match? Regexp.escape("  # system('bin/yarn')\n")
-    gsub_file(setup_path, "# system('bin/yarn')", "system!('bin/yarn')")
+    gsub_file(setup_path, "# system('bin/yarn')", "system!('#{native_install_command}')")
   else
     # Due to the inconsistency of quotation usage in Rails 7 compared to
     # earlier versions, we check both single and double quotations here.
@@ -55,30 +72,58 @@
     string_to_add = <<-RUBY
 
   # Install JavaScript dependencies
-  system!("bin/yarn")
-RUBY
+  system!("#{native_install_command}")
+    RUBY
 
     if File.read(setup_path).match? pattern
       insert_into_file(setup_path, string_to_add, after: pattern)
     else
       say <<~MSG, :red
         It seems your `bin/setup` file doesn't have the expected content.
-        Please review the file and manually add `system!("bin/yarn")` before any
+        Please review the file and manually add `system!("#{native_install_command}")` before any
         other command that requires JavaScript dependencies being already installed.
       MSG
     end
   end
 end
 
-results = []
+def add_dependencies(dependencies, type)
+  return package_json.manager.add!(dependencies, type: type) if Shakapacker::Utils::Misc.use_package_json_gem
+
+  # TODO: check that run actually errors
+  run("yarn add #{dependencies.join(" ")}") if type == :production
+  run("yarn add --dev #{dependencies.join(" ")}") if type == :dev
+rescue PackageJson::Error
+  say "Shakapacker installation failed 😭 See above for details.", :red
+  exit 1
+end
+
+def fetch_peer_dependencies
+  if Shakapacker::Utils::Misc.use_package_json_gem
+    return PackageJson.read("#{__dir__}/../../").fetch("peerDependencies")
+  end
+
+  package_json = File.read("#{__dir__}/../../package.json")
+  JSON.parse(package_json)["peerDependencies"]
+end
 
 Dir.chdir(Rails.root) do
   npm_version = Shakapacker::Utils::VersionSyntaxConverter.new.rubygem_to_npm(Shakapacker::VERSION)
   say "Installing shakapacker@#{npm_version}"
-  results << run("yarn add shakapacker@#{npm_version} --exact")
+  add_dependencies(["shakapacker@#{npm_version}"], :production)
+
+  if Shakapacker::Utils::Misc.use_package_json_gem
+    package_json.merge! do |pj|
+      {
+        "dependencies" => pj["dependencies"].merge({
+          # TODO: workaround for test suite - long-run need to actually account for diff pkg manager behaviour
+          "shakapacker" => pj["dependencies"]["shakapacker"].delete_prefix("^")
+        })
+      }
+    end
+  end
 
-  package_json = File.read("#{__dir__}/../../package.json")
-  peers = JSON.parse(package_json)["peerDependencies"]
+  peers = fetch_peer_dependencies
   dev_dependency_packages = ["webpack-dev-server"]
 
   dependencies_to_add = []
@@ -96,13 +141,8 @@
   end
 
   say "Adding shakapacker peerDependencies"
-  results << run("yarn add #{dependencies_to_add.join(' ')}")
+  add_dependencies(dependencies_to_add, :production)
 
   say "Installing webpack-dev-server for live reloading as a development dependency"
-  results << run("yarn add --dev #{dev_dependencies_to_add.join(' ')}")
-end
-
-unless results.all?
-  say "Shakapacker installation failed 😭 See above for details.", :red
-  exit 1
+  add_dependencies(dev_dependencies_to_add, :dev)
 end

lib/shakapacker/dev_server_runner.rb

@@ -68,11 +68,7 @@ def execute_cmd
         env["WEBPACK_SERVE"] = "true"
         env["NODE_OPTIONS"] = ENV["NODE_OPTIONS"] || ""
 
-        cmd = if node_modules_bin_exist?
-          ["#{@node_modules_bin_path}/webpack", "serve"]
-        else
-          ["yarn", "webpack", "serve"]
-        end
+        cmd = build_cmd
 
         if @argv.include?("--debug-webpacker")
           Shakapacker.puts_deprecation_message(
@@ -98,6 +94,16 @@ def execute_cmd
         end
       end
 
+      def build_cmd
+        if Shakapacker::Utils::Misc.use_package_json_gem
+          return package_json.manager.native_exec_command("webpack", ["serve"])
+        end
+
+        return ["#{@node_modules_bin_path}/webpack", "serve"] if node_modules_bin_exist?
+
+        ["yarn", "webpack", "serve"]
+      end
+
       def node_modules_bin_exist?
         File.exist?("#{@node_modules_bin_path}/webpack-dev-server")
       end

lib/shakapacker/runner.rb

@@ -1,3 +1,5 @@
+require "shakapacker/utils/misc"
+
 module Shakapacker
   class Runner
     def self.run(argv)
@@ -14,7 +16,7 @@ def initialize(argv)
 
       Shakapacker.set_shakapacker_env_variables_for_backward_compatibility
 
-      @node_modules_bin_path = ENV["SHAKAPACKER_NODE_MODULES_BIN_PATH"] || `yarn bin`.chomp
+      @node_modules_bin_path = fetch_node_modules_bin_path
       @shakapacker_config    = ENV["SHAKAPACKER_CONFIG"] || File.join(@app_path, "config/shakapacker.yml")
 
       @shakapacker_config = Shakapacker.get_config_file_path_with_backward_compatibility(@shakapacker_config)
@@ -24,5 +26,21 @@ def initialize(argv)
         exit!
       end
     end
+
+    def fetch_node_modules_bin_path
+      return nil if Shakapacker::Utils::Misc.use_package_json_gem
+
+      ENV["SHAKAPACKER_NODE_MODULES_BIN_PATH"] || `yarn bin`.chomp
+    end
+
+    def package_json
+      if @package_json.nil?
+        Shakapacker::Utils::Misc.require_package_json_gem
+
+        @package_json = PackageJson.read(@app_path)
+      end
+
+      @package_json
+    end
   end
 end

lib/shakapacker/utils/misc.rb

@@ -8,6 +8,18 @@ module Utils
     class Misc
       extend FileUtils
 
+      def self.use_package_json_gem
+        ENV.fetch("SHAKAPACKER_USE_PACKAGE_JSON_GEM", "false").casecmp("true").zero?
+      end
+
+      def self.require_package_json_gem
+        unless use_package_json_gem
+          raise "PackageJson should not be used unless SHAKAPACKER_USE_PACKAGE_JSON_GEM is true"
+        end
+
+        require "package_json"
+      end
+
       def self.uncommitted_changes?(message_handler)
         return false if ENV["COVERAGE"] == "true"
 

lib/shakapacker/webpack_runner.rb

@@ -21,11 +21,7 @@ def run
       env["SHAKAPACKER_CONFIG"] = @shakapacker_config
       env["NODE_OPTIONS"] = ENV["NODE_OPTIONS"] || ""
 
-      cmd = if node_modules_bin_exist?
-        ["#{@node_modules_bin_path}/webpack"]
-      else
-        ["yarn", "webpack"]
-      end
+      cmd = build_cmd
 
       if @argv.include?("--debug-webpacker")
         Shakapacker.puts_deprecation_message(
@@ -61,6 +57,17 @@ def run
     end
 
     private
+
+      def build_cmd
+        if Shakapacker::Utils::Misc.use_package_json_gem
+          return package_json.manager.native_exec_command("webpack")
+        end
+
+        return ["#{@node_modules_bin_path}/webpack"] if node_modules_bin_exist?
+
+        ["yarn", "webpack"]
+      end
+
       def node_modules_bin_exist?
         File.exist?("#{@node_modules_bin_path}/webpack")
       end

lib/tasks/shakapacker.rake

@@ -1,11 +1,11 @@
 tasks = {
   "shakapacker:info"                    => "Provides information on Shakapacker's environment",
-  "shakapacker:install"                 => "Installs and setup webpack with Yarn",
+  "shakapacker:install"                 => "Installs and setup webpack",
   "shakapacker:compile"                 => "Compiles webpack bundles based on environment",
   "shakapacker:clean"                   => "Remove old compiled bundles",
   "shakapacker:clobber"                 => "Removes the webpack compiled output directory",
   "shakapacker:check_node"              => "Verifies if Node.js is installed",
-  "shakapacker:check_yarn"              => "Verifies if Yarn is installed",
+  "shakapacker:check_manager"           => "Verifies if the expected JS package manager is available",
   "shakapacker:check_binstubs"          => "Verifies that bin/shakapacker is present",
   "shakapacker:binstubs"                => "Installs Shakapacker binstubs in this application",
   "shakapacker:verify_install"          => "Verifies if Shakapacker is installed",

lib/tasks/shakapacker/binstubs.rake

@@ -3,7 +3,7 @@ bin_path = ENV["BUNDLE_BIN"] || Rails.root.join("bin")
 
 namespace :shakapacker do
   desc "Installs Shakapacker binstubs in this application"
-  task binstubs: [:check_node, :check_yarn] do |task|
+  task binstubs: [:check_node, :check_manager] do |task|
     prefix = task.name.split(/#|shakapacker:binstubs/).first
 
     if Rails::VERSION::MAJOR >= 5

lib/tasks/shakapacker/check_manager.rake

@@ -0,0 +1,27 @@
+require "shakapacker/utils/misc"
+
+namespace :shakapacker do
+  desc "Verifies if the expected JS package manager is installed"
+  task :check_manager do |task|
+    unless Shakapacker::Utils::Misc.use_package_json_gem
+      prefix = task.name.split(/#|shakapacker:/).first
+      Rake::Task["#{prefix}shakapacker:check_manager"].invoke
+      next
+    end
+
+    require "package_json"
+
+    package_json = PackageJson.read
+    pm = package_json.manager.binary
+
+    begin
+      version = package_json.manager.version
+
+      $stdout.puts "using #{pm}@#{version} to manage dependencies and scripts in package.json"
+    rescue PackageJson::Error
+      $stderr.puts "#{pm} not installed - please ensure it is installed before trying again"
+      $stderr.puts "Exiting!"
+      exit!
+    end
+  end
+end

lib/tasks/shakapacker/check_yarn.rake

@@ -1,7 +1,8 @@
-require "semantic_range"
 namespace :shakapacker do
   desc "Verifies if Yarn is installed"
   task :check_yarn do
+    require "semantic_range"
+
     begin
       which_command = Gem.win_platform? ? "where" : "which"
       raise Errno::ENOENT if `#{which_command} yarn`.strip.empty?