Release v2.17.0

CHANGELOG.md

@@ -1,116 +1,137 @@
-# Changelog for Oban v2.16
+# Changelog for Oban v2.17
 
-_🌟 Looking for changes to Web or Pro? Check the [Oban.Pro Changelog][opc] or
-the [Oban.Web Changelog][owc]. 🌟_
+_🌟 Looking for changes to Web or Pro? Check the [Oban.Pro Changelog][opc] or the [Oban.Web
+Changelog][owc]. 🌟_
 
-## 🐑 Oban Instance Module
+This release includes an optional database migration to disable triggers and relax priority
+checks. See the [v2.17 upgrade guide](v2-17.md) for step-by-step instructions.
 
-New facade modules allow you to call `Oban` functions on instances with custom names, e.g. not
-`Oban`, without passing a `t:Oban.name/0` as the first argument.
+## 📟 Universal Insert Notifications
 
-For example, rather than calling `Oban.config/1` you'd call `MyOban.config/0`:
+Historically, Oban used database triggers to emit a notification after a job is inserted. That
+allowed jobs to execute sooner, without waiting up to a second until the next poll event. Those
+triggers and subsequent notifications added some overhead to database operations bulk inserts into
+the same queue, despite deduplication logic in the trigger. Even worse, trigger notifications
+didn't work behind connection poolers and were restricted to the Postgres notifier.
 
-```elixir
-MyOban.config()
-```
+Now insert notifications have moved out of the database and into application code, so it's
+possible to disable triggers without running database migrations, and they work for _any_
+notifier, not just Postgres.
 
-It also makes piping into Oban functions far more convenient: 
+Disable notifications with the `insert_trigger` option if sub-second job execution isn't important
+or you'd like to reduce PubSub chatter:
 
 ```elixir
-%{some: :args}
-|> MyWorker.new()
-|> MyOban.insert()
+config :my_app, Oban,
+  insert_trigger: false,
+  ...
 ```
 
-## 🧩 Partial Matches in Testing Assertions
+## 🧑‍🏭 Worker Conveniences
 
-It's now possible to match a subset of fields on args or meta with `all_enqueued`,
-`assert_enqueued`, and `refute_enqueued`. For example, the following assertion will now pass:
+Workers received a few quality of life improvements to make defining `unique` behaviour more
+expressive and intuitive.
 
-```elixir
-# Given a job with these args: %{id: 123, mode: "active"}
+First, it's now possible to define a job's unique period with time units like `{1, :minute}` or
+`{2, :hours}`, just like a job's `:schedule_in` option:
 
-assert_enqueued args: %{id: 123} #=> true
-assert_enqueued args: %{mode: "active"} #=> true
-assert_enqueued args: %{id: 321, mode: "active"} #=> false
+```elixir
+use Oban.Worker, unique: [period: {5, :minutes}]
 ```
 
-The change applies to `args` and `meta` queries for `all_enqueued/2`, `assert_enqueued/2` and
-`refute_enqueued/2` helpers.
+Second, you can set the `replace` option in `use Oban.Worker` rather than in an overridden `new/2`
+or as a runtime option. For example, to enable updating a job's `scheduled_at` value on unique
+conflict:
 
-## ⏲️ Unique Timestamp Option
+```elixir
+use Oban.Worker, unique: [period: 60], replace: [scheduled: [:scheduled_at]]
+```
 
-Jobs are frequently scheduled for a time far in the future and it's often desirable for to
-consider `scheduled` jobs for uniqueness, but unique jobs only checked the `:inserted_at`
-timestamp.
+## 🐦‍🔥 Oban Phoenix Notifier
 
-Now `unique` has a `timestamp` option that allows checking the `:scheduled_at` timestamp instead:
+The new [`oban_notifiers_phoenix` package][onp] allows Oban to share a Phoenix application's
+PubSub for notifications. In addition to centralizing PubSub communications, it opens up the
+possible transports to all PubSub adapters. As Oban already provides `Postgres` and `PG`
+(Distributed Erlang) notifiers, the new package primarily enables Redis notifications.
 
 ```elixir
-use Oban.Worker, unique: [period: 120, timestamp: :scheduled_at]
+config :my_app, Oban,
+  notifier: {Oban.Notifiers.Phoenix, pubsub: MyApp.PubSub},
+  ...
 ```
 
-## v2.16.3 — 2023-10-26
+[onp]: https://github.com/sorentwo/oban_notifiers_phoenix
 
-### Bug Fixes
+## 🎚️ Ten Levels of Job Priority
 
-- [Oban] Start `Peer` and `Stager` after `Queue` supervisor
+Job priority may now be set to values between 0 (highest) and 9 (lowest). This increases the
+range from 4 to 10 possible priorities, giving applications much finer control over execution
+order.
 
-  The queue supervisor blocks shutdown to give jobs time to shut down gracefully. During that
-  time, the Peer could obtain or retain leadership despite all of the plugins having stopped. Now
-  the Peer and Stager (which is only active on the leader) stop before the queue supervisor.
+```elixir
+args
+|> MyApp.PrioritizedWorker.new(priority: 9)
+|> Oban.insert()
+```
 
-- [Testing] Cast timestamp to utc_datetime in testing queries
+## v2.17.0 — 2023-12-08
 
-  Timestamps with a timezone are now cast to `:utc_datetime` via a changeset before running
-  `Oban.Testing` queries.
+### Enhancements
 
-## v2.16.2 — 2023-10-03
+- [Oban] Add `Oban.pause_all_queues/2` and `Oban.resume_all_queues/2`.
 
-### Bug Fixes
+  Pause and resume all queues with a single function call and a single notification signal, rather
+  than manually looping through all queues and issuing separate calls.
 
-- [Testing] Match args/meta patterns in Elixir rather than the database
+- [Cron] Add non-raising `Expression.parse/2` for use in `Cron.parse/2` and shared validations.
 
-  The containment operators, `@>` and `<@`, used for pattern matching in tests are only available
-  in Postgres and have some quirks. Most notably, containment considers matching any value in a
-  list a successful match, which isn't intuitive or desirable.
+  Multiple locations used `parse!` and converted a raised exception into an error tuple. That was
+  inefficient, repetitive, and violated the common practice of avoiding exceptions for flow
+  control.
 
-  The other issue with using a containment operator in tests is that SQLite doesn't have those
-  operators available and test helpers are shared between all engines.
+- [Validation] Use schema based validation for workers, plugins, and config. 
 
-### Enhancements
+  Validations are now simpler and more consistent, and behaviour based notifiers such as Engine,
+  Repo, and Peer are more descriptive.
 
-- [Testing] Support wildcard matcher in patterns for args/meta
+- [Engine] Expand telemetry meta for all engine callbacks events.
 
-  Now that we match in Elixir, it's simple to support wildcard matching with a `:_` to assert that
-  a key is present in a json field without specifying an exact value.
+  All callbacks now include every argument in telemetry event metadata. In some situations, e.g.
+  `:init`, this simplifies testing and can be used to eliminate the need to poll a supervision
+  tree to see which queues started.
 
-  ```elixir
-  assert_enqueued args: %{batch_id: :_, callback: true}
-  ```
+- [Notifier] Add `Isolated` notifier for local use and simplified testing.
 
-## v2.16.1 — 2023-09-25
+  Using PG for async tests has occasional flakes due to its eventually consistent nature. In tests
+  and single node systems, we don't need to broadcast messages between instances or nodes, and a
+  simplified "isolated" mechanism is ideal.
+
+- [Repo] Add `Repo.query!/4` for `Ecto.Repo` parity
+
+- [Migration] Configure a third-party engine's migrator using the repo's `config` map.
 
 ### Bug Fixes
 
-- [Testing] Restore splitting out all config options in helpers.
+- [Cron] Guard against invalid cron range expressions where the left side is greater than the
+  right, e.g. `SAT-FRI`.
 
-  Splitting all configuration keys is necessary when using `perform_job/3` with non-job options
-  such as `:engine`.
+- [Testing] Disable the `prefix` by default in generated testing helpers.
 
-## v2.16.0 — 2023-09-22
+  A prefix is only necessary when it's not the standard "public" prefix, which is rarely the case
+  in testing helpers. This makes it easier to use testing helpers with the `Lite` engine.
 
-### Bug Fixes
+- [Testing] Remove `prefix` segment from `assert_enqueued` error messages.
 
-- [Reindexer] Correct relname match for reindexer plugin
+  Not all engines support a prefix and the assert/refute message in testing helpers is confusing
+  when the prefix is `nil`.
 
-  We can safely assume all indexes start with `oban_jobs`. The previous pattern was based on an
-  outdated index format from older migrations.
+### Deprecations
 
-- [Testing] Support `repo`, `prefix`, and `log` query options in `use Oban.Testing`
+- [Gossip] The Gossip plugin is no longer needed, and shouldn't be used, by applications running
+  Oban Web v2.10 or above.
 
-For changes prior to v2.16 see the [v2.15][prv] docs.
+For changes prior to v2.17 see the [v2.16][prv] docs.
 
 [opc]: https://getoban.pro/docs/pro/changelog.html
 [owc]: https://getoban.pro/docs/web/changelog.html
-[prv]: https://hexdocs.pm/oban/2.15.2/changelog.html
+[prv]: https://hexdocs.pm/oban/2.16.3/changelog.html

guides/troubleshooting.md

@@ -30,8 +30,7 @@ systems with multi-node setups where "web" nodes only enqueue jobs while "worker
 configured to run queues and plugins. Most plugins require leadership to function, so When a "web"
 node becomes leader the plugins go dormant.
 
-The solution is to disable leadership with `peer: false` on any node that doesn't run plugins, or
-only runs decentralized plugins like `Gossip`:
+The solution is to disable leadership with `peer: false` on any node that doesn't run plugins:
 
 ```elixir
 config :my_app, Oban, peer: false, ...

guides/upgrading/v2.17.md

@@ -0,0 +1,67 @@
+# Upgrading to v2.17
+
+This Oban release includes an optional, but recommended migration.
+
+> #### Prevent Duplicate Insert Notifications {: .warning}
+>
+> You must either [run the v12 migrations](#run-oban-migrations-for-v12-optional) or [disable
+> insert triggers](#disable-insert-notifications-optional) in your Oban configuration, otherwise
+> you'll receive duplicate insert notifications for each job.
+
+## Bump Your Deps
+
+Update Oban (and optionally Pro) to the latest versions:
+
+```elixir
+[
+  {:oban, "~> 2.17"},
+  {:oban_pro, "~> 1.2", repo: "oban"}
+]
+```
+
+## Run Oban.Migrations for v12 (Optional)
+
+The v12 migration removes insert triggers and relaxes the `priority` column's check constraint to
+allow values in the new range of `0..9`.
+
+To get started, create a migration to create the table:
+
+```bash
+$ mix ecto.gen.migration upgrade_oban_jobs_to_v12
+```
+
+Within the generated migration module:
+
+```elixir
+use Ecto.Migration
+
+def up, do: Oban.Migrations.up(version: 12)
+
+def down, do: Oban.Migrations.down(version: 11)
+```
+
+If you have multiple Oban instances, or use an alternate prefix, you'll need to run the migration
+for each prefix.
+
+## Disable Insert Notifications (Optional)
+
+If you opt not to run the v12 migration to disable Postgres triggers, then you should disable
+insert notifications in your configuration:
+
+```diff
+ config :my_app, Oban,
++  insert_trigger: false,
+   ...
+```
+
+## Remove the Gossip Plugin
+
+The Gossip plugin is no longer used by Oban Web and now useless. You can safely remove it from
+your configuration:
+
+```diff
+ config :my_app, Oban,
+   plugins: [
+-    Oban.Plugins.Gossip,
+   ]
+```

mix.exs

@@ -2,7 +2,7 @@ defmodule Oban.MixProject do
   use Mix.Project
 
   @source_url "https://github.com/sorentwo/oban"
-  @version "2.16.3"
+  @version "2.17.0"
 
   def project do
     [
@@ -74,6 +74,7 @@ defmodule Oban.MixProject do
       "guides/upgrading/v2.11.md",
       "guides/upgrading/v2.12.md",
       "guides/upgrading/v2.14.md",
+      "guides/upgrading/v2.17.md",
 
       # Recipes
       "guides/recipes/recursive-jobs.md",