Setting a serializer in redis options causes queue to re-run completed jobs

[b-t-927]

Laravel Version

10.13.5

PHP Version

8.2.7

Database Driver & Version

Server: Redis 7.0.9 container on aarch64 k3s instance phpredis: 5.3.7

Description

When using redis via phpredis as a queue database, setting the serializers option in database.php causes complete jobs to be retried even after completing successfully after the retry_value has been met. Eventually the job will be marked as failed based on whatever value is being used. This occurs regardless of if you use queue:work or horizon for managing the queue.

Before I worked out the root cause of my jobs repeating, I ruled out anything in the job class itself or how it's dispatched as making a difference. Even a completely empty job that had just been created using artisan would exhibit this behaviour.

This also seems to occur regardless of what the serializer option is actually set to, just having it set causes the jobs to be retried.

Although my development environment is an apple silicon mac running PHP 8.2 installed from brew, I was able to recreate this on both an aarch64 and x86_64 linux environment as well (connecting to the same redis server instance).

Steps To Reproduce

• Set a serializer under options in the redis connection in database.php
• Dispatch a job to a queue using redis as the driver

[leopado]

I have the same issue with this configuration

Laravel Version 10.13.5

PHP 8.2.5

Redis 6.0.16

When I dispatch the job to a queue it runs and complete with success.
Then if I run php artisan queue:monitor the queue counter doesn't decrease and after the retry_after time has passed the job runs again but this time fails immediatly because the maximum time from the dispatch time has been elapsed.

This is my configuration

'redis' => [

        'client' => env('REDIS_CLIENT', 'phpredis'),

        'options' => [
            'cluster' => env('REDIS_CLUSTER', 'redis'),
            'prefix' => env('REDIS_PREFIX', Str::slug(env('APP_NAME', 'laravel'), '_').'_database_'),
            'password' => env('REDIS_PASSWORD', null),
            //'serializer' => env('REDIS_SERIALIZER', 2),//2=Redis::SERIALIZER_IGBINARY
           'compression' => env('REDIS_COMPRESSION', 3),//3=Redis::COMPRESSION_LZ4,
        ],

If i set 'compression'=>null everything goes as expected.

[driesvints]

This is expected. You need to make sure your queues are empty before changing these options.

[tarkis]

Sorry @driesvints, but what do you mean by "This is expected"? Is it expected for queue with redis driver do not work if it has anything else set on "serializer" or "compression" option?

I'm facing same issue. Queues do not work if you use serializer or compression.

How to reproduce:

1. Download latest laravel and use sail with redis client as phpredis.
2. Update .env QUEUE_CONNECTION=redis
3. Create new test job sail artisan make:job TestJob
4. Update config/database.php by setting 'serializer' => Redis::SERIALIZER_IGBINARY to redis.options
5. Dispatch TestJob via tinker (new App\Jobs\TestJob())->dispatch()
   After dispatch you can check redis queues:default and you will see queue payload:

{"uuid":"9d079093-4e4d-4040-894e-8f678665e05b","displayName":"App\\Jobs\\TestJob","job":"Illuminate\\Queue\\CallQueuedHandler@call","maxTries":null,"maxExceptions":null,"failOnTimeout":false,"backoff":null,"timeout":null,"retryUntil":null,"data":{"commandName":"App\\Jobs\\TestJob","command":"O:16:\"App\\Jobs\\TestJob\":0:{}"},"id":"0EbqYPuCTpxtdGe7HHw39CVefAPjp8cG","attempts":0}

6. Run sail artisan queue:work

 INFO  Processing jobs from the [default] queue.

  2023-08-23 16:01:02 App\Jobs\TestJob ................. RUNNING
  2023-08-23 16:01:02 App\Jobs\TestJob .................. 9.38ms DONE

7. Check redis and you will see new sorted set: queues:default:reserved with a job inside it that has increased number of attempts:

{"uuid":"9d079093-4e4d-4040-894e-8f678665e05b","timeout":null,"id":"0EbqYPuCTpxtdGe7HHw39CVefAPjp8cG","backoff":null,"displayName":"App\\Jobs\\TestJob","maxTries":null,"failOnTimeout":false,"maxExceptions":null,"retryUntil":null,"data":{"command":"O:16:\"App\\Jobs\\TestJob\":0:{}","commandName":"App\\Jobs\\TestJob"},"job":"Illuminate\\Queue\\CallQueuedHandler@call","attempts":1}

8. Rerun sail artisan queue:work and you will get MaxAttemptsExceededException

[SudoGetBeer]

I have the same weird problem like @tarkis.
@driesvints and I cleared the whole redis db after changing the config.

[driesvints]

I'm sorry but I fail to understand how it can re-run completed jobs if the database is empty? Are you 100% sure it's entirely cleared?

[SudoGetBeer]

Okay maybe I was not clear enough.
After changing the serializer and compression configs I emptied the redis db and when I then run jobs again I get the following:

As you can see the jobs will be retried until they throw MaxAttemptsExceededException and even then they will be retried.
This is with serializer = SERIALIZER_IGBINARY and compression = COMPRESSION_LZF . When disabling both it works fine.

[github-actions]

Thank you for reporting this issue!

As Laravel is an open source project, we rely on the community to help us diagnose and fix issues as it is not possible to research and fix every issue reported to us via GitHub.

If possible, please make a pull request fixing the issue you have described, along with corresponding tests. All pull requests are promptly reviewed by the Laravel team.

Thank you!

[driesvints]

Since it seems we added a warning about this in the docs, I'm closing this one.

laravel/docs#9000