[Bug] Import stales when error in one import data record

[fashxp]

Expected behavior

To be discussed

Actual behavior

For example, I get following error in the logs:

 CRITICAL  [messenger] Error thrown while handling message Pimcore\Bundle\DataImporterBundle\Messenger\DataImporterMessage. Removing from transport after 3 retries. Error: "Handling "Pimcore\Bundle\DataImporterBundle\Messenger\DataImporterMessage" failed: Pimcore\Bundle\DataImporterBundle\Tool\DataObjectLoader::loadByAttribute(): Argument #3 ($identifier) must be of type string, null given, called in /var/www/html/dev/pimcore/data-importer/src/Resolver/Load/AttributeStrategy.php on line 70" ["class" => "Pimcore\Bundle\DataImporterBundle\Messenger\DataImporterMessage","retryCount" => 3,"error" => "Handling "Pimcore\Bundle\DataImporterBundle\Messenger\DataImporterMessage" failed: Pimcore\Bundle\DataImporterBundle\Tool\DataObjectLoader::loadByAttribute(): Argument #3 ($identifier) must be of type string, null given, called in /var/www/html/dev/pimcore/data-importer/src/Resolver/Load/AttributeStrategy.php on line 70","exception" => Symfony\Component\Messenger\Exception\HandlerFailedException { …}]

no messages in application log, but import is stuck as tmp-store worker entry is not cleared properly.

Steps to reproduce

import CSV/xls with empty rows

[mcop1]

I could reproduce the problem, a possible solution could be:

We could catch the exception either directly in processQueueItems method or in the methods that are calling processQueueItems. Log the exception message to the application logger.

We could introduce a flag "continue on error" which is set by default. If the flag is set, we catch and log the exception, after that continue with the next row. If it is not set, we catch and log the exception, after that stop import. This could be important if you have imports or rows that are depended of each other (e.g. parent child relations).
Or maybe we could even decide this based on which process configuration was set (parallel or sequential). But a flag/checkbox would be the most flexible option.

[mcop1]

See #263 (comment) for more information on replicating the issue.

[mcop1]

I could reproduce the problem, a possible solution could be:

We could catch the exception either directly in processQueueItems method or in the methods that are calling processQueueItems. Log the exception message to the application logger.

We could introduce a flag "continue on error" which is set by default. If the flag is set, we catch and log the exception, after that continue with the next row. If it is not set, we catch and log the exception, after that stop import. This could be important if you have imports or rows that are depended of each other (e.g. parent child relations). Or maybe we could even decide this based on which process configuration was set (parallel or sequential). But a flag/checkbox would be the most flexible option.

Implement:

• Add exception for the mentioned error
• Catch all exceptions in processQueueItem and remove the row from the queue table

Follow up feature:
#307