Intermittent index failure on dev.nceas

[taojing2002]

Justin reported a index failure at this ticket. Robyn and Juliet reported a index failure on dev as well.
Two of them have the same error:

[ERROR]: IndexWorker.indexOjbect - cannot index the task for identifier urn:uuid:d0194568-ab15-4212-8998-670da97cb065 since Could not deserialize the org.dataone.service.types.v2.SystemMetadata: null [org.dataone.cn.indexer.IndexWorker:indexOjbect:425]

Both clients are R. It only happened on dev.nceas

[taojing2002]

We saw some error like:

ataone-indexer 20240424-23:10:35: [INFO]: response httpCode: 200 [org.dataone.service.util.ExceptionHandler:filterErrors:93]
dataone-indexer 20240424-23:10:35: [ERROR]: IndexWorker.indexOjbect - cannot index the task for identifier doi:10.6085/AA/VALXXX_015MTBD009R00_20051117.50.9 since Could not deserialize the org.dataone.service.types.v2.SystemMetadata: null [org.dataone.cn.indexer.IndexWorker:indexOjbect:425]
org.dataone.service.exceptions.ServiceFailure: Could not deserialize the org.dataone.service.types.v2.SystemMetadata: null
	at org.dataone.client.rest.MultipartD1Node.deserializeServiceType(MultipartD1Node.java:873) ~[dataone-index-worker-3.0.0-SNAPSHOT-shaded.jar:?]
	at org.dataone.client.v2.impl.MultipartD1Node.getSystemMetadata(MultipartD1Node.java:96) ~[dataone-index-worker-3.0.0-SNAPSHOT-shaded.jar:?]
	at org.dataone.client.v2.impl.MultipartMNode.getSystemMetadata(MultipartMNode.java:337) ~[dataone-index-worker-3.0.0-SNAPSHOT-shaded.jar:?]
	at org.dataone.cn.indexer.object.ObjectManager.getSystemMetadata(ObjectManager.java:187) ~[dataone-index-worker-3.0.0-SNAPSHOT-shaded.jar:?]
	at org.dataone.cn.indexer.SolrIndex.update(SolrIndex.java:503) ~[dataone-index-worker-3.0.0-SNAPSHOT-shaded.jar:?]
	at org.dataone.cn.indexer.IndexWorker.indexOjbect(IndexWorker.java:396) [dataone-index-worker-3.0.0-SNAPSHOT-shaded.jar:?]
	at org.dataone.cn.indexer.IndexWorker$1$1.run(IndexWorker.java:341) [dataone-index-worker-3.0.0-SNAPSHOT-shaded.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
dataone-indexer 20240424-23:10:35: [INFO]: IndexWorker.consumer.indexOjbect by multiple thread? true, with the thread id 43 - Received the index task from the index queue with the identifier: doi:10.6085/AA/SCPXXX_015MTBD009R0