Update ORT training to be compatible with transformers 4.31

[JingyaHuang]

What does this PR do?

As per title.

• ORTTrainer
• ORTTrainingArguments
• ORTSeq2SeqTrainer
• ORTSeq2SeqTrainingArguments

Maybe in the next PRs

These PRs are for the inference (evaluation / prediction) of trainer APIs. Given that the Trainer APIs are mainly used for training, and the ORT inference can always be done with ORTModel(s) in the optimum library, there is no priority nor eta for the following PRs. If anyone is interested in contributing, please feel free to open a PR and tag me for review. I am willing to handle them, but I have no vision of when I can have the bandwidth for them...

• Improve the export of models
• Support merged decoder
• Register more tasks supported by Optimum right now

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[prathikr]

Related Issue: #1133

[JingyaHuang]

Gently pinging @pacman100 for the context.

Current issue I met with the optimizer while using deepspeed zero stage 2

----------------------------------------------------------------------
Traceback (most recent call last):
  File "/workspace/optimum/test_onnxruntime_train.py", line 135, in test_ort_trainer_encoder
    train_result = trainer.train()
  File "/workspace/optimum/optimum/onnxruntime/trainer.py", line 455, in train
    return inner_training_loop(
  File "/workspace/optimum/optimum/onnxruntime/trainer.py", line 815, in _inner_training_loop
    self.optimizer.step()
AttributeError: 'DummyOptim' object has no attribute 'step'

----------------------------------------------------------------------

[prathikr]

@JingyaHuang any updates?

[kshama-msft]

@JingyaHuang @pacman100 we are waiting for this issue to help unblock integrations within our team. It would be great if this could be fasttracked. Thanks!

[JingyaHuang]

Thanks @pacman100 for helping out!

So @prathikr, @kshama-msft, with the help of @pacman100 from the accelerate team, we are having ORTTrainer compatible with transformers 4.31 and accelerate 0.10. Can you review and also try out my branch to ensure the fix? Thanks!

[prathikr]

Thank you @JingyaHuang this PR resolved my issue. Please merge ASAP.

[kshama-msft]

Thanks Jingya for the prompt fix!