Training hang detection based on XPU Timer metric.

[BalaBalaYi]

What changes were proposed in this pull request?

1. Implement CheckTrainingHangOperator based on XPU Timer metric to detect training hang.
2. Implement ResolveTrainingHangOperator to determine what to do if training hang.
3. Use limited Deque instead of Queue in DiagnosisDataManager.
4. Add lock on 'diagnosis data' operation.

Why are the changes needed?

Training hang detection POC.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

UT.

[codecov]

Codecov Report

Attention: Patch coverage is 97.27520% with 10 lines in your changes missing coverage. Please review.

Project coverage is 81.02%. Comparing base (a6153a6) to head (3c0a334).
Report is 3 commits behind head on master.

Files with missing lines	Patch %	Lines
...eoperator/observer/check_training_hang_operator.py	95.94%	3 Missing ⚠️
dlrover/python/master/diagnosis/diagnosis.py	93.54%	2 Missing ⚠️
...nosis/inferencechain/inferenceoperator/operator.py	88.88%	1 Missing ⚠️
...perator/resolver/resolve_training_hang_operator.py	95.00%	1 Missing ⚠️
dlrover/python/master/node/dist_job_manager.py	87.50%	1 Missing ⚠️
dlrover/python/tests/test_diagnosis_coordinator.py	95.65%	1 Missing ⚠️
dlrover/python/tests/test_inference.py	99.20%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1288      +/-   ##
==========================================
+ Coverage   80.86%   81.02%   +0.15%     
==========================================
  Files         229      230       +1     
  Lines       21540    21788     +248     
==========================================
+ Hits        17418    17653     +235     
- Misses       4122     4135      +13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.