Hackathon 2023 bug in tensorrt 9.0.0

[yuanjiechen]

Reproduce code links: https://drive.google.com/file/d/1Kyw98JTrIokfcXkJBTv71lJOtwGcr2QF/view?usp=drive_link

Environment

If applicable, please include the following:
CPU architecture: x86_64
GPU properties
GPU name: NVIDIA A10
Clock frequencies used: None
Libraries
TensorRT branch: 9.0.0
TensorRT LLM: 0.1.3
Cuda: 12.1.66
Cudnn: unknown
Container: registry.cn-hangzhou.aliyuncs.com%2ftrt-hackathon%2ftrt-hackathon%3afinal_v1
NVIDIA driver version: 525.105.17
OS: Ubuntu 5.15.0-73

Reproduction Steps

Run sh preprocess.sh, build trt fp16 engine and compare with torch outputs

Expected Behaviour

Norm value < 100

Actural Behaviour

Norm value = 1316

Additional Notes

I have 20% time build good enging with norm < 100. I provide build logs in package: good.txt and bad.txt.