Issue in transceiver eeprom during OIR

[nazeerhussainf]

problem with "show interfaces transceiver eeprom -d"

We now have .06 installed on the switch and I have found a new, but seemingly related, issue.

When we installed .06 and rebooted/reloaded the switch, the eeprom data from the ONet AOCs seemed to be correct. Then, today, I continued with my testing and started getting test failures during out hotswap test. This is the test where we check the function of the optics before and after unplugging and re-inserting the optics.

Here is an example of what is happening.
Before unplugging the optic we have what looks like correct data.
Ethernet72: SFP EEPROM detected
Application Advertisement: 400GAUI-8 C2M (Annex 120E) - Active Cable assembly with BER < 2.6x10^-4 200GAUI-4 C2M (Annex 120E) - Active Cable assembly with BER < 2.6x10^-4
Connector: No separable connector
Encoding: Not supported
Extended Identifier: Power Class 1(10.0W Max)
Extended RateSelect Compliance: Not supported
Identifier: QSFP-DD Double Density 8X Pluggable Transceiver
Length Cable Assembly(m): 3.0
Nominal Bit Rate(100Mbs): Not supported
Specification compliance: active_cable_media_interface
Vendor Date Code(YYYY-MM-DD Lot): 2021-02-05 01
Vendor Name: O-NET
Vendor OUI: 34-78-77
Vendor PN: 1AT-5QAM03XX-10A
Vendor Rev: A
Vendor SN: 4QA-0000022
ChannelMonitorValues:
RX1Power: 1.4476dBm
RX2Power: 1.4826dBm
RX3Power: 1.502dBm
RX4Power: 1.3437dBm
RX5Power: 1.3792dBm
RX6Power: 1.4461dBm
RX7Power: 1.2031dBm
RX8Power: 1.213dBm
TX1Bias: 5.986mA
TX1Power: 1.9011dBm
TX2Bias: 6.27mA
TX2Power: 2.1024dBm
TX3Bias: 6.178mA
TX3Power: 2.0382dBm
TX4Bias: 6.104mA
TX4Power: 1.986dBm
TX5Bias: 6.058mA
TX5Power: 1.9529dBm
TX6Bias: 6.152mA
TX6Power: 2.02dBm
TX7Bias: 5.892mA
TX7Power: 1.8324dBm
TX8Bias: 6.082mA
TX8Power: 1.97dBm
ChannelThresholdValues:
RxPowerHighAlarm : 4.7699dBm
RxPowerHighWarning: 3.9799dBm
RxPowerLowAlarm : -6.9919dBm
RxPowerLowWarning : -6.0206dBm
TxBiasHighAlarm : 9.0000mA
TxBiasHighWarning : 8.5000mA
TxBiasLowAlarm : 4.0000mA
TxBiasLowWarning : 4.5000mA
TxPowerHighAlarm : 4.7699dBm
TxPowerHighWarning: 3.9799dBm
TxPowerLowAlarm : -6.9919dBm
TxPowerLowWarning : -6.0206dBm
ModuleMonitorValues:
Temperature: 43.3555C
Vcc: 3.3803Volts
ModuleThresholdValues:
TempHighAlarm : 75.0000C
TempHighWarning: 70.0000C
TempLowAlarm : -5.0000C
TempLowWarning : 0.0000C
VccHighAlarm : 3.5700Volts
VccHighWarning : 3.4650Volts
VccLowAlarm : 3.0400Volts
VccLowWarning : 3.1350Volts

Now, I unplug the optics and re-insert them.
Afterward, the data from the optic shows changes.

Ethernet72: SFP EEPROM detected
Application Advertisement: 400GAUI-8 C2M (Annex 120E) - Active Cable assembly with BER < 2.6x10^-4 200GAUI-4 C2M (Annex 120E) - Active Cable assembly with BER < 2.6x10^-4
Connector: N/A
Encoding: Not supported
Extended Identifier: N/A
Extended RateSelect Compliance: Not supported
Identifier: N/A
Length Cable Assembly(m): N/A
Nominal Bit Rate(100Mbs): Not supported
Specification compliance:
N/A
Vendor Date Code(YYYY-MM-DD Lot): N/A
Vendor Name: N/A
Vendor OUI: 34-78-77
Vendor PN: 1AT-5QAM03XX-10A
Vendor Rev: A
Vendor SN: 4QA-0000022
MonitorData:
RXPower: 1.3985dBm
TXBias: 5.978mA
TXPower: 1.919dBm
Temperature: 41.7695C
Vcc: 3.3795Volts
ThresholdData:
TempHighAlarm : 75.0000C
TempHighWarning: 70.0000C
TempLowAlarm : -5.0000C
TempLowWarning : 0.0000C
VccHighAlarm : 3.5700Volts
VccHighWarning : 3.4650Volts
VccLowAlarm : 3.0400Volts
VccLowWarning : 3.1350Volts
RxPowerHighAlarm : 4.7699dBm
RxPowerHighWarning: 3.9799dBm
RxPowerLowAlarm : -6.9919dBm
RxPowerLowWarning : -6.0206dBm
TxBiasHighAlarm : 9.0000mA
TxBiasHighWarning : 8.5000mA
TxBiasLowAlarm : 4.0000mA
TxBiasLowWarning : 4.5000mA
TxPowerHighAlarm : 4.7699dBm
TxPowerHighWarning: 3.9799dBm
TxPowerLowAlarm : -6.9919dBm
TxPowerLowWarning : -6.0206dBm

So, it looks like there remains some instability in the I2C reads in this switch.
Also, the whole structure of the output changes.

Things I know:

It doesn’t always happen the same way. Different fields can be affected on different iterations.
It seems that rebooting has a pretty good chance of clearing up the problem. I am not 100% sure this works all the time.
It happens on more than one port, but doesn’t seem to be guaranteed to happen on any given port.
Things I don’t know:

I don’t know if optics from any other vendor show the same behavior.
I don’t know if it happened before on older version of OS, because we had problems with the before measurement
I don’t know if rebooting solves the problem all the time.