Skip to content
This repository has been archived by the owner on Jan 26, 2022. It is now read-only.

Handle NaN loss #110

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Handle NaN loss #110

wants to merge 4 commits into from

Conversation

MFattouh
Copy link

To avoid division by zero or passing zeros or negative values to log in bbox_transform_inv I replace those values with cfg.EPS. In addition, the NaN mask loss which rises after a negative area calculation is handled by discarding the entire batch.

@heinzermch
Copy link

This might be a tempting fix, but in my opinion if you have a NaN loss somewhere, then there is a serious problem in your network which deserves a deeper investigation.

There is probably a good reason why FAIR did not implement it that way in the first place.

@MFattouh
Copy link
Author

well, you're absolutely right. NaN should not "normally" appear. Otherwise, A deeper investigation is required. Yet, this could help if NaN loss somehow appeared a few times. A warning message would help to notify the user in such cases.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants