Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ensure scalars are float32 in test cases for dropout #174

Merged
merged 1 commit into from
Aug 23, 2024

Conversation

iclementine
Copy link
Collaborator

@iclementine iclementine commented Aug 22, 2024

Ensure scalars are float32 in test cases for dropout. Now dropout is computed in the following formula:

out = tensor * mask / (1-p)

In our test cases, the reference is also compute in thay.

But in some cases, 1-p have different byte representations depending on how it is computed.

  1. (1 - p) when p is a native python floating point number then cast to float32;
  2. (1 - p) when p and 1 are both numpy 32bit floating point number then cast to float32;

Thus, tensor / (1-p) may have have different results when 1-p have different representations.

Copy link
Contributor

@tongxin tongxin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LG

Copy link
Collaborator

@Bowen12992 Bowen12992 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@iclementine iclementine merged commit abbc798 into FlagOpen:master Aug 23, 2024
2 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants