We should use relative comparing in testing model

[tk26eng]

blueoil/blueoil/converter/templates/include/dlk_test.h

Lines 41 to 57 in 107c4fc

	static const T_FLOAT tolerance = 0.00001;
	template<class T_IN, class T_RES>
	inline bool same(T_IN input, T_RES result, T_IN& diff)
	{
	T_INT diff_ = input - T_IN(result);
	diff = diff_;
	return diff_ == 0;
	}
	template<>
	inline bool same(T_FLOAT input, T_FLOAT result, T_FLOAT& diff)
	{
	T_FLOAT diff_ = std::abs(input - result);
	diff = (tolerance > diff_) ? 0 : diff_;
	return diff == 0;
	}

In this above source, absolute error is used in comparing ground truth values and calculated values for elf binaries like lm_x86.elf. This comparison with absolute error works well when the values are normalized and range from 0.0 to 1.0.
But sometimes in some network, they are not normalized and range from 0 to 255 especially when output is image data. Then the absolute error of 0.00001 is too strict (small) for 32bit floating point which is valid within about 6 to 7 digit by decimal value.

Maybe, absolute error of 0.00001 is OK for lm_x86.elf because it doesn't use 32bit floating point.
But other elf binaries will use SIMD operations with 32bit floating point in some functions for faster calculation. So they might occur error of validation.

For example, the ground truth is 100.0 and calculated value is 100.000005, originally the test should pass. However, the comparison will fail because 32bit floating point has accuracy of 0.0001 (6digits of 100.0).

So we should use relative error for this problem.
This change from absolute error to relative one is very easy to modify. But this part has big effect to validation of model.

What do you think about changing absolute error to relative one ?

[primenumber]

We should consider both relative and absolute error, because of loss of significance...

[tk26eng]

@primenumber
How about using threshold like this ?

float val_gt; // ground truth
float val_calc; // calculated value
// Certain value is set to val_gt and val_calc here
abs_tol = 0.00001; // Same threshold as original code for smaller value
rel_tol = 0.00001; // 6 digits should be matched for bigger value
if(std::abs(val_calc - val_gt) < max(abs_tol, rel_tol * val_gt)){
  std::cout << "value is same" << std::endl;
}
else{
  std::cout << "value is different" << std::endl;
}