v0.2.0

Full Changelog: v0.1.3...v0.2.0

---

Add Global Normalization

Code: normalization_layer.py

Changes:

1. Use Cumulative Sum and Sum of Squares Across Samples:

   • Instead of relying solely on mean and variance within a single batch, the normalization now utilizes the cumulative sum and sum of squares across all samples.

2. Synchronize Across Nodes After Accumulating Partial Batches:

   • After accumulating a portion of batches, the statistics are synchronized across all nodes to ensure consistency.

3. Update BN Table Statistics Per Batch:

   • For each batch, the statistics in the Batch Normalization (BN) table are updated. The overall mean and variance are computed, and the results are outputted accordingly.

4. Store Mean and Variance in Checkpoint:

   • The mean and variance are stored within the checkpoint, serving as parameters for prediction.
   • The statistics are also stored in HDFS for future reference and usage.

---

Ref

Keras BN
TF Synchronized BN

---

Local Testing