Why is my Learning Rate Curve Flat?

[advaithgit]

Dataset: fashion_mnist

Model Code:

from tensorflow.keras.datasets import fashion_mnist

(train_data,train_labels),(test_data,test_labels)=fashion_mnist.load_data()
class_name=["T-shirt/top","Trouser","Pullover","Dress","Coat","Sandal","Shirt","Sneaker","Bag","Ankle boot"]

train_data.min(),train_data.max()

train_data.min(),train_data.max()

#Since we know min and max value...

traindatanorm1=train_data/255.0
testdatanorm1=test_data/255.0

#Let's find optimalLearning Rate

tf.random.set_seed(42)

model2=tf.keras.Sequential([
tf.keras.layers.Flatten(input_shape=(28,28)),
tf.keras.layers.Dense(100,activation="relu"),
tf.keras.layers.Dense(100,activation="relu"),
tf.keras.layers.Dense(100,activation="relu"),
tf.keras.layers.Dense(10,name="OutputLayer",activation="softmax")
])

model2.compile(loss=tf.keras.losses.SparseCategoricalCrossentropy(),
optimizer=tf.keras.optimizers.Adam(),
metrics=["accuracy"])

lr_scheduler=tf.keras.callbacks.LearningRateScheduler(lambda epoch:1e-3*10**(epoch/20))

norm_history=model2.fit(traindatanorm,train_labels,epochs=40,validation_data=(testdatanorm,test_labels),callbacks=[lr_scheduler])

Epoch 1/40
1875/1875 [==============================] - 7s 3ms/step - loss: 1.1140 - accuracy: 0.5841 - val_loss: 0.7648 - val_accuracy: 0.7444 - lr: 0.0010
Epoch 2/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.6910 - accuracy: 0.7648 - val_loss: 0.6720 - val_accuracy: 0.7741 - lr: 0.0011
Epoch 3/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.6298 - accuracy: 0.7790 - val_loss: 0.6475 - val_accuracy: 0.7752 - lr: 0.0013
Epoch 4/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.6005 - accuracy: 0.7869 - val_loss: 0.6167 - val_accuracy: 0.7864 - lr: 0.0014
Epoch 5/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.5847 - accuracy: 0.7902 - val_loss: 0.6405 - val_accuracy: 0.7779 - lr: 0.0016
Epoch 6/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.5718 - accuracy: 0.7936 - val_loss: 0.6058 - val_accuracy: 0.7901 - lr: 0.0018
Epoch 7/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.5655 - accuracy: 0.7966 - val_loss: 0.5979 - val_accuracy: 0.7889 - lr: 0.0020
Epoch 8/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.5633 - accuracy: 0.7980 - val_loss: 0.5804 - val_accuracy: 0.7968 - lr: 0.0022
Epoch 9/40
1875/1875 [==============================] - 7s 4ms/step - loss: 0.5574 - accuracy: 0.7988 - val_loss: 0.6009 - val_accuracy: 0.7884 - lr: 0.0025
Epoch 10/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.5587 - accuracy: 0.7994 - val_loss: 0.5853 - val_accuracy: 0.7924 - lr: 0.0028
Epoch 11/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.5555 - accuracy: 0.8010 - val_loss: 0.6070 - val_accuracy: 0.7875 - lr: 0.0032
Epoch 12/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.5528 - accuracy: 0.8007 - val_loss: 0.5752 - val_accuracy: 0.7960 - lr: 0.0035
Epoch 13/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.5573 - accuracy: 0.8004 - val_loss: 0.5985 - val_accuracy: 0.7942 - lr: 0.0040
Epoch 14/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.5544 - accuracy: 0.8033 - val_loss: 0.5820 - val_accuracy: 0.7925 - lr: 0.0045
Epoch 15/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.5569 - accuracy: 0.8004 - val_loss: 0.6305 - val_accuracy: 0.7843 - lr: 0.0050
Epoch 16/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.5581 - accuracy: 0.8008 - val_loss: 0.5849 - val_accuracy: 0.7962 - lr: 0.0056
Epoch 17/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.5580 - accuracy: 0.8027 - val_loss: 0.5759 - val_accuracy: 0.8008 - lr: 0.0063
Epoch 18/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.5623 - accuracy: 0.8009 - val_loss: 0.5838 - val_accuracy: 0.7968 - lr: 0.0071
Epoch 19/40
1875/1875 [==============================] - 7s 3ms/step - loss: 0.5643 - accuracy: 0.7997 - val_loss: 0.5968 - val_accuracy: 0.7961 - lr: 0.0079
Epoch 20/40
1875/1875 [==============================] - 5s 3ms/step - loss: 0.5686 - accuracy: 0.7988 - val_loss: 0.6266 - val_accuracy: 0.7896 - lr: 0.0089
Epoch 21/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.5800 - accuracy: 0.7981 - val_loss: 0.6125 - val_accuracy: 0.7832 - lr: 0.0100
Epoch 22/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.5767 - accuracy: 0.7985 - val_loss: 0.6110 - val_accuracy: 0.7891 - lr: 0.0112
Epoch 23/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.5855 - accuracy: 0.7961 - val_loss: 0.8235 - val_accuracy: 0.6991 - lr: 0.0126
Epoch 24/40
1875/1875 [==============================] - 7s 4ms/step - loss: 0.5973 - accuracy: 0.7938 - val_loss: 0.6205 - val_accuracy: 0.7901 - lr: 0.0141
Epoch 25/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.5986 - accuracy: 0.7932 - val_loss: 0.6916 - val_accuracy: 0.7593 - lr: 0.0158
Epoch 26/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.6159 - accuracy: 0.7890 - val_loss: 0.7374 - val_accuracy: 0.7707 - lr: 0.0178
Epoch 27/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.6259 - accuracy: 0.7839 - val_loss: 0.6307 - val_accuracy: 0.7819 - lr: 0.0200
Epoch 28/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.6408 - accuracy: 0.7815 - val_loss: 0.6523 - val_accuracy: 0.7903 - lr: 0.0224
Epoch 29/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.6542 - accuracy: 0.7785 - val_loss: 0.6754 - val_accuracy: 0.7690 - lr: 0.0251
Epoch 30/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.6920 - accuracy: 0.7658 - val_loss: 0.7127 - val_accuracy: 0.7606 - lr: 0.0282
Epoch 31/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.7334 - accuracy: 0.7429 - val_loss: 1.0184 - val_accuracy: 0.6664 - lr: 0.0316
Epoch 32/40
1875/1875 [==============================] - 7s 3ms/step - loss: 0.8681 - accuracy: 0.6802 - val_loss: 1.0220 - val_accuracy: 0.6153 - lr: 0.0355
Epoch 33/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.9163 - accuracy: 0.6411 - val_loss: 0.9461 - val_accuracy: 0.5839 - lr: 0.0398
Epoch 34/40
1875/1875 [==============================] - 6s 3ms/step - loss: 0.9782 - accuracy: 0.6256 - val_loss: 1.1679 - val_accuracy: 0.5173 - lr: 0.0447
Epoch 35/40
1875/1875 [==============================] - 6s 3ms/step - loss: 1.0384 - accuracy: 0.5566 - val_loss: 0.9975 - val_accuracy: 0.5726 - lr: 0.0501
Epoch 36/40
1875/1875 [==============================] - 6s 3ms/step - loss: 1.1233 - accuracy: 0.5116 - val_loss: 1.0799 - val_accuracy: 0.5564 - lr: 0.0562
Epoch 37/40
1875/1875 [==============================] - 7s 4ms/step - loss: 1.2897 - accuracy: 0.4607 - val_loss: 1.3470 - val_accuracy: 0.3795 - lr: 0.0631
Epoch 38/40
1875/1875 [==============================] - 7s 3ms/step - loss: 1.3910 - accuracy: 0.3768 - val_loss: 1.7291 - val_accuracy: 0.2980 - lr: 0.0708
Epoch 39/40
1875/1875 [==============================] - 6s 3ms/step - loss: 1.7454 - accuracy: 0.2235 - val_loss: 1.7503 - val_accuracy: 0.1974 - lr: 0.0794
Epoch 40/40
1875/1875 [==============================] - 6s 3ms/step - loss: 1.7731 - accuracy: 0.1992 - val_loss: 1.7340 - val_accuracy: 0.1989 - lr: 0.0891

import numpy as np
import matplotlib.pyplot as plt

lrs=1e-3*(10**tf.range(40)/20)
plt.semilogx(lrs,norm_history.history["loss"])
plt.xlabel("Learning Rate")
plt.ylabel("Loss")
plt.title("Finding optimal Learning Rate")

Could someone comment on my two screenshots uploaded?