02 - Missing piece non-linearity: Different outcomes with binary crossentropy

[kmirijan]

So, I'm following along with the videos for the classification notebook, and we get to the part where we enable Binary Crossentropy. Now everything worked fine, but I noticed something. I didn't need to define the sigmoid activation on my output layer when using the default Binary_Crossentropy() loss function.

tf.random.set_seed(42)

model_5 = tf.keras.Sequential([
    tf.keras.layers.Dense(16, activation='relu'),
    tf.keras.layers.Dense(4, activation='relu'),
    tf.keras.layers.Dense(1)
])

model_5.compile(loss=tf.losses.BinaryCrossentropy(),
                optimizer='adam',
                metrics=['accuracy'])
history = model_5.fit(X,y,epochs=100)

I got 99.9% test accuracy but my decision boundary plot looked like this.

So, I went back and used a sigmoid activation on the output layer. This required me to set from_logits=true for the binary crossentropy loss, and I still got perfect accuracy, but the decision boundary looked significantly more firm. The code and image is below.

tf.random.set_seed(42)

model_5 = tf.keras.Sequential([
    tf.keras.layers.Dense(16, activation='relu'),
    tf.keras.layers.Dense(4, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

model_5.compile(loss=tf.losses.BinaryCrossentropy(from_logits=True),
                optimizer='adam',
                metrics=['accuracy'])
history = model_5.fit(X,y,epochs=100)

So what is going ne here? I'm using tf 2.9.0, so is there something maybe going on there? Or is something going on with the plotting function. How should I go about exploring this difference?