When does terminated get triggered?

[behradkhadem]

Hello @maxspahn
I'm trying to run a RL algorithm on a sample robotic arm inside this package, (I think I get some results next week) but something caught my eye. When does terminated (previously done) gets triggered? I looked at the code and saw that terminated gets triggered when something gets wrong (exception-wise), but I didn't see any limitations on number of steps for each episode.

Can you elaborate on this matter?

Thanks.

[behradkhadem]

Aaand doesn't it make more sense to use truncated for what we're doing right now with terminated?

[maxspahn]

Hi @behradkhadem ,

Good question, I wasn't sure about the difference, because I don't use it for RL. According to https://www.gymlibrary.dev/api/core/#:~:text=terminated%20(bool)%20%E2%80%93%20whether%20a,of%20the%20MDP%20is%20satisfied., we should use truncated for out of bounds observations and terminated for goal reaching.

[behradkhadem]

Hi @behradkhadem ,

Good question, I wasn't sure about the difference, because I don't use it for RL. According to https://www.gymlibrary.dev/api/core/#:~:text=terminated%20(bool)%20%E2%80%93%20whether%20a,of%20the%20MDP%20is%20satisfied., we should use truncated for out of bounds observations and terminated for goal reaching.

How are we handling goal reaching? And what about the limitations on the maximum number of steps?

[maxspahn]

Goal reaching is not handled so far. Could be done similar to your reward Function. Number of steps is done outside of the environment. Is it common to have that inside the environment, @behradkhadem ?

[behradkhadem]

Goal reaching is not handled so far. Could be done similar to your reward Function. Number of steps is done outside of the environment. Is it common to have that inside the environment, @behradkhadem ?

Ummm we have goals inside examples, can't we use them?
I think that implementing the reward object outside the env has caused computational load in training sessions and I don't want to do the same for the terminated too. And in my limited tests, I was able to increase the reward without implementing the terminated or so, (it took 2.5 hours equivalent to 60000 episodes to increase the reward from 15.3 to 15.8 inside colab!) and I hope I don't need it!
And yes, it is common to do so in order to decrease training time.

[maxspahn]

Ok, sounds that there are two steps to take. I'll create the corresponding issues for that. Feel free to address them.

Best,
Max.

[maxspahn]

Created issues #227 and #228 according to the discussion.