Questions about the dataset split

[nachiket92]

Hi, first of all, congratulations! This (along with Y-net) is very impressive work. Also thank you for making the code available.

I have been working on a related model and am also using / have used the Stanford drone dataset for evaluation. I had a few questions regarding the dataset split used and reported in the PECnet and Y-Net papers so as to conduct a fair comparison.

1. Are the results reported on just pedestrian trajectories in both papers, or are all agents such as bicyclists, skateboarders and vehicles considered? Non-pedestrians tend to be fast moving and can potentially have higher ADE/FDE values. The Y-Net paper suggests that only pedestrians were considered while prior work (including mine - P2TIRL) seems to consider all agents.

2. SDD has a lot of tracks that temporarily go missing/ get occluded. The raw annotation files have a flag indicating this. The annotation files show the last known location for a missing track for all timestamps that the track went missing. From the standpoint of a prediction model, this would effectively be a stationary agent (which should be trivial to the model). Were these agents filtered during evaluation?

3. Finally, the Y-net paper suggests that all short trajectories (shorter than np + nf) were filtered. However np + nf is 35 seconds for training Y-net, while the baselines have all reported results for np + nf = 8 seconds. Are trajectories of lengths 8 to 35 seconds in the test set discarded?

Looking forward to your response!

-- Nachiket

[HarshayuGirase]

Hi Nachiket,

Thanks for your interest and questions!

I believe there might have been some confusion in questions (1) and (3). The PECNet paper considers the standard 3.2s in/4.8s out (8 frames in/12 frames out) prediction setting. For this we use the same data as in the TrajNet challenge and use the same splits as in prior works (S-GAN, SoPhie, etc.). The agents considered are all agents (no filtering). In the Y-net paper we benchmark with this same dataset (all agents considered) for the short term setting.

For our long-term setting (in the Y-net paper), we only consider pedestrians and combine shorter trajectories from the same person into longer ones. If we want to predict for 30 seconds we need to make sure the trajectories in the dataset are at least 35 seconds (5in, 30out) and so discard anything shorter. Please let us know if you still have questions.

Regarding (2) we use the same processed data that was provided. There are a few cases where the agent is actually relatively stationary (standing in a certain location) for the entire trajectory length; if this happens, it is still a valid trajectory.

[nachiket92]

Thanks for the clarifications, this is helpful :)

There's just one small issue that ties in to (1): the TrajNet challenge considers only a small subset of all trajectories in the raw SDD annotation files. Based on my communication with the authors of SoPhie, MATF and CF-VAE, prior work considers all trajectories in the scenes corresponding to the train/val/test sets of TrajNet. This is the setting I followed in P2T_IRL as well.

This is significant mainly because the subset of trajectories used in the TrajNet challenge is almost all pedestrians, while the scenes have a large number of bicyclists, skaters and vehicles. The complete set of trajectories in the test scenes is a slightly more challenging dataset. As things stand, I think our papers may have considered different dataset splits.

That said, I'll be happy to update the P2T paper with two separate result tables for the two different splits.

[os1a]

Hi @nachiket92 @HarshayuGirase,

Since the website of the trajNet benchmark is not working anymore: http://trajnet.stanford.edu/
I am wondering if you can share with me the train/val split of the SDD dataset. I have all training scenes with 31 scenes, but not sure which are used for training and which for validation.

Thanks,

[os1a]

An update to my previous question, I think I figured out that the 31 scenes are all training. But I do not have the validation scenes of the TrajNet benchmark for SDD. Could you please share these files with me?

Thanks,