tsdf.show() or display(tsdf) only displays the first 5 rows

[nikhilmakan02]

Could somebody tell me if I am missing something here. Really simple code I am using in a Databricks notebook.
Tempo version 0.1.23

%pip install dbl-tempo

from tempo import *
import pandas as pd

df = pd.DataFrame({'ts':pd.date_range(start='2018-04-24 00:00:00', end='2018-04-25 12:00:00',freq='1H')}).assign(val=3)
dfs = spark.createDataFrame(df)
dfs.show() # -----> This shows me 20 rows

test_tsdf = TSDF(dfs, ts_col="ts")
test_tsdf.show(20) # -----> This only shows 5 rows.

display(test_tsdf) # -----> This only shows 5 rows.

Any thoughts?

[tnixon]

Oh, that's strange. Yes, this is definitely inconsistent behavior @nikhilmakan02 - thanks for letting us know!
I'll see if I can put together a quick fix for this so that the show and display methods are more consistent with the standard Spark DF show method.

[nikhilmakan02]

Thanks @tnixon I have renamed the issue to more accurately describe the problem as I initially thought it was the conversion from a Spark Dataframe to TSDF that only returned 5 rows.

Workaround is just calling the '.df.show()' on the tsdf object and it works fine.

Thanks.

[Melissari1997]

Hi @tnixon
I took a look at the repo for a personal project and I would like to try to contribute
Do you mind if I try to solve this problem?

[tnixon]

Hi @Melissari1997 - if you want to fix the issue, please go ahead. We welcome outside contributions. Just submit a PR and we'll take a look at it.