Streaming CSV processing/plotting

[breznak]

Fixes #26

• CSV parser (and dygraph) use streaming mode (in batches of BUFFER_SIZE=1000)
• cleanup code
• faster, less browser-killing loading of large CSV files
• most of work for I want to be able to visualize (streaming) data online, as the (NuPIC) model is running  #17

[jefffohl]

@breznak - I ran a test of this, and it did not seem to work. When I tried to load a file nothing happens, and I got a lot of stuff logged to the console.

[breznak]

I'm not sure what changed (broken merge, the newer libs) but I'm getting this error too. Will try to investigate.

[breznak]

This needs some more love, but I got it working.

• Not sure if really much faster, but should hog less the browser on large files.
  • we should validate with a 2M points file
• a strange error!: opening a 2nd file keeps the CSV header settings from the former, thus those errors on "Mismatched count ...". A solution is refresh the browser. @jefffohl any idea how to cleanup properly?
• after each step(s) when you see console "draw", I'd like to paint a new graph. Why does it not happen? Or is the rendering just that slow so a new batch arrives before it finished?

When ready, this could be a good way to #17

[breznak]

@jefffohl @rhyolight can you take a look at c2c14a7 please? I want to load a small(er) batch, render it quickly, repeate.
But I don't know how exactly to redraw the graph?
Also, do DyGraphs support something like updates? That I keep what is rendered (just shrink it) and append a few of new data?

[jefffohl]

@breznak You can update the graph data by calling the updateOptions function. You can see this being called here: https://github.com/nupic-community/nupic.visualizations/blob/master/client/src/app/appCtrl.js#L192

I can start working on this if you would like - but I don't want to work in parallel with you - so let me know.

[jefffohl]

FYI - here is the Dygraphs API reference: http://dygraphs.com/jsdoc/symbols/Dygraph.html

[breznak]

I can start working on this if you would like - but I don't want to work in parallel with you - so let me know.

That would be great @jefffohl ! I'll take a break and sleep now :) DyGraph.updateOption looks like what we need - called from Papa.step() with the small batch payload. Would be awesome if we can get the "streaming data" done! For now I think we could assume NuPIC runs slower than the data appears, so we don't have to bother with polling and sleeps.

[jefffohl]

@breznak - I started on this today, but didn't get too far yet. I will work on it again tomorrow, unless you are eager to jump back in. If so - let me know.

[jefffohl]

@breznak - due to requests from @rhyolight i am prioritizing #40. i still plan on working on this though. let me know your thoughts.

[jefffohl]

@breznak PR #54 could possibly replace this. Thoughts?

[breznak]

@jefffohl thanks for #54 (you made a PR to merge into this PR, not master, which confused me..sorry.) Anyway, this is your work from #54.
I've added a new example file examples/CSV/huge_2M.csv and it uncovered a bug in the current implementation:

• the first chunk seems to determine the x-axis - reading data > CHUNK_SIZE does not extend the axis, instead, the data is written over the existing one.

[jefffohl]

@breznak - ok, taking a look.

[jefffohl]

@breznak - this is happening only for files without a timestamp. Working on a fix.

One other thing I noticed with very large files is that there is a limit as to what DyGraphs can handle. My system locked up trying to render the 60MB file. For files this large, it seems we will need to figure out a way to enforce windowing or something.

[breznak]

this is happening only for files without a timestamp. Working on a fix.

Thanks @jefffohl ! I've noticed another thing, when the timestamp is not monotonic the graph is rendered over the exiting values (i think that is an OK behavior, as we expect time to be monotonically increasing only)- but this happens when you append 2 files, maybe we could raise a warning "Your timestamp is not monothonic." ?

My system locked up trying to render the 60MB file. For files this large, it seems we will need to figure out a way to enforce windowing or something.

• maybe one approach could be Preprocess/Compress when too much data, so the presentation is clear #43
• we already have such functionality - the appConfig.SLIDING_WINDOW, so we can set some var like MAX_POINTS and if the actual points is >, just change the sliding window and switch from "append" to "slide" mode? (also trigger a warning on this occasion?)

[jefffohl]

I made a new PR to the speedup_csv_parse branch from my version of that branch, which fixes the problem of series without timestamps overwriting the iteration value.

[jefffohl]

@breznak - I am going to merge this into master, if that is OK with you. For your recent comments regarding enforced windowing, and monotonic alerts, lets make some new issues for those.

[breznak]

Thank you Jeff! merged the branch and I'll make the issues.