Potentially inefficient pipeline implementation

[mf12ca]

Hi all,

I'm currently trying to dive deeper into the library and I noted something I find quite inefficient about the implementation of the pipeline functionality. It seems to me - please correct me if I am wrong or if there is a way to control this behavior - that the pipeline always pulls all data available for all assets in the database, no matter if a screen is set or not. So if I have a large universe of stocks, but I only ever backtest on 'AAPL' data, say, the pipeline would always pull the data for all stocks first, and then only filter the data for Apple after...

I get that this is the case due to the dependency structure of terms (handled in the graph in the back), but wouldn't it make more sense to calculate only those terms the screen/universe depends on first and then passing down a mask variable that only requires pulling data for the assets actually needed?

Would be great to discuss this a bit since I am currently planning on implementing a graph that handles the dependencies differently, if there is a screen given...

Best,
Chris

[MBounouar]

Hi,
I can't remember if all data for all assets is pulled from the database in one go, but I do agree that a "lazier" approach would make thing more efficient.

By all means, if you have ideas on how to improve the code, please have a go and share.

The issue I often encountered when trying to implement changes into zipline, is that functionalities are often more intertwined than one would imagine in the first place, which makes things quite hard.

Cheers,
MBounouar