Future direction for backtesting.py

[eervin123]

@kernc what direction would you like to see backtesting.py head in the future?

Examples:

• Multiple assets for 'portfolio' capabilities, vs. single asset strategy but wicked fast and robust,
• next-level machine learning and optimizations,
• improved visualization, and interactivity to better analyze different backtest strategies,
• going from strategy backtesting to live trading deployment.
• etc.

I suppose I'm asking about the "outcome" you are hoping for and the ideal "user" I know based on some of your answers to different questions or issues generally where you are guiding it, but I don't think I've ever seen you share your thoughts explicitly perhaps because you don't quite know yet, you are letting it grow organically based on users asking for features, which is fine as well.

I just ask because I see some similarities between different frameworks eg. bt and backtesting so I'm curious which features you would like to work on incorporating and which features you would view as superfluous to the core objective of backtesting.py

[kernc]

I'm looking forward to seeing #81 implemented. It's an important aspect and seems a fairly low-hanging fruit relative to potential value.

• Multiple assets for 'portfolio' capabilities, vs. single asset strategy but wicked fast and robust,

Single-asset strategy but wicked fast and simple is what we sell by. Undecided on API in #20. #104 is related. I feel bt maybe already does a good job at portfolios.

• next-level machine learning and optimizations,

High-level API for general optimization (maximization) like we have, sure. But specific ML methods are out-of-scope, and it should rather be made easy to plug in your own.

• improved visualization, and interactivity to better analyze different backtest strategies

Insightful visualizations are invaluable! I guess whatever fits in backtesting.lib.

• going from strategy backtesting to live trading deployment.

#81 would be nice to have.

[eervin123]

Single-asset strategy but wicked fast and simple is what we sell by. Undecided on API in #20. #104 is related. I feel bt maybe already does a good job at portfolios.

Having just worked a bit with bt, and it's cousin ffn, I tend to agree, for quick work with portfolios bt is pretty robust. Though, I think you @kernc could do a much better job creating interesting visualizations. (note I haven't really thought much about what is missing, but the more I work with them both I think I'll be able to zero in on some concrete ideas.)

I've been working on some projects where I flip back and forth using all three libraries in the workflow. backtesting.py to build out single asset strategies, ffn to better explore the strategy performance and present findings, then bt to build and optimize a portfolio of strategies. It would be nice if some things carried over like trades, and equity, etc but the combined analysis has to be done manually after the fact by combining the different dataframes from each strategy.

I've been meaning to share this stuff with you but currently they have a bunch of proprietary stuff tangled up in them so I haven't gotten around to creating generic versions. When I do, I think you will be able to have a look and quickly identify areas that might be low hanging fruit for you to incorporate into backtesting.py.

[garyfritzz]

I started using Tradestation in 1997. Used it successfully for many years -- but I eventually dropped it because the single-asset model is too limiting. I want to work with baskets of symbols, rebalance the basket, scan through a list of symbols to decide which to hold now, hold long/short positions, etc.

I like the look of backtesting.py. But if it only tests one symbol, it's not very useful to me.

[qacollective]

I was originally attracted to backtesting.py because of its simple, clean and well focused design and implementation and the great interactive output graph. So I know this may be heracy (or out of scope - same thing 😆 ), but I'd suggest the idea of a tick level trading mode for greater accuracy when testing strategies that rely very much on tick level data. Either that or perhaps make it more explicit that this won't be super accurate on tick level strategies ...

My story:
I did see a this comment and this comment by @kernc on issues suggesting that setting O=H=L=C could approximate tick level data.

So, I have made a stub for our live trading platform that redirects all autotrader actions to use backtesting.py in stead of going on market.

Following on from my work on #83, I set out on comparing 1 full day of actual past trading on market with the results that backtesting.py produced, being sure to use the exact input tick data that was recorded by the same trading platform on that day...

...the results of this analysis have been quite impressive. Trades that actually occurred on market appear in the list of backtesting.py trades with almost second accurate entry and exit timestamps 👍

Unfortunately though, in my setup, there is no escaping: even though a trading algorithm might make decisions based on OHLCV data, the real decision point on whether a broker will convert an order to a trade is made on tick data: datetime, bid, ask.

As a result, there are always more trades produced that don't reflect what actually happened on that day. This still happens even when I reduce the granularity of OHLCV data to 1 second. Further investigation revealed that backtesting.py is working as expected, but that despite what I read in the issue comments linked above, setting O=H=L=C doesn't seem to produce a set of trades that would actually happen in a production scenario (nor does setting OHLC as b+a/2,ask,bid,b+a/2 nor other variants).

The most accurate I could get backtesting.py was to set O=H=L=C at the tick level and then run 2 backtests where O=H=L=C was the bid price and another where O=H=L=C was the ask price and then discard the buys and sells respectively and consolidate. Of course, that's madness - and also ignoring the fact that if this were scaled up past, say one week or month, Bokeh would likely fall over trying to graph this and probably backtesting.py shortly after, requiring all pandas data to be loaded into memory.

So ... I've unfortunately had to stop using backtesting.py. 😭 I almost laughed when I started 'rolling my own' as per the backtesting.py front page ... but as a POC, it got results ... it gets near millisecond accurate trade entry/exit times and 100% accurate trade deals done when comparing it with actual production data and it runs so fast, I had to stop using Pandas because it was too damn slow.

What this all could mean for backtesting.py is uncertain, so I found this discussion and typed all this up. I half considered re-writing the backtesting.py broker, but of course its bigger than that ... its also changes to data structures (although I could dynamically produce an OHLC for downstream reporting input?) and data input has to be streamed, as I don't have enough RAM to fit in 400,000,000 rows 😃

These are just my ponderings, do with them what you will.

[crazy25000]

Adding examples with memory metrics to the documentation for users would be helpful, not add it to the project.

[kernc]

I agree the iterative-stitching-results approach would be helpful to have an example of. Have you got it working?

Note, the examples are tested:

backtesting.py/backtesting/test/_test.py

Lines 907 to 913 in 73e1534

	def test_examples(self):
	examples = glob(os.path.join(self.DOCS_DIR, 'examples', '*.py'))
	self.assertGreaterEqual(len(examples), 4)
	with chdir(gettempdir()):
	for file in examples:
	with self.subTest(example=os.path.basename(file)):
	run_path(file)

and test dependencies installed:

backtesting.py/setup.py

Lines 46 to 51 in 73e1534

	'test': [
	'seaborn',
	'matplotlib',
	'scikit-learn',
	'scikit-optimize',
	],

so it would make sense to iter over the built-in toy example data. I certainly don't want anything crazy like SQL! 🤡

[crazy25000]

Ran into an issue with moving compute_stats to lib.py, it causes a circular import since it imports Strategy

backtesting.py/backtesting/lib.py

Line 23 in 73e1534

from .backtesting import Strategy

I'm trying to keep the current functionality here so that it still calculates the stats + give us the ability to explicitly call compute_stats from lib.py.

backtesting.py/backtesting/backtesting.py

Line 1180 in 73e1534

self._results = self._compute_stats(broker, strategy)

I will look at it again tonight.

[crazy25000]

Unless we're okay with removing it from self._results = self._compute_stats(broker, strategy) and just let those who want the stats to call it only if needed.

[kernc]

A way to break circular imports is to import locally. I think it'd be reasonable to refactor stats code, now that's it's grown, into its own _stats.py file and import the new function where needed. Best to discuss in #194, though.