final PnL in pips incorrect?

[jjphung]

In an attempt to see PnL in terms of pips when dealing with forex, I stumbled upon this thread: #8. In this thread @kernc suggested: pnl_in_pips = self.position.pl / self.position.size / self.data.pip in order to calculate PnL in terms of pips and I inserted into a strategy along the lines of:

class BaseStrategy(Strategy)
    total_pips_gained = 0
    # ......    
    def next(self):
        super().next()
        exit_portion = self.__exit_signal[-1]
        print("Total pips gained: " ,self.total_pips_gained)
        if exit_portion > 0:
            for trade in self.trades:
                if trade.is_long:
                    pnl_in_pips = self.position.pl / self.position.size / self.data.pip
                    self.total_pips_gained += pnl_in_pips
                    trade.close(exit_portion)
        elif exit_portion < 0:
            for trade in self.trades:
                if trade.is_short:
                    pnl_in_pips = self.position.pl / self.position.size / self.data.pip
                    self.total_pips_gained += pnl_in_pips
                    trade.close(-exit_portion)

With the idea that we'd want to calculate PnL of pips when we close our position.

However, stats = bt.run() outputs Return [%]: 1.6968, while the last print statement for total_pips_gained prints out -1033.8971649107916. Am calculating pnl_in_pips correctly? How is it that return % overall is positive, but total_pips_gained is very negative.

UPDATE:

For shorts: it should be: self.total_pips_gained += (pnl_in_pips * -1)? Since it is the opposite direction! Is this logic correct?

Follow up question: I was not able to find self.data.pip anywhere in the codebase on how it is determined. My forex data is in OHCLV format.

Thank you!

[kernc]

Not sure it's related, but you likely should reset total_pips_gained in init():

    def init(self):
        self.total_pips_gained = 0

Let me know if that does anything.

I was not able to find self.data.pip anywhere in the codebase on how it is determined.

It's here:

backtesting.py/backtesting/_util.py

Lines 141 to 146 in 5e1accc

	@property
	def pip(self) -> float:
	if self.__pip is None:
	self.__pip = 10**-np.median([len(s.partition('.')[-1])
	for s in self.__arrays['Close'].astype(str)])
	return self.__pip

[jjphung]

Thank you!

Let me know if that does anything.

It does not.

[kernc]

What about if you stop dividing by self.position.size?

[jjphung]

Interesting. If I don't divide by self.position.size, pips become an absurd number.. Each trade is in the multi-millions on the EUR_USD pair.

Ex:

SELL
Total pips gained:  -8888523.600199785

[kernc]

Let's extend the simple SmaCross strategy from the tests:

backtesting.py/backtesting/test/_test.py

Lines 54 to 69 in 73e1534

	class SmaCross(Strategy):
	# NOTE: These values are also used on the website!
	fast = 10
	slow = 30
	def init(self):
	self.sma1 = self.I(SMA, self.data.Close, self.fast)
	self.sma2 = self.I(SMA, self.data.Close, self.slow)
	def next(self):
	if crossover(self.sma1, self.sma2):
	self.position.close()
	self.buy()
	elif crossover(self.sma2, self.sma1):
	self.position.close()
	self.sell()

We override it such that self.position.close() calls that the strategy uses also extract and save pnl_in_pips:

from backtesting import Backtest, Strategy
from backtesting.test import GOOG
from backtesting.test._test import SmaCross

class S(SmaCross):
    def init(self):
        super().init()        
        self.total_pips = 0
        
        # Patch self.position.close()
        old_position_close = self.position.close
        def new_position_close(*args, **kwargs):
            if self.position:
                pnl_in_pips = self.position.pl / self.data.pip
                print(pnl_in_pips, self.trades[-1])
                self.total_pips += pnl_in_pips
            old_position_close(*args, **kwargs)
        self.position.close = new_position_close
            
bt = Backtest(GOOG.iloc[:100], S, cash=10_000, trade_on_close=True)
stats = bt.run()
stats._strategy.total_pips
stats._trades
stats[['Equity Final [$]', 'Return [%]']]

With this, I get:

The numbers all add up. (There is one additional trade forcefully closed at the end of the backtest run, and that closing doesn't call our patched method, hence the missing output.)

Notice the use of trade_on_close=True. Without it, I get a slightly different output:

a result of the fact trades aren't closed instantly, but rather with the next bar's open.
bt.plot() confirms.

[jjphung]

Wow thank you for your dedication to answer my question and clarifying. Much appreciated!

[kernc]

For shorts: it should be: self.total_pips_gained += (pnl_in_pips * -1)? Since it is the opposite direction! Is this logic correct?

Best just look in the source or in docs:

backtesting.py/backtesting/backtesting.py

Lines 600 to 604 in 73e1534

	@property
	def pl(self):
	"""Trade profit (positive) or loss (negative) in cash units."""
	price = self.__exit_price or self.__broker.last_price
	return self.__size * (price - self.__entry_price)

size (volume) is the only attribute whose sign corresponds with direction, afaik. And yes, you are using it. 😁