Merge pull request #1327 from bug-or-feature/doc_links

Docs improvements

docs/backtesting.md

@@ -5,6 +5,7 @@ Related documents:
 - [Storing futures and spot FX data](/docs/data.md)
 - [Using pysystemtrade as a production trading environment](/docs/production.md)
 - [Connecting pysystemtrade to interactive brokers](/docs/IB.md)
+- [Recent undocumented changes](/docs/recent_changes.md)
 
 
 This guide is divided into four parts. The first ['How do I?'](#how_do_i)

docs/installation.md

@@ -2,7 +2,7 @@
 
 ## Introduction
 
-This project has some quirks relating to installation and dependencies. Mostly, they're to do with the reliance on Arctic. That project has very specific requirements with some versions of dependencies which are becoming seriously out of date. It makes most sense to have a specific Python installation for this project, with the dependencies isolated from those used by other projects. This guide shows the quickest and easiest way to manage that
+This guide shows the quickest and easiest way to install the project in a virtual environment
 
 
 ## pyenv
@@ -15,12 +15,12 @@ Installation instructions for pyenv are here:
 
 https://github.com/pyenv/pyenv#installation
 
-## Python 3.8
+## Python 3.10
 
-pysystemtrade currently requires Python 3.8, so once pyenv is installed, the first step is to get that. Get the latest 3.8.x version, at the time of writing it is 3.8.16
+pysystemtrade currently requires Python 3.10, so once pyenv is installed, the first step is to get that. Get the latest 3.10.x version, at the time of writing it is 3.10.13
 
 ```
-$ pyenv install 3.8.16
+$ pyenv install 3.10
 ```
 
 Once complete you should be able to see the new version in the output of `pyenv versions`
@@ -33,11 +33,12 @@ $ pyenv versions
   3.8.6
   3.8.10
   3.8.13
-* 3.8.15
+  3.8.15
   3.8.16
   3.9.6
   3.9.13
   3.10.4
+* 3.10.13
 ```
 
 Your output will be different, it's just an example
@@ -59,47 +60,46 @@ otherwise, you'll want the main repo
 git clone https://github.com/robcarver17/pysystemtrade.git
 ```
 
-Now we will want to let pyenv know that we want to use Python 3.8 for this project
+Now we will want to let pyenv know that we want to use Python 3.10 for this project
 
 ```
 cd pysystemtrade
-pyenv local 3.8.16
+pyenv local 3.10.13
 ```
 
-this creates a file at the top level of the project `.python-version` that lets the Python execution environment know to use version 3.8.16. We can check this by running python
+this creates a file at the top level of the project `.python-version` that lets the Python execution environment know to use version 3.10.13. We can check this by running python
 
 ```
 $ python
-Python 3.8.16 (default, Mar 19 2023, 11:38:42) 
-[Clang 14.0.0 (clang-1400.0.29.202)] 
+Python 3.10.13 (main, Nov 27 2023, 11:13:49) [Clang 14.0.0 (clang-1400.0.29.202)]
 Type "help", "copyright", "credits" or "license" for more information.
 >>> 
 < ctrl-D to exit >
 ```
 
 ## venv
 
-https://docs.python.org/3.8/library/venv.html
+https://docs.python.org/3.10/library/venv.html
 
-Now we want to create a virtual env (venv) for the project. Doing this will keep all the dependencies for pysystemtrade (some of which are pretty old) separate from your other python projects
+Now we want to create a virtual env (venv) for the project. Doing this will keep all the dependencies for pysystemtrade separate from your other python projects
 
 ```
-$ python -m venv venv/3.8.16
+$ python -m venv venv/3.10.13
 ```
 
 This will create a brand new, isolated Python environment *inside the pysystemtrade project* at the directory
-`<your_path>/pysystemtrade/venv/3.8.6`. You can give your environment any name (the *venv/3.8.6* bit).
+`<your_path>/pysystemtrade/venv/3.10.13`. You can give your environment any name (the *venv/3.10.13* bit).
 
 Now activate the virtual environment
 
 ```
-source venv/3.8.16/bin/activate
+source venv/3.10.13/bin/activate
 ```
 
 Once your virtual env is activated, the prompt will change. It will look something like 
 
 ```
-(3.8.16) $
+(3.10.13) $
 ```
 This reminds you that you're in a venv. (You can exit the venv at any time by running `deactivate`)
 
@@ -109,29 +109,15 @@ This reminds you that you're in a venv. (You can exit the venv at any time by ru
 Now it's time to start setting up the venv. First check to see what is there 
 
 ```
-(3.8.16) $ pip list
+(3.10.13) $ pip list
 ```
 
 You will probably be prompted to update pip at this time. Do whatever command it suggests.
 
-Now install *wheel*
-
-```
-(3.8.16) $ pip install wheel
-```
-
-### Linux, Windows, MacOS (Intel)
-
-Install *cython* 
-
-```
-(3.8.16) $ pip install cython
-```
-
 And now install the dependencies
 
 ```
-(3.8.16) $ pip install -r requirements.txt
+(3.10.13) $ pip install -r requirements.txt
 ```
 
 ### MacOS (ARM)
@@ -141,88 +127,79 @@ If you're running MacOS on one of the new ARM chips, the process is more complex
 ```
 chmod u+x install_dependencies_apple_silicon.sh
 ./install_dependencies_apple_silicon.sh
+
 ```
 Note: this may (unfortunately) become out of date and require some tweaking.
 
-### Check dependencies, all OSs
+### Check dependencies
 
 Check what is installed, should look something like
 
 ```
-(3.8.16) $ pip list
-Package               Version
---------------------- -----------
-arctic                1.79.2
-attrs                 22.2.0
-backports.zoneinfo    0.2.1
-click                 8.1.3
-contourpy             1.0.7
-cycler                0.11.0
-Cython                0.29.33
-decorator             5.1.1
-enum-compat           0.0.3
-eventkit              1.0.0
-exceptiongroup        1.1.1
-Flask                 2.2.3
-fonttools             4.39.2
-ib-insync             0.9.70
-importlib-metadata    6.1.0
-importlib-resources   5.12.0
-iniconfig             2.0.0
-itsdangerous          2.1.2
-Jinja2                3.1.2
-kiwisolver            1.4.4
-lz4                   4.3.2
-MarkupSafe            2.1.2
-matplotlib            3.7.1
-mockextras            1.0.2
-nest-asyncio          1.5.6
-numpy                 1.23.5
-packaging             23.0
-pandas                1.0.5
-patsy                 0.5.3
-Pillow                9.4.0
-pip                   23.0.1
-pluggy                1.0.0
-psutil                5.6.6
-pymongo               3.9.0
-pyparsing             3.0.9
-PyPDF2                3.0.1
-pytest                7.2.2
-python-dateutil       2.8.2
-pytz                  2022.7.1
-pytz-deprecation-shim 0.1.0.post0
-PyYAML                5.4
-scipy                 1.10.1
-setuptools            56.0.0
-six                   1.16.0
-statsmodels           0.12.2
-tomli                 2.0.1
-typing_extensions     4.5.0
-tzdata                2022.7
-tzlocal               4.3
-Werkzeug              2.2.3
-wheel                 0.40.0
-zipp                  3.15.0
+(3.10.13) % pip list
+Package         Version
+--------------- ------------
+blinker         1.7.0
+click           8.1.7
+contourpy       1.2.0
+cycler          0.12.1
+eventkit        1.0.3
+exceptiongroup  1.2.0
+Flask           3.0.1
+fonttools       4.47.2
+ib-insync       0.9.86
+iniconfig       2.0.0
+itsdangerous    2.1.2
+Jinja2          3.1.3
+joblib          1.3.2
+kiwisolver      1.4.5
+MarkupSafe      2.1.4
+matplotlib      3.8.2
+nest-asyncio    1.6.0
+numpy           1.26.3
+packaging       23.2
+pandas          2.1.3
+patsy           0.5.6
+pillow          10.2.0
+pip             23.3.2
+pluggy          1.3.0
+psutil          5.6.6
+pyarrow         15.0.0
+pymongo         3.11.3
+pyparsing       3.1.1
+PyPDF2          3.0.1
+pytest          7.4.4
+python-dateutil 2.8.2
+pytz            2023.3.post1
+PyYAML          5.3.1
+scikit-learn    1.4.0
+scipy           1.12.0
+setuptools      65.5.0
+six             1.16.0
+statsmodels     0.14.0
+threadpoolctl   3.2.0
+tomli           2.0.1
+tzdata          2023.4
+Werkzeug        3.0.1
 ```
 
 ## pysystemtrade
 
 And finally, install the project itself
 
 ```
-(3.8.16) $ python setup.py develop
+(3.10.13) $ python setup.py develop
 ```
 
 Check stuff works
 
 ```
-(3.8.16) $ python
+(3.10.13) $ python
 >>>
 >>> from sysdata.sim.csv_futures_sim_data import csvFuturesSimData
+Configuring sim logging
 >>> data=csvFuturesSimData()
-2023-03-19 12:29:18 {'type': 'csvFuturesSimData'} [Warning] No datapaths provided for .csv, will use defaults  (may break in production, should be fine in sim)
 >>> data
-csvFuturesSimData object with 208 instruments
+csvFuturesSimData object with 249 instruments
 >>> 
 ```

docs/production.md

@@ -18,6 +18,7 @@ And documents you should read after this one:
 - [Instruments](/docs/instruments.md)
 - [Dashboard and monitor](/docs/dashboard_and_monitor.md)
 - [Production strategy changes](/docs/production_strategy_changes.md)
+- [Recent undocumented changes](/docs/recent_changes.md)
 
 *IMPORTANT: Make sure you know what you are doing. All financial trading offers the possibility of loss. Leveraged trading, such as futures trading, may result in you losing all your money, and still owing more. Backtested results are no guarantee of future performance. No warranty is offered or implied for this software. I can take no responsibility for any losses caused by live trading using pysystemtrade. Use at your own risk.*
 
@@ -50,9 +51,15 @@ Table of Contents
       * [Echos: stdout output](#echos-stdout-output)
          * [Cleaning old echo files](#cleaning-old-echo-files)
       * [Logging](#logging)
+      * [socket](#socket)
+         * [socket server as a service](#socket-server-as-a-service)
+      * [console](#console)
+      * [email](#email)
          * [Adding logging to your code](#adding-logging-to-your-code)
-         * [Getting log data back](#getting-log-data-back)
+         * [Examples](#examples)
          * [Cleaning old logs](#cleaning-old-logs)
+            * [Echos](#echos)
+            * [Logs](#logs)
       * [Reporting](#reporting)
 * [Positions and order levels](#positions-and-order-levels)
    * [Instrument level](#instrument-level)
@@ -61,12 +68,9 @@ Table of Contents
 * [The journey of an order](#the-journey-of-an-order)
    * [Optimal positions](#optimal-positions)
       * [Optimal position for roll orders](#optimal-position-for-roll-orders)
-      * [Optimal positions for intra-instrument spread orders](#optimal-positions-for-intra-instrument-spread-orders)
-      * [Optimal positions for intra-instrument spread orders](#optimal-positions-for-intra-instrument-spread-orders-1)
    * [Strategy order handling](#strategy-order-handling)
       * [Instrument orders in detail:](#instrument-orders-in-detail)
       * [Strategy order handling for roll orders](#strategy-order-handling-for-roll-orders)
-      * [Strategy order handling for spread orders](#strategy-order-handling-for-spread-orders)
       * [Overrides](#overrides)
    * [Stack handler](#stack-handler)
    * [Instrument order netting (to be implemented)](#instrument-order-netting-to-be-implemented)
@@ -75,10 +79,6 @@ Table of Contents
       * [Contract order creation - conditional orders](#contract-order-creation---conditional-orders)
       * [Contract order creation - passive roll status](#contract-order-creation---passive-roll-status)
       * [Instrument and contract order creation - active roll orders](#instrument-and-contract-order-creation---active-roll-orders)
-      * [Contract order creation: Intra market spread](#contract-order-creation-intra-market-spread)
-         * [Intra market spreads and rolls](#intra-market-spreads-and-rolls)
-      * [Contract order creation: Inter market spread](#contract-order-creation-inter-market-spread)
-         * [Inter market spreads and rolls](#inter-market-spreads-and-rolls)
    * [Manual trades](#manual-trades)
    * [Broker order creation and execution](#broker-order-creation-and-execution)
       * [Before an order is traded](#before-an-order-is-traded)
@@ -110,6 +110,7 @@ Table of Contents
       * [Get spot FX data from interactive brokers, write to MongoDB (Daily)](#get-spot-fx-data-from-interactive-brokers-write-to-mongodb-daily)
       * [Update sampled contracts (Daily)](#update-sampled-contracts-daily)
       * [Update futures contract historical price data (Daily)](#update-futures-contract-historical-price-data-daily)
+         * [Set times when different regions download prices](#set-times-when-different-regions-download-prices)
       * [Update multiple and adjusted prices (Daily)](#update-multiple-and-adjusted-prices-daily)
       * [Update capital and p&l by polling brokerage account](#update-capital-and-pl-by-polling-brokerage-account)
       * [Allocate capital to strategies](#allocate-capital-to-strategies)
@@ -135,8 +136,8 @@ Table of Contents
             * [View processes](#view-processes)
             * [Change status of process](#change-status-of-process)
             * [Global status change](#global-status-change)
-            * [Mark as close](#mark-as-finished)
-            * [Mark all dead processes as close](#mark-all-dead-processes-as-finished)
+            * [Mark as close](#mark-as-close)
+            * [Mark all dead processes as close](#mark-all-dead-processes-as-close)
             * [View process configuration](#view-process-configuration)
          * [Update configuration](#update-configuration)
       * [Interactive diagnostics](#interactive-diagnostics)
@@ -148,8 +149,6 @@ Table of Contents
          * [Reports](#reports)
          * [Logs, errors, emails](#logs-errors-emails)
             * [View stored emails](#view-stored-emails)
-            * [View errors](#view-errors)
-            * [View logs](#view-logs)
          * [View prices](#view-prices)
          * [View capital](#view-capital)
          * [Positions and orders](#positions-and-orders)
@@ -186,7 +185,7 @@ Table of Contents
       * [Backup state files](#backup-state-files)
       * [Backup mongo dump](#backup-mongo-dump)
       * [Start up script](#start-up-script)
-   * [Scripts under other (non\-linux) operating systems](#scripts-under-other-non-linux-operating-systems)
+   * [Scripts under other (non-linux) operating systems](#scripts-under-other-non-linux-operating-systems)
 * [Scheduling](#scheduling)
    * [Issues to consider when constructing the schedule](#issues-to-consider-when-constructing-the-schedule)
    * [Choice of scheduling systems](#choice-of-scheduling-systems)
@@ -208,7 +207,8 @@ Table of Contents
       * [System backtest .yaml config file(s)](#system-backtest-yaml-config-files)
       * [Control config files](#control-config-files)
       * [Broker and data source specific configuration files](#broker-and-data-source-specific-configuration-files)
-      * [Only used when setting up the system](#only-used-when-setting-up-the-system)
+      * [Instrument and roll configuration](#instrument-and-roll-configuration)
+      * [Set up configuration](#set-up-configuration)
    * [Capital](#capital)
       * [Large changes in capital](#large-changes-in-capital)
       * [Withdrawals and deposits of cash or stock](#withdrawals-and-deposits-of-cash-or-stock)
@@ -235,6 +235,7 @@ Table of Contents
       * [Risk report](#risk-report)
       * [Liquidity report](#liquidity-report)
       * [Costs report](#costs-report)
+   * [Customize report generation in the run_report process](#customize-report-generation-in-the-run_report-process)
 
 Created by [gh-md-toc](https://github.com/ekalinin/github-markdown-toc)