Merge pull request #454 from rd4398/doc-dirs

Document output directories generated by fromager

docs/files.md

@@ -93,3 +93,133 @@ The following example shows `graph.json` file for the top-level dependency `whee
   }
 }
 ```
+
+## Output Directories
+
+During the wheel building process, fromager generates multiple output directories namely `sdists-repo`, `wheels-repo` and `work-dir`. These directories contain important information related to the wheel build.
+
+### sdist-repo
+
+This directory contains the source distributions for the package and its dependencies that we are building. The directory structure of `sdists-repo` is as follows
+
+```
+sdists-repo
+├── builds
+└── downloads
+```
+
+The `builds` and `downloads` sub-directories contain the source distributions that are built and downloaded respectively. The `downloads` directory includes the original sdist from upstream and the `builds` directory contains the sdist created by fromager after any patches are applied. For example, the `sdists-repo` for `stevedore` package looks as follows:
+
+```
+sdists-repo
+├── builds
+│   ├── pbr-6.1.0.tar.gz
+│   ├── setuptools-75.1.0.tar.gz
+│   └── stevedore-5.3.0.tar.gz
+└── downloads
+    ├── pbr-6.1.0.tar.gz
+    ├── setuptools-75.1.0.tar.gz
+    └── stevedore-5.3.0.tar.gz
+
+```
+
+We can see source distributions for `pbr` and `setuptools` since these are dependencies of `stevedore`.
+
+### wheels-repo
+
+This directory contains the wheels for a package and its dependencies that are built by fromager, used as prebuilt and the ones that are downloaded from indices. The directory structure of `wheels-repo` is as follows
+
+```
+wheels-repo
+├── build
+├── downloads
+├── prebuilt
+└── simple
+```
+
+* The `build` sub-directoy holds temporary builds. We use it as the output directory when building the wheel because we can't predict the filename, and so using an empty directory with a name we know gives us a way to find the file and move it into the `downloads` directory after it's built
+* The `downloads` sub-directory contains the wheels in `.whl` format that fromager builds combined with the pre-built wheels so we can create a local package index in `simple`
+* The `prebuilt` sub-directory contains wheels that are being used as prebuilt
+* The `simple` sub-directory is managed by [pypi-mirror](https://pypi.org/project/pypi-mirror/) to create a local wheel index.
+
+For example, the `wheels-repo` for `stevedore` package looks as follows:
+
+```
+wheels-repo
+├── build
+├── downloads
+│   ├── pbr-6.1.0-0-py2.py3-none-any.whl
+│   ├── setuptools-75.1.0-0-py3-none-any.whl
+│   └── stevedore-5.3.0-0-py3-none-any.whl
+├── prebuilt
+└── simple
+    ├── index.html
+    ├── pbr
+    │   ├── index.html
+    │   └── pbr-6.1.0-0-py2.py3-none-any.whl
+    ├── setuptools
+    │   ├── index.html
+    │   └── setuptools-75.1.0-0-py3-none-any.whl
+    └── stevedore
+        ├── index.html
+        └── stevedore-5.3.0-0-py3-none-any.whl
+
+```
+
+### work-dir
+
+This directory contains information that is required during the wheel build process. This information includes logs, constraints, requirements, graph for dependency resolution and the order in which the package and its dependencies will be built. The directory structure of `work-dir` is as follows:
+
+```
+work-dir
+├── build-order.json
+├── constraints.txt
+├── graph.json
+├── logs
+├── sample-package-foo
+   ├── simple-package-foo
+   ├── build-backend-requirements.txt
+   ├── build.log
+   ├── build-meta.json
+   ├── build-sdist-requirements.txt
+   ├── build-system-requirements.txt
+   └── requirements.txt
+
+```
+
+* The `build-order.json` file is an output file that contains the bottom-up order in which the dependencies need to be built for a specific wheel. You can find more details [here](https://fromager.readthedocs.io/en/latest/files.html#build-order-json)
+* The `constraints.txt` is the output file, produced by fromager, showing all of the versions of the packages that are install-time dependencies of the top-level items
+* The `graph.json` is an output file that contains all the paths fromager can take to resolve a dependency during building the wheel. You can find more details [here](https://fromager.readthedocs.io/en/latest/files.html#graph-json)
+* The `logs` sub-directory contains detailed logs for fromager's `build-sequence` command including various settings and overrides for each individual package and its dependencies whose wheel was built by fromager. Each log file also contains information about build-backend dependencies if present for a given package
+* The `work-dir` also includes sub-directories for the package and its dependencies. These sub-directories include various types of requirements files including `build-backend-requirements.txt`, `build-sdists-requirements.txt`, `build-system-requirements.txt` and the general `requirements.txt`. Files like `build.log` which store the logs generated by pip and `build-meta.json` that stores the metadata for the build are also located in `work-dir`. These sub-directories also include all the other relevant information for a particular package. Each sub-directory of the package will also contain the unpacked source code of each wheel that is used for the build if `--no-cleanup` option of fromager is used. For example, in the above directory structure, for `simple-package-foo` requirement, we will have a subdirectory titled `simple-package-foo` that holds the unpacked source code
+
+For example, the `work-dir` for `stevedore` package after `bootstrap` command looks as follows:
+
+```
+work-dir
+├── build-order.json
+├── constraints.txt
+├── graph.json
+├── logs
+├── pbr-6.1.0
+│   ├── build-backend-requirements.txt
+│   ├── build.log
+│   ├── build-meta.json
+│   ├── build-sdist-requirements.txt
+│   ├── build-system-requirements.txt
+│   └── requirements.txt
+├── setuptools-75.1.0
+│   ├── build-backend-requirements.txt
+│   ├── build.log
+│   ├── build-meta.json
+│   ├── build-sdist-requirements.txt
+│   ├── build-system-requirements.txt
+│   └── requirements.txt
+└── stevedore-5.3.0
+    ├── build-backend-requirements.txt
+    ├── build.log
+    ├── build-meta.json
+    ├── build-sdist-requirements.txt
+    ├── build-system-requirements.txt
+    └── requirements.txt
+```