Skip to content

Commit

Permalink
Doc: jobs option and package build setting
Browse files Browse the repository at this point in the history
  • Loading branch information
rd4398 committed Nov 5, 2024
1 parent 70e212e commit 9ea42dc
Show file tree
Hide file tree
Showing 2 changed files with 22 additions and 0 deletions.
21 changes: 21 additions & 0 deletions docs/how-to.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# How To

## Run parallel jobs, allocate cpu cores per job and allocate memory per job

The `--jobs` option of fromager allows to set the maximum number of wheel build jobs to run in parallel. Below is an example which uses the `--jobs` option along with the bootstrap command

`fromager --jobs 4 bootstrap torch`

For this example, the maximum number of jobs fromager will run in parallel is 4.

The `cpu_cores_per_job` is a package setting that allows to scale parallel jobs by available CPU cores. The default value is set to 1 which indicates as many parallel jobs as CPU logical cores.
Example: `cpu_cores_per_job = 2` indicates allocating 2 cores per job
This setting should always have value greater than or equal to 1

The `memory_per_job_gb` is a package setting that allows to scale parallel jobs by available virtual memory without swap. The default value is set to 1.0 which indicates that each parallel job requires 1 GB virtual memory
Example: `memory_per_job_gb = 0.5` indicates that each parallel job requires 512 MB virtual memory
This setting should always have value greater than or equal to 0.1

The above mentioned options are related to build systems and can be used as a per package setting when multiple cores and significant amount of memory is available on a build system.
On the other hand, the `--jobs` overrides the default calculations based on the other settings. For example, when someone passes `--jobs 4` then at most 4 processes will run in parallel when building a given wheel, but if they don't use this option, fromager computes a number of jobs and that value can be influenced based on the per-package settings.
Note that the jobs are all within the context of building a single wheel.
1 change: 1 addition & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ those special cases directly into fromager.
using.md
getting-started.rst
customization.md
how-to.md
hooks.rst
files.md
config-reference.rst
Expand Down

0 comments on commit 9ea42dc

Please sign in to comment.