-
Notifications
You must be signed in to change notification settings - Fork 11
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Doc: jobs option and package build setting
- Loading branch information
Showing
2 changed files
with
22 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
# How To | ||
|
||
## Run parallel jobs, allocate cpu cores per job and allocate memory per job | ||
|
||
The `--jobs` option of fromager allows to set the maximum number of wheel build jobs to run in parallel. Below is an example which uses the `--jobs` option along with the bootstrap command | ||
|
||
`fromager --jobs 4 bootstrap torch` | ||
|
||
For this example, the maximum number of jobs fromager will run in parallel is 4. | ||
|
||
The `cpu_cores_per_job` is a package setting that allows to scale parallel jobs by available CPU cores. The default value is set to 1 which indicates as many parallel jobs as CPU logical cores. | ||
Example: `cpu_cores_per_job = 2` indicates allocating 2 cores per job | ||
This setting should always have value greater than or equal to 1 | ||
|
||
The `memory_per_job_gb` is a package setting that allows to scale parallel jobs by available virtual memory without swap. The default value is set to 1.0 which indicates that each parallel job requires 1 GB virtual memory | ||
Example: `memory_per_job_gb = 0.5` indicates that each parallel job requires 512 MB virtual memory | ||
This setting should always have value greater than or equal to 0.1 | ||
|
||
The above mentioned options are related to build systems and can be used as a per package setting when multiple cores and significant amount of memory is available on a build system. | ||
On the other hand, the `--jobs` overrides the default calculations based on the other settings. For example, when someone passes `--jobs 4` then at most 4 processes will run in parallel when building a given wheel, but if they don't use this option, fromager computes a number of jobs and that value can be influenced based on the per-package settings. | ||
Note that the jobs are all within the context of building a single wheel. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters