From d7c02547189e54e88772e4abd252adaf683de6fc Mon Sep 17 00:00:00 2001 From: Eric Mehl Date: Fri, 25 Jun 2021 10:39:54 -0400 Subject: [PATCH 1/3] Deadline Plugin : add configuration for Gaffer 0.60.0.0 --- custom/Gaffer/Gaffer.param | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/custom/Gaffer/Gaffer.param b/custom/Gaffer/Gaffer.param index abc33f5..9d045c3 100644 --- a/custom/Gaffer/Gaffer.param +++ b/custom/Gaffer/Gaffer.param @@ -85,6 +85,16 @@ Label=Gaffer 0.59.0.0 Render Executable Default=%HOME%/gaffer_0.59.0.0;~/gaffer_0.59.0.0 Description=The path to the Gaffer 0.59.0.0 executable (gaffer.bat on Windows) file used for executing. Enter alternative paths on separate lines. +[Executable0_60_0_0] +Type=multilinemultifilename +Category=Gaffer 0.60.0.0 Executables +CategoryOrder=0 +CategoryIndex=0 +Label=Gaffer 0.60.0.0 Render Executable +Default=%HOME%/gaffer_0.60.0.0;~/gaffer_0.60.0.0 +Description=The path to the Gaffer 0.60.0.0 executable (gaffer.bat on Windows) file used for executing. Enter alternative paths on separate lines. + + [EnablePathMapping] Type=boolean Category=Path Mapping (For Mixed Farms) From 324a596d986adbb52940db0a84df74ada6b715df Mon Sep 17 00:00:00 2001 From: Eric Mehl Date: Fri, 25 Jun 2021 10:40:24 -0400 Subject: [PATCH 2/3] README : improve formatting - Add section about running test with Python < 3.3 --- README.md | 36 +++++++++++++++++++----------------- 1 file changed, 19 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 302e15a..de8a1ee 100644 --- a/README.md +++ b/README.md @@ -9,18 +9,18 @@ It is tested on Linux and the beta Windows Gaffer build. OS X compatibility is u ## Installing ## 1. Extract the archive / clone the repository to a directory accessible to Gaffer. -2. Move the "Gaffer" subdirectory to your Deadline repository "custom/plugins" directory. This is the Deadline plugin that will run Gaffer jobs on your render farm. -3. Add the directory where you extracted / cloned the repository to the GAFFER_EXTENSION_PATHS environment variable before running Gaffer. -4. Move the gaffer_batch_dependency.py file to a location where all of your Deadline Workers and Pulse machines can access the file. Deadline will run that script according to your repository settings to check for tasks that can be released from pending status based on their dependencies being completed. +2. Move the `Gaffer` subdirectory to your Deadline repository `custom/plugins` directory. This is the Deadline plugin that will run Gaffer jobs on your render farm. +3. Add the directory where you extracted / cloned the repository to the `GAFFER_EXTENSION_PATHS` environment variable before running Gaffer. +4. Move the `gaffer_batch_dependency.py` file to a location where all of your Deadline Workers and Pulse machines can access the file. Deadline will run that script according to your repository settings to check for tasks that can be released from pending status based on their dependencies being completed. If you have multiple operating systems in your Deadline installation, you will likely need to set up path mapping for machines to locate the script. -5. Set the DEADLINE_DEPENDENCY_SCRIPT_PATH environment variable to the full path (including filename) where you saved the gaffer_batch_dependency.py file before running Gaffer. GafferDeadline dispatcher uses this variable as the location for the dependency script when submitting jobs to Deadline. -6. Ensure that the DEADLINE_PATH environment variable is set to the directory where the "deadlinecommand" executable lives. This is typically set system-wide when you install the Deadline Client. GafferDeadline uses this environment variable to locate "deadlinecommand" for interacting with your Deadline repository. +5. Set the `DEADLINE_DEPENDENCY_SCRIPT_PATH` environment variable to the full path (including filename) where you saved the `gaffer_batch_dependency.py` file before running Gaffer. GafferDeadline dispatcher uses this variable as the location for the dependency script when submitting jobs to Deadline. +6. Ensure that the `DEADLINE_PATH` environment variable is set to the directory where the `deadlinecommand` executable lives. This is typically set system-wide when you install the Deadline Client. GafferDeadline uses this environment variable to locate `deadlinecommand` for interacting with your Deadline repository. ## Deadline Setup ## -Once you have the Gaffer plugin copied to your Deadline "custom/plugins" directory you need to setup the Gaffer plugin in Deadline. In Super-User Mode in Deadline Monitor, go to Tools -> Configure Plugins. Select Gaffer from the list and add the paths to your Gaffer executables as they are found on your workstations and rendering machines. To support more than one Gaffer version in a single studio, GafferDeadline adds the version of Gaffer that is submitting the job to the Deadline Job settings. The Gaffer plugin looks for an executable of the same version and will fail if it is not found. If you are using a Gaffer version that GafferDeadline is not configured for, it's easy to add that version: -1. Open the gaffer.param file from your \/custom/plugins/gaffer directory. -2. Copy and paste the block starting with [Executable0_53_0_0] (or similar) to add a new entry. -3. Change the version number to the exact version of Gaffer you are running: [Executable0_51_0_1] corresponds to Gaffer version 0.51.0.1. +Once you have the Gaffer plugin copied to your Deadline `custom/plugins` directory you need to setup the Gaffer plugin in Deadline. In Super-User Mode in Deadline Monitor, go to Tools -> Configure Plugins. Select Gaffer from the list and add the paths to your Gaffer executables as they are found on your workstations and rendering machines. To support more than one Gaffer version in a single studio, GafferDeadline adds the version of Gaffer that is submitting the job to the Deadline Job settings. The Gaffer plugin looks for an executable of the same version and will fail if it is not found. If you are using a Gaffer version that GafferDeadline is not configured for, it's easy to add that version: +1. Open the `gaffer.param` file from your `/custom/plugins/gaffer` directory. +2. Copy and paste the block starting with `[Executable0_53_0_0]` (or similar) to add a new entry. +3. Change the version number to the exact version of Gaffer you are running: `[Executable0_51_0_1]` corresponds to Gaffer version 0.51.0.1. 4. Save the file and open up Configure Plugins again and the new version will be available to set the Gaffer binary locations. ## Using ## @@ -30,28 +30,30 @@ When you are ready to submit the node(s) press the node's "Execute" button and s You need a Deadline Client installed and connected to your repository on the machines you will be running GafferDeadline from. GafferDeadline uses the Deadline installation on the host machine, similar to other integrated submitters Deadline includes for Nuke, Houdini, etc. -The Deadline settings in Gaffer include an override for the dependency method for that node. This override controls downstream Task Nodes that depend on the node on which it was set. Most of the time it should be left on Auto to let the dispatcher determine the most efficient method. If you know a node needs to be handled in a particular way, you can force its dependency method with the override plug. Usually the "Full Job" setting will be the safest but least flexible because downstream tasks will wait for all frames of that job to complete before being released. +The Deadline settings in Gaffer include an override for the dependency method for that node. This override controls downstream Task Nodes that depend on the node on which it was set. Most of the time it should be left on "Auto" to let the dispatcher determine the most efficient method. If you know a node needs to be handled in a particular way, you can force its dependency method with the override plug. Usually the "Full Job" setting will be the safest but least flexible because downstream tasks will wait for all frames of that job to complete before being released. ### Auxiliary Files ### GafferDeadline can submit auxiliary files along with the dispatched job at submission time. These files will be uploaded to the Deadline repository and downloaded to each Deadline Worker when it dequeues a task. -The GafferDeadline Deadline plugin sets an environment variable called AUXFILEDIRECTORY to the local directory on the Deadline Worker where the files are downloaded. This environment variable can then be used in any of Gaffer's usual string substitutions to point files to the auxiliary file location. +The GafferDeadline Deadline plugin sets an environment variable called `AUXFILEDIRECTORY` to the local directory on the Deadline Worker where the files are downloaded. This environment variable can then be used in any of Gaffer's usual string substitutions to point files to the auxiliary file location. ### Render Threads and GPU Affinity ### -GafferDeadline sets the environment variable CPUTHREAD the Deadline Worker's render thread. GafferDeadline also sets the GPUAFFINITY environment variable to a comma-separated list of GPU Threads configured for that Worker. More information on setting up GPU Affinity can be found at https://www.awsthinkbox.com/blog/cpu-and-gpu-affinity-in-deadline. +GafferDeadline sets the environment variable `CPUTHREAD` the Deadline Worker's render thread. GafferDeadline also sets the `GPUAFFINITY` environment variable to a comma-separated list of GPU Threads configured for that Worker. More information on setting up GPU Affinity can be found at https://www.awsthinkbox.com/blog/cpu-and-gpu-affinity-in-deadline. ## Running Unit Tests ## You don't need to run the unit tests for normal use of GafferDeadline, but if you want to make customizations it is recommended that you add unit tests as appropriate and run the existing tests to ensure compatibility. To run the unit tests, you need to have an installation of Gaffer. -- All OS : set the GAFFER_EXTENSION_PATHS environment variable to the directory for GafferDeadline. -- Linux : set the GAFFER_ROOT environment variable to your Gaffer installation directory. From the GAFFER_ROOT/bin directory, run - ./gaffer test GafferDeadlineTest GafferDeadlineUITest -- Windows : You don't need to set the GAFFER_ROOT environment variable. From your Gaffer installation "bin" subdirectory, - run gaffer.bat test GafferDeadlineTest GafferDeadlineUITest +- All OS : set the `GAFFER_EXTENSION_PATHS` environment variable to the directory for GafferDeadline. +- Linux : set the `GAFFER_ROOT` environment variable to your Gaffer installation directory. From the GAFFER_ROOT/bin directory, run `./gaffer test GafferDeadlineTest GafferDeadlineUITest` +- Windows : You don't need to set the GAFFER_ROOT environment variable. From your Gaffer installation "bin" subdirectory, run `gaffer.bat test GafferDeadlineTest GafferDeadlineUITest` There is also a Visual Studio Code environment included that may be helpful. +### Testing on Python Versions Lower than 3.3 ### +GafferDeadline uses the `mock` library to avoid sending actual jobs to Deadline. If you are testing on with Python versions lower than 3.3 you will need to install `mock` for the Python distribution for Gaffer. This can be done by running +`gaffer env python -m pip install mock` + ## Contributing ## Feedback and pull requests are welcome! If you have ideas about how to improve the dispatcher, find bugs or would like to submit improvements, please create an issue on GitHub for discussion or a pull request. From b8953dfa563c182cfe3939d2df126fc29cbc18e9 Mon Sep 17 00:00:00 2001 From: Eric Mehl Date: Fri, 25 Jun 2021 10:40:45 -0400 Subject: [PATCH 3/3] DeadlineDispatcherTest : support Python versions < 3.3 --- python/GafferDeadlineTest/DeadlineDispatcherTest.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/python/GafferDeadlineTest/DeadlineDispatcherTest.py b/python/GafferDeadlineTest/DeadlineDispatcherTest.py index 1ffeb06..bcb0699 100644 --- a/python/GafferDeadlineTest/DeadlineDispatcherTest.py +++ b/python/GafferDeadlineTest/DeadlineDispatcherTest.py @@ -36,7 +36,11 @@ import os import unittest -from unittest import mock +import sys +if sys.version_info >= (3, 3): + from unittest import mock +else: + import mock import IECore