From 90cf5dd35d435209ad4005d8a178ee32c97a94d5 Mon Sep 17 00:00:00 2001 From: Christopher Gallo Date: Tue, 17 Oct 2023 16:37:24 -0500 Subject: [PATCH 1/3] Added an article about threads for python --- content/python/mtSpeedup.md | 221 ++++++++++++++++++++++++++++++++++++ 1 file changed, 221 insertions(+) create mode 100644 content/python/mtSpeedup.md diff --git a/content/python/mtSpeedup.md b/content/python/mtSpeedup.md new file mode 100644 index 0000000000..8cc23ef91a --- /dev/null +++ b/content/python/mtSpeedup.md @@ -0,0 +1,221 @@ +--- +title: "Threads for improved API performance" +description: "Describes how to use Python's Thread library to improve how quickly API commands can be executed by doing them in parallel instead of in series." +date: "2023-10-16" +classes: + - "SoftLayer_Account" +tags: + - "article" + - "examples" + - "objectfilter" + - "slcli" +--- + + +# Basic of Threads +If you are unfamiliar with Threads these examples will use the [concurrent.futures](https://docs.python.org/3/library/concurrent.futures.html#module-concurrent.futures) library within the python standard library (available after python3.2) which is fairly simple to get setup. + +For some technical details on how python handles Threads the following are good reading on the topic +- [Python Threading Tutorial: Run Code Concurrently Using the Threading Module](https://youtu.be/IEEhzQoKtQU?si=5F9NvQcf6km9QHan) +- [An Intro to Threading in Python](https://realpython.com/intro-to-python-threading/) + +The basic pattern will look something like this + +```python +import concurrent.futures as cf +import SoftLayer + +client = SoftLayer.Client() +with cf.ThreadPoolExecutor(max_workers=10) as executor: + # Adds this API call to the ThreadPool, to be executed at some point in the future + api_call = executor.submit( + client.call, 'SoftLayer_Hardware_Server', 'getObject', + id=9999, mask="mask[id,hostname]" + ) + # Waits for the API call to return, then prints out the result + print(api_call.result()) +``` + +With regards to the SoftLayer API, threads are useful in two situations. First is Pagination, when you make a series of API calls with a [resultLimit](/article/using-result-limits-softlayer-api/) to get chunks of data from a long list of results. Instead of making 1 api call, then another and another, you can make them all at the same time (up to your max_works limit). Also threads are great to break up a single API call that might take a long time to complete, into multiple smaller api calls that each execute quickly, thus making the overall execution time faster. + +# Threads for Pagination + +[v6.2.0](https://github.com/softlayer/softlayer-python/releases/tag/v6.2.0) implements a new feature to easily enable threaded API calls to page through data in the client.cf_call() method. Here is that code and I will explain a bit on how it works. + +```python +def cf_call(self, service, method, *args, **kwargs): + """Uses threads to iterate through API calls. + + :param service: the name of the SoftLayer API service + :param method: the method to call on the service + :param integer limit: result size for each API call (defaults to 100) + :param \\*args: same optional arguments that ``Service.call`` takes + :param \\*\\*kwargs: same optional keyword arguments that ``Service.call`` takes + """ + limit = kwargs.pop('limit', 100) + offset = kwargs.pop('offset', 0) + + if limit <= 0: + raise AttributeError("Limit size should be greater than zero.") + # This initial API call is to determine how many API calls we need to make after this first one. + first_call = self.call(service, method, offset=offset, limit=limit, *args, **kwargs) + + # This was not a list result, just return it. + if not isinstance(first_call, transports.SoftLayerListResult): + return first_call + # How many more API calls we have to make + api_calls = math.ceil((first_call.total_count - limit) / limit) + + + def this_api(offset): + """Used to easily call executor.map() on this fuction""" + return self.call(service, method, offset=offset, limit=limit, *args, **kwargs) + + with cf.ThreadPoolExecutor(max_workers=10) as executor: + future_results = {} + offset_map = [x * limit for x in range(1, api_calls)] + future_results = list(executor.map(this_api, offset_map)) + # Append the results in the order they were called + for call_result in future_results: + first_call = first_call + call_result + return first_call +``` + +First, we need to get (or set a default) for the `limit` and `offset` properties. + +This removes them from the kwargs (if any other properties like mask/filter are set, we keep those), and will set them to a default value if they do not exist. +```python +limit = kwargs.pop('limit', 100) +offset = kwargs.pop('offset', 0) +``` + + +From there, we need to make the first api call so we can determine how many total results we have to get. API results will have a special header called `softlayer-total-items` that we make use of here for that count. If the result is a List, the transport will set the result to a transports.SoftLayerListResult that has the `total_count` property with this header in it. + +```python +first_call = self.call(service, method, offset=offset, limit=limit, *args, **kwargs) +# This was not a list result, just return it. +if not isinstance(first_call, transports.SoftLayerListResult): + return first_call +# How many more API calls we have to make +api_calls = math.ceil((first_call.total_count - limit) / limit) +``` + +Once we have `total_count`, we subtract the `limit` (we don't want to get those results again), and divide by the limit (rounding up to the nearest whole number), and that gives us the count of how many API calls to make. + +Next we need to define a small function we can use with the ThreadPoolExecutor to make things a little easier. The only variable that changes here is the offset, so that is our only parameter. The rest are set to whatever was passed in originally. + +```python +def this_api(offset): + """Used to easily call executor.map() on this fuction""" + return self.call(service, method, offset=offset, limit=limit, *args, **kwargs) +``` + +Then we add the function calls to the ThreadPoolExecutor as follows + +```python +with cf.ThreadPoolExecutor(max_workers=10) as executor: + offset_map = [x * limit for x in range(1, api_calls)] + future_results = list(executor.map(this_api, offset_map)) +``` + +`offset_map` is just a list of ints we will pass into our `this_api` function, defining our offset. We start at `1 * limit` since we already have the `0 -> limit` results from our first api call. [Executor.map()](https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Executor.map) will add a call to `this_api()` for each element in `offset_map` + +Finally we leave the `with cf.ThreadPoolExecutor` block once all the API calls have executed, and we can simply add them together and return the result. + +```python +for call_result in future_results: + first_call = first_call + call_result +return first_call +``` + +# Threads for large API calls + +For API calls that get a single resources (for example, SoftLayer_Hardware_Server::GetObject()), it can still be possible to use threading to improve performance if you are getting a lot of relational properties about the object. [get_hardware_fast()](https://github.com/softlayer/softlayer-python/blob/master/SoftLayer/managers/hardware.py#L281) is a good example of this pattern. + +The basic process is to remove any relational properties from your object mask that have corresponding methods, and call those methods instead. + + +```python +def get_hardware_fast(self, hardware_id): + """Get details about a hardware device. Similar to get_hardware() but this uses threads + + :param integer id: the hardware ID + :returns: A dictionary containing a large amount of information about the specified server. + """ + + hw_mask = ( + 'id, globalIdentifier, fullyQualifiedDomainName, hostname, domain,' + 'provisionDate, hardwareStatus, bareMetalInstanceFlag, processorPhysicalCoreAmount,' + 'memoryCapacity, notes, privateNetworkOnlyFlag, primaryBackendIpAddress,' + 'primaryIpAddress, networkManagementIpAddress, userData, datacenter, hourlyBillingFlag,' + 'lastTransaction[transactionGroup], hardwareChassis[id,name]' + ) + server = self.client.call('SoftLayer_Hardware_Server', 'getObject', id=hardware_id, mask=hw_mask) + with cf.ThreadPoolExecutor(max_workers=10) as executor: + networkComponents = executor.submit( + self.client.call, 'SoftLayer_Hardware_Server', 'getNetworkComponents', id=hardware_id + ) + activeComponents = executor.submit( + self.client.call, 'SoftLayer_Hardware_Server', 'getActiveComponents', id=hardware_id + ) + activeTransaction = executor.submit( + self.client.call, 'SoftLayer_Hardware_Server', 'getActiveTransaction', id=hardware_id + ) + operatingSystem = executor.submit( + self.client.call, 'SoftLayer_Hardware_Server', 'getOperatingSystem', id=hardware_id + ) + softwareComponents = executor.submit( + self.client.call, 'SoftLayer_Hardware_Server', 'getSoftwareComponents', id=hardware_id + ) + billingItem = executor.submit( + self.client.call, 'SoftLayer_Hardware_Server', 'getBillingItem', id=hardware_id + ) + networkVlans = executor.submit( + self.client.call, 'SoftLayer_Hardware_Server', 'getNetworkVlans', id=hardware_id + ) + remoteManagementAccounts = executor.submit( + self.client.call, 'SoftLayer_Hardware_Server', 'getRemoteManagementAccounts', id=hardware_id + ) + + server['networkComponents'] = networkComponents.result() + server['activeComponents'] = activeComponents.result() + server['activeTransaction'] = activeTransaction.result() + server['operatingSystem'] = operatingSystem.result() + server['softwareComponents'] = softwareComponents.result() + server['billingItem'] = billingItem.result() + server['networkVlans'] = networkVlans.result() + server['remoteManagementAccounts'] = remoteManagementAccounts.result() + server['tagReferences'] = tagReferences.result() + + return server +``` + +So instead of having `networkComponents` in the Hardware_Server objectMask, you remove it and make its own API call to `Hardware_Server::getNetworkComponents`, and put the result in the `networkComponents` property of the Hardware_Server object. Then do the same for all the other properties as shown above. + +This can overall be faster than a single API call because with all the properties we are selecting originally, the database has to join in quite a few different tables and that can be very complex for the database. However each individual query is much simpler and can be returned very quickly. Since we are making all the seperate calls at the same time, the overall API execution time is much faster. + +# Threads for two stage API calls + +For this pattern, we will get a list of object Ids with one api call, then use Threads to get the details of each of those Ids. For example, instead of using SoftLayer_Account::getVirtualGuests to get all the details about all of the Virtual_Guests on your account, you would use SoftLayer_Account::getVirtualGuests to simply get the Ids of each guest, then a bunch of calls to SoftLayer_Virtual_Guest::getObject(id=guest_id) to get all of the details. For accounts with a large number of guests this can be quite a lot faster than a single API call. + +[slcli bandwidth pools](https://github.com/softlayer/softlayer-python/blob/master/SoftLayer/CLI/bandwidth/pools.py#L48) is a good example of this pattern. First it gets all the bandwidth pool Ids, then uses threads to get the information about each pool (which can be a fairly long API call). + +```python +def cli(env): + + # Initialize the AccountManager + manager = AccountManager(env.client) + # Get a lit of bandwidth pools, including the ID + items = manager.get_bandwidth_pools() + + # Setup ThreadPool + with cf.ThreadPoolExecutor(max_workers=5) as executor: + # List of pool ids, to be passed into AccountManager.get_bandwidth_pool_counts() + item_ids = [item.get('id') for item in items] + # Builds the map, returning a tuple of the item, and the result of manager.get_bandwidth_pool_counts + for item, servers in zip(items, executor.map(manager.get_bandwidth_pool_counts, item_ids)): + id_bandwidth = item.get('id') + name = item.get('name') + print(f"{id_bandwidth}, {name}, {servers}") +``` \ No newline at end of file From bebc25640a37385f4da65258092830d96a2f10f7 Mon Sep 17 00:00:00 2001 From: Christopher Gallo Date: Fri, 20 Oct 2023 15:05:00 -0500 Subject: [PATCH 2/3] MThreads article revisions --- content/python/mtSpeedup.md | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/content/python/mtSpeedup.md b/content/python/mtSpeedup.md index 8cc23ef91a..56bc2bca39 100644 --- a/content/python/mtSpeedup.md +++ b/content/python/mtSpeedup.md @@ -13,11 +13,12 @@ tags: # Basic of Threads -If you are unfamiliar with Threads these examples will use the [concurrent.futures](https://docs.python.org/3/library/concurrent.futures.html#module-concurrent.futures) library within the python standard library (available after python3.2) which is fairly simple to get setup. +If you are unfamiliar with Threads, these examples will use the [concurrent.futures](https://docs.python.org/3/library/concurrent.futures.html#module-concurrent.futures) library within python (available after python3.2) which is fairly simple to get setup. I wont really go over HOW it works on a technical level, only how to best use it threads to improve performance when making SoftLayer API requests. For some technical details on how python handles Threads the following are good reading on the topic - [Python Threading Tutorial: Run Code Concurrently Using the Threading Module](https://youtu.be/IEEhzQoKtQU?si=5F9NvQcf6km9QHan) - [An Intro to Threading in Python](https://realpython.com/intro-to-python-threading/) +- [What Is Python concurrent.futures? (with examples)](https://www.packetswitch.co.uk/what-is-concurrent-futures-and-how-can-it-boost-your-python-performance/) The basic pattern will look something like this @@ -36,11 +37,11 @@ with cf.ThreadPoolExecutor(max_workers=10) as executor: print(api_call.result()) ``` -With regards to the SoftLayer API, threads are useful in two situations. First is Pagination, when you make a series of API calls with a [resultLimit](/article/using-result-limits-softlayer-api/) to get chunks of data from a long list of results. Instead of making 1 api call, then another and another, you can make them all at the same time (up to your max_works limit). Also threads are great to break up a single API call that might take a long time to complete, into multiple smaller api calls that each execute quickly, thus making the overall execution time faster. +With regards to the SoftLayer API, threads are useful in three situations. First is Pagination, when you make a series of identical API calls with a [resultLimit](/article/using-result-limits-softlayer-api/) to get chunks of data from a long list of results. Instead of making 1 api call, then another and another, you can make them all at the same time (up to your max_works limit). Second, threads are great to break up a single API call that might take a long time to complete, into multiple smaller api calls that each execute quickly, thus making the overall execution time faster. Third is a sort of combination of the two previous. You will make a single API call to get a large list of Ids, then threads to pull each individual item and the verbose details you want about it. # Threads for Pagination -[v6.2.0](https://github.com/softlayer/softlayer-python/releases/tag/v6.2.0) implements a new feature to easily enable threaded API calls to page through data in the client.cf_call() method. Here is that code and I will explain a bit on how it works. +[softlayer-python v6.2.0](https://github.com/softlayer/softlayer-python/releases/tag/v6.2.0) implements a new feature to easily enable threaded API calls to page through data in the client.cf_call() method. Here is that code and I will explain a bit on how it works. ```python def cf_call(self, service, method, *args, **kwargs): @@ -131,7 +132,7 @@ return first_call # Threads for large API calls -For API calls that get a single resources (for example, SoftLayer_Hardware_Server::GetObject()), it can still be possible to use threading to improve performance if you are getting a lot of relational properties about the object. [get_hardware_fast()](https://github.com/softlayer/softlayer-python/blob/master/SoftLayer/managers/hardware.py#L281) is a good example of this pattern. +For API calls that get a single resources (for example, SoftLayer_Hardware_Server::GetObject()), it can still be possible to use threading to improve performance if you are getting a lot of relational properties about the object. The function [get_hardware_fast()](https://github.com/softlayer/softlayer-python/blob/master/SoftLayer/managers/hardware.py#L281) is a good example of this pattern. The basic process is to remove any relational properties from your object mask that have corresponding methods, and call those methods instead. @@ -144,13 +145,7 @@ def get_hardware_fast(self, hardware_id): :returns: A dictionary containing a large amount of information about the specified server. """ - hw_mask = ( - 'id, globalIdentifier, fullyQualifiedDomainName, hostname, domain,' - 'provisionDate, hardwareStatus, bareMetalInstanceFlag, processorPhysicalCoreAmount,' - 'memoryCapacity, notes, privateNetworkOnlyFlag, primaryBackendIpAddress,' - 'primaryIpAddress, networkManagementIpAddress, userData, datacenter, hourlyBillingFlag,' - 'lastTransaction[transactionGroup], hardwareChassis[id,name]' - ) + hw_mask = "mask[id, globalIdentifier, fullyQualifiedDomainName, hostname, domain]" server = self.client.call('SoftLayer_Hardware_Server', 'getObject', id=hardware_id, mask=hw_mask) with cf.ThreadPoolExecutor(max_workers=10) as executor: networkComponents = executor.submit( @@ -193,7 +188,9 @@ def get_hardware_fast(self, hardware_id): So instead of having `networkComponents` in the Hardware_Server objectMask, you remove it and make its own API call to `Hardware_Server::getNetworkComponents`, and put the result in the `networkComponents` property of the Hardware_Server object. Then do the same for all the other properties as shown above. -This can overall be faster than a single API call because with all the properties we are selecting originally, the database has to join in quite a few different tables and that can be very complex for the database. However each individual query is much simpler and can be returned very quickly. Since we are making all the seperate calls at the same time, the overall API execution time is much faster. +This can overall be faster than a single API call because with all the properties we are selecting originally, the database has to join in quite a few different tables and that can be very complex for the database. However each individual query is much simpler and can be returned very quickly. Since we are making all the seperate calls at the same time, the overall API execution time is much faster. + +For this specific example, getting all the relational properties in a single api call takes about 4-9s. When breaking it up, each api call takes about 1-2s, and since they execute all at the same time, our overall execution time is just a bit over 2s. # Threads for two stage API calls From 04c02899b7f95986dcc9b7099d2f6e9c8ba729aa Mon Sep 17 00:00:00 2001 From: Christopher Gallo Date: Fri, 20 Oct 2023 15:05:37 -0500 Subject: [PATCH 3/3] 20231020 release updates --- .../manageLicenses.md | 16 ++++++++++ .../getPrecheckStatus.md | 2 +- data/sldn_metadata.json | 31 ++++++++++++++++--- 3 files changed, 44 insertions(+), 5 deletions(-) create mode 100644 content/reference/services/SoftLayer_Network_Gateway/manageLicenses.md diff --git a/content/reference/services/SoftLayer_Network_Gateway/manageLicenses.md b/content/reference/services/SoftLayer_Network_Gateway/manageLicenses.md new file mode 100644 index 0000000000..07a191ed17 --- /dev/null +++ b/content/reference/services/SoftLayer_Network_Gateway/manageLicenses.md @@ -0,0 +1,16 @@ +--- +title: "manageLicenses" +description: "Used to manage gateway require and add on licenses. If license request is valid for the gateway type a Gateway License Manage process will be created if licenses need to be adjusted on the gateway. + +" +date: "2018-02-12" +tags: + - "method" + - "sldn" + - "Network" +classes: + - "SoftLayer_Network_Gateway" +type: "reference" +layout: "method" +mainService : "SoftLayer_Network_Gateway" +--- diff --git a/content/reference/services/SoftLayer_Network_Gateway_Precheck/getPrecheckStatus.md b/content/reference/services/SoftLayer_Network_Gateway_Precheck/getPrecheckStatus.md index a25cef76c9..f4cf0d8f65 100644 --- a/content/reference/services/SoftLayer_Network_Gateway_Precheck/getPrecheckStatus.md +++ b/content/reference/services/SoftLayer_Network_Gateway_Precheck/getPrecheckStatus.md @@ -1,6 +1,6 @@ --- title: "getPrecheckStatus" -description: "Get the precheck status for all Virtual (Juniper, Fortigate vFSA Gateway Action categories which require a readiness check before executing. Reference cloud.ibm.com documentation for more details. +description: "Get the precheck status for all Virtual (Juniper, Fortigate vFSA) Gateway Action categories which require a readiness check before executing. Reference cloud.ibm.com documentation for more details. Possible precheck readiness values include: diff --git a/data/sldn_metadata.json b/data/sldn_metadata.json index 762ba50c07..fbe6309e97 100644 --- a/data/sldn_metadata.json +++ b/data/sldn_metadata.json @@ -130201,6 +130201,23 @@ "type": "boolean", "doc": "Returns true if rollback is allowed. \n\n" }, + "manageLicenses": { + "name": "manageLicenses", + "type": "boolean", + "doc": "Used to manage gateway require and add on licenses. If license request is valid for the gateway type a Gateway License Manage process will be created if licenses need to be adjusted on the gateway. \n\n", + "parameters": [ + { + "name": "requiredItemKeyName", + "type": "string", + "doc": "Item Key Name of the required license to be used on the gateway" + }, + { + "name": "addOnLicenses", + "type": "string", + "doc": "<<< EOT" + } + ] + }, "rebuildHACluster": { "name": "rebuildHACluster", "type": "boolean", @@ -130829,11 +130846,11 @@ "form": "local", "doc": "Indicates if the member has been upgraded. " }, - "lastvSRXVersion": { - "name": "lastvSRXVersion", + "lastVersion": { + "name": "lastVersion", "type": "string", "form": "local", - "doc": "The previous vSRX version of the gateway software " + "doc": "The previous version of the gateway software " }, "licenseExpirationDate": { "name": "licenseExpirationDate", @@ -130882,6 +130899,12 @@ "form": "local", "doc": "The vSRX version of the gateway software " }, + "version": { + "name": "version", + "type": "string", + "form": "local", + "doc": "The version of the gateway software " + }, "warningCode": { "name": "warningCode", "type": "int", @@ -130974,7 +130997,7 @@ "name": "getPrecheckStatus", "type": "SoftLayer_Network_Gateway_Precheck", "typeArray": true, - "doc": "Get the precheck status for all Virtual (Juniper, Fortigate vFSA Gateway Action categories which require a readiness check before executing. Reference cloud.ibm.com documentation for more details. \n\nPossible precheck readiness values include: \n\nReady (0): The member or Gateway category is ready. The only state that will be allowed to execute the Action. Not Ready (1): The member or Gateway category is not ready. This could occur because of several reasons. Either a precheck error occur, or the precheck has not run within the precheck timeout window. Check the returnCode for details on the specific error. Reference the cloud.ibm.com documentation for recovery details. Running (2): The precheck is currently running with no errors. Incomplete (3): The other member in the Gateway failed, therefore the current member could not complete it's precheck. Unsupported (4): The category is unsupported for the given member or Gateway. Expired (5) : The precheck record has expired so will need to be run again. Unchecked (6) : The precheck for the category has never been run. Current (7) : The gateway state is current so running precheck is not required. This commonly relates to version upgrade if gateway is in most update version. \n\nReturn Values: Array of objects \n\nObject Definition: \n\ncategory : String : The precheck category which corresponds to one or more executeable actions. \n\nCurrent categories include: upgrade_precheck : Required for major and minor upgrade version actions. license_precheck : Required for license upgrade and downgrade actions. reload_precheck : Required for OS Reload action. rollback_precheck : Optional and related to upgrade_precheck. Only returned if getRollbackPrecheck is provided and set to True (1). \n\n\n\nmemberId : Integer : The softlayer member id. memberReadinessValue : String : The precheck readiness state for the member. See possible readiness values above. gatewayReadinessValue : String : The precheck readiness state for the gateway : See possible readiness values above. returnCode : Integer : The return code. 0 if no error. Reference cloud.ibm.com documentation for details. \n\n", + "doc": "Get the precheck status for all Virtual (Juniper, Fortigate vFSA) Gateway Action categories which require a readiness check before executing. Reference cloud.ibm.com documentation for more details. \n\nPossible precheck readiness values include: \n\nReady (0): The member or Gateway category is ready. The only state that will be allowed to execute the Action. Not Ready (1): The member or Gateway category is not ready. This could occur because of several reasons. Either a precheck error occur, or the precheck has not run within the precheck timeout window. Check the returnCode for details on the specific error. Reference the cloud.ibm.com documentation for recovery details. Running (2): The precheck is currently running with no errors. Incomplete (3): The other member in the Gateway failed, therefore the current member could not complete it's precheck. Unsupported (4): The category is unsupported for the given member or Gateway. Expired (5) : The precheck record has expired so will need to be run again. Unchecked (6) : The precheck for the category has never been run. Current (7) : The gateway state is current so running precheck is not required. This commonly relates to version upgrade if gateway is in most update version. \n\nReturn Values: Array of objects \n\nObject Definition: \n\ncategory : String : The precheck category which corresponds to one or more executeable actions. \n\nCurrent categories include: upgrade_precheck : Required for major and minor upgrade version actions. license_precheck : Required for license upgrade and downgrade actions. reload_precheck : Required for OS Reload action. rollback_precheck : Optional and related to upgrade_precheck. Only returned if getRollbackPrecheck is provided and set to True (1). \n\n\n\nmemberId : Integer : The softlayer member id. memberReadinessValue : String : The precheck readiness state for the member. See possible readiness values above. gatewayReadinessValue : String : The precheck readiness state for the gateway : See possible readiness values above. returnCode : Integer : The return code. 0 if no error. Reference cloud.ibm.com documentation for details. \n\n", "docOverview": "Get Precheck status for Gateway", "static": true, "maskable": true,