CONJUBILEE-2783: SaaS Connectors: Support for Partitioning Aggregation #60
Conversation
| await Promise.all(promises); | ||
| } else { | ||
| return await this.partitionAdapter.list(context, input, res, undefined); | ||
| } |
There was a problem hiding this comment.
Can we add logging here to allow troubleshooters to determine if:
- We used partition aggregation or not.
- How long retrieving the partitions took.
- The number of partitions returned.
- Might also be useful to log when we finishing aggregating a specific partition.
There was a problem hiding this comment.
Point 1-3 addressed.
Point 4 - Shouldn't this get logged at the connector level? I think we do not have control to check when a particular partition is finishing.
There was a problem hiding this comment.
For 4 I was thinking we could do something like:
let partitionsDone = 0
for (const partition of partitionList!) {
promises.push(async () => {
let startTime = new Date().getTime();
await this.partitionAdapter.list(context, input, res, partition);
partitionsDone += 1
this.childLogger.info(`Partition ${} of ${} done after ${}`)
})
}
await Promise.all()The reason I think this is valuable at the library level is because it gives us a good idea of how good of a job the connector is doing at creating equally sized partitions. I remember seeing a case of partitioning on java connectors where one of the partitions ended up being much larger than the others so we weren't taking advantage of as much parallelism as we could have.
There was a problem hiding this comment.
Makes sense. Updated the code.
Description
What is the intent of this change and why is it being made?
How Has This Been Tested?
What testing have you done to verify this change?