Translate proc optimizations #4824
Conversation
Signed-off-by: Krishna Kondaka <[email protected]>
Signed-off-by: Krishna Kondaka <[email protected]>
Signed-off-by: Krishna Kondaka <[email protected]>
kkondaka
requested review from
chenqi0805,
engechas,
graytaylor0,
dinujoh,
KarstenSchnitter,
dlvenable and
oeyh
as code owners
| @@ -107,14 +106,15 @@ private void translateSource(Object sourceObject, Event recordEvent, TargetsPara | |||
| } | |||
|
|
|||
| String rootField = jsonExtractor.getRootField(commonPath); | |||
| if(!recordObject.containsKey(rootField)){ | |||
| Object rootFieldObject = recordEvent.get(rootField, Object.class); | |||
There was a problem hiding this comment.
Is this faster? If so, can we optimize containsKey instead of making this change?
There was a problem hiding this comment.
Biggest performance issue is calling toMap(). I wanted to avoid calling until it is actually needed. Worked with a customer provided 50000 line yaml and this change seems to be helping.
There was a problem hiding this comment.
So is change to get doing anything to help with performance? If not, I'd suggest reverting to containsKey. If so, let's fix containsKey in JacksonEvent and then revert to use containsKey here.
There was a problem hiding this comment.
@dlvenable the profiler showed that a lot of time was spent in event.toMap() Not sure how fixing containsKey would help here.
There was a problem hiding this comment.
Data Prepper Event has a containsKey() method.
if(!recordEvent.containsKey(rootField)) {
...
There was a problem hiding this comment.
Oh Ok. So, you are referring to line 110 and not 92. I did not try with containsKey. I can try that and get back to you.
There was a problem hiding this comment.
@dlvenable looking at the code, both get() and containsKey() code is almost same. Definitely using containsKey would help even more.
Signed-off-by: Krishna Kondaka <[email protected]>
Description
Translate processor is taking a lot of cpu cycles while doing
event.toMap()when it is not necessary in some cases.This change optimizes it by refactoring the code.
Issues Resolved
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.