How to Debug Native Compile Times

[justinpapezsonos]

How should one go about debugging / diagnosing / resolving exuberant native compile times? The company that I work for recently migrated from NDK r21 to NDK r24 and some developers have seen their compile times increase on average from 4 minutes to 1 hour. These developers have noticed that the compile time regression occurs when building with NDK r23+ (they have tested with NDK r21, r22, r23, r24, & r25). The native library being compiled is roughly 1,200 files. Unfortunately, we are not sure how to diagnose what the root issue may be. We'd like to use NDK r23+ as it supports M1 machines, something that is important to our company. Any suggestions or help as to how we can proceed in order to diagnose the issue would be greatly appreciated. Thanks in advance!

[pirama-arumuga-nainar]

tl;dr: Antivirus was slowing things down. See the full thread for details. A jump from a few minutes to an hour seems surprising. There was no noticeable slowdown in the Android platform or kernel build times. Does anything stand out when observing build progress? For e.g., is the slowdown uniform across all sources or is it just a few files that are bottlenecked?

…

On Wed, Apr 20, 2022 at 2:45 PM Justin Papez ***@***.***> wrote: How should one go about debugging / diagnosing / resolving exuberant native compile times? The company that I work for recently migrated from NDK r21 to NDK r24 and some developers have seen their compile times increase on average from 4 minutes to 1 hour. These developers have noticed that the compile time regression occurs when building with NDK r23+ (they have tested with NDK r21, r22, r23, r24, & r25). The native library being compiled is roughly 1,200 files. Unfortunately, we are not sure how to diagnose what the root issue may be. We'd like to use NDK r23+ as it supports M1 machines, something that is important to our company. Any suggestions or help as to how we can proceed in order to diagnose the issue would be greatly appreciated. Thanks in advance! — Reply to this email directly, view it on GitHub <#1703>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEZPQ6ASH6ZXG3Q3IJ7JUP3VGB3IJANCNFSM5T5LEJ5A> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[justinpapezsonos]

Which r21? If you go back far enough (r21b or older) the binaries won't be signed/notarized. If gatekeeper is for some reason blocking all those processes only when they're signed/notarized, that's a difference. I am inclined to blame gatekeeper here, since this does look very similar to the behavior you'd find on Windows when a process is blocked waiting on AV.
...
Another possible split would be the settings on each user's machine. If they have different security settings (maybe some of them told gatekeeper to go away and others didn't), maybe that'd explain the behavior difference?

I had a fellow developer who is experiencing exuberant build times disable gatekeeper via sudo spctl --master-disable. Sadly, re-building with gatekeeper disabled had no impact on performance time.

[DanAlbert]

That's the original r21 release, so it is pre-signing. You're missing everything described in r21b, c, d, and e: https://github.com/android/ndk/wiki/Changelog-r21

Would be interesting if upgrading to a newer r21 causes the same issue.

[justinpapezsonos]

That's the original r21 release, so it is pre-signing. You're missing everything described in r21b, c, d, and e: https://github.com/android/ndk/wiki/Changelog-r21

Would be interesting if upgrading to a newer r21 causes the same issue.

As you mentioned, we have not yet tried compiling against a later version of NDK r21 (such as b, c, d, or e). However, it may be worth mentioning that we have compiled against NDK r22, and none of our developers experienced increased build times.

[justinpapezsonos]

Just wanted to close the loop, your hunch was right @DanAlbert, the root issue is our corporate AV program. Thanks again for all the responses & help!

[DanAlbert]

Great, thanks for following up. I edited Pirama's initial comment a bit to make the thread more readable (seems only the first comment in a thread will be attached to the question as the "answer" when selected).