feat: refactor to maintain policy state in-memory

[cmaclaughlin]

Re: #25

The capacity scheduling plugin was used for frame of reference. A collection of policy information is kept in-memory on the PlacementPolicyManager. Instead of annotations, the policy info in the collection includes 2 sets:

• all pods that qualify for / fall under the policy
• all pods assigned to nodes in accordance with the policy ("managed by")

An event handler was attached to the pod informer so pod deletion can trigger updating the in-memory version of the policy accordingly. Additionally, the custom info object is included in the state object added to cycle state as a part of PreFilter and PreScore stages so that Filter and Score stages have access to it as well.

[helayoty]

The way you implemented it is great. Thank you for your efforts!

[aramase]

Thanks for the PR!

I've reviewed core.go and policy.go mostly and added comments. I'll review the other files next and add comments if any.

[aramase]

Do you know if Delete from the set is thread safe?

[cmaclaughlin]

I don't think it is since it is implemented via map[string]struct{} and delete(map, key) is not thread safe.

[cmaclaughlin]

Would you like a thread safe implementation or leave it as-is?

[aramase]

Added few more comments!

One thing I noticed is that PreScore now depends on cycle state set by the PreFilter phase. If a user wants to to use the scheduler plugin only for the preScore and score extension points and not configure it for preFilter and filter the plugin will now not work. IMO we should not depend on state being set by a different extension point.

[cmaclaughlin]

One thing I noticed is that PreScore now depends on cycle state set by the PreFilter phase. If a user wants to to use the scheduler plugin only for the preScore and score extension points and not configure it for preFilter and filter the plugin will now not work. IMO we should not depend on state being set by a different extension point.

I've removed that dependency now. For context, I had been looking at the Volume Binding plugin which sets state data in PreFilter and then reads/consumes it in all other extension points (Filter, Score, Reserve and PreBind). Out of curiosity, though, if the user only wants to use PreScore and Score and disables PreFilter and Filter, any policy with a Strict enforcement mode is essentially ignored. Is this the intent?

[helayoty]

@aramase what is the status of the PR? Can we merge it soon?