Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Import - new modules #6350

Merged
merged 23 commits into from
Feb 20, 2024
Merged

Import - new modules #6350

merged 23 commits into from
Feb 20, 2024

Conversation

wildum
Copy link
Contributor

@wildum wildum commented Feb 9, 2024

PR Description

This PR adds the import functionality of the new modules with import.string and import.file.

The functionality is described in #5547

The change introduces a new type of node: ImportConfigNode.
It imports modules from an ImportSource to retrieve declare and import blocks.
For every import block, it creates an ImportConfigNode child. The child is evaluated and runs inside of its parent.
When the ImportSource retrieves new content, the ImportConfigNode updates its declares and import node children. Then it propagates an update to the root ImportConfigNode which will notify the controller that it has been updated.
The loader will then update the CustomComponentRegistry with new content and will trigger the dependants of the ImportConfigNode for reevaluation.

Notes to the Reviewer

The PR can be reviewed commit by commit.

PR Checklist

  • Tests updated

An import source retrieves a module from a source.
Add the ImportFile to retrieve a module from a file.
Add the ImportString to retrieve a module from a string.
Add a new config node that retrieves a module via an import source.
It will parse the module to collect declare and import blocks.
For every imported import block it will create ImportConfigNode children.
The children are evaluated and ran by the parent.
By navigating through the import tree, it's possible to access all imported declares.
When an ImportConfigNode is evaluated, the custom component registry is updated.
Custom components have edges to import node that they depend on.
The componentNodeManager can search in the customComponentRegistry for imported declares.
ImportConfigNodes are runnable nodes. They are ran by the controller.
When a custom component is instantiated with an imported declare, it will have the customComponentRegistry associated with the scope of the import.
@wildum wildum requested a review from rfratto February 9, 2024 17:11
Copy link
Member

@rfratto rfratto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks William! I don't have any huge comments, just a few nits, questions, and recommendations for helping making the tests shine.

pkg/flow/internal/controller/node_config_import.go Outdated Show resolved Hide resolved
pkg/flow/internal/controller/node_config_import.go Outdated Show resolved Hide resolved
pkg/flow/internal/controller/node_config_import.go Outdated Show resolved Hide resolved
pkg/flow/internal/controller/node_config_import.go Outdated Show resolved Hide resolved
pkg/flow/internal/controller/node_config_import.go Outdated Show resolved Hide resolved
pkg/flow/internal/controller/metrics.go Outdated Show resolved Hide resolved
pkg/flow/internal/controller/node_custom_component.go Outdated Show resolved Hide resolved
pkg/flow/import_test.go Show resolved Hide resolved
pkg/flow/import_test.go Show resolved Hide resolved
pkg/flow/import_test.go Outdated Show resolved Hide resolved
@wildum wildum requested a review from rfratto February 14, 2024 15:42
import.file should not have "isSecret" arg because the imported module is not exported.
It should then not use the same args as local.file
Copy link
Member

@rfratto rfratto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partial review; I wanted to leave a comment on something I noticed, but I still need to spend more time doing a full review of the current state.

pkg/flow/internal/controller/node_config_import.go Outdated Show resolved Hide resolved
Copy link
Member

@rfratto rfratto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks William! I left some final comments. All my comments here in this review are nits, but my feedback from before about the errChan is a little more firm and I'd like to ideally see that resolved at least before merging.

pkg/flow/import_test.go Outdated Show resolved Hide resolved
pkg/flow/import_test.txtar Outdated Show resolved Hide resolved
pkg/flow/internal/controller/component_node_manager.go Outdated Show resolved Hide resolved
pkg/flow/internal/controller/node_config_import.go Outdated Show resolved Hide resolved
pkg/flow/internal/controller/node_custom_component.go Outdated Show resolved Hide resolved
@wildum wildum requested a review from rfratto February 20, 2024 08:42
Copy link
Member

@rfratto rfratto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still LGTM (the new tests look great, thank you). After you change the Equals method as discussed feel free to merge :)

@wildum wildum merged commit 8967576 into main Feb 20, 2024
10 checks passed
@wildum wildum deleted the import-new-modules-2 branch February 20, 2024 14:54
@github-actions github-actions bot added the frozen-due-to-age Locked due to a period of inactivity. Please open new issues or PRs if more discussion is needed. label Mar 24, 2024
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 24, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
frozen-due-to-age Locked due to a period of inactivity. Please open new issues or PRs if more discussion is needed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants