Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement new and improved terraform-base deployment #3089

Merged
merged 18 commits into from
Mar 12, 2024
Merged

Implement new and improved terraform-base deployment #3089

merged 18 commits into from
Mar 12, 2024

Conversation

frrist
Copy link
Member

@frrist frrist commented Dec 11, 2023 •
edited
Loading

@frrist frrist linked an issue Dec 11, 2023 that may be closed by this pull request
Refactor Terrafrorm to support modules, dynamic config, and configurable number of compute nodes #3090
Closed
@frrist frrist changed the title Frrist/tf Improved Bacalhau Terraform Deployment Dec 11, 2023
@frrist frrist changed the title Improved Bacalhau Terraform Deployment WIP: Improved Bacalhau Terraform Deployment Jan 24, 2024
@frrist frrist self-assigned this Jan 24, 2024
@frrist frrist force-pushed the frrist/tf branch 3 times, most recently from 0bf6b49 to d6a85d5 Compare January 30, 2024 22:01
@frrist frrist force-pushed the frrist/tf branch 2 times, most recently from e390372 to 8f7e391 Compare February 8, 2024 20:47
@frrist frrist marked this pull request as ready for review February 8, 2024 21:50
@frrist frrist changed the title WIP: Improved Bacalhau Terraform Deployment Implement new and improved terraform-base deployment Feb 8, 2024
@frrist frrist added type/enhancement Type: New features or enhancements to existing features type/tech-debt Type: Issues meant to address technical debt feature and removed do-not-merge wip labels Feb 8, 2024
This was referenced Feb 9, 2024
Closed
Closed
@frrist frrist mentioned this pull request Feb 14, 2024
Open
simonwo
simonwo reviewed Feb 19, 2024
View reviewed changes
Copy link
Contributor

@simonwo simonwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This broadly looks good to me... will this mean we have new two terraform folders now? terraform and tf? Are we aiming to replace the former?

@frrist
Copy link
Member Author

frrist commented Feb 19, 2024 •
edited
Loading

will this mean we have new two terraform folders now? terraform and tf?

I think it depends. The features we have in our deployments to development/staging/production are specific to our needs as operators of a public network. e.g. limiting the HTTP endpoints jobs can query when executed by a compute node, promtail agent, ipfs running as a separate service, pre-defined IP addresses (for DNS), hard-coded network size, lack of auth, etc.

The terraform in tf which will be used for marketplace deployments doesn't/cannot have some of these custom things.

I think the ideal scenario would be us using the same code for our networks as users use for MP deployments, perhaps we can merge these things together in the future? WDYT?

simonwo
simonwo approved these changes Feb 29, 2024
View reviewed changes
Copy link
Contributor

@simonwo simonwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think in the future it would be nice to use one as a base for the other. But we can work forward to that from here.

Maybe a more descriptive folder name would be good? I.e. marketplace or something?

js-ts and others added 17 commits March 12, 2024 12:08
@js-ts
Make changes to incorporate VMI
bdd71bd
feat: enable aws s3 access via env vars in service file
92d60bd
feat: add support for enabling networed jobs
8598b6b
feat: move bacalhau repo to own disk
99e2de0
feat: support local directory publisher
ff9f379
feat: add otel agent to collect bacalhau telemetry
82b9a0e 
- agent currently logs telemetry. later can be configured to pusblish to
  an endpoint: either one we host, or something like google stackdriver
feat: implement centralized otel collector via terraform
3f35596 
- closes #expanso-planning/issues/429
feat: implement token-based authn/z & NATs
03a20c8 
- still need HTTPS
feat: allow different versions of bacalhau to be installed
248131c 
gracefully handle version upgrades, ensure the attached disks re-attach
fix: configure otel collector correctly
75d396c 
- don't export tracing data
fix: terraform ignores state changes to attached disks
fe72e47 
also set correct env vars to install bacalhau from source
fix: set bacalhau_otel_collector_endpoint in vars
734ba65
feat: better install script, install via commit
98ab93e
feat: wire up tokens for compute nodes
256e952
refactor: simplify variables
29c4f70 
- do not repeat yourself terraform practices
fix: add earthly to deps in install script
c86e8e4
rename terraform for marketplace
ea39a5f
@coderabbitai coderabbitai
Copy link

coderabbitai bot commented Mar 12, 2024

Important

Auto Review Skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository.

To trigger a single review, invoke the @coderabbitai review command.

Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit-tests for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit tests for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit tests.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • The JSON schema for the configuration file is available here.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json

CodeRabbit Discord Community

Join our Discord Community to get help, request features, and share feedback.

@frrist
Copy link
Member Author

frrist commented Mar 12, 2024

Maybe a more descriptive folder name would be good? I.e. marketplace or something?

Done. merging once CI passes.

@frrist frrist merged commit a705550 into main Mar 12, 2024
12 checks passed
@frrist frrist deleted the frrist/tf branch March 12, 2024 20:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@simonwo simonwo simonwo approved these changes

@rossjones rossjones Awaiting requested review from rossjones

@wdbaruni wdbaruni Awaiting requested review from wdbaruni

Assignees

@frrist frrist

Labels
type/enhancement Type: New features or enhancements to existing features type/tech-debt Type: Issues meant to address technical debt
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Refactor Terrafrorm to support modules, dynamic config, and configurable number of compute nodes
4 participants
@frrist @simonwo @aronchick @js-ts