Add the openfn package to platform. Also fixed CU-86bzwyh69 #322
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| # Package Documentation | ||
|
|
||
| ## Introduction | ||
|
|
||
| Welcome to the documentation for the `openfn` package! This package is designed to provide a platform for seamless integration and automation of data workflows. Whether you are a developer, data analyst, or data scientist, this package will help you streamline your data processing tasks. | ||
|
|
||
| ## Usage | ||
|
|
||
| Once you have added the `openfn` package, you can start using it in your projects. Here is how to instantiate the package | ||
|
|
||
| `instant package init -n openfn --dev` | ||
|
|
||
| ## Demo | ||
|
|
||
| To get a hands-on experience with the `openfn` package, you can try out the demo. The demo showcases the capabilities of the package and provides. | ||
|
|
||
| ### Getting Started | ||
|
|
||
| To access the demo, follow these steps: | ||
|
|
||
| 1. Visit the [OpenFn Demo](http://localhost:4000) website. | ||
| 2. Use the following demo credentials | ||
|
|
||
| ``` | ||
| username: [email protected] | ||
| password: welcome123 | ||
| ``` | ||
|
|
||
| ### Documentation | ||
|
|
||
| For more detailed information on the `openfn` package and its functionalities, please refer to the [official documentation](https://github.com/openfn/docs). The documentation covers various topics, including installation instructions, usage guidelines, and advanced features. | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,184 @@ | ||
| ## Environment Variables | ||
|
|
||
| <table> | ||
| <thead> | ||
| <tr> | ||
| <th>Variable Name</th> | ||
| <th>Description</th> | ||
| <th>Type</th> | ||
| <th>Relevance</th> | ||
| <th>Required</th> | ||
| <th>Default</th> | ||
| </tr> | ||
| </thead> | ||
| <tbody> | ||
| <tr> | ||
| <td>DATABASE_URL</td> | ||
| <td>The URL of the PostgreSQL database</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>DISABLE_DB_SSL</td> | ||
| <td>Whether to disable SSL for the database connection</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>IS_RESETTABLE_DEMO</td> | ||
| <td>Whether the application is running in resettable demo mode</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>LISTEN_ADDRESS</td> | ||
| <td>The IP address to listen on</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>LOG_LEVEL</td> | ||
| <td>The log level for the application</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>ORIGINS</td> | ||
| <td>The allowed origins for CORS</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>PRIMARY_ENCRYPTION_KEY</td> | ||
| <td>The primary encryption key</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>SECRET_KEY_BASE</td> | ||
| <td>The secret key base</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>WORKER_RUNS_PRIVATE_KEY</td> | ||
| <td>The private key for worker runs</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>POSTGRES_USER</td> | ||
| <td>The username for the PostgreSQL database</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>POSTGRES_SERVICE</td> | ||
| <td>The service name for the PostgreSQL database</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>POSTGRES_DATABASE</td> | ||
| <td>The name of the PostgreSQL database</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>POSTGRES_PASSWORD</td> | ||
| <td>The password for the PostgreSQL database</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>POSTGRES_PORT</td> | ||
| <td>The port number for the PostgreSQL database</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>OpenFn_POSTGRESQL_DB</td> | ||
| <td>The name of the OpenFn PostgreSQL database</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>OpenFn_POSTGRESQL_USERNAME</td> | ||
| <td>The username for the OpenFn PostgreSQL database</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>OpenFn_POSTGRESQL_PASSWORD</td> | ||
| <td>The password for the OpenFn PostgreSQL database</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>WORKER_LIGHTNING_PUBLIC_KEY</td> | ||
| <td>The public key for the worker lightning</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>WORKER_SECRET</td> | ||
| <td>The secret key for the worker</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>OpenFn_IMAGE</td> | ||
| <td>The image name for OpenFn</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>OpenFn_WORKER_IMAGE</td> | ||
| <td>The image name for OpenFn worker</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| </tbody> | ||
| </table> | ||
|
Comment on lines
+1
to
+184
There was a problem hiding this comment. Complete the environment variables documentation. The table is missing information in several columns: Type, Relevance, Required, and Default. Providing this information will help users understand how to configure these variables. | Variable Name | Description | Type | Relevance | Required | Default |
|-------------------------|-------------------------------------------------|---------|-----------|----------|-------------------|
| DATABASE_URL | The URL of the PostgreSQL database | String | Critical | Yes | None |
| DISABLE_DB_SSL | Whether to disable SSL for the database connection | Boolean | Optional | No | false |
| ... | ... | ... | ... | ... | ... | |
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| version: '3.9' | ||
|
|
||
| services: | ||
| openfn: | ||
| ports: | ||
| - target: 4000 | ||
| published: 4000 | ||
| mode: host |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| version: '3.9' | ||
|
|
||
| services: | ||
| openfn: | ||
| image: ${OpenFn_IMAGE} | ||
| command: sh -c '/app/bin/lightning eval "Lightning.Release.migrate()" && /app/bin/lightning eval "Lightning.Demo.reset_demo()" && /app/bin/lightning start' | ||
| deploy: | ||
| resources: | ||
| limits: | ||
| cpus: '${DOCKER_WEB_CPUS:-0}' | ||
| memory: '${DOCKER_WEB_MEMORY:-0}' | ||
| environment: | ||
| - DATABASE_URL=${DATABASE_URL} | ||
| - DISABLE_DB_SSL=${DISABLE_DB_SSL} | ||
| - IS_RESETTABLE_DEMO=${IS_RESETTABLE_DEMO} | ||
| - LISTEN_ADDRESS=${LISTEN_ADDRESS} | ||
| - LOG_LEVEL=${LOG_LEVEL} | ||
| - ORIGINS=${ORIGINS} | ||
| - PRIMARY_ENCRYPTION_KEY=${PRIMARY_ENCRYPTION_KEY} | ||
| - SECRET_KEY_BASE=${SECRET_KEY_BASE} | ||
| - WORKER_RUNS_PRIVATE_KEY=${WORKER_RUNS_PRIVATE_KEY} | ||
| - WORKER_SECRET=${WORKER_SECRET} | ||
| - KAFKA_TRIGGERS_ENABLED=${KAFKA_TRIGGERS_ENABLED} | ||
| healthcheck: | ||
| test: '${DOCKER_WEB_HEALTHCHECK_TEST:-curl localhost:4000/health_check}' | ||
| interval: '10s' | ||
| timeout: '3s' | ||
| start_period: '5s' | ||
| retries: 3 | ||
| networks: | ||
| - kafka_public | ||
| - postgres | ||
|
|
||
| networks: | ||
| kafka_public: | ||
| name: kafka_public | ||
| external: true | ||
| postgres: | ||
| name: postgres_public | ||
| external: true |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| const { Pool } = require("pg"); | ||
|
|
||
| const user = process.env.POSTGRES_USER || "postgres"; | ||
| const host = process.env.POSTGRES_SERVICE || "postgres-1"; | ||
| const database = process.env.POSTGRES_DATABASE || "postgres"; | ||
| const password = process.env.POSTGRES_PASSWORD || "instant101"; | ||
| const port = process.env.POSTGRES_PORT || 5432; | ||
| const newDb = process.env.NEW_DATABASE_NAME || "openfn"; | ||
| const newUser = process.env.NEW_DATABASE_USER || "openfn"; | ||
| const newUserPassword = process.env.NEW_DATABASE_PASSWORD || "instant101"; | ||
|
|
||
| const pool = new Pool({ | ||
| user, | ||
| host, | ||
| database, | ||
| password, | ||
| port, | ||
| }); | ||
|
|
||
| const tableQueries = []; | ||
| const insertQueries = []; | ||
|
|
||
| (async () => { | ||
| const client = await pool.connect(); | ||
|
|
||
| const createDb = async (db) => { | ||
| //Check db exists before creating | ||
|
|
||
| const result = await client.query( | ||
| "SELECT 1 FROM pg_database WHERE datname = $1", | ||
| [db] | ||
| ); | ||
|
|
||
| if (!result.rows.length) { | ||
| await client.query('CREATE DATABASE $1;', [db]); | ||
|
|
||
| console.log(`Database '${db}' created successfully`); | ||
| } else { | ||
| console.log(`Database '${db}' already exists`); | ||
| } | ||
| }; | ||
|
|
||
| const createUser = async () => { | ||
| const user = await client.query( | ||
| "SELECT 1 FROM pg_user WHERE usename = $1", | ||
| [newUser] | ||
| ); | ||
|
|
||
| if (!user.rows.length) { | ||
| await client.query( | ||
| 'CREATE USER $1 WITH ENCRYPTED PASSWORD $2;', [newUser, newUserPassword] | ||
| await client.query( | ||
| `GRANT ALL PRIVILEGES ON DATABASE ${newDb} TO ${newUser};` | ||
drizzentic marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ); | ||
| console.log(`User ${newUser} created`); | ||
| } | ||
| }; | ||
|
|
||
| try { | ||
| await createDb(newDb); | ||
|
|
||
| await createUser(); | ||
| await Promise.all(tableQueries.map((query) => client.query(query))); | ||
|
|
||
| await Promise.all(insertQueries.map((query) => client.query(query))); | ||
| } catch (error) { | ||
| console.error("Error in db operations:", error.message); | ||
| } finally { | ||
| client.release(); | ||
| pool.end(); | ||
| } | ||
| })(); | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Update the demo URL
The current demo URL (http://localhost:4000) appears to be a local development URL. For the public documentation, this should be updated to the actual public demo URL.
Please replace
http://localhost:4000with the correct public demo URL.