Add the openfn package to platform. Also fixed CU-86bzwyh69 #322
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| # Package Documentation | ||
|
|
||
| ## Introduction | ||
|
|
||
| Welcome to the documentation for the `openfn` package! This package is designed to provide a platform for seamless integration and automation of data workflows. Whether you are a developer, data analyst, or data scientist, this package will help you streamline your data processing tasks. | ||
|
|
||
| ## Usage | ||
|
|
||
| Once you have added the `openfn` package, you can start using it in your projects. Here is how to instantiate the package | ||
|
|
||
| `instant package init -n openfn --dev` | ||
|
|
||
| ## Demo | ||
|
|
||
| To get a hands-on experience with the `openfn` package, try out the demo. The demo showcases the package's capabilities and provides a sample project used to export data from CDR to NDR with transformations. It utilizes a Kafka queue and a custom adapter to map Bundles to be compliant with the FHIR Implementation Guide (IG). | ||
|
|
||
| ### Getting Started | ||
|
|
||
| To access the demo, follow these steps: | ||
|
|
||
| 1. Visit the [OpenFn Demo](http://localhost:4000) website. | ||
|
There was a problem hiding this comment. Update the demo URL The current demo URL (http://localhost:4000) appears to be a local development URL. For the public documentation, this should be updated to the actual public demo URL. Please replace |
||
| 2. Use the following demo credentials | ||
|
|
||
| ``` | ||
| username: [email protected] | ||
| password: instant101 | ||
| ``` | ||
|
Comment on lines
+24
to
+27
There was a problem hiding this comment. Security concern: Exposed credentials Exposing login credentials in public documentation is a security risk, even if they are for a demo account. Consider one of the following alternatives:
🧰 Tools🪛 Markdownlint
|
||
|
|
||
| 3. Configure the Kafka trigger | ||
| Change the trigger type from webhook to “Kafka Consumer” | ||
| Enter in configuration details → see [docs](https://docs.google.com/document/d/1cefHnp6IS6zvFwqQs8EsMQo5D4npsoE-S33R4OBpQjI/edit?usp=sharing) | ||
| Kafka topic: {whichever you want to use} (e.g., “cdr-ndr”) | ||
| Hosts: {cdr host name} | ||
| Initial offset reset policy: earliest | ||
| Connection timeout: 30 (default value, but can be adjusted) | ||
| Warning: Check Disable this trigger to ensure that consumption doesn’t start until you are ready to run the workflow! Once unchecked, it will immediately start consuming messages off the topic. | ||
|
|
||
| ### Documentation | ||
|
|
||
| For more detailed information on the `openfn` package and its functionalities, please refer to the [official documentation](https://github.com/openfn/docs). The documentation covers various topics, including installation instructions, usage guidelines, and advanced features. | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,184 @@ | ||
| ## Environment Variables | ||
|
|
||
| <table> | ||
| <thead> | ||
| <tr> | ||
| <th>Variable Name</th> | ||
| <th>Description</th> | ||
| <th>Type</th> | ||
| <th>Relevance</th> | ||
| <th>Required</th> | ||
| <th>Default</th> | ||
| </tr> | ||
| </thead> | ||
| <tbody> | ||
| <tr> | ||
| <td>DATABASE_URL</td> | ||
| <td>The URL of the PostgreSQL database</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>DISABLE_DB_SSL</td> | ||
| <td>Whether to disable SSL for the database connection</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>IS_RESETTABLE_DEMO</td> | ||
| <td>Whether the application is running in resettable demo mode</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>LISTEN_ADDRESS</td> | ||
| <td>The IP address to listen on</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>LOG_LEVEL</td> | ||
| <td>The log level for the application</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>ORIGINS</td> | ||
| <td>The allowed origins for CORS</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>PRIMARY_ENCRYPTION_KEY</td> | ||
| <td>The primary encryption key</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>SECRET_KEY_BASE</td> | ||
| <td>The secret key base</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>WORKER_RUNS_PRIVATE_KEY</td> | ||
| <td>The private key for worker runs</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>POSTGRES_USER</td> | ||
| <td>The username for the PostgreSQL database</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>POSTGRES_SERVICE</td> | ||
| <td>The service name for the PostgreSQL database</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>POSTGRES_DATABASE</td> | ||
| <td>The name of the PostgreSQL database</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>POSTGRES_PASSWORD</td> | ||
| <td>The password for the PostgreSQL database</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>POSTGRES_PORT</td> | ||
| <td>The port number for the PostgreSQL database</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>OpenFn_POSTGRESQL_DB</td> | ||
| <td>The name of the OpenFn PostgreSQL database</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>OpenFn_POSTGRESQL_USERNAME</td> | ||
| <td>The username for the OpenFn PostgreSQL database</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>OpenFn_POSTGRESQL_PASSWORD</td> | ||
| <td>The password for the OpenFn PostgreSQL database</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>WORKER_LIGHTNING_PUBLIC_KEY</td> | ||
| <td>The public key for the worker lightning</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>WORKER_SECRET</td> | ||
| <td>The secret key for the worker</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>OpenFn_IMAGE</td> | ||
| <td>The image name for OpenFn</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| <tr> | ||
| <td>OpenFn_WORKER_IMAGE</td> | ||
| <td>The image name for OpenFn worker</td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| <td></td> | ||
| </tr> | ||
| </tbody> | ||
| </table> | ||
|
Comment on lines
+1
to
+184
There was a problem hiding this comment. Complete the environment variables documentation. The table is missing information in several columns: Type, Relevance, Required, and Default. Providing this information will help users understand how to configure these variables. | Variable Name | Description | Type | Relevance | Required | Default |
|-------------------------|-------------------------------------------------|---------|-----------|----------|-------------------|
| DATABASE_URL | The URL of the PostgreSQL database | String | Critical | Yes | None |
| DISABLE_DB_SSL | Whether to disable SSL for the database connection | Boolean | Optional | No | false |
| ... | ... | ... | ... | ... | ... | |
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| version: '3.9' | ||
|
|
||
| services: | ||
| openfn: | ||
| ports: | ||
| - target: 4000 | ||
| published: 4000 | ||
| mode: host | ||
| worker: | ||
| ports: | ||
| - target: 2222 | ||
| published: 2222 | ||
| mode: host |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| version: '3.9' | ||
|
|
||
| services: | ||
| openfn: | ||
| image: ${OpenFn_IMAGE} | ||
| # This command runs a shell script that performs the following actions: | ||
| # 1. Executes the Lightning.Release.migrate() function to handle database migrations. | ||
| # 2. Sets up a user with the provided first name, last name, email, password, and role using the Lightning.Setup.setup_user function. | ||
| # - The user details are hardcoded with the first name "Test", last name "User", email "[email protected]", and password "instant101". | ||
| # - The role assigned to the user is "superuser". | ||
| # - The function also takes an API key from the environment variable ${OPENFN_API_KEY}. | ||
| # - Additionally, it sets up a schema with the name "openhim ndr" and type "http", including credentials and a base URL from environment variables. | ||
| # 3. Starts the Lightning application using the /app/bin/lightning start command. | ||
| command: > | ||
| sh -c "/app/bin/lightning eval 'Lightning.Release.migrate()' && /app/bin/lightning eval 'Lightning.Setup.setup_user(%{first_name: \"Test\",last_name: \"User\",email: \"[email protected]\",password: \"instant101\", role: :superuser}, \"${OPENFN_API_KEY}\" ,[%{name: \"openhim ndr\", schema: \"http\", body: %{username: \"${FHIR_SERVER_USERNAME}\", password: \"${FHIR_SERVER_PASSWORD}\", baseUrl: \"${FHIR_SERVER_BASE_URL}\"}}])' && /app/bin/lightning start" | ||
| deploy: | ||
| resources: | ||
| limits: | ||
| cpus: '${DOCKER_WEB_CPUS:-0}' | ||
| memory: '${DOCKER_WEB_MEMORY:-0}' | ||
| environment: | ||
| - DATABASE_URL=${DATABASE_URL} | ||
| - DISABLE_DB_SSL=${DISABLE_DB_SSL} | ||
| - IS_RESETTABLE_DEMO=${IS_RESETTABLE_DEMO} | ||
| - LISTEN_ADDRESS=${LISTEN_ADDRESS} | ||
| - LOG_LEVEL=${LOG_LEVEL} | ||
| - ORIGINS=${ORIGINS} | ||
| - PRIMARY_ENCRYPTION_KEY=${PRIMARY_ENCRYPTION_KEY} | ||
| - SECRET_KEY_BASE=${SECRET_KEY_BASE} | ||
| - WORKER_RUNS_PRIVATE_KEY=${WORKER_RUNS_PRIVATE_KEY} | ||
| - WORKER_SECRET=${WORKER_SECRET} | ||
| - KAFKA_TRIGGERS_ENABLED=${KAFKA_TRIGGERS_ENABLED} | ||
| healthcheck: | ||
| test: '${DOCKER_WEB_HEALTHCHECK_TEST:-curl localhost:4000/health_check}' | ||
| interval: '10s' | ||
| timeout: '3s' | ||
| start_period: '5s' | ||
| retries: 3 | ||
| networks: | ||
| - kafka_public | ||
| - postgres | ||
| worker: | ||
| image: ${OpenFn_WORKER_IMAGE} | ||
| deploy: | ||
| resources: | ||
| limits: | ||
| cpus: '${DOCKER_WORKER_CPUS:-0}' | ||
| memory: '${DOCKER_WORKER_MEMORY:-0}' | ||
| environment: | ||
| - WORKER_LIGHTNING_PUBLIC_KEY=${WORKER_LIGHTNING_PUBLIC_KEY} | ||
| - WORKER_SECRET=${WORKER_SECRET} | ||
| - NODE_ENV=production | ||
| command: [ 'pnpm', 'start:prod', '-l', 'ws://openfn:${URL_PORT-4000}/worker' ] | ||
| networks: | ||
| - kafka_public | ||
| - postgres | ||
|
|
||
| networks: | ||
| kafka_public: | ||
| name: kafka_public | ||
| external: true | ||
| postgres: | ||
| name: postgres_public | ||
| external: true |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think this should be deployed by default for CDR, let's leave it as optional.