-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add the openfn package to platform. Also fixed CU-86bzwyh69 #322
base: main
Are you sure you want to change the base?
Changes from 8 commits
40f1191
6259af7
f2ecd0f
e3b9bd8
86ebef3
9819edb
8f63cb3
01114fe
97daf98
ea58c2e
2490ccd
88d086c
e67847f
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
# Package Documentation | ||
|
||
## Introduction | ||
|
||
Welcome to the documentation for the `openfn` package! This package is designed to provide a platform for seamless integration and automation of data workflows. Whether you are a developer, data analyst, or data scientist, this package will help you streamline your data processing tasks. | ||
|
||
## Usage | ||
|
||
Once you have added the `openfn` package, you can start using it in your projects. Here is how to instantiate the package | ||
|
||
`instant package init -n openfn --dev` | ||
|
||
## Demo | ||
|
||
To get a hands-on experience with the `openfn` package, you can try out the demo. The demo showcases the capabilities of the package and provides. | ||
|
||
### Getting Started | ||
|
||
To access the demo, follow these steps: | ||
|
||
1. Visit the [OpenFn Demo](http://localhost:4000) website. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Update the demo URL The current demo URL (http://localhost:4000) appears to be a local development URL. For the public documentation, this should be updated to the actual public demo URL. Please replace |
||
2. Use the following demo credentials | ||
|
||
``` | ||
username: [email protected] | ||
password: welcome123 | ||
``` | ||
|
||
### Documentation | ||
|
||
For more detailed information on the `openfn` package and its functionalities, please refer to the [official documentation](https://github.com/openfn/docs). The documentation covers various topics, including installation instructions, usage guidelines, and advanced features. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,184 @@ | ||
## Environment Variables | ||
|
||
<table> | ||
<thead> | ||
<tr> | ||
<th>Variable Name</th> | ||
<th>Description</th> | ||
<th>Type</th> | ||
<th>Relevance</th> | ||
<th>Required</th> | ||
<th>Default</th> | ||
</tr> | ||
</thead> | ||
<tbody> | ||
<tr> | ||
<td>DATABASE_URL</td> | ||
<td>The URL of the PostgreSQL database</td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
</tr> | ||
<tr> | ||
<td>DISABLE_DB_SSL</td> | ||
<td>Whether to disable SSL for the database connection</td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
</tr> | ||
<tr> | ||
<td>IS_RESETTABLE_DEMO</td> | ||
<td>Whether the application is running in resettable demo mode</td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
</tr> | ||
<tr> | ||
<td>LISTEN_ADDRESS</td> | ||
<td>The IP address to listen on</td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
</tr> | ||
<tr> | ||
<td>LOG_LEVEL</td> | ||
<td>The log level for the application</td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
</tr> | ||
<tr> | ||
<td>ORIGINS</td> | ||
<td>The allowed origins for CORS</td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
</tr> | ||
<tr> | ||
<td>PRIMARY_ENCRYPTION_KEY</td> | ||
<td>The primary encryption key</td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
</tr> | ||
<tr> | ||
<td>SECRET_KEY_BASE</td> | ||
<td>The secret key base</td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
</tr> | ||
<tr> | ||
<td>WORKER_RUNS_PRIVATE_KEY</td> | ||
<td>The private key for worker runs</td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
</tr> | ||
<tr> | ||
<td>POSTGRES_USER</td> | ||
<td>The username for the PostgreSQL database</td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
</tr> | ||
<tr> | ||
<td>POSTGRES_SERVICE</td> | ||
<td>The service name for the PostgreSQL database</td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
</tr> | ||
<tr> | ||
<td>POSTGRES_DATABASE</td> | ||
<td>The name of the PostgreSQL database</td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
</tr> | ||
<tr> | ||
<td>POSTGRES_PASSWORD</td> | ||
<td>The password for the PostgreSQL database</td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
</tr> | ||
<tr> | ||
<td>POSTGRES_PORT</td> | ||
<td>The port number for the PostgreSQL database</td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
</tr> | ||
<tr> | ||
<td>OpenFn_POSTGRESQL_DB</td> | ||
<td>The name of the OpenFn PostgreSQL database</td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
</tr> | ||
<tr> | ||
<td>OpenFn_POSTGRESQL_USERNAME</td> | ||
<td>The username for the OpenFn PostgreSQL database</td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
</tr> | ||
<tr> | ||
<td>OpenFn_POSTGRESQL_PASSWORD</td> | ||
<td>The password for the OpenFn PostgreSQL database</td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
</tr> | ||
<tr> | ||
<td>WORKER_LIGHTNING_PUBLIC_KEY</td> | ||
<td>The public key for the worker lightning</td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
</tr> | ||
<tr> | ||
<td>WORKER_SECRET</td> | ||
<td>The secret key for the worker</td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
</tr> | ||
<tr> | ||
<td>OpenFn_IMAGE</td> | ||
<td>The image name for OpenFn</td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
</tr> | ||
<tr> | ||
<td>OpenFn_WORKER_IMAGE</td> | ||
<td>The image name for OpenFn worker</td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
<td></td> | ||
</tr> | ||
</tbody> | ||
</table> | ||
Comment on lines
+1
to
+184
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Complete the environment variables documentation. The table is missing information in several columns: Type, Relevance, Required, and Default. Providing this information will help users understand how to configure these variables. | Variable Name | Description | Type | Relevance | Required | Default |
|-------------------------|-------------------------------------------------|---------|-----------|----------|-------------------|
| DATABASE_URL | The URL of the PostgreSQL database | String | Critical | Yes | None |
| DISABLE_DB_SSL | Whether to disable SSL for the database connection | Boolean | Optional | No | false |
| ... | ... | ... | ... | ... | ... | |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
version: '3.9' | ||
|
||
services: | ||
openfn: | ||
ports: | ||
- target: 4000 | ||
published: 4000 | ||
mode: host | ||
worker: | ||
ports: | ||
- target: 2222 | ||
published: 2222 | ||
mode: host |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,55 @@ | ||
version: '3.9' | ||
|
||
services: | ||
openfn: | ||
image: ${OpenFn_IMAGE} | ||
command: sh -c '/app/bin/lightning eval "Lightning.Release.migrate()" && /app/bin/lightning eval "Lightning.Demo.reset_demo()" && /app/bin/lightning start' | ||
deploy: | ||
resources: | ||
limits: | ||
cpus: '${DOCKER_WEB_CPUS:-0}' | ||
memory: '${DOCKER_WEB_MEMORY:-0}' | ||
environment: | ||
- DATABASE_URL=${DATABASE_URL} | ||
- DISABLE_DB_SSL=${DISABLE_DB_SSL} | ||
- IS_RESETTABLE_DEMO=${IS_RESETTABLE_DEMO} | ||
- LISTEN_ADDRESS=${LISTEN_ADDRESS} | ||
- LOG_LEVEL=${LOG_LEVEL} | ||
- ORIGINS=${ORIGINS} | ||
- PRIMARY_ENCRYPTION_KEY=${PRIMARY_ENCRYPTION_KEY} | ||
- SECRET_KEY_BASE=${SECRET_KEY_BASE} | ||
- WORKER_RUNS_PRIVATE_KEY=${WORKER_RUNS_PRIVATE_KEY} | ||
- WORKER_SECRET=${WORKER_SECRET} | ||
- KAFKA_TRIGGERS_ENABLED=${KAFKA_TRIGGERS_ENABLED} | ||
healthcheck: | ||
test: '${DOCKER_WEB_HEALTHCHECK_TEST:-curl localhost:4000/health_check}' | ||
interval: '10s' | ||
timeout: '3s' | ||
start_period: '5s' | ||
retries: 3 | ||
networks: | ||
- kafka_public | ||
- postgres | ||
worker: | ||
image: ${OpenFn_WORKER_IMAGE} | ||
deploy: | ||
resources: | ||
limits: | ||
cpus: '${DOCKER_WORKER_CPUS:-0}' | ||
memory: '${DOCKER_WEB_MEMORY:-0}' | ||
environment: | ||
- WORKER_LIGHTNING_PUBLIC_KEY=${WORKER_LIGHTNING_PUBLIC_KEY} | ||
- WORKER_SECRET=${WORKER_SECRET} | ||
- NODE_ENV=production | ||
command: [ 'pnpm', 'start:prod', '-l', 'ws://openfn:${URL_PORT-4000}/worker' ] | ||
networks: | ||
- kafka_public | ||
- postgres | ||
|
||
networks: | ||
kafka_public: | ||
name: kafka_public | ||
external: true | ||
postgres: | ||
name: postgres_public | ||
external: true |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,73 @@ | ||
const { Pool } = require("pg"); | ||
|
||
const user = process.env.POSTGRES_USER || "postgres"; | ||
const host = process.env.POSTGRES_SERVICE || "postgres-1"; | ||
const database = process.env.POSTGRES_DATABASE || "postgres"; | ||
const password = process.env.POSTGRES_PASSWORD || "instant101"; | ||
const port = process.env.POSTGRES_PORT || 5432; | ||
const newDb = process.env.NEW_DATABASE_NAME || "openfn"; | ||
const newUser = process.env.NEW_DATABASE_USER || "openfn"; | ||
const newUserPassword = process.env.NEW_DATABASE_PASSWORD || "instant101"; | ||
|
||
const pool = new Pool({ | ||
user, | ||
host, | ||
database, | ||
password, | ||
port, | ||
}); | ||
|
||
const tableQueries = []; | ||
const insertQueries = []; | ||
|
||
(async () => { | ||
const client = await pool.connect(); | ||
|
||
const createDb = async (db) => { | ||
//Check db exists before creating | ||
|
||
const result = await client.query( | ||
"SELECT 1 FROM pg_database WHERE datname = $1", | ||
[db] | ||
); | ||
|
||
if (!result.rows.length) { | ||
await client.query(`CREATE DATABASE ${db};`); | ||
drizzentic marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
console.log(`Database '${db}' created successfully`); | ||
} else { | ||
console.log(`Database '${db}' already exists`); | ||
} | ||
}; | ||
|
||
const createUser = async () => { | ||
const user = await client.query( | ||
"SELECT 1 FROM pg_user WHERE usename = $1", | ||
[newUser] | ||
); | ||
|
||
if (!user.rows.length) { | ||
await client.query( | ||
`CREATE USER ${newUser} WITH ENCRYPTED PASSWORD '${newUserPassword}';` | ||
); | ||
drizzentic marked this conversation as resolved.
Show resolved
Hide resolved
|
||
await client.query( | ||
`GRANT ALL PRIVILEGES ON DATABASE ${newDb} TO ${newUser};` | ||
drizzentic marked this conversation as resolved.
Show resolved
Hide resolved
|
||
); | ||
console.log(`User ${newUser} created`); | ||
} | ||
}; | ||
|
||
try { | ||
await createDb(newDb); | ||
|
||
await createUser(); | ||
await Promise.all(tableQueries.map((query) => client.query(query))); | ||
|
||
await Promise.all(insertQueries.map((query) => client.query(query))); | ||
} catch (error) { | ||
console.error("Error in db operations:", error.message); | ||
} finally { | ||
client.release(); | ||
pool.end(); | ||
} | ||
})(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think this should be deployed by default for CDR, let's leave it as optional.