Merge branch 'main' into prometheus

CONTRIBUTING.md

@@ -1,84 +1,172 @@
 # Contributing Guide
 
-welcome, or something
+Requirements:
 
-# Setting up the dev environment
-## PDM
-We use `pdm` to manage dev dependencies.
-You can install `pdm` here [https://pdm.fming.dev/latest/#recommended-installation-method](https://pdm.fming.dev/latest/#recommended-installation-method).
+-   [Python 3.11](https://www.python.org/downloads/)
+-   [Git](https://git-scm.com/downloads)
+-   [PDM](https://pdm.fming.dev/latest/#recommended-installation-method)
+-   [Docker](https://docs.docker.com/engine/install/)
 
-Once installed, use `pdm install -d` to install dev dependencies.
+# Getting Started
+
+If you are a member of the organization, create a branch and a pull request when you are finished.
+
+```sh
+git clone https://github.com/vipyrsec/dragonfly-mainframe.git
+```
+
+Otherwise, work from a fork of the repository and create a pull request when you're done.
+Once you've forked the repository, go ahead and clone that fork with the following command.
+
+```sh
+git clone https://github.com/<YourUsernameHere>/dragonfly-mainframe.git
+```
+
+making sure to replace `<YourUsernameHere>` with your GitHub username.
+
+Now that you have your hands on the code, you can run
+
+```sh
+cd dragonfly-mainframe
+```
+
+in the directory which you cloned your repository into.
+
+Next, you'll want to install the project dependencies, with `pdm`. You can do so with the following command.
+
+```sh
+pdm install
+```
+
+You can activate the virtual environment with one of the following commands.
+
+```sh
+eval $(pdm venv activate for-test)  # If you're using bash/csh/zsh
+eval (pdm venv activate for-test)  # If you're using fish
+Invoke-Expression (pdm venv activate for-test)  # If you're using powershell
+```
+
+We use `pre-commit` to manage this project's pre-commit hooks, which are run before each commit you make to ensure your code follows linting and style rules. You can view the configuration in `.pre-commit-config.yaml`. You'll want to install the hooks with the following command.
+
+```sh
+pdm run pre-commit install
+```
+
+You can run
+
+```sh
+pdm run precommit
+```
+
+to have the pre-commit hooks run at any time.
+
+We recommend using `Docker` and `docker compose` for running the API, as that handles most of the setup in regards to environment variables and such for you. You can run the API with
+
+```sh
+pdm start
+```
+
+which will invoke `docker compose` for you.
+
+# Tests
+
+## Writing tests
 
-## Tests
-### Writing tests
 We use `pytest` to run our tests. Tests go in the `tests/` directory.
 The tests for each python module should go in a separate tests file.
 
 We use `requests` for making requests to the API. Use the fixture `api_url` for the URL to make requests to.
 For example:
+
 ```py
 def test_root(api_url: str):
     r = requests.get(api_url)
     assert r.status_code == 200
 ```
+
 Additionally, we can connect to the database to check if our request had the desired effect.
 The `db_session` fixture gives a `sqlalchemy.orm.Session` that you can use to make queries.
 The database is preloaded with some test data.
 For example:
+
 ```py
 def test_query(api_url: str, db_session: Session):
     r = requests.get(api_url + "/package?name=a&version=0.1.0")
     data = r.json()
     assert r["name"] == "a"
 ```
+
 All database changes are rolled back after each test, so you are given a fresh database with the original test data every time.
 
-### Running the tests
-#### Method 1 - Recommended
+## Running the tests
+
+### Method 1 - Recommended
+
 Use `pdm test`. This should be the go-to method.
 
-#### Method 2
-Alternatively you can run Postgresql locally or in a container, then run the server using `pdm run python -m uvicorn src.mainframe.server:app`.
+### Method 2
+
+Alternatively you can run PostgreSQL locally or in a container, then run the server using `pdm run python -m uvicorn src.mainframe.server:app`.
 To run the tests, use `pdm run pytest`.
 If you choose to use this method, be sure to set the environment variable `DB_URL` to the appropriate value, so that the tests can connect to the database.
 You can also use manual testing via a browser, or `curl`, for example, but this requires some additional setup, described in the Database Migrations section below.
 
-# Database Migrations
-We use `Alembic` to manage database migrations.
-Migrations handle changes between versions of a database schema.
-You only need to worry about this if you want to run manual tests, or you have made changes to the database schema.
+# Running the API
 
-## Generating revisions
-For each schema change, you should create a revision file using `pdm run alembic revision --autogenerate -m "<short change message>"`.
+## Method 1 - Recommended
 
-**Important**: You must then edit the created file and make sure it is correct.
+Use `docker` and `docker compose` to run the API, as that handles most of the setup in regards to environment variables and such for you. You can do so through running
 
-## Running migrations
-For manual testing, run `pdm run alembic upgrade head` in order to set up the database. If you get an error along the lines of `relation "packages" does not exist`, then you probably did not run the migration.
+```sh
+pdm start
+```
 
-## Environment Variables
-The following table illustrates configuration options in the form of environment variables that may be set.
+which is an alias for `docker compose up --build`.
 
+## Method 2 - Manual setup
 
-| Environment Variable      | Type | Default                                                 | Description                                                                                                                                                 |
+Alternatively, you'll want to run PostgreSQL locally or in a container, and run the API manually by invoking `entrypoint.sh`.
+
+```sh
+./entrypoint.sh
+```
+
+You'll need to have the following environment variables set.
+| Environment Variable | Type | Default | Description |
 |---------------------------|------|---------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `AUTH0_DOMAIN`            | str  | "vipyrsec.us.auth0.com"                                 | Authentication domain for Auth0                                                                                                                             |
-| `AUTH0_AUDIENCE`          | str  | "dragonfly.vipyrsec.com"                                | Audience field for Auth0                                                                                                                                    |
-| `DRAGONFLY_GITHUB_TOKEN`  | str  |                                                         | Github PAT for accessing YARA rules in the security-intelligence repository                                                                                 |
-| `JOB_TIMEOUT`             | int  | 60 * 2                                                  | The maximum time to wait for clients to respond with job results. After this time has elapsed, the server will begin distributing this job to other clients |
-|                           |      |                                                         |                                                                                                                                                             |
-| `EMAIL_RECIPIENT`         | str  | "[email protected]"                                     | The recipient address of report emails                                                                                                                      |
-| `BCC_RECIPIENTS`          | set  | set()                                                   | Additional addresses that should be BCC'd in email reports. Defaults to an empty set.                                                                       |
-| `DB_URL`                  | str  | "postgresql+asyncpg://postgres:postgres@localhost:5432" | PostgreSQL database connection string                                                                                                                       |
-|                           |      |                                                         |                                                                                                                                                             |
-| `SENTRY_DSN`              | str  | ""                                                      | Sentry Data Source Name (DSN)                                                                                                                               |
-| `SENTRY_ENVIRONMENT`      | str  | ""                                                      | Sentry environment                                                                                                                                          |
-| `SENTRY_RELEASE_PREFIX`   | str  | ""                                                      | Sentry release prefix                                                                                                                                       |
-|                           |      |                                                         |                                                                                                                                                             |
-| `MICROSOFT_TENANT_ID`     | str  |                                                         | Microsoft tenant ID for automated emails                                                                                                                    |
-| `MICROSOFT_CLIENT_ID`     | str  |                                                         | Microsoft client ID for automated emails                                                                                                                    |
-| `MICROSOFT_CLIENT_SECRET` | str  |                                                         | Microsoft client secret for automated emails                                                                                                                |
+| `AUTH0_DOMAIN` | str | "vipyrsec.us.auth0.com" | Authentication domain for Auth0 |
+| `AUTH0_AUDIENCE` | str | "dragonfly.vipyrsec.com" | Audience field for Auth0 |
+| `DRAGONFLY_GITHUB_TOKEN` | str | | Github PAT for accessing YARA rules in the security-intelligence repository |
+| `JOB_TIMEOUT` | int | 60 \* 2 | The maximum time to wait for clients to respond with job results. After this time has elapsed, the server will begin distributing this job to other clients |
+| | | | |
+| `EMAIL_RECIPIENT` | str | "[email protected]" | The recipient address of report emails |
+| `BCC_RECIPIENTS` | set | set() | Additional addresses that should be BCC'd in email reports. Defaults to an empty set. |
+| `DB_URL` | str | "postgresql+asyncpg://postgres:postgres@localhost:5432" | PostgreSQL database connection string |
+| | | | |
+| `SENTRY_DSN` | str | "" | Sentry Data Source Name (DSN) |
+| `SENTRY_ENVIRONMENT` | str | "" | Sentry environment |
+| `SENTRY_RELEASE_PREFIX` | str | "" | Sentry release prefix |
+| | | | |
+| `MICROSOFT_TENANT_ID` | str | | Microsoft tenant ID for automated emails |
+| `MICROSOFT_CLIENT_ID` | str | | Microsoft client ID for automated emails |
+| `MICROSOFT_CLIENT_SECRET` | str | | Microsoft client secret for automated emails |
 
 **NOTE**: Environment variables where the `default` column is empty are required for the application to startup
 
 **NOTE**: Environment variables with type `str` and `default` `""` are not required for the application to startup but may cause the application to function incorrectly
+
+# Database Migrations
+
+We use `Alembic` to manage database migrations.
+Migrations handle changes between versions of a database schema.
+You only need to worry about this if you want to run manual tests, or you have made changes to the database schema.
+
+## Generating revisions
+
+For each schema change, you should create a revision file using `pdm run alembic revision --autogenerate -m "<short change message>"`.
+
+**Always** check the generated migrations to see if they are accurate. There are many situations where alembic is unable to generate them correctly. You can refer to [this page](https://alembic.sqlalchemy.org/en/latest/autogenerate.html#what-does-autogenerate-detect-and-what-does-it-not-detect) for more details.
+
+## Running migrations
+
+For manual testing, run `pdm run alembic upgrade head` in order to set up the database.

Dockerfile

@@ -34,6 +34,6 @@ COPY src/ src/
 COPY entrypoint.sh ./
 RUN chmod +x entrypoint.sh
 
-CMD ["./entrypoint.sh"]
+CMD ["sh", "./entrypoint.sh"]
 
 EXPOSE 8000

docker-compose-tests.yaml

@@ -0,0 +1,8 @@
+version: "3.9"
+
+services:
+  mainframe:
+    build:
+      target: test
+      args:
+        GIT_SHA: testing

docker-compose.yaml

@@ -3,16 +3,14 @@ version: "3.9"
 services:
   mainframe:
     build:
-      target: test
       context: .
-      args:
-        GIT_SHA: testing
     ports:
       - "8000:8000"
     depends_on:
       db:
         condition: service_healthy
     tty: true
+    restart: always
     environment:
       # disable prepared statement cache because we want to be able to create and destroy tables for each test
       # when enabled, any schema change will invalidate the cache, causing an error
@@ -23,6 +21,8 @@ services:
       MICROSOFT_CLIENT_SECRET: client_secret
       EMAIL_RECIPIENT: email_recipient
       DRAGONFLY_GITHUB_TOKEN: test
+    volumes:
+      - "./src:/app/src"
 
   db:
     image: postgres

entrypoint.sh

@@ -1,2 +1,6 @@
 #!/bin/sh
-python -m alembic upgrade head && python -m uvicorn src.mainframe.server:app --host 0.0.0.0 && exec "$0"
+if [ "$GIT_SHA" = "development" ] || [ "$GIT_SHA" = "testing" ] ; then
+    python -m alembic upgrade head && python -m uvicorn src.mainframe.server:app --host 0.0.0.0 --reload && exec "$0"
+else
+    python -m alembic upgrade head && python -m uvicorn src.mainframe.server:app --host 0.0.0.0 && exec "$0"
+fi

pyproject.toml

@@ -43,12 +43,11 @@ logs = ["rich>=13.4.2"]
 [tool.pdm.scripts]
 format = { shell = "black . && isort ." }
 lint = { shell = "pyright; ruff ." }
-test = { shell = "docker compose up --build --exit-code-from mainframe" }
+test = { shell = "docker compose -f docker-compose.yaml -f docker-compose-tests.yaml up --build --exit-code-from mainframe" }
 precommit = { shell = "pre-commit run --all-files" }
 all = { composite = ["format", "lint", "test"] }
 
-start = { cmd = "uvicorn src.mainframe.server:app" }
-dev = { cmd = "uvicorn src.mainframe.server:app --reload" }
+start = { cmd = "docker compose up --build" }
 
 [tool.isort]
 profile = "black"

src/mainframe/constants.py

@@ -22,7 +22,7 @@ class Mainframe(EnvConfig):
     auth0_domain: str = ""
     auth0_audience: str = ""
 
-    email_recipient: str
+    email_recipient: str = "[email protected]"
     bcc_recipients: set[str] = set()
 
     db_url: str = "postgresql+asyncpg://postgres:postgres@localhost:5432"

src/mainframe/database.py

@@ -4,7 +4,7 @@
 
 from mainframe.constants import mainframe_settings
 
-engine = create_async_engine(mainframe_settings.db_url)
+engine = create_async_engine(mainframe_settings.db_url, pool_size=25)
 async_session = async_sessionmaker(bind=engine, expire_on_commit=False)