Add Gmail client and update email functions (#944)

## Fixes issue #927 ## Description of Changes Add the ability to send emails from a Google Workspace account using a GCP service account and update the feature's respective documentation. ## Notes for Deployment - Need `service_account_key.json` file that is stored in LPL document storage. --------- Co-authored-by: abandoned-prototype <[email protected]>
lucyparsons · Jul 7, 2023 · c964a94 · c964a94
1 parent a3a6bd8
commit c964a94
Show file tree

Hide file tree

Showing 21 changed files with 261 additions and 209 deletions.
diff --git a/CONTRIB.md b/CONTRIB.md
@@ -1,9 +1,7 @@
 # Contributing Guide
-
 First, thanks for being interested in helping us out! If you find an issue you're interested in, feel free to make a comment about how you're thinking of approaching implementing it in the issue and we can give you feedback.  Please also read our [code of conduct](/CODE_OF_CONDUCT.md) before getting started.
 
 ## Submitting a Pull Request (PR)
-
 When you come to implement your new feature, clone the repository and then create a branch off `develop` locally and add commits to implement your feature.
 
 If your git history is not so clean, please do rewrite before you submit your PR - if you're not sure if you need to do this, go ahead and submit and we can let you know when you submit.
@@ -21,13 +19,11 @@ git config user.name "<your-github-username>"
 This will make sure that all commits you make locally are associated with your github account and do not contain any additional identifying information. More detailed information on this topic can be found [here](https://docs.github.com/en/free-pro-team@latest/github/setting-up-and-managing-your-github-user-account/setting-your-commit-email-address).
 
 ### Linting / Style Checks
+We use [pre-commit](https://pre-commit.com/) for automated linting and style checks. Be sure to [install pre-commit](https://pre-commit.com/#installation) and run `pre-commit install` in your local version of the repository to install our pre-commit checks. This will make sure your commits are always formatted correctly.
 
- We use [pre-commit](https://pre-commit.com/) for automated linting and style checks. Be sure to [install pre-commit](https://pre-commit.com/#installation) and run `pre-commit install` in your local version of the repository to install our pre-commit checks. This will make sure your commits are always formatted correctly.
-
- You can run `pre-commit run --all-files` or `make lint` to run pre-commit over your local codebase, or `pre-commit run` to run it only over the currently stages files.
+You can run `pre-commit run --all-files` or `make lint` to run pre-commit over your local codebase, or `pre-commit run` to run it only over the currently stages files.
 
 ## Development Environment
-
 You can use our Docker-compose environment to stand up a development OpenOversight.
 
 You will need to have Docker installed in order to use the Docker development environment.
@@ -53,8 +49,24 @@ $ docker exec -it openoversight_web_1 /bin/bash
 
 Once you're done, `make stop` and `make clean` to stop and remove the containers respectively.
 
-## Testing S3 Functionality
+## Gmail Requirements
+**NOTE:** If you are running on dev and do not currently have a `service_account_key.json` file, create one and leave it empty. The email client will then default to an empty object and simulate emails in the logs.
+
+For the application to work properly, you will need a [Google Cloud Platform service account](https://cloud.google.com/iam/docs/service-account-overview) that is attached to a GSuite email address. Here are some general tips for working with service accounts: [Link](https://support.google.com/a/answer/7378726?hl=en).
+We would suggest that you do not use a personal email address, but instead one that is used strictly for sending out OpenOversight emails.
+
+You will need to do these two things for the service account to work as a Gmail bot:
+1. Enable domain-wide delegation for the service account: [Link](https://support.google.com/a/answer/162106?hl=en)
+2. Enable the `https://www.googleapis.com/auth/gmail.send` scope in the Gmail API for your service account: [Link](https://developers.google.com/gmail/api/auth/scopes#scopes)
+3. Save the service account key file in OpenOversight's base folder as `service_account_key.json`. The file is in the `.gitignore` file GitHub will not allow you to save it, provided you've named it correctly.
+4. For production, save the email address associated with your service account to a variable named `OO_SERVICE_EMAIL` in a `.env` file in the base directory of this repository. For development and testing, update the `OO_SERVICE_EMAIL` variable in the `docker-compose.yml` file.
 
+Example `.env` variable:
+```bash
+OO_SERVICE_EMAIL="[email protected]"
+```
+
+## Testing S3 Functionality
 We use an S3 bucket for image uploads. If you are working on functionality involving image uploads,
 then you should follow the "S3 Image Hosting" section in [DEPLOY.md](/DEPLOY.md) to make a test S3 bucket
 on Amazon Web Services.
@@ -72,7 +84,6 @@ Now when you run `make dev` as usual in the same session, you will be able to su
 your test bucket.
 
 ## Database commands
-
 Running `make dev` will create the database and persist it into your local filesystem.
 
 You can access your PostgreSQL development database via psql using:
@@ -89,7 +100,6 @@ or
 `$ python test_data.py --cleanup` to delete the data
 
 ### Migrating the Database
-
 If you e.g. add a new column or table, you'll need to migrate the database using the Flask CLI. First we need to 'stamp' the current version of the database:
 
 ```sh
@@ -152,7 +162,6 @@ pip install -r dev-requirements.txt
 ```
 
 ## OpenOversight Management Interface
-
 In addition to generating database migrations, the Flask CLI can be used to run additional commands:
 
 ```sh

diff --git a/Makefile b/Makefile
@@ -3,19 +3,19 @@ export UID=$(shell id -u)
 default: build start create_db populate test stop clean
 
 .PHONY: build
-build:  ## Build containers
+build: ## Build containers
 	docker-compose build
 
 .PHONY: build_with_version
-build_with_version:
-	docker-compose build --build-arg TRAVIS_PYTHON_VERSION=$(PYTHON_VERSION)
+build_with_version: create_empty_secret create_empty_env
+	docker-compose build --build-arg MAKE_PYTHON_VERSION=$(PYTHON_VERSION)
 
 .PHONY: test_with_version
 test_with_version: build_with_version assets
 	FLASK_ENV=testing docker-compose run --rm web pytest --cov=OpenOversight --cov-report xml:OpenOversight/tests/coverage.xml --doctest-modules -n 4 --dist=loadfile -v OpenOversight/tests/
 
 .PHONY: start
-start: build  ## Run containers
+start: build ## Run containers
 	docker-compose up -d
 
 .PHONY: create_db
@@ -36,7 +36,7 @@ assets:
 dev: build start create_db populate
 
 .PHONY: populate
-populate: create_db  ## Build and run containers
+populate: create_db ## Build and run containers
 	@until docker-compose exec postgres psql -h localhost -U openoversight -c '\l' postgres &>/dev/null; do \
 		echo "Postgres is unavailable - sleeping..."; \
 		sleep 1; \
@@ -46,27 +46,27 @@ populate: create_db  ## Build and run containers
 	docker-compose run --rm web python ./test_data.py -p
 
 .PHONY: test
-test: start  ## Run tests
+test: start ## Run tests
 	if [ -z "$(name)" ]; then \
 		FLASK_ENV=testing docker-compose run --rm web pytest --cov --doctest-modules -n auto --dist=loadfile -v OpenOversight/tests/; \
 	else \
-	    FLASK_ENV=testing docker-compose run --rm web pytest --cov --doctest-modules -v OpenOversight/tests/ -k $(name); \
+		FLASK_ENV=testing docker-compose run --rm web pytest --cov --doctest-modules -v OpenOversight/tests/ -k $(name); \
 	fi
 
 .PHONY: lint
 lint:
 	pre-commit run --all-files
 
-.PHONY: cleanassets
-cleanassets:
+.PHONY: clean_assets
+clean_assets:
 	rm -rf ./OpenOversight/app/static/dist/
 
 .PHONY: stop
-stop:  ## Stop containers
+stop: ## Stop containers
 	docker-compose stop
 
 .PHONY: clean
-clean: cleanassets stop  ## Remove containers
+clean: clean_assets stop ## Remove containers
 	docker-compose rm -f
 
 .PHONY: clean_all
@@ -88,3 +88,18 @@ help: ## Print this message and exit
 .PHONY: attach
 attach:
 	docker-compose exec postgres psql -h localhost -U openoversight openoversight-dev
+
+# TODO: These two commands are the same with the exception of the file name, this should be addressed at some point
+.PHONY: create_empty_secret
+create_empty_secret: # This is needed to make sure docker doesn't create an empty directory, or delete that directory first
+	touch service_account_key.json || \
+	(echo "Need to delete that empty directory first"; \
+	 sudo rm -d service_account_key.json/; \
+	 touch service_account_key.json)
+
+.PHONY: create_empty_env
+create_empty_env: # This is needed to make sure docker doesn't create an empty directory, or delete that directory first
+	touch .env || \
+	(echo "Need to delete that empty directory first"; \
+	 sudo rm -d .env/; \
+	 touch .env)
diff --git a/OpenOversight/app/__init__.py b/OpenOversight/app/__init__.py
@@ -12,17 +12,17 @@
 from flask_limiter import Limiter
 from flask_limiter.util import get_remote_address
 from flask_login import LoginManager
-from flask_mail import Mail
 from flask_migrate import Migrate
 from flask_sitemap import Sitemap
 from flask_wtf.csrf import CSRFProtect
 from markupsafe import Markup
 
-from .config import config
+from OpenOversight.app.config import config
+from OpenOversight.app.email_client import EmailClient
+from OpenOversight.app.utils.constants import MEGABYTE, SERVICE_ACCOUNT_FILE
 
 
 bootstrap = Bootstrap()
-mail = Mail()
 
 login_manager = LoginManager()
 login_manager.session_protection = "strong"
@@ -43,12 +43,15 @@ def create_app(config_name="default"):
     from .models import db
 
     bootstrap.init_app(app)
-    mail.init_app(app)
+    csrf.init_app(app)
     db.init_app(app)
-    login_manager.init_app(app)
+    # This allows the application to run without creating an email client if it is
+    # in testing or dev mode and the service account file is empty.
+    service_account_file_size = os.path.getsize(SERVICE_ACCOUNT_FILE)
+    EmailClient(dev=app.debug and service_account_file_size == 0, testing=app.testing)
     limiter.init_app(app)
+    login_manager.init_app(app)
     sitemap.init_app(app)
-    csrf.init_app(app)
 
     from .main import main as main_blueprint
 
@@ -58,7 +61,7 @@ def create_app(config_name="default"):
 
     app.register_blueprint(auth_blueprint, url_prefix="/auth")
 
-    max_log_size = 10 * 1024 * 1024  # start new log file after 10 MB
+    max_log_size = 10 * MEGABYTE  # start new log file after 10 MB
     num_logs_to_keep = 5
     file_handler = RotatingFileHandler(
         "/tmp/openoversight.log", "a", max_log_size, num_logs_to_keep

diff --git a/OpenOversight/app/auth/utils.py b/OpenOversight/app/auth/utils.py
@@ -4,6 +4,7 @@
 from flask_login import current_user
 
 
+# TODO: Move these functions to the utils package
 def admin_required(f):
     @wraps(f)
     def decorated_function(*args, **kwargs):

diff --git a/OpenOversight/app/auth/views.py b/OpenOversight/app/auth/views.py
@@ -11,13 +11,20 @@
 )
 from flask_login import current_user, login_required, login_user, logout_user
 
+from OpenOversight.app.email_client import (
+    AdministratorApprovalEmail,
+    ChangeEmailAddressEmail,
+    ConfirmAccountEmail,
+    ConfirmedUserEmail,
+    EmailClient,
+    ResetPasswordEmail,
+)
+from OpenOversight.app.models import User, db
 from OpenOversight.app.utils.constants import HTTP_METHOD_GET, HTTP_METHOD_POST
 from OpenOversight.app.utils.forms import set_dynamic_default
 from OpenOversight.app.utils.general import validate_redirect_url
 
 from .. import sitemap
-from ..email import send_email
-from ..models import User, db
 from . import auth
 from .forms import (
     ChangeDefaultDepartmentForm,
@@ -99,7 +106,7 @@ def logout():
 @sitemap_include
 @auth.route("/register", methods=[HTTP_METHOD_GET, HTTP_METHOD_POST])
 def register():
-    jsloads = ["js/zxcvbn.js", "js/password.js"]
+    js_loads = ["js/zxcvbn.js", "js/password.js"]
     form = RegistrationForm()
     if form.validate_on_submit():
         user = User(
@@ -113,31 +120,23 @@ def register():
         if current_app.config["APPROVE_REGISTRATIONS"]:
             admins = User.query.filter_by(is_administrator=True).all()
             for admin in admins:
-                send_email(
-                    admin.email,
-                    "New user registered",
-                    "auth/email/new_registration",
-                    user=user,
-                    admin=admin,
+                EmailClient.send_email(
+                    AdministratorApprovalEmail(admin.email, user=user, admin=admin)
                 )
             flash(
                 "Once an administrator approves your registration, you will "
                 "receive a confirmation email to activate your account."
             )
         else:
             token = user.generate_confirmation_token()
-            send_email(
-                user.email,
-                "Confirm Your Account",
-                "auth/email/confirm",
-                user=user,
-                token=token,
+            EmailClient.send_email(
+                ConfirmAccountEmail(user.email, user=user, token=token)
             )
             flash("A confirmation email has been sent to you.")
         return redirect(url_for("auth.login"))
     else:
         current_app.logger.info(form.errors)
-    return render_template("auth/register.html", form=form, jsloads=jsloads)
+    return render_template("auth/register.html", form=form, jsloads=js_loads)
 
 
 @auth.route("/confirm/<token>", methods=[HTTP_METHOD_GET])
@@ -148,12 +147,8 @@ def confirm(token):
     if current_user.confirm(token):
         admins = User.query.filter_by(is_administrator=True).all()
         for admin in admins:
-            send_email(
-                admin.email,
-                "New user confirmed",
-                "auth/email/new_confirmation",
-                user=current_user,
-                admin=admin,
+            EmailClient.send_email(
+                ConfirmedUserEmail(admin.email, user=current_user, admin=admin)
             )
         flash("You have confirmed your account. Thanks!")
     else:
@@ -165,12 +160,8 @@ def confirm(token):
 @login_required
 def resend_confirmation():
     token = current_user.generate_confirmation_token()
-    send_email(
-        current_user.email,
-        "Confirm Your Account",
-        "auth/email/confirm",
-        user=current_user,
-        token=token,
+    EmailClient.send_email(
+        ConfirmAccountEmail(current_user.email, user=current_user, token=token)
     )
     flash("A new confirmation email has been sent to you.")
     return redirect(url_for("main.index"))
@@ -179,7 +170,7 @@ def resend_confirmation():
 @auth.route("/change-password", methods=[HTTP_METHOD_GET, HTTP_METHOD_POST])
 @login_required
 def change_password():
-    jsloads = ["js/zxcvbn.js", "js/password.js"]
+    js_loads = ["js/zxcvbn.js", "js/password.js"]
     form = ChangePasswordForm()
     if form.validate_on_submit():
         if current_user.verify_password(form.old_password.data):
@@ -192,7 +183,7 @@ def change_password():
             flash("Invalid password.")
     else:
         current_app.logger.info(form.errors)
-    return render_template("auth/change_password.html", form=form, jsloads=jsloads)
+    return render_template("auth/change_password.html", form=form, jsloads=js_loads)
 
 
 @auth.route("/reset", methods=[HTTP_METHOD_GET, HTTP_METHOD_POST])
@@ -204,12 +195,8 @@ def password_reset_request():
         user = User.by_email(form.email.data).first()
         if user:
             token = user.generate_reset_token()
-            send_email(
-                user.email,
-                "Reset Your Password",
-                "auth/email/reset_password",
-                user=user,
-                token=token,
+            EmailClient.send_email(
+                ResetPasswordEmail(user.email, user=user, token=token)
             )
         flash("An email with instructions to reset your password has been sent to you.")
         return redirect(url_for("auth.login"))
@@ -245,12 +232,8 @@ def change_email_request():
         if current_user.verify_password(form.password.data):
             new_email = form.email.data
             token = current_user.generate_email_change_token(new_email)
-            send_email(
-                new_email,
-                "Confirm your email address",
-                "auth/email/change_email",
-                user=current_user,
-                token=token,
+            EmailClient.send_email(
+                ChangeEmailAddressEmail(new_email, user=current_user, token=token)
             )
             flash(
                 "An email with instructions to confirm your new email "
@@ -338,7 +321,8 @@ def edit_user(user_id):
                 db.session.add(user)
                 db.session.commit()
 
-                # automatically send a confirmation email when approving an unconfirmed user
+                # automatically send a confirmation email when approving an
+                # unconfirmed user
                 if (
                     current_app.config["APPROVE_REGISTRATIONS"]
                     and not already_approved
@@ -376,12 +360,6 @@ def admin_resend_confirmation(user):
         flash("User {} is already confirmed.".format(user.username))
     else:
         token = user.generate_confirmation_token()
-        send_email(
-            user.email,
-            "Confirm Your Account",
-            "auth/email/confirm",
-            user=user,
-            token=token,
-        )
+        EmailClient.send_email(ConfirmAccountEmail(user.email, user=user, token=token))
         flash("A new confirmation email has been sent to {}.".format(user.email))
     return redirect(url_for("auth.get_users"))