Solr Search ETL

Atlas BI Library ETL | Solr Search Engine

Website • Demo • Documentation • Chat

Load Atlas metadata into the Solr search engine that powers Atlas' search.

🏃 Getting Started

In order to use these scripts you should already have Atlas published, with Solr started. See the Atlas BI Libary docs.

Dependencies

Java

For development purposes, sorl search can be started directly from the Atlas source code.

1. Install Java JRE
2. Add a system environment variable called JAVA_HOME with the path to java, for example C:\Program Files\Java\jdk-17.0.1.
3. In your terminal navigate to /web/solr/ in the Atlas source code. Run ./bin/solr start to start solr.

Python

This ETL uses python > 3.8. Python can be installed from https://www.python.org/downloads/

C++ build tools are needed on Windows OS.

ODBC Driver for SQL Server is required for connecting to the database.

Install Packages

This ETL uses poetry as the package manager. Alternatively, you can use pip to install the dependencies listed in pyproject.toml/dependencies.

poetry install

Create a .env file

Variables can either be set in the environment, or added to a .env file.

SOLRURL=http://localhost:8983/solr/atlas
SOLRLOOKUPURL=http://localhost:8983/solr/atlas_lookups
ATLASDATABASE=DRIVER={ODBC Driver 18 for SQL Server};SERVER=server_name;DATABASE=atlas;UID=user_name;PWD=password;TrustServerCertificate=Yes;"

# Optional for bookstack etl
BOOKSTACKURL=https://docs.example.com
BOOKSTACKTOKENID=123456
BOOKSTACKTOKENSECRET=78910111213

Running

delete.py script should be run once daily to empty Solr.

The remaining atlas_*.py scripts can be run periodically through the day to keep search results current.

poetry run python delete.py
poetry run python atlas_collections.py
poetry run python atlas_initiatives.py
poetry run python atlas_groups.py
poetry run python atlas_terms.py
poetry run python atlas_lookups.py
poetry run python atlas_users.py
poetry run python atlas_reports.py

# Optional etl to load documents from bookstack. Use this as an example etl for loading external content into search!
poetry run python atlas_bookstack.py

🎁 Contributing

This repository uses pre-commit and commitzen. Please commit npm run commit && git push.