From 8809bb219bc934f7648b636f1813160c376a1b2b Mon Sep 17 00:00:00 2001 From: eNDdy Date: Tue, 22 Oct 2024 11:14:45 -0700 Subject: [PATCH] chore(docs): update content-watcher docs Update Content-Watcher-Service Documentation structure and content. Update Content-Publisher Docs running test. --- Makefile | 9 + developer-docs/content-publishing/README.md | 11 +- developer-docs/content-watcher/README.md | 416 ++++++-------------- 3 files changed, 139 insertions(+), 297 deletions(-) diff --git a/Makefile b/Makefile index 62dce39f..0be7df8b 100644 --- a/Makefile +++ b/Makefile @@ -90,3 +90,12 @@ start-graph-worker: update-packages .PHONY: setup-account setup-account: @cd apps/account-api/test/setup && npm install && npm run main + + +.PHONY: test-content-watcher-services-start +test-content-watcher-services-start: + @scripts/content-watcher/local-setup.sh -n cw-e2e -i + +.PHONY: test-content-watcher-services-stop +test-content-watcher-services-stop: + @scripts/content-watcher/local-setup.sh -n cw-e2e -d diff --git a/developer-docs/content-publishing/README.md b/developer-docs/content-publishing/README.md index 25d818e1..e5f54bbb 100644 --- a/developer-docs/content-publishing/README.md +++ b/developer-docs/content-publishing/README.md @@ -70,7 +70,7 @@ Ensure you have the following installed: 3. Set up environment variables: ```bash - cp env-files/content-publishing.template.env .env + cp env-files/content-publishing.template.env .env.content-publishing ``` Configure the environment variables according to your needs. @@ -110,8 +110,6 @@ Ensure you have the following installed: ### Verification -### Verification - #### Docker Setup - Access Swagger UI: [http://localhost:3012/docs/swagger](http://localhost:3012/docs/swagger) - View and manage queues: [http://localhost:3012/queues](http://localhost:3012/queues) @@ -128,11 +126,16 @@ Ensure you have the following installed: ### Testing -Run the test suite: +Run the test: ```bash npm test:content-publising ``` +Run E2E tests: +```bash +npm run test:e2e:content-publishing +``` + ### Linting and Formatting Run linter: diff --git a/developer-docs/content-watcher/README.md b/developer-docs/content-watcher/README.md index 873fd850..54ad99d5 100644 --- a/developer-docs/content-watcher/README.md +++ b/developer-docs/content-watcher/README.md @@ -1,356 +1,186 @@ -# Content Watcher +# Content Watcher Service - +The Content Watcher Service is a crucial component of the Gateway suite, providing a familiar callback API to retrieve content and publishing announcements from the Frequency blockchain. This document provides an overview of the service, its architecture, and guides for setup and usage. -# 📗 Table of Contents +## 📗 Table of Contents - [📖 About the Project](#about-project) -- [🔍 Arch Map](#-arch-maps) -- [🛠 Built With](#-built-with) - - [Tech Stack](#tech-stack) - - [Key Features](#key-features) -- [🚀 Live OpenAPI Docs](#-live-docs) -- [💻 Getting Started](#-getting-started) - - [Prerequisites](#prerequisites) - - [Environment Variables](#environment-variables) - - [Setup](#setup) - - [Install](#install) - - [Usage](#usage) - - [Swagger](#swagger-ui) - - [Queue Management](#queue-management) - - [Run tests](#run-tests) - - [Linting](#linting) - - [Formatting](#auto-format) -- [🤝 Contributing](#-contributing) -- [⭐️ Show your support](#-support) -- [🙏 Acknowledgements](#-acknowledgements) -- [❓FAQ](#faq) -- [📝 License](#-license) - - - -# 📖 Content Watcher Service - -The Content Watcher Service is part of the "Social Gateway" collection of services that provides a familiar callback API to retrieve content and publishing announcements from the Frequency chain. The service handles all of the necessary blockchain interaction and allows clients to interact using a familiar, Web2-friendly interface. - - - -## 🔭 Arch Maps - -The Content Watcher Service provides a webhook system for receiving notice of new content. -Even with a crawl request, the results are not retreived, but pushed to the webhook. - -![Content Watcher Service Arch](./docs/content_watcher_service_arch.drawio.png) +- [🔍 Architecture Overview](#architecture-overview) +- [🔑 Key Features](#key-features) +- [💻 Getting Started](#getting-started) +- [🚀 API Documentation](#api-documentation) +- [🛠 Development](#development) +- [🤝 Contributing](#contributing) +- [❓ FAQ](#faq) +- [📝 License](#license) -

(back to top)

- -## 🛠 Built With - -### Tech Stack - -
- Server - -
- -
- Data store - -
- -
- Frameworks and Libraries - -
- -
- Polkadot and DSNP Integration - -
- -
- Testing - -
- -
- Formatting - -
- -
- Build and Deployment - -
- - - -### Key Features - -#### API - -- **Parse DSNP Messages on Frequency** -- **Send content to registered webhooks** -- **Simple Schema and MSA Id based filtering** - -#### Scanner API - -- **Start, Stop, and restart scanning** +## 📖 About the Project -

(back to top)

+The Content Watcher Service is part of the Gateway suite that provides a Web2-friendly interface for monitoring and retrieving blockchain content. It implements a webhook system that pushes updates to registered endpoints, making it easy to stay synchronized with chain events. - +## 🔍 Architecture Overview -## 🚀 Live Docs +The Content Watcher Service provides a webhook-based system for receiving notifications about new content. Even with a crawl request, results are pushed to the webhook rather than retrieved directly. -- [Live Docs](https://projectlibertylabs.github.io/content-watcher-service/) -- [API Documentation](https://projectlibertylabs.github.io/content-watcher-service/) -- [GitHub](https://github.com/ProjectLibertyLabs/content-watcher-service) +![Content Watcher Service Architecture](./content_watcher_service_arch.drawio.png) -

(back to top)

+## 🔑 Key Features - +### Content Monitoring +- **Parse DSNP Messages**: Monitor and parse messages on Frequency +- **Webhook Integration**: Send content to registered webhooks +- **Flexible Filtering**: Simple Schema and MSA Id based filtering -## 💻 Getting Started +### Scanner Operations +- **Start/Stop Control**: Manage scanning operations +- **Restart Capability**: Reset and restart scanning processes +- **Block Range Scanning**: Specify custom block ranges for content retrieval -This guide is tailored for developers working in the code base for the Content Watcher Service itself. For a more tutorial tailored more for developers wanting to deploy the Content Watcher Service as part of the broader Social Gateway in order to develop their own Social Gateway app, visit [Live Docs](https://projectlibertylabs.github.io/gateway/). +## 💻 Getting Started -To prepare and run a local instance of the Content Watcher Service for local development, follow the guide below. +This section guides you through setting up the Content Watcher Service for both development and deployment. ### Prerequisites -In order to run this project you need: - -- [Nodejs](https://nodejs.org) -- [Docker](https://www.docker.com) or Docker-compatible container system for running Gateway Services - - (note, Docker is not strictly required; all of the services described below may be installed or built & run locally, but that is outside the scope of this guide) - -### Environment Variables - -Use the provided [env.template](./env.template) file to create an initial environment for the application, and edit as desired. Additional documentation on the complete set of environment variables is provided in the [ENVIRONMENT.md](./ENVIRONMENT.md) file. - -1. For running locally, copy to the template file to `.env` and update as needed. - -```sh -cp env.template .env -``` - -2. Configure the environment variable values according to your environment. - - - Docker: `.env.docker.dev` - - Local: `.env` - -### Setup - -Clone this repository to your desired folder: - -Example commands: - -```sh - git clone git@github.com:ProjectLibertyLabs/content-watcher-service.git - cd content-watcher-service -``` - -### Install - -Install NPM Dependencies: - -```sh - npm install -``` - -### Usage - -To run the project, execute the following command: - -#### 1. Start the required auxiliary services - -Frequency node, Redis, IPFS - -```sh -docker compose up -d frequency redis ipfs -``` - -#### 2. [Optional] Start the publishing services - -Content Publishing Service - -```sh -docker compose up -d content-publishing-service-api content-publishing-service-worker -``` - -#### 3. Start the application services - -Each of the application services may be run either under Docker or bare-metal, depending on your preferred development workflow. - -#### Running locally - -```sh -npm run start:api:dev -``` - --- or -- - -#### Running under Docker - -```sh -docker compose up [-d] content-watcher-service -``` - -#### 4. [Optional] Setup a basic publisher - -Setup provider and users for the publishing service. - -```sh -npm run local:init -``` - -#### 5. [Optional] Webhook registration - -Start a simple webhook that will just echo out responses to the console. - -```sh -npm run local:webhook -``` - -#### 5. [Optional] Trigger Content - -Publishes some random content through the Content Publishing Service in Docker. -Can be run more than once. - -```sh -npm run local:publish -``` - -#### 5. Check the job in BullUI, to monitor job progress based on defined tests. - -### Swagger UI +Ensure you have the following installed: +- [Node.js](https://nodejs.org) +- [Docker](https://docs.docker.com/get-docker/) -Check out the Swagger UI hosted on the app instance at http://localhost:3000/api/docs/swagger to view the API documentation and submit requests to the service. +### Quick Start with Docker -### Queue Management - -You may also view and manage the application's queue at http://localhost:3000/queues. - -For the Content Publishing Service Queue go to http://localhost:3001/queues. - -### Run unit tests +1. Clone the repository and navigate to the project directory: + ```bash + git clone https://github.com/ProjectLibertyLabs/gateway + cd gateway + ``` -To run unit tests, run the following command: +2. Install dependencies: + ```bash + npm install + ``` -```sh - npm test -``` +3. Start auxiliary services: + ```bash + docker compose up -d frequency redis ipfs + ``` -### Run e2e tests: +4. Start the Content Watcher Service: + ```bash + docker compose up -d content-watcher-service + ``` -1. Execute the following `make` command to deploy the entire stack: +### Local Development Setup +1. Start required services: ```bash - make test-services-start + docker compose up -d frequency redis ipfs ``` -2. Run the following `make` command to execute the content watcher tests: +2. Set up environment variables: + ```bash + cp env-files/content-watcher.template .env.content-watcher + ``` + Configure the environment variables according to your needs. +2. Start Content-Watcher ```bash - make test-e2e + npm run start:content-watcher:dev ``` -3. Alternatively, create a `.env` file, run `npm run start:api` to start the content watcher as a standalone service, register a webhook with the content watcher using [swagger](http://0.0.0.0:3000/api/docs/swagger#), and try the following scenarios: +### Optional Setup Steps -- **Reset Scanner:** This action will reset the scanner to start from the beginning of the chain or whichever block is chosen to start with. Upon successful parsing, a respective announcement will be made to the webhook. -- **Put a Search Request:** This action will put a search request on the queue. The request requires a start block and end block. Upon successful parsing, a respective announcement will be made to the webhook. +1. Set up publishing services: + ```bash + docker compose up -d content-publishing-service-api content-publishing-service-worker + ``` -### Linting: +2. Initialize basic publisher: + ```bash + npm run setup:content-publishing:chain + ``` -```sh - npm run lint -``` +3. Start test webhook: + ```bash + npm run setup:content-watcher:webhook + ``` -### Auto-format: +4. Generate test content: + ```bash + npm run setup:content-watcher:publish + ``` -```sh - npm run format -``` +### Verification -

(back to top)

+#### Docker Setup +- Access Swagger UI: [http://localhost:3011/api/docs/swagger](http://localhost:3014/api/docs/swagger) +- View and manage queues: [http://localhost:3011/queues](http://localhost:3014/queues) - +#### Local Development Setup +- Access Swagger UI: [http://localhost:3000/api/docs/swagger](http://localhost:3000/api/docs/swagger) +- View and manage queues: [http://localhost:3000/queues](http://localhost:3000/queues) -## 🤝 Contributing +## 🚀 API Documentation -Contributions, issues, and feature requests are welcome! +- [Live API Documentation](https://projectlibertylabs.github.io/gateway/) -- [Contributing Guidelines](https://github.com/ProjectLibertyLabs/gateway/blob/main/CONTRIBUTING.md) -- [Open Issues](https://github.com/ProjectLibertyLabs/content-watcher-service/issues) -

(back to top)

+## 🛠 Development - +### Testing -## ⭐️ Show your support +Run unit tests: +```bash +npm run test:content-watcher +``` -If you would like to explore contributing bug fixes or enhancements, issues with the label `good-first-issue` can be a good place to start. +Run E2E tests: +```bash +make test-content-watcher-services-start -

(back to top)

+npm run test:e2e:content-watcher +``` - +### Linting and Formatting -## 🙏 Acknowledgements +Run linter: +```bash +npm run lint +``` -Thank you to [Frequency](https://www.frequency.xyz) for assistance and documentation making this possible. +Auto-format code: +```bash +npm run format +``` - +### Built With -## ❓FAQ +- **Server Framework**: NestJS, Node.js, TypeScript +- **Data Store**: Redis (ioredis) +- **Queue System**: BullMQ +- **Blockchain Integration**: Polkadot API, DSNP +- **Testing**: Jest, Supertest +- **Documentation**: Swagger +- **Containerization**: Docker, Docker Compose -- **Can I use this service in my production social app?** +## 🤝 Contributing - - Yes. All the Gateway Services are intended to be ready-to-use out of the box as part of the fabric of your own social media app using DSNP on Frequency. +We welcome contributions! Please check our [Contributing Guidelines](https://github.com/ProjectLibertyLabs/gateway/blob/main/CONTRIBUTING.md) and [open issues](https://github.com/ProjectLibertyLabs/gateway/issues). -- **Does this service index content?** +## ❓ FAQ - - No. This can be used by your own indexing service to get access to the content, but the service is intentially limited to getting the content and further customization is open to you. +**Q: Can I use this service in my production social app?** +A: Yes, Gateway Services are designed to be ready-to-use out of the box as part of your social media app using DSNP on Frequency. -- **Does this service filter content?** +**Q: Does this service index content?** - - No. This can be used by your own content filtering service to get new content and then have your custom service process them. +*No. This can be used by your own indexing service to get access to the content, but the service is intentionally limited to getting the content and further customization is open to you.* -

(back to top)

+**Q: Does this service filter content?** - +*No. This can be used by your own content filtering service to get new content and then have your custom service process them.* -## 📝 License +## 📝 License -This project is [Apache 2.0](./LICENSE) licensed. +This project is licensed under the [Apache 2.0 License](./LICENSE).

(back to top)