Add a high-level overview of classes in the repository (#588)

* add docs Signed-off-by: Kartavya Vashishtha <[email protected]> * rename to overview Signed-off-by: Kartavya Vashishtha <[email protected]> * use newer architecture photo in reader.md Signed-off-by: Kartavya Vashishtha <[email protected]> --------- Signed-off-by: Kartavya Vashishtha <[email protected]>
opensearch-project · Nov 14, 2023 · 5ab3345 · 5ab3345
1 parent c5700d6
commit 5ab3345
Show file tree

Hide file tree

Showing 4 changed files with 36 additions and 4 deletions.
diff --git a/README.md b/README.md
@@ -102,6 +102,10 @@ See the [design doc](https://github.com/opensearch-project/performance-analyzer-
 
 Please refer to the [technical documentation](https://opensearch.org/docs/monitoring-plugins/pa/index/) for detailed information on installing and configuring Performance Analyzer.
 
+The [Architecture](docs/READER.md) document provides a high-level overview of the Performance Analyzer architecture.
+
+You can find a high-level overview of the project file structure [here](docs/OVERVIEW.md).
+
 ## Contributing
 
 See [developer guide](DEVELOPER_GUIDE.md) and [how to contribute to this project](CONTRIBUTING.md).

diff --git a/docs/OVERVIEW.md b/docs/OVERVIEW.md
@@ -0,0 +1,30 @@
+In `src/main/java/org/opensearch/performanceanalyzer/`:
+
+- `PerformanceAnalyzerPlugin`'s constructor does the following:
+    1. Creates a `scheduledMetricCollectorsExecutor`.
+    2. `scheduledMetricCollectorsExecutor.addScheduledMetricCollector(new XYZMetricsCollector())` is called for all collectors.
+    3. `scheduledMetricCollectorsExecutor.start()` is called and then `EventLogQueueProcessor.scheduleExecutor()`.
+
+- Methods in `PerformanceAnalyzerPlugin` interface with the OpenSearch plugin architecture 
+    - `onIndexModule`, `onDiscovery`, etc. are all called by OpenSearch when their corresponding events occur and the plugin can act on them.
+    - For example:
+        - `getActionFilters` provides OpenSearch with a list of classes that implement `ActionFilter`.
+        - `action/PerformanceAnalyzerActionFilter` is the only class currently returned to OpenSearch as an `ActionFilter`.
+        - when a BulkRequest or SearchRequest is recieved by OpenSearch, `action/PerformanceAnalyzerActionFilter` logs a start event and creates a listener (`action/PerformanceAnalyzerActionListener`) which waits to record the corresponding end event.
+    - `PerformanceAnalyzerPlugin.getRestHandlers` returns all the classes that can handle REST requests to OpenSearch.
+-  The classes in `http_action/config` define all the public API routes for Performance Analyzer.
+    - `http_action/config/RestConfig` defines `PA_BASE_URI = "/_plugins/_performanceanalyzer"`
+    - `PerformanceAnalyzerResourceProvider` defines the `metrics`, `rca`, `batch` and `actions` routes.
+    -  `PerformanceAnalyzerResourceProvider.SUPPORTED_REDIRECTIONS = ("rca", "metrics", "batch", "actions")`
+- `listener/PerformanceAnalyzerSearchListener` hooks into OpenSearch core to emit search operation related metrics.
+
+
+- `writer/EventLogQueueProcessor`:
+    -  contains `purgeQueueAndPersist` which drains `PerformanceAnalyzerMetrics.metricQueue` into a file that contains all events for a certain time bucket. It also removes old events. Uses `event_process.EventLogFileHandler` in Performance Analyzer Commons for file writing logic.
+    - `scheduleExecutor` periodically runs `purgeQueueAndPersist`.
+
+- Classes in `collectors` extend `PerformanceAnalyzerMetricsCollector` and implement `MetricsProcessor`.
+	- `PerformanceAnalyzerMetricsCollector` implements `Runnable` and contains common variables like `value` where a collector stores serialized metrics.
+	- A collector is triggered through  `PerformanceAnalyzerMetricsCollector.collectMetrics`.
+		1. The collector will store serialized data in the `value` variable and then call `MetricsProcessor.saveMetricValues`.
+		2. `saveMetricValues` calls `PerformanceAnalyzerMetrics.emitMetric` that creates an `Event` from the serialized data and adds it to a static queue (`PerformanceAnalyzerMetrics.metricQueue`) shared across all collectors. 
diff --git a/docs/READER.md b/docs/READER.md
@@ -5,20 +5,18 @@ Performance Analyzer is an agent and REST API that allows you to query numerous
 * Performance analyzer plugin aka writer
 * Performance analyzer application aka reader
 
-![alt text][performance analyzer]
+![alt text](images/arch.png "Performance Analyzer Architecture")
 
-[performance analyzer]: https://github.com/opensearch-project/performance-analyzer/blob/main/docs/images/pa.png "Performance Analyzer Architecture diagram"
 # Performance Analyzer plugin
 
 The performance analyzer plugin captures important events in OpenSearch and writes them to a shared memory. These events are then analyzed separately by the reader. Having separate processes, gives us better isolation between OpenSearch and metrics collection/aggregation.
 
-
 # Performance Analyzer Application
 
 The performance analyzer application is written in Java, and this allows us to share code with the writer easily.  Java libraries like jdbc and jooq made it very easy to generate sql programmatically.
 ![alt text][reader]
 
-[reader]: https://github.com/opensearch-project/performance-analyzer/blob/main/docs/images/reader.png "Reader Architecture diagram"
+[reader]: images/reader.png "Reader Architecture diagram"
 * MetricsProcessor - Periodically (every 5 seconds) processes all the events and statistics generated by the writer and generates metrics out of them. These metrics are then stored in metricsDB.
 * MetricsDB - Sqlite database that contains all the metrics that can be queried by the client. We create a new db every 5 seconds. 
 * Webservice - A http server that serves client API requests. It services client requests by querying metricsDB for the relevant metrics requested by the client. 

diff --git a/docs/images/arch.png b/docs/images/arch.png