diff --git a/README.md b/README.md index 7c79a9e..fc7e531 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,8 @@ > > Configuration for the Application Study Tool has changed significantly in the v0.6.0 release. To update a legacy configuration, see [pages/config_migration.md](pages/config_migration.md). +> +> Before you start, make sure to backup the /config/big-ips.json file! ## Overview @@ -215,6 +217,10 @@ This will write 2 new files in the services/otel_collector directory: * `pipelines.yaml` - The final pipeline configs that map receievers to output destinations (prometheus, and optionally F5). +## Adding New Devices or Updating Configs +To add new devices or update the config after changes to the ast_defaults.yaml or receivers.yaml files, +re-run the config helper script as shown above and then restart the otel collector container. + ## Account Permissions The vast majority of telemetry data can be collected with read-only access to the BigIP. Some granular stats are only available as output to a iControl Rest 'bash' shell command, and diff --git a/pages/config_management.md b/pages/config_management.md index c074453..b619c6a 100644 --- a/pages/config_management.md +++ b/pages/config_management.md @@ -4,23 +4,23 @@ In the post v0.6.0 management scheme, users can choose from one of the below options to manage the AST Otel Collector configs: -1. Using the [/src/config_helper.py](/src/config_helper.py) script to generate full Otel Collector +1. Using the [/src/config_helper.py](https://github.com/f5devcentral/application-study-tool/blob/main/src/config_helper.py) script to generate full Otel Collector Config files from a small set of configuration for each device plus a set of defaults (recommended for most users, and includes migration path from old big-ips.json configs) 2. Manual maintenance of the Otel Collector Config files in -[/services/otel_collector/pipelines.yaml](/services/otel_collector/pipelines.yaml) -and [/services/otel_collector/receivers.yaml](/services/otel_collector/receivers.yaml) +[/services/otel_collector/pipelines.yaml](https://github.com/f5devcentral/application-study-tool/blob/main/services/otel_collector/pipelines.yaml) +and [/services/otel_collector/receivers.yaml](https://github.com/f5devcentral/application-study-tool/blob/main/services/otel_collector/receivers.yaml) 3. Manual maintenance of the Otel Collector Config files in -[/services/otel_collector/defaults](/services/otel_collector/defaults) +[/services/otel_collector/defaults](https://github.com/f5devcentral/application-study-tool/blob/main/services/otel_collector/defaults) ### Using config_helper.py (Recommended For Most Users) -With the included python script in [/src/config_helper.py](/src/config_helper.py), AST collector +With the included python script in [/src/config_helper.py](https://github.com/f5devcentral/application-study-tool/blob/main/src/config_helper.py), AST collector configuration is managed through 2 primary files: 1. A default file which contains settings that should be applied for each BigIP unless overriden (see below) -in [/config/ast_defaults.yaml](/config/ast_defaults.yaml): +in [/config/ast_defaults.yaml](https://github.com/f5devcentral/application-study-tool/blob/main/config/ast_defaults.yaml): ```yaml # These configs are applied to each entry in the bigip_receivers file @@ -83,7 +83,7 @@ pipelines: ``` 2. A file which contains settings that should override (or have no default) at the individual bigip level -in [/config/bigip_receivers.yaml](/config/bigip_receivers.yaml): +in [/config/bigip_receivers.yaml](https://github.com/f5devcentral/application-study-tool/blob/main/config/bigip_receivers.yaml): ```yaml # This file contains the list of BigIP receivers (scrape jobs). # Each item must have a unique key (e.g. bigip/1, bigip/2, etc). @@ -109,9 +109,9 @@ bigip/1: ``` When the config_helper script is run with the --generate-configs option, 2 new files are written out -to the [/services/otel_collector](/services/otel_collector) directory: +to the [/services/otel_collector](https://github.com/f5devcentral/application-study-tool/blob/main/services/otel_collector) directory: -1. The first contains the OTEL Collector pipelines configuration [/services/otel_collector/pipelines.yaml](/services/otel_collector/pipelines.yaml) which is basically the contents of the default config pipelines section plus the list of receivers (bigip scrape jobs): +1. The first contains the OTEL Collector pipelines configuration [/services/otel_collector/pipelines.yaml](https://github.com/f5devcentral/application-study-tool/blob/main/services/otel_collector/pipelines.yaml) which is basically the contents of the default config pipelines section plus the list of receivers (bigip scrape jobs): ```yaml metrics/bigip: exporters: @@ -126,7 +126,7 @@ metrics/bigip: ``` 2. The second contains the OTEL Collector receivers configuration -[/services/otel_collector/receivers.yaml](/services/otel_collector/receivers.yaml) +[/services/otel_collector/receivers.yaml](https://github.com/f5devcentral/application-study-tool/blob/main/services/otel_collector/receivers.yaml) which is the merged contents of the default config settings and the per-device settings: ```yaml bigip/1: @@ -145,19 +145,33 @@ bigip/1: ``` When the OTEL container is run, the default configs in -[/services/otel_collector/defaults/](/services/otel_collector/defaults/) merge these files into +[/services/otel_collector/defaults/](https://github.com/f5devcentral/application-study-tool/blob/main/services/otel_collector/defaults/) merge these files into the final configuration the OTEL Collector needs to run correctly. +#### Run The Configuration Helper +The config helper script can be run natively or via docker from the project root directory +to merge the default and device level configs into the final OTEL Collector config as follows: +```shell +# Run the configuration generator from the project root directory +docker run --rm -it -w /app -v ${PWD}:/app --entrypoint /app/src/bin/init_entrypoint.sh python:3.12.6-slim-bookworm --generate-config +``` + +This will write 2 new files in the services/otel_collector directory: + +* `receivers.yaml` - The final list of scraper configs and their settings. +* `pipelines.yaml` - The final pipeline configs that map receievers to output destinations +(prometheus). + ### Manual Maintenance Of Receiver and Pipeline Files The files mentioned above can be managed directly by users if they want to skip the config_helper script in favor of their own automation / templating. In this case, you just need to update the files: -* [/services/otel_collector/receivers.yaml](/services/otel_collector/receivers.yaml) -* [/services/otel_collector/pipelines.yaml](/services/otel_collector/pipelines.yaml) +* [/services/otel_collector/receivers.yaml](https://github.com/f5devcentral/application-study-tool/blob/main/services/otel_collector/receivers.yaml) +* [/services/otel_collector/pipelines.yaml](https://github.com/f5devcentral/application-study-tool/blob/main/services/otel_collector/pipelines.yaml) These are mapped into the final OTEL Collector config via the "file" directives in the receivers and services.pipelines section of the -[/services/otel_collector/defaults/bigip-scraper-config.yaml]() file: +[/services/otel_collector/defaults/bigip-scraper-config.yaml](https://github.com/f5devcentral/application-study-tool/blob/main/services/otel_collector/defaults/bigip-scraper-config.yaml) file: ``` receivers: ${file:/etc/otel-collector-config/receivers.yaml} diff --git a/pages/config_migration.md b/pages/config_migration.md index f379f76..bba9995 100644 --- a/pages/config_migration.md +++ b/pages/config_migration.md @@ -4,6 +4,8 @@ The configuration mangagement for AST Otel Collector is being updated to allow for more flexible configuration, and to simplify configuration for advanced usecases. +> Before you start, make sure to backup the /config/big-ips.json file! + The old configuration process relied on a docker container that would run each time the AST docker-compose instance was started. The process wrote the generated configs to an internal volume where users were unable to view and modify the files to tune parameters for their deployment. @@ -11,20 +13,22 @@ where users were unable to view and modify the files to tune parameters for thei In the new process, the raw otel configs are exposed in the /services/otel_collector directory where they can be managed manually, or through continued use of a refactored config_helper script. -For additional detail on configuration management options in the post v0.6.0 scheme, please see [Config Management](./config_management.md) +For additional detail on configuration management options in the post v0.6.0 scheme, please see [Config Management](https://github.com/f5devcentral/application-study-tool/blob/main/config_management.md) ## Migrating From pre v0.6.0 Configs There's a python script in /src/config_helper.py which will convert the original big-ips.json schema into the new management format. Assuming you have an existing list of configured BigIPs in /config/big-ips.json, migration is a 2 step process: -1. Make sure the default values in [/config/ast_defaults.yaml](/config/ast_defaults.yaml) match your +0. Backup the existing /config/big-ips.json file. +1. Make sure the default values in [/config/ast_defaults.yaml](https://github.com/f5devcentral/application-study-tool/blob/main/config/ast_defaults.yaml) match your desired default settings. 2. Run the migration script. +3. Run the config generation script. ## Verify Default Settings -The default settings in [/config/ast_defaults.yaml](/config/ast_defaults.yaml) are merged with your existing values in big-ips.json by the script. +The default settings in [/config/ast_defaults.yaml](https://github.com/f5devcentral/application-study-tool/blob/main/config/ast_defaults.yaml) are merged with your existing values in big-ips.json by the script. You can reduce the amount of repetitive yaml in the output by making sure these values match your common values (e.g. if you use the username: "telemetry", updating that value in the defaults @@ -83,7 +87,7 @@ pipelines: ## Run The Conversion Script -The /src/config_helper.py script can be run on a system with python or via docker image as follows: +The [/src/config_helper.py](https://github.com/f5devcentral/application-study-tool/blob/main/src/config_helper.py) script can be run on a system with python or via docker image as follows: ### Conversion Run Via Docker If you don't have an environment with python handy, you can run the script via @@ -106,7 +110,7 @@ bigip/2: ``` If the planned output looks correct, you can run again without the --dry-run to update -the contents of [./config/bigip_receivers.yaml](./config/bigip_receivers.yaml) +the contents of [./config/bigip_receivers.yaml](https://github.com/f5devcentral/application-study-tool/blob/main/config/bigip_receivers.yaml) ```shell docker run --rm -it -w /app -v ${PWD}:/app --entrypoint /app/src/bin/init_entrypoint.sh python:3.12.6-slim-bookworm --convert-legacy-config ``` @@ -137,7 +141,7 @@ bigip/2: ``` If the planned output looks correct, you can run again without the --dry-run to update -the contents of [./config/bigip_receivers.yaml](./config/bigip_receivers.yaml) +the contents of [./config/bigip_receivers.yaml](https://github.com/f5devcentral/application-study-tool/blob/main/config/bigip_receivers.yaml) ```shell python ./src/config_helper.py --convert-legacy-config ``` @@ -147,4 +151,20 @@ Output: 2024-09-25 17:06:29,897 - INFO - Successfully wrote data to './config/bigip_receivers.yaml'. ``` -### Adding New Devices \ No newline at end of file +## Run The Configuration Helper To Generate New Configs +The config helper script can be run natively or via docker from the project root directory +to merge the default and device level configs into the final OTEL Collector config as follows: +```shell +# Run the configuration generator from the project root directory +docker run --rm -it -w /app -v ${PWD}:/app --entrypoint /app/src/bin/init_entrypoint.sh python:3.12.6-slim-bookworm --generate-config +``` + +This will write 2 new files in the services/otel_collector directory: + +* `receivers.yaml` - The final list of scraper configs and their settings. +* `pipelines.yaml` - The final pipeline configs that map receievers to output destinations +(prometheus). + +### Adding New Devices +To add new devices or update the config after changes to the ast_defaults.yaml or receivers.yaml files, +re-run the config helper script as shown above. \ No newline at end of file diff --git a/pages/dashboard.md b/pages/dashboard.md index 64fa8e2..acd25be 100644 --- a/pages/dashboard.md +++ b/pages/dashboard.md @@ -1,6 +1,7 @@ # Dashboards -We provide a variety of dashboards to help monitor your BigIP devices, these dashboards are built in Grafana which makes it super flexible to apply changes. Below you will find all the dashboards you will get access to when you deploy the application. +Application Study Tool includes a variety of starter dashboards to help monitor your BigIP devices. +These dashboards are built in Grafana and can be customized to suit your unique organizational needs. ## Table of Contents - [Fleet](#fleet-dashboards) diff --git a/pages/index.md b/pages/index.md index d2e1739..57f13bf 100644 --- a/pages/index.md +++ b/pages/index.md @@ -2,7 +2,7 @@ layout: home --- -# Seven Layer Cake +# Application Study Took ### Overview The Application Study Tool is intended to provide enhanced insights into (classic) BIG-IP products, leveraging best in class diff --git a/pages/metric_obfuscation.md b/pages/metric_obfuscation.md index 669fd72..62a27d6 100644 --- a/pages/metric_obfuscation.md +++ b/pages/metric_obfuscation.md @@ -18,7 +18,7 @@ The attributes processor can be used to hash metric attributes before they're ex To enable this functionality for data being exported to F5: 1. Add an attributes processor config as shown to -[services/otel_collector/defaults/bigip-scraper-config.yaml](services/otel_collector/defaults/bigip-scraper-config.yaml) +[services/otel_collector/defaults/bigip-scraper-config.yaml](https://github.com/f5devcentral/application-study-tool/blob/main/services/otel_collector/defaults/bigip-scraper-config.yaml) (This example will mask the f5.instance.management_ip and f5.node.ip_address fields): @@ -33,7 +33,7 @@ processors: action: hash ``` -2. Edit the [config/ast_defaults.yaml](config/ast_defaults.yaml) file to include the new processor +2. Edit the [config/ast_defaults.yaml](https://github.com/f5devcentral/application-study-tool/blob/main/config/ast_defaults.yaml) file to include the new processor on the F5 Datafabric pipeline: ```yaml pipelines: diff --git a/pages/receiver_readme.md b/pages/receiver_readme.md index 6d16490..e346bf3 100644 --- a/pages/receiver_readme.md +++ b/pages/receiver_readme.md @@ -1,17 +1,5 @@ # F5 Big-IP Receiver - -| Status | | -| ------------- |-----------| -| Stability | [beta]: metrics | -| Distributions | [contrib] | -| Issues | [![Open issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aopen%20label%3Areceiver%2Fbigip%20&label=open&color=orange&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Areceiver%2Fbigip) [![Closed issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aclosed%20label%3Areceiver%2Fbigip%20&label=closed&color=blue&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Areceiver%2Fbigip) | -| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@djaglowski](https://www.github.com/djaglowski), [@StefanKurek](https://www.github.com/StefanKurek) \| Seeking more code owners! | - -[beta]: https://github.com/open-telemetry/opentelemetry-collector#beta -[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib - - This receiver fetches stats from a F5 Big-IP node using F5's [iControl REST API](https://clouddocs.f5.com/api/icontrol-rest). ## Prerequisites