1.2.0 - June 14th 2022

Version 1.2.0 (14JUN22)

• Overall

  • [CHANGE] - Support for Kubernetes 1.20 and OpenStack 4.7 has been dropped.

• Monitoring

  • [CHANGE] - create_elasticsearch_datasource script has been changed to create_logging_datasource and supports both Opensearch and Elasticsearch data source creation.
  • [FIX] - Fixed issue where namespace dropdown on "PostgreSQL" and "PostgreSQL Database" dashboards would not correctly filter information on the dashboards.

• Logging

  • [CHANGE] - OpenSearch 1.3.2 replaces Open Distro for Elasticsearch as the search technology used for log monitoring. This change includes replacing Elasticsearch with OpenSearch and Kibana with OpenSearch Dashboards. This involves changes in the names and format of the yaml files used with Helm. See Differences between Open Distro for Elasticsearch and OpenSearch for details. Documentation has been updated to reflect the move to OpenSearch.
  • [CHANGE] - Script and files names related to the deployment of log monitoring components were simplified and standardized.

1.1.8 - May 17th 2022

Version 1.1.8 (17MAY22)

• Overall

  • [FIX] - Fixed some Kubernetes-related messages were being displayed without log levels.
  • [CHANGE] The minimum supported version of OpenShift is now 4.9.

• Monitoring

  • [FIX] - Updated SAS Viya logo on the Grafana Welcome screen so that it is easier to see on darker background.
  • [FIX] - Fixed an issue that pod names were concatenated when multiple instances of tenant monitoring have been deployed.
  • [CHANGE] - Combined the functionality of the create_elasticsearch_datasource_cluster.sh and create_elasticsearch_datasource_tenant.sh scripts into one script (create_elasticsearch_datasource.sh).

• Logging

  • [ANNOUNCEMENT] - In our next release, we expect to move to using OpenSearch rather than Open Distro for Elasticseach as the search back-end supporting our log monitoring capabilities. While this change will have only minor impact on the user interface (primarily some cosmetic changes), it will have a more significant impact on the deployment process. Therefore, this should be considered a breaking change. It will involve:
    • changes to many script names, including the names of the primary deployment and removal scripts;
    • changes to the names and structure of the user value (yaml) files used with the Helm charts;
    • changes to the topology and configuration of search pod; and,
    • changes in product/application terminology (OpenSearch replacing Elasticsearch; OpenSearch Dashboards replacing Kibana).
  • Organizations wanting to get more familiar with the new technology stack are encouraged to deploy it on an experimental basis on a test cluster separate from their current "production" cluster. This can be done by running the deploy_logging_opensearch.sh script in the logging/bin sub-directory. Customizations to the OpenSearch and OpenSearch Dashboards configuration can be provided via the user-values-elasticsearch-opensearch.yaml and user-values-osd-opensearch.yaml files, respectively, in the logging sub-directory of the directory identified via the USER_DIR environment variable. We have OpenSearch-specific files in the Azure Deployment, Ingress and TLS samples to handle two of the most common customization scenarios. Please note that file names may change as this moves from experimental to production status.
  • The deployment process handles migrating the current set of collected log messages in the many scenarios. However, migration is not supported in some scenarios, such as the configuration documented in the min-logging sample (i.e. sample/min-logging) in the project repository.
  • [CHANGE] - Improved handling of log messages emitted by SingleStore.

1.1.7 - April 19th 2022

Version 1.1.7 (19APR22)

• Overall

  • [FEATURE] - Added a document to the project repository, IMAGE_INVENTORY.md, listing all container images used by pods when deploying this project using default values and deploying all components.
  • [FIX] - Modified naming of Helm releases of the internal V4M chart used to capture deployment information to support deploying log and metric monitoring components in the same namespace.
  • [EXPERIMENTAL] - A Docker file to allow you to work with this project in a containerized environment. This eliminates virtually all pre-requisites and concerns about properly configuring your environment. See the associated README.md for more information.

• Monitoring

  • [EXPERIMENTAL] - A new script, create_elasticsearch_datasource.sh, that creates datasource(s) which allow collected log messages collected to be viewed within Grafana.

• Logging

  • [FEATURE] - New role-based access controls and roles are created during initial deployment and the onboarding process to facilitate the creation of datasource(s) which allow collected log messages collected to be viewed in Grafana.
  • [FIX] - Corrected annotations on Grafana ingress objects in the Azure Deployment sample. (Fixes #318)
  • [EXPERIMENTAL] - Running logging/bin/deploy_logging_opensearch.sh instead of logging/bin/deploy_logging_open.sh will deploy log monitoring with OpenSearch 1.3.1 (instead of Open Distro for Elasticsearch 1.13.3) as the search back-end. OpenSearch will become the default (only) back-end in a coming release. The files user-values-elasticsearch-opensearch.yaml
    and user-values-osd-opensearch.yaml replace the user-values-elasticsearch.yaml file for providing user-supplied values during the Helm deployment process and use a different set of keys.

1.1.6 - March 15th 2022

Version 1.1.6 (15MAR22)

• Monitoring

  • [CHANGE] Monitoring component versions were updated
    • kube-prometheus-stack 19.0.3 -> 32.0.0
    • Prometheus Operator 0.51.2 -> 0.54.0
    • Prometheus 2.30.3 -> 2.33.1
    • Grafana 8.2.1 -> 8.4.1
    • node-exporter 1.2.2 -> 1.3.1
    • kube-state-metrics 2.2.1 -> 2.3.0
  • [CHANGE] The PostgreSQL Database and two RabbitMQ dashboards were refreshed from upstream
  • [FIX] The Grafana data source was not configured properly when TLS was enabled and using path-based ingress
  • [DOC] Comments were added to help users specify retention time and size for Prometheus

• Logging

  • [FIX] Removed hard-coded references to a specific namespace
  • [FIX] Eliminated a spurious ERROR message generated by trap function
  • [CHANGE] Removed references to deprecated NODE_NAME environment variable from scripts
  • [CHANGE] Replaced hard-coded 90 second wait for Elasticsearch PVC binding with a loop that checks its status
    to allow for quicker execution.

1.1.5 - February 2022

Version 1.1.5 (15FEB22)

• Monitoring

  • [CHANGE] - The SAS Viya Welcome dashboard has been updated with a
    cleaner design and an improved layout
  • [EXPERIMENTAL] - Setting ELASTICSEARCH_DATASOURCE=true when running
    monitoring/bin/deploy_monitoring_cluster.sh will configure an
    Elasicsearch data source for Grafana that enables viewing and querying of
    logs from Grafana dashboards. Logging must already be deployed for this
    to function properly.
  • [EXPERIMENTAL] - Enhanced dashboards for SAS Go services, SAS Java
    services, and CAS support viewing logs in Grafana. Setting
    ELASTICSEARCH_DATASOURCE=true or VIYA_LOGS_DASH=true will deploy
    the updated dashboards to replace the prior versions.

• Logging

  • [FEATURE] - Enhanced multi-line support is now enabled by default in
    Fluent Bit

  • [FEATURE] - logging/bin/change_internal_password.sh now supports the
    recently added logadm user

  • [CHANGE] - The deprecated KB_TLS_ENABLE flag has been removed. Kibana TLS
    is now controlled via the normal TLS_ENABLE and LOG_TLS_ENABLE flags

  • [FIX] - Several status check in scripts have been simplified to use
    kubectl wait

  • [FIX] - All logging components now specify Kubernetes resource requests

  • [FIX] - Event router is now properly deployed to the logging namespace
    when setting NODE_PLACEMENT_ENABLE=true or LOG_NODE_PLACEMENT_ENABLE=true

  • [DOC] - The documentation on how to adjust log retention
    has been revised to improve clarity and correct errors. Fixes #261.

1.1.4 - January 31st 2022

Version 1.1.4 (31JAN22)

• Logging
  • [SECURITY] Move to use Open Distro for Elasticsearch 1.13.3 (addresses LOG4J security vulnerability)

1.1.3 - January 14th 2022

Version 1.1.3 (14JAN22)

• Overall

  • [CHANGE] The ingress sample is deprecated in favor of
    the TLS sample
  • [FIX] The TLS Sample is now more consistent across
    monitoring/logging and host/path-based ingress
  • [FIX] The CloudWatch sample has been updated to support
    IMDSv2, which is used by viya4-iac-aws
  • [CHANGE] Samples have been reviewed and updated as needed for
    consistency and correctness

• Monitoring

  • No changes this release

• Logging

  • [FEATURE] A new Kibana user logadm has been created. This user is intended
    to be the primary Kibana used for routine day-to-day log monitoring. See
    The logadm User and Its Access Controls.
  • [CHANGE] Documentation on security and controling access to log messages
    has been revised extensively. See Limiting Access to Logs
  • [CHANGE] The Event Router component is now deployed to the logging ($LOG_NS) namespace
    instead of to the kube-system namespace. During upgrades of existing deployments,
    Event Router will be removed from the kube-system namespace and redeployed in the
    logging ($LOG_NS) namespace.
  • [FIX] Upgrade of an existing deployment using Open Distro for Elasticsearch 1.7.0 to the
    current release (which uses Open Distro for Elasticsearch 1.13.2) no longer fails.

1.1.2 - December 13th 2021

Version 1.1.2 (13DEC21)

• Overall

  • [CHANGE] Samples now use Ingress v1 for Kubernetes 1.22 compatibility

• Monitoring

  • [CHANGE] Monitoring components now use Ingress v1 for Kubernetes 1.22 compatibility
  • [FIX] The SAS Jobs dashboards properly handle large numbers of jobs
  • [FIX] The network metric recording rule for SAS Jobs has been fixed to support kube-state-metrics 2.x
  • [FIX] Using LOG_COLOR_ENABLE=false now shows log levels in output
  • [FIX] Deployments without an active TERM now run properly again
  • [FIX] Perf/Utilization dashboard metrics display properly again

• Logging

  • [SECURITY] Moved to Open Distro for Elasticsearch 1.13.3 (addresses LOG4J security vulnerability)
  • [FEATURE] Access controls supporting a new class of users with access to all log messages are now created during the deployment process.
  • [FEATURE] Kibana content in a given directory is loaded as a single 'batch' rather than individually during the deployment process.
  • [TASK] Feature-flag logic controlling enablement of Kibana tenant spaces and other application multi-tenancy related capabilities has been removed since these capabilities are no longer optional.
  • [FIX] The path-based ingress sample for accessing Kibana after the move to Open Distro for Elasticsearch 1.13.2 now works.
  • [FIX] Improvements for handling failures when deploying specific components made in logging deployment scripts
  • [FIX] New Fluent Bit configuration setting to prevent "stale" Kubernetes metadata being added to collected log messages.

1.1.1 - November 18th 2021

Version 1.1.1 (18NOV21)

• Overall

  • [FIX] Running in a non-interactive shell (no $TERM) caused automated
    deployments to fail

• Known Issues

  • On Openshift clusters, upgrading an existing deployment using Open Distro
    for Elasticsearch 1.7.0 to this release (which uses Open Distro for
    Elasticsearch 1.13.2) fails. Deploying this release onto a new OpenShift
    cluster is possible.

1.1.0 - November 15th 2021

Version 1.1.0 (15NOV21)

• Overall

  • [FEATURE] A new flag LOG_VERBOSE_ENABLE is now available to suppress detailed logging during script execution. The default setting of this flag is true.

• Monitoring

  • [CHANGE] Most monitoring component versions have been updated
    • kube-prometheus-stack Helm chart upgraded from 15.0.0 to 19.0.3
    • Prometheus Operatator upgraded from 0.47.0 to 0.51.2
    • Prometheus upgraded from 2.26.1 to 2.30.3
    • Alertmanager upgraded from 0.21.0 to 0.23.0
    • Grafana upgraded from 7.5.4 to 8.2.1
    • Node Exporter upgraded from 1.0.1 to 1.2.2
    • kube-state-metrics upgraded from 1.9.8 to 2.2.1
  • [FIX] Several dashboards were fixed to adjust to the kube-state-metrics 2.x metrics
  • [FIX] The KubeHpaMaxedOut alert has been patched to not fire when max instances == current instances == 1

• Logging

  • [CHANGE] Open Distro for Elasticsearch (i.e. Elasticsearch and Kibana) upgraded to version 1.13.2. This includes significant changes to Kibana user-interface, see Important Information About Kibana in the New Release for details.

  • [FEATURE] A significant number of changes to support application multi-tenancy in SAS Viya; including the ability to limit users to log messages from a specific Viya deployment and tenant. See Tenant Logging for details.

• Known Issues

  • On Openshift clusters, upgrading an existing deployment using Open Distro for Elasticsearch 1.7.0 to this release (which uses Open Distro for Elasticsearch 1.13.2) fails. Deploying this release onto a new OpenShift cluster is possible.