From f7c4ce7d51b9d05997b86552c2f2a91477bd0730 Mon Sep 17 00:00:00 2001 From: "Liu, Rachel A" Date: Mon, 30 Oct 2023 09:07:48 +0800 Subject: [PATCH] Reference Architecture 23.10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New Components/Features: - Extended support for 5th Gen Intel® Xeon® Scalable processors to VMRA, which requires the NDA version of Intel® QuickAssist Technology (Intel® QAT) drivers - Support for Intel® Trust Domain Extensions (Intel® TDX) on 5th Gen Intel® Xeon® Scalable processors with limitations - Support for 12th Gen Intel® Core™ processor for IoT Edge - Support for Edge AI Box 3.1 on 12th Gen Intel® Core™ desktop processors, 12th Gen Intel® Core™ processor for IoT Edge - Extended support for Software Defined Factory use-case on Intel® Core™ and Intel Atom® platforms to include an additional platform to manage the deployment of controller node via Kubernetes - Support deployment of a Machine Controller use case via VMRA - Support ECDSA keys for 5G O-RAN security with NETCONF server/client authentication (Intel® Software Guard Extensions (Intel® SGX)) - Added boot guard and secure boot to 5G O-RAN security - Use of Vector Packet Processing (VPP) (23.06) dataplane within Calico - Integrated Intent-Driven Orchestration Release v0.2 with BMRA - Support for Intel® Infrastructure Processing Unit (Intel® IPU) ASIC E2000 card and Infrastructure Programmer Development Kit (IPDK) Networking Recipe - Support for virtual machine and bare metal mixed Kubernetes deployment - Support for generic virtual machine type - Integrated Intel® Media Transport Library library - Integrated Intel® XPU Manager (Intel® XPUM) for GPU health and telemetry monitoring - Extend telemetry stack support to container only environment for AIBox - Initial support of Collection, other consumers can directly import and use just roles from this collection - Added Rocky Linux 9.2 as base OS Updates/Changes: - Version upgraded for the vast majority of RA components - Notable updates: FlexRAN™ to v23.07 Kubernetes* to v1.27.1 Intel® Containerized Telegraf to 1.3 Service Mesh Istio to v1.19.0 Intel® Managed Distribution of Istio* Service Mesh to v1.19.0-intel.0 Open vSwitch with DPDK to 3.2 Intel Multi-Buffer Crypto for IPsec library to v1.4 Intel® QAT Engine for OpenSSL to v1.3.1 Intel® Power Manager to v2.3.1 Data Plane Development Kit (DPDK) to v23.07 Intel® Ethernet Operator v23.08 SR-IOV FEC Operator to v2.7.1 OpenVINO™ v2023.1 Key Management Reference Application (KMRA) v2.4 Kubernetes to v1.27 for Cloud RA on Microsoft* Azure Kubernetes Service (AKS) and Amazon Elastic Kubernetes Service* (Amazon EKS) New Hardware (Platforms/CPUs/GPUs/Accelerators): - 12th Gen Intel® Core™ desktop processors with Intel® Arc™ Discrete Graphics GPU A380 - 12th Gen Intel® Core™ processor for IoT Edge with Intel® Iris® Xe Integrated Graphics - 13th Gen Intel® Core™ mobile processor with Intel® Iris® Xe Integrated Graphics Removed Support: - Discontinued support for Rocky 9.1 as base OS Known Limitations/Restrictions: - Intel® Data Center GPU Flex Series, CPU Control Plane Plugin for Kubernetes*, Intel® Media SDK (only Docker runtime) are only supported on Ubuntu OS - MinIO* is supported only with CRI-O runtime - Only in-tree Intel® QuickAssist Technology (Intel® QAT) drivers supported on RHEL 9.2 and Rocky 9.2 - Intel® Data Streaming Accelerator (Intel® DSA) may not work on some older (earlier stepping) CPUs on RHEL 9.2 and Rocky Linux 9.2 - Work-around applied to fix default Intel® QAT driver 4.23.0 install fail in VMRA mode on 3rd Gen Intel® Xeon® Scalable processors - UserSpace CNI with VPP is not supported - Intel® Trust Domain Extensions (Intel® TDX) on VMRA does not support Intel® Dynamic Load Balancer (Intel® DLB), Intel® DSA, Intel QAT, or network adapter device passthrough due to Intel® TDX driver security concerns - Intel Ethernet Operator DDP update feature might not work in rare cases; use legacy DDP update feature instead - Mixed deployment on_prem_sw_defined_factory profile with 12th Gen Intel® Core™ or later CPU should disable hyper thread - Power Manager fail to set target frequency with frequency_scaling_driver set as "acpi_cpufreq" Co-authored-by: Alek Du Co-authored-by: Ali Shah, Syed Faraz Co-authored-by: Benedikt, Jan Co-authored-by: Fiala, Jiri Co-authored-by: Hu, Hao Co-authored-by: Jiang, Renzhi Co-authored-by: Kasanic, Denis Co-authored-by: Liu, GuangyuX Co-authored-by: Liu, Rachel A Co-authored-by: Long, Zhifang Co-authored-by: Pedersen, Michael Co-authored-by: Prokes, Jiri Co-authored-by: Ren, Shu Co-authored-by: Vrana, Roman Co-authored-by: Xu, Guoshu Co-authored-by: Zenov, Mykyta Co-authored-by: Zhang, Yunzhe1 --- Pipfile | 20 +- README.md | 4 +- action_plugins/cpupin.py | 2 +- ansible.cfg | 2 +- cloud/README.md | 10 +- cloud/cwdf_example_aws.yaml | 2 +- cloud/cwdf_example_azure.yaml | 2 +- cloud/cwdf_util/config.py | 4 +- cloud/cwdf_util/main.py | 4 +- .../cloudcli/aws/aws_cloudcli_deploy.sh.j2 | 15 + .../azure/azure_cloudcli_deploy.sh.j2 | 15 + .../terraform/aws/ansible_host.tf.jinja | 13 + .../templates/terraform/aws/provider.tf.jinja | 6 +- .../templates/terraform/azure/aks.tf.jinja | 2 +- .../terraform/azure/ansible_host.tf.jinja | 13 + .../terraform/azure/provider.tf.jinja | 4 +- cloud/deployer.py | 10 +- cloud/discovery/profiles.yml | 33 +- cloud/ssh_connector.py | 26 +- cloud/sw_deployment/docker_management.py | 2 +- cloud/sw_deployment/sw_deployment_tool.py | 47 +- collections/requirements.yml | 2 +- collections/share/README.md | 118 + collections/share/changelogs/changelog.yaml | 6 + collections/share/galaxy.yml | 60 + .../roles/configure_dpdk/defaults/main.yml | 6 + .../files/cek_config_dpdk.service | 0 .../configure_dpdk/files/cek_config_dpdk.sh | 2 +- .../files/cek_config_dpdk_bind.py | 19 +- .../files/cek_config_dpdk_link.py | 0 .../files/cek_config_dpdk_rebind.py | 0 .../files/cek_config_dpdk_unbind.py | 0 .../files/cek_config_dpdk_util.py | 0 .../roles}/configure_dpdk/tasks/cleanup.yml | 5 +- .../roles}/configure_dpdk/tasks/main.yml | 63 +- .../install_gpu_driver/defaults/main.yml | 113 + .../files/cek_detect_gpu_type.py | 11 +- .../files/cek_get_latest_gpu_pkgs.sh | 4 +- .../install_gpu_driver/tasks/debian.yml | 68 +- .../roles/install_gpu_driver/tasks/main.yml | 59 + .../roles/install_gpu_driver/tasks/rhel.yml | 3 + docs/calico_vpp.md | 132 + docs/eci_guide.md | 80 + docs/emr.md | 134 +- docs/flexran_guide.md | 197 + docs/generate_profiles.md | 1 + docs/images/tdx-bios-configure.png | Bin 0 -> 53341 bytes docs/images/tdx-host-check.png | Bin 0 -> 216364 bytes docs/intent_driven_orchestration.md | 172 + docs/ipu_setup.md | 186 +- docs/monitoring.md | 60 + docs/profile_overview/BMRA.pdf | Bin 0 -> 54198 bytes docs/profile_overview/CloudRA.pdf | Bin 0 -> 34370 bytes docs/profile_overview/VMRA.pdf | Bin 0 -> 52146 bytes docs/storage.md | 63 +- docs/vm_bm_mixed_config_guide.md | 101 + docs/vm_cluster_expansion_guide.md | 43 +- generate/playbook_templates/infra_playbook.j2 | 90 +- generate/playbook_templates/intel_playbook.j2 | 77 +- generate/playbook_templates/main_playbook.j2 | 9 +- .../profiles_templates/cloud/profiles.yml | 52 +- .../profiles_templates/common/group_vars.j2 | 134 +- .../profiles_templates/common/host_vars.j2 | 214 +- generate/profiles_templates/k8s/profiles.yml | 146 +- generate/profiles_templates/vm/inventory.j2 | 4 + .../vm/vm_host_profiles.yml | 141 +- .../profiles_templates/vm/vms_profiles.yml | 141 +- library/check_nic_firmware.py | 39 +- playbooks/dockerfiles.yml | 1 + playbooks/dyna_config.yml | 67 +- playbooks/infra/prepare_ipu.yml | 2 + playbooks/infra/prepare_vms.yml | 48 +- playbooks/intel/eci_basic.yml | 48 + playbooks/intel/tdx.yml | 44 + playbooks/k8s/k8s.yml | 56 +- playbooks/k8s/patch_kubespray.yml | 10 - playbooks/k8s/templates/rke2_config.yaml.j2 | 13 +- playbooks/preflight.yml | 573 +- playbooks/redeploy_cleanup.yml | 1 - playbooks/versions.yml | 36 +- requirements.txt | 18 +- roles/bond_cni_install/defaults/main.yml | 2 +- .../auto_detect_qat_devices/tasks/main.yml | 12 +- .../defaults}/main.yml | 10 +- .../files/ra_loopdevices.service | 18 + .../tasks/configure_loopdevices.yml | 109 + .../bootstrap/configure_disks/tasks/main.yml | 56 + .../configure_disks/tasks/mount_disks.yml | 44 + .../configure_disks/tasks/setup_fake_pvs.yml | 30 + .../templates/loopdevice_bind.j2 | 24 + .../bootstrap/configure_dlb/defaults/main.yml | 6 +- roles/bootstrap/configure_dlb/tasks/main.yml | 8 +- .../tasks/configure_drivers.yml | 59 + .../configure_kpm_drivers/tasks/main.yml | 17 + .../configure_openssl/defaults/main.yml | 2 +- .../configure_openssl/tasks/main.yml | 5 +- .../configure_qat/tasks/check_qat_status.yml | 71 +- .../templates/cek_sriov_qat_init.service.j2 | 2 + .../bootstrap/configure_sgx/defaults/main.yml | 20 +- .../bootstrap/configure_sgx/tasks/ubuntu.yml | 8 + .../golang_install/defaults/main.yml | 4 +- .../install_gpu_driver/defaults/main.yml | 130 - .../install_gpu_driver/tasks/main.yml | 69 - .../install_packages/tasks/debian.yml | 17 + .../defaults/main.yml | 12 +- .../tasks/main.yml | 194 +- .../tasks/qat_oot_driver_install.yml | 229 + .../defaults/main.yml | 2 +- .../tasks/main.yml | 38 +- .../vars/main.yml | 0 .../install_tdx_drivers/defaults/main.yml} | 21 +- .../install_tdx_drivers/tasks/main.yml | 21 + .../tasks/tdx_preflight.yml} | 26 +- .../install_tdx_drivers/tasks/ubuntu.yml | 135 + .../vars/main.yml | 31 +- .../bootstrap/reset_qat_option/tasks/main.yml | 32 - .../tasks/main.yml | 82 +- .../set_sriov_kernel_flags/tasks/main.yml | 4 +- .../tasks/setup_sriov_kernel_flags.yml | 4 +- .../set_tdx_kernel_flags/defaults/main.yml | 4 +- .../tasks/main.yml} | 22 +- .../update_nic_drivers/defaults/main.yml | 8 +- .../update_nic_drivers/tasks/ice.yml | 34 +- .../update_nic_firmware/defaults/main.yml | 12 +- roles/cadvisor_install/defaults/main.yaml | 15 +- roles/cadvisor_install/tasks/cleanup.yml | 20 +- roles/cadvisor_install/tasks/install.yml | 47 +- roles/cadvisor_install/tasks/main.yml | 34 +- .../tasks/perf_events_config.yml | 18 +- roles/cadvisor_install/tasks/preflight.yml | 2 +- .../templates/cadvisor_custom_values.yml.j2 | 102 - .../templates/overlay/configmap.yaml.j2 | 8 + .../templates/overlay/daemonset.yaml.j2 | 48 + .../templates/overlay/kustomization.yaml.j2 | 11 + .../templates/overlay/rbac-proxy.yaml.j2 | 29 + .../templates/overlay/service.yaml.j2 | 13 + roles/calico_vpp_install/defaults/main.yml | 36 + .../tasks/calico_vpp_preflight.yml | 50 + roles/calico_vpp_install/tasks/calicoctl.yml | 25 + roles/calico_vpp_install/tasks/calivppctl.yml | 25 + roles/calico_vpp_install/tasks/main.yml | 139 + .../templates/calico-vpp.yaml.j2 | 335 + .../templates/calico.yaml.j2 | 27 + roles/check_machine_type/vars/main.yml | 2 + roles/cluster_defaults/defaults/main.yml | 2 +- .../containerd/defaults/main.yml | 6 +- .../container_engine/crictl/defaults/main.yml | 4 +- roles/container_engine/crio/defaults/main.yml | 10 +- .../crio/templates/crio.conf.j2 | 3 - .../container_engine/docker/defaults/main.yml | 2 +- roles/container_engine/runc/defaults/main.yml | 4 +- roles/container_registry/defaults/main.yml | 4 +- roles/elasticsearch_install/tasks/main.yml | 17 + .../templates/elasticsearch_settings.yml.j2 | 86 + roles/ffmpeg_install/defaults/main.yml | 8 +- .../tasks/ffmpeg_archive_patch.yml | 2 +- roles/ffmpeg_install/tasks/ffmpeg_install.yml | 14 +- roles/imtl_install/defaults/main.yml | 31 + roles/imtl_install/tasks/download.yml | 21 + roles/imtl_install/tasks/dpdk_patch.yml | 127 + roles/imtl_install/tasks/ice_drv_patch.yml | 80 + .../tasks/main.yml} | 32 +- roles/imtl_install/tasks/preflight.yml | 78 + roles/imtl_install/tasks/redhat_deps.yml | 95 + roles/imtl_install/vars/main.yml | 45 + roles/install_ddp_pkgs/defaults/main.yml | 87 +- roles/install_ddp_pkgs/tasks/download.yml | 54 + roles/install_ddp_pkgs/tasks/install.yml | 58 + .../install_ddp_pkgs/tasks/install_a_pkg.yml | 96 - roles/install_ddp_pkgs/tasks/main.yml | 64 +- roles/install_dependencies/tasks/main.yml | 5 +- .../install_dpdk/tasks/install_dpdk_meson.yml | 2 +- roles/install_dpdk/tasks/main.yml | 6 + roles/intel_base_container/defaults/main.yml | 100 +- .../files/install_dlstreamer.sh | 22 + .../files/install_ffmpeg.sh | 12 +- .../files/install_gpu_stack.sh | 168 +- .../files/install_opencv.sh | 12 +- .../files/install_openvino.sh | 51 +- .../files/install_openvino_dev.sh | 36 +- .../files/test_base_devel_entry.sh | 40 - .../files/test_dlstreamer_entry.sh | 72 +- .../files/test_ffmpeg_entry.sh | 43 +- .../files/test_gpu_entry.sh | 10 +- .../files/test_opencv_entry.sh | 36 +- .../files/test_openvino_dev_entry.sh | 41 + .../files/test_openvino_entry.sh | 60 +- roles/intel_base_container/tasks/main.yml | 7 +- .../templates/Dockerfile.j2 | 23 +- .../templates/Dockerfile_test.j2 | 4 +- roles/intel_csl_excat/defaults/main.yml | 27 + roles/intel_csl_excat/files/rdt-config.yaml | 34 + .../files/sys-fs-resctrl.mount | 12 + roles/intel_csl_excat/tasks/cleanup.yml | 38 + roles/intel_csl_excat/tasks/main.yml | 182 + roles/intel_csl_excat/tasks/preflight.yml | 44 + roles/intel_dp_operator/tasks/main.yml | 4 +- roles/intel_eci/tasks/eci_host.yml | 41 + roles/intel_eci/tasks/handle-RT.yml | 45 + roles/intel_eci/tasks/main.yml | 67 +- roles/intel_eci/tasks/openplc.yml | 125 + roles/intel_eci/vars/main.yml | 1 + .../intel_ethernet_operator/defaults/main.yml | 50 +- .../tasks/cache_server.yml | 59 +- roles/intel_ethernet_operator/tasks/ddp.yml | 16 +- .../tasks/ethernet_operator.yml | 21 +- .../templates/cache-server-svc.yml.j2 | 1 + .../templates/cache-server.yml.j2 | 1 + .../templates/catalog.yml.j2 | 3 + .../templates/ddp-update.yml.j2 | 11 +- .../templates/firmware-update.yml.j2 | 5 +- .../templates/operator-group.yml.j2 | 3 + .../templates/subscription.yml.j2 | 3 + roles/intel_flexran/defaults/main.yml | 15 +- .../intel_flexran/files/kernel_cmdline_gen.sh | 43 +- roles/intel_flexran/tasks/bind_fec.yml | 35 +- roles/intel_flexran/tasks/fec_acc.yml | 20 +- .../intel_flexran/tasks/flexran_preflight.yml | 23 +- roles/intel_flexran/tasks/main.yml | 18 + roles/intel_flexran/tasks/pod.yml | 29 +- ...an_pod_timer_mode_spr_ee_non_root.yaml.j2} | 48 +- ...ran_pod_xran_mode_spr_ee_non_root.yaml.j2} | 30 +- .../templates/Dockerfile.j2 | 7 + roles/intel_oneapi_install/defaults/main.yml | 12 +- roles/intel_power_manager/defaults/main.yml | 24 - .../templates/global_shared_profile.yaml.j2 | 12 - .../defaults/main.yml | 8 +- .../tasks/check_sriov_fec_operator.yml | 31 + .../tasks/sriov_fec_operator.yml | 10 +- .../templates/acc200-cr.yaml.j2 | 4 + roles/intel_xpumanager/defaults/main.yml | 28 + roles/intel_xpumanager/tasks/main.yml | 125 + .../tasks/xpumanager_cleanup.yml | 55 + .../tasks/xpumanager_preflight.yml | 66 + .../templates/prometheus_role.yml.j2 | 23 + .../templates/prometheus_rolebinding.yml.j2 | 13 + .../templates/xpumanager_daemonset.yml.j2 | 163 + .../templates/xpumanager_service.yml.j2 | 18 + .../xpumanager_servicemonitor.yml.j2 | 23 + .../defaults/main.yml | 47 + .../files/example_deployment.yaml | 35 + .../tasks/cleanup.yml | 86 + .../tasks/install_ido.yml | 156 + .../tasks/linkerd_viz_install.yml | 49 + .../tasks/main.yml | 22 + .../templates/linkerd-viz-auth.yaml.j2 | 14 + .../acc/defaults/main.yml} | 10 +- roles/ipu/acc/tasks/main.yml | 222 +- roles/ipu/acc/templates/es2k_skip_p4.conf.j2 | 41 + roles/ipu/common/defaults/main.yml | 14 +- roles/ipu/common/tasks/main.yml | 26 +- roles/ipu/flash_ipu_nvm/defaults/main.yml | 10 +- roles/ipu/flash_ipu_nvm/tasks/main.yml | 147 +- roles/ipu/flash_ipu_ssd/defaults/main.yml | 2 +- roles/ipu/flash_ipu_ssd/tasks/main.yml | 21 +- roles/ipu/imc/tasks/main.yml | 47 +- roles/ipu/imc/templates/l2-fwd_lem.sh.j2 | 16 + roles/ipu/prepare_ipu_linkp/tasks/main.yml | 12 +- roles/ipu/prepare_ipu_linkp/vars/main.yml | 2 + .../files/istio-netfilter.conf | 7 + roles/istio_service_mesh/tasks/cleanup.yml | 6 + roles/istio_service_mesh/tasks/main.yml | 33 +- roles/istio_service_mesh/vars/main.yml | 4 +- roles/jaeger_install/defaults/main.yml | 2 +- .../kmra-apphsm-config-configmap.yaml | 1 + .../templates/kmra-apphsm-deployment.yml | 24 +- .../templates/kmra-apphsm-env-configmap.yml | 2 +- .../templates/kmra-ctk-loadkey-configmap.yml | 1 + .../kmra-oran-netopeer2-client-deployment.yml | 5 +- ...ra-oran-netopeer2-client-env-configmap.yml | 1 + .../kmra-oran-netopeer2-server-deployment.yml | 5 +- ...ra-oran-netopeer2-server-env-configmap.yml | 1 + roles/kmra_install/defaults/main/main.yml | 8 +- roles/kmra_install/defaults/main/oran.yml | 36 +- roles/kmra_install/files/oran/Dockerfile | 37 +- roles/kmra_install/files/oran/openssl.cnf.j2 | 6 +- .../kmra_install/files/oran/oran_commands.sh | 9 +- roles/kmra_install/tasks/cleanup.yml | 15 +- .../tasks/create_cosign_tls_secrets.yml | 7 +- .../tasks/create_custom_tls_configmap.yml | 200 +- .../tasks/create_sysrepo_keystore.yml | 41 + .../kmra_install/tasks/create_tls_secrets.yml | 17 +- roles/kmra_install/tasks/main.yml | 58 +- .../templates/kmra-apphsm-values.yaml.j2 | 1 + .../templates/kmra-ctk-values.yaml.j2 | 1 + .../kmra-oran-netopeer2-client-values.yaml.j2 | 3 +- .../kmra-oran-netopeer2-server-values.yaml.j2 | 7 +- .../templates/tls_keystore.xml.j2 | 16 + .../kmra_install/templates/tls_listen.xml.j2 | 38 + .../templates/tls_truststore.xml.j2 | 16 + .../defaults/main.yml | 24 + .../tasks/deploy_features.yml | 10 +- .../tasks/deploy_sample_pods.yml | 8 +- .../tasks/deploy_shared_resources.yml | 12 +- .../tasks/kpm_preflight.yml | 76 + .../tasks/main.yml | 18 +- .../tasks/power_manager.yml | 54 +- .../tasks/power_pod_template_helper.yml | 2 +- .../templates/controller_manager.yaml.j2 | 12 +- .../templates/cstates.yaml.j2 | 2 +- .../templates/global_shared_profile.yaml.j2 | 13 + .../templates/local_shared_profile.yaml.j2 | 5 +- .../templates/power_config.yaml.j2 | 2 +- .../templates/sample_power_pod.yaml.j2 | 2 +- .../templates/shared_workload.yaml.j2 | 2 +- .../templates/uncore_frequency.yaml.j2 | 2 +- .../vars/main.yml | 8 +- .../files/kubespray_dnsstublistener.patch | 13 - roles/kubespray_patch/tasks/main.yml | 7 - roles/linkerd_service_mesh/defaults/main.yml | 2 +- roles/linkerd_service_mesh/tasks/main.yml | 1 - .../templates/cert-manager-objects.yml.j2 | 4 +- .../tasks/cleanup_minio_filesystems.yml | 93 - .../tasks/cleanup_minio_main.yml | 2 - .../tasks/create_nvme_partition.yml | 46 - .../tasks/create_persistentvolumes.yml | 4 +- .../minio_install/tasks/file_blockdevice.yml | 44 - .../tasks/format_blockdevicefiles.yml | 2 +- roles/minio_install/tasks/main.yml | 16 - .../minio_install/tasks/nvme_blockdevice.yml | 26 - .../tasks/preflight_minio_config.yml | 4 +- ...minio_tenant_localpersistentvolumes.yml.j2 | 2 +- .../openssl_engine_install/defaults/main.yml | 4 +- .../tasks/openssl_engine_config.yml | 11 + roles/opentelemetry_install/defaults/main.yml | 2 +- .../files/otel-agent-cadvisor.yaml | 6 +- .../defaults/main.yml | 4 +- .../container_prometheus/defaults/main.yml | 30 + .../container_prometheus/files/dashboard.yml | 12 + .../files/dashboard_node_exporter.json | 23379 ++++++++++++++++ .../files/dashboard_xpumanager.json | 2762 ++ .../container_prometheus/files/grafana.ini | 1544 + .../files/node_web_config.yml | 3 + .../files/prom_web_config.yml | 3 + .../container_prometheus/tasks/cleanup.yml | 49 + .../container_prometheus/tasks/main.yml | 196 + .../templates/datasource.yml.j2 | 26 + .../templates/prometheus.yml.j2 | 45 + roles/prometheus_install/defaults/main.yml | 18 + .../kube_prometheus/defaults/main.yml | 9 +- .../files/dashboards/collectd-dashboard.yml | 0 .../kubernetes-mixin-dashboards.yml | 0 .../files/dashboards/telegraf-dashboard.yml | 0 .../files/dashboards/xpumanager-dashboard.yml | 2776 ++ ...managerConfigCustomResourceDefinition.yaml | 0 ...0alertmanagerCustomResourceDefinition.yaml | 0 .../0podmonitorCustomResourceDefinition.yaml | 0 .../crds/0probeCustomResourceDefinition.yaml | 0 .../0prometheusCustomResourceDefinition.yaml | 0 ...rometheusruleCustomResourceDefinition.yaml | 0 ...ervicemonitorCustomResourceDefinition.yaml | 0 .../0thanosrulerCustomResourceDefinition.yaml | 0 .../kube-prometheus-storage-class.yml | 0 .../kubePrometheus-prometheusRule.yaml | 0 .../kubeStateMetrics-clusterRole.yaml | 0 .../kubeStateMetrics-clusterRoleBinding.yaml | 0 .../kubeStateMetrics-prometheusRule.yaml | 0 .../kubeStateMetrics-service.yaml | 0 .../kubeStateMetrics-serviceAccount.yaml | 0 .../kubeStateMetrics-serviceMonitor.yaml | 0 ...kubernetesControlPlane-prometheusRule.yaml | 0 ...sControlPlane-serviceMonitorApiserver.yaml | 0 ...tesControlPlane-serviceMonitorCoreDNS.yaml | 0 ...e-serviceMonitorKubeControllerManager.yaml | 0 ...trolPlane-serviceMonitorKubeScheduler.yaml | 0 ...tesControlPlane-serviceMonitorKubelet.yaml | 0 .../nodeExporter-prometheusRule.yaml | 0 .../prometheus-clusterRole.yaml | 0 .../prometheus-clusterRoleBinding.yaml | 0 .../prometheus-prometheusRule.yaml | 0 .../kube-prometheus-stack/prometheus-pvc.yml | 0 .../prometheus-roleBindingConfig.yaml | 0 ...metheus-roleBindingSpecificNamespaces.yaml | 0 .../prometheus-roleConfig.yaml | 0 .../prometheus-roleSpecificNamespaces.yaml | 0 .../prometheus-service.yaml | 0 .../prometheus-serviceAccount.yaml | 0 .../prometheus-serviceMonitor.yaml | 0 .../prometheusAdapter-clusterRole.yaml | 0 .../prometheusAdapter-clusterRoleBinding.yaml | 0 ...usAdapter-clusterRoleBindingDelegator.yaml | 0 .../prometheusAdapter-configMap.yaml | 0 .../prometheusAdapter-deployment.yaml | 0 ...ometheusAdapter-roleBindingAuthReader.yaml | 0 .../prometheusAdapter-service.yaml | 0 .../prometheusAdapter-serviceAccount.yaml | 0 .../prometheusAdapter-serviceMonitor.yaml | 0 .../prometheusOperator-clusterRole.yaml | 0 ...prometheusOperator-clusterRoleBinding.yaml | 0 .../prometheusOperator-prometheusRule.yaml | 0 .../prometheusOperator-service.yaml | 0 .../prometheusOperator-serviceAccount.yaml | 0 .../prometheusOperator-serviceMonitor.yaml | 0 .../kube_prometheus/tasks/cleanup.yml | 0 .../tasks/create-tas-demo-policy.yml | 0 .../kube_prometheus/tasks/kube-prometheus.yml | 4 + .../kube_prometheus/tasks/main.yml | 0 .../templates/grafana-config.yaml.j2 | 0 .../grafana-dashboardDatasources.yaml.j2 | 0 .../grafana-dashboardSources.yaml.j2 | 0 .../templates/grafana-deployment.yml.j2 | 22 + .../templates/grafana-prometheusRule.yaml.j2 | 0 .../templates/grafana-pv.yml.j2 | 0 .../templates/grafana-pvc.yml.j2 | 0 .../templates/grafana-service.yaml.j2 | 0 .../templates/grafana-serviceAccount.yaml.j2 | 0 .../templates/grafana-serviceMonitor.yaml.j2 | 0 .../kube-prometheus-stack-certs.yml.j2 | 0 .../kubeStateMetrics-deployment.yaml.j2 | 2 +- .../templates/node-exporter-config.yml.j2 | 0 .../nodeExporter-clusterRole.yaml.j2 | 0 .../nodeExporter-clusterRoleBinding.yaml.j2 | 0 .../templates/nodeExporter-daemonset.yml.j2 | 0 .../templates/nodeExporter-service.yaml.j2 | 0 .../nodeExporter-serviceAccount.yaml.j2 | 0 .../nodeExporter-serviceMonitor.yaml.j2 | 0 .../templates/prometheus-prometheus.yml.j2 | 0 .../templates/prometheus-pv.yml.j2 | 0 .../prometheusAdapter-apiService.yml.j2 | 0 .../prometheusOperator-deployment.yaml.j2 | 0 .../prometheus-srv-nginx-configmap.yaml.j2 | 0 .../prometheus-srv-secret.yaml.j2 | 4 +- .../prometheus-srv-service.yaml.j2 | 4 +- .../kube_prometheus/vars/main.yml | 0 roles/prometheus_install/meta/main.yml | 23 + roles/redeploy_cleanup/defaults/main.yml | 2 + .../redeploy_cleanup/tasks/intel_cleanup.yml | 50 + roles/redeploy_cleanup/tasks/k8s_cleanup.yml | 9 +- roles/redeploy_cleanup/tasks/main.yml | 140 +- roles/redeploy_cleanup/tasks/main_k8s.yml | 170 + roles/redeploy_cleanup/tasks/main_no_k8s.yml | 31 + roles/redeploy_cleanup/tasks/tdx_cleanup.yml | 27 + .../helm/defaults/main.yml | 2 +- roles/rook_install/tasks/cleanup_rook.yml | 49 +- .../tasks/install_rook_helmchart.yml | 68 + .../templates/block-storageclass.yaml.j2 | 57 + .../templates/cephfs-storageclass.yaml.j2 | 37 + roles/rook_install/templates/cluster.yaml.j2 | 22 +- .../templates/object-storageclass.yaml.j2 | 17 + .../defaults/main.yml | 2 +- .../tasks/preflight.yml | 2 +- roles/telegraf_install/defaults/main.yml | 18 +- roles/telegraf_install/tasks/cleanup.yml | 10 + roles/telegraf_install/tasks/main.yml | 10 + roles/telegraf_install/tasks/pmu_events.yml | 2 +- roles/telegraf_install/tasks/telegraf.yml | 2 +- .../compile_libvirt/tasks/compile_libvirt.yml | 5 +- roles/vm/compile_libvirt/tasks/main.yml | 4 +- roles/vm/compile_libvirt/vars/main.yml | 2 + roles/vm/conf_libvirt/tasks/main.yml | 1 + .../install_bm_libvirt/defaults/main.yml} | 3 +- .../install_bm_libvirt/tasks/main.yml} | 27 +- roles/vm/install_bm_libvirt/vars/main.yml | 27 + roles/vm/manage_bridges/tasks/main.yml | 3 + .../tasks/vxlan_interface_name.yml | 6 +- .../vm/manage_bridges/tasks/vxlan_unicast.yml | 26 + .../templates/manage-vxlan.sh.j2 | 9 +- .../templates/simple-bridge.xml.j2 | 5 + .../templates/vxlan-interface.j2 | 2 +- roles/vm/manage_imgs/tasks/prepare_vm_img.yml | 77 +- roles/vm/manage_imgs/templates/user-data.j2 | 42 +- .../defaults => vm/manage_imgs/vars}/main.yml | 12 +- roles/vm/manage_vms/defaults/main.yml | 20 + roles/vm/manage_vms/tasks/start_vm.yml | 143 +- .../tasks/main.yml | 16 +- .../tasks/main.yml | 10 +- .../tasks/main.yml | 111 + roles/vm/prepare_cek/tasks/main.yml | 19 +- roles/vm/prepare_cek_vxlan/tasks/main.yml | 6 +- roles/vm/prepare_vm_inventory/tasks/main.yml | 8 +- .../wait_for_kubernetes_ready/tasks/main.yml | 2 +- .../intel_media_transport_lib/README.md | 78 + .../power_manager/README.md | 8 + .../topology_manager/README.md | 2 - 474 files changed, 41383 insertions(+), 3045 deletions(-) create mode 100644 collections/share/README.md create mode 100644 collections/share/changelogs/changelog.yaml create mode 100644 collections/share/galaxy.yml create mode 100644 collections/share/roles/configure_dpdk/defaults/main.yml rename {roles => collections/share/roles}/configure_dpdk/files/cek_config_dpdk.service (100%) rename {roles => collections/share/roles}/configure_dpdk/files/cek_config_dpdk.sh (64%) rename {roles => collections/share/roles}/configure_dpdk/files/cek_config_dpdk_bind.py (94%) rename {roles => collections/share/roles}/configure_dpdk/files/cek_config_dpdk_link.py (100%) rename {roles => collections/share/roles}/configure_dpdk/files/cek_config_dpdk_rebind.py (100%) rename {roles => collections/share/roles}/configure_dpdk/files/cek_config_dpdk_unbind.py (100%) rename {roles => collections/share/roles}/configure_dpdk/files/cek_config_dpdk_util.py (100%) rename {roles => collections/share/roles}/configure_dpdk/tasks/cleanup.yml (91%) rename {roles => collections/share/roles}/configure_dpdk/tasks/main.yml (63%) create mode 100644 collections/share/roles/install_gpu_driver/defaults/main.yml rename {roles/bootstrap => collections/share/roles}/install_gpu_driver/files/cek_detect_gpu_type.py (85%) rename {roles/bootstrap => collections/share/roles}/install_gpu_driver/files/cek_get_latest_gpu_pkgs.sh (94%) rename {roles/bootstrap => collections/share/roles}/install_gpu_driver/tasks/debian.yml (68%) create mode 100644 collections/share/roles/install_gpu_driver/tasks/main.yml create mode 100644 collections/share/roles/install_gpu_driver/tasks/rhel.yml create mode 100644 docs/calico_vpp.md create mode 100644 docs/eci_guide.md create mode 100644 docs/images/tdx-bios-configure.png create mode 100644 docs/images/tdx-host-check.png create mode 100644 docs/intent_driven_orchestration.md create mode 100644 docs/monitoring.md create mode 100644 docs/profile_overview/BMRA.pdf create mode 100644 docs/profile_overview/CloudRA.pdf create mode 100644 docs/profile_overview/VMRA.pdf create mode 100644 docs/vm_bm_mixed_config_guide.md create mode 100644 playbooks/intel/eci_basic.yml create mode 100644 playbooks/intel/tdx.yml rename roles/bootstrap/{configure_intel_pstate/tasks => configure_disks/defaults}/main.yml (82%) create mode 100644 roles/bootstrap/configure_disks/files/ra_loopdevices.service create mode 100644 roles/bootstrap/configure_disks/tasks/configure_loopdevices.yml create mode 100644 roles/bootstrap/configure_disks/tasks/main.yml create mode 100644 roles/bootstrap/configure_disks/tasks/mount_disks.yml create mode 100644 roles/bootstrap/configure_disks/tasks/setup_fake_pvs.yml create mode 100644 roles/bootstrap/configure_disks/templates/loopdevice_bind.j2 create mode 100644 roles/bootstrap/configure_kpm_drivers/tasks/configure_drivers.yml create mode 100644 roles/bootstrap/configure_kpm_drivers/tasks/main.yml delete mode 100644 roles/bootstrap/install_gpu_driver/defaults/main.yml delete mode 100644 roles/bootstrap/install_gpu_driver/tasks/main.yml create mode 100644 roles/bootstrap/install_qat_drivers_services/tasks/qat_oot_driver_install.yml rename roles/bootstrap/{install-qatlibs => install_qatlibs}/defaults/main.yml (95%) rename roles/bootstrap/{install-qatlibs => install_qatlibs}/tasks/main.yml (85%) rename roles/bootstrap/{install-qatlibs => install_qatlibs}/vars/main.yml (100%) rename roles/{install_ddp_pkgs/tasks/install_pkgs.yml => bootstrap/install_tdx_drivers/defaults/main.yml} (64%) create mode 100644 roles/bootstrap/install_tdx_drivers/tasks/main.yml rename roles/{minio_install/tasks/mount_loopdevices.yml => bootstrap/install_tdx_drivers/tasks/tdx_preflight.yml} (54%) create mode 100644 roles/bootstrap/install_tdx_drivers/tasks/ubuntu.yml rename roles/bootstrap/{configure_intel_pstate => install_tdx_drivers}/vars/main.yml (71%) delete mode 100644 roles/bootstrap/reset_qat_option/tasks/main.yml rename playbooks/intel/base_container.yml => roles/bootstrap/set_tdx_kernel_flags/defaults/main.yml (90%) rename roles/bootstrap/{configure_intel_pstate/tasks/setup_intel_pstate.yml => set_tdx_kernel_flags/tasks/main.yml} (57%) delete mode 100644 roles/cadvisor_install/templates/cadvisor_custom_values.yml.j2 create mode 100644 roles/cadvisor_install/templates/overlay/configmap.yaml.j2 create mode 100644 roles/cadvisor_install/templates/overlay/daemonset.yaml.j2 create mode 100644 roles/cadvisor_install/templates/overlay/kustomization.yaml.j2 create mode 100644 roles/cadvisor_install/templates/overlay/rbac-proxy.yaml.j2 create mode 100644 roles/cadvisor_install/templates/overlay/service.yaml.j2 create mode 100644 roles/calico_vpp_install/defaults/main.yml create mode 100644 roles/calico_vpp_install/tasks/calico_vpp_preflight.yml create mode 100644 roles/calico_vpp_install/tasks/calicoctl.yml create mode 100644 roles/calico_vpp_install/tasks/calivppctl.yml create mode 100644 roles/calico_vpp_install/tasks/main.yml create mode 100644 roles/calico_vpp_install/templates/calico-vpp.yaml.j2 create mode 100644 roles/calico_vpp_install/templates/calico.yaml.j2 create mode 100644 roles/elasticsearch_install/templates/elasticsearch_settings.yml.j2 create mode 100644 roles/imtl_install/defaults/main.yml create mode 100644 roles/imtl_install/tasks/download.yml create mode 100644 roles/imtl_install/tasks/dpdk_patch.yml create mode 100644 roles/imtl_install/tasks/ice_drv_patch.yml rename roles/{intel_power_manager/tasks/enable_drivers.yml => imtl_install/tasks/main.yml} (51%) create mode 100644 roles/imtl_install/tasks/preflight.yml create mode 100644 roles/imtl_install/tasks/redhat_deps.yml create mode 100644 roles/imtl_install/vars/main.yml create mode 100644 roles/install_ddp_pkgs/tasks/download.yml create mode 100644 roles/install_ddp_pkgs/tasks/install.yml delete mode 100644 roles/install_ddp_pkgs/tasks/install_a_pkg.yml delete mode 100755 roles/intel_base_container/files/test_base_devel_entry.sh create mode 100755 roles/intel_base_container/files/test_openvino_dev_entry.sh create mode 100644 roles/intel_csl_excat/defaults/main.yml create mode 100644 roles/intel_csl_excat/files/rdt-config.yaml create mode 100644 roles/intel_csl_excat/files/sys-fs-resctrl.mount create mode 100644 roles/intel_csl_excat/tasks/cleanup.yml create mode 100644 roles/intel_csl_excat/tasks/main.yml create mode 100644 roles/intel_csl_excat/tasks/preflight.yml create mode 100644 roles/intel_eci/tasks/eci_host.yml create mode 100644 roles/intel_eci/tasks/handle-RT.yml create mode 100644 roles/intel_eci/tasks/openplc.yml rename roles/intel_flexran/templates/{intel_flexran_pod_timer_mode_spr_ee.yaml.j2 => intel_flexran_pod_timer_mode_spr_ee_non_root.yaml.j2} (58%) rename roles/intel_flexran/templates/{intel_flexran_pod_xran_mode_spr_ee.yaml.j2 => intel_flexran_pod_xran_mode_spr_ee_non_root.yaml.j2} (74%) delete mode 100644 roles/intel_power_manager/defaults/main.yml delete mode 100644 roles/intel_power_manager/templates/global_shared_profile.yaml.j2 create mode 100644 roles/intel_xpumanager/defaults/main.yml create mode 100644 roles/intel_xpumanager/tasks/main.yml create mode 100644 roles/intel_xpumanager/tasks/xpumanager_cleanup.yml create mode 100644 roles/intel_xpumanager/tasks/xpumanager_preflight.yml create mode 100644 roles/intel_xpumanager/templates/prometheus_role.yml.j2 create mode 100644 roles/intel_xpumanager/templates/prometheus_rolebinding.yml.j2 create mode 100644 roles/intel_xpumanager/templates/xpumanager_daemonset.yml.j2 create mode 100644 roles/intel_xpumanager/templates/xpumanager_service.yml.j2 create mode 100644 roles/intel_xpumanager/templates/xpumanager_servicemonitor.yml.j2 create mode 100644 roles/intent_driven_orchestration/defaults/main.yml create mode 100644 roles/intent_driven_orchestration/files/example_deployment.yaml create mode 100644 roles/intent_driven_orchestration/tasks/cleanup.yml create mode 100644 roles/intent_driven_orchestration/tasks/install_ido.yml create mode 100644 roles/intent_driven_orchestration/tasks/linkerd_viz_install.yml create mode 100644 roles/intent_driven_orchestration/tasks/main.yml create mode 100644 roles/intent_driven_orchestration/templates/linkerd-viz-auth.yaml.j2 rename roles/{minio_install/tasks/setup_loopdevices.yml => ipu/acc/defaults/main.yml} (78%) create mode 100644 roles/ipu/acc/templates/es2k_skip_p4.conf.j2 create mode 100644 roles/ipu/imc/templates/l2-fwd_lem.sh.j2 create mode 100644 roles/istio_service_mesh/files/istio-netfilter.conf create mode 100644 roles/kmra_install/tasks/create_sysrepo_keystore.yml create mode 100644 roles/kmra_install/templates/tls_keystore.xml.j2 create mode 100644 roles/kmra_install/templates/tls_listen.xml.j2 create mode 100644 roles/kmra_install/templates/tls_truststore.xml.j2 create mode 100644 roles/kubernetes_power_manager/defaults/main.yml rename roles/{intel_power_manager => kubernetes_power_manager}/tasks/deploy_features.yml (84%) rename roles/{intel_power_manager => kubernetes_power_manager}/tasks/deploy_sample_pods.yml (82%) rename roles/{intel_power_manager => kubernetes_power_manager}/tasks/deploy_shared_resources.yml (78%) create mode 100644 roles/kubernetes_power_manager/tasks/kpm_preflight.yml rename roles/{intel_power_manager => kubernetes_power_manager}/tasks/main.yml (79%) rename roles/{intel_power_manager => kubernetes_power_manager}/tasks/power_manager.yml (73%) rename roles/{intel_power_manager => kubernetes_power_manager}/tasks/power_pod_template_helper.yml (87%) rename roles/{intel_power_manager => kubernetes_power_manager}/templates/controller_manager.yaml.j2 (71%) rename roles/{intel_power_manager => kubernetes_power_manager}/templates/cstates.yaml.j2 (78%) create mode 100644 roles/kubernetes_power_manager/templates/global_shared_profile.yaml.j2 rename roles/{intel_power_manager => kubernetes_power_manager}/templates/local_shared_profile.yaml.j2 (68%) rename roles/{intel_power_manager => kubernetes_power_manager}/templates/power_config.yaml.j2 (87%) rename roles/{intel_power_manager => kubernetes_power_manager}/templates/sample_power_pod.yaml.j2 (93%) rename roles/{intel_power_manager => kubernetes_power_manager}/templates/shared_workload.yaml.j2 (89%) rename roles/{intel_power_manager => kubernetes_power_manager}/templates/uncore_frequency.yaml.j2 (75%) rename roles/{intel_power_manager => kubernetes_power_manager}/vars/main.yml (72%) delete mode 100644 roles/kubespray_patch/files/kubespray_dnsstublistener.patch delete mode 100644 roles/minio_install/tasks/cleanup_minio_filesystems.yml delete mode 100644 roles/minio_install/tasks/create_nvme_partition.yml delete mode 100644 roles/minio_install/tasks/file_blockdevice.yml delete mode 100644 roles/minio_install/tasks/nvme_blockdevice.yml create mode 100644 roles/prometheus_install/container_prometheus/defaults/main.yml create mode 100644 roles/prometheus_install/container_prometheus/files/dashboard.yml create mode 100644 roles/prometheus_install/container_prometheus/files/dashboard_node_exporter.json create mode 100644 roles/prometheus_install/container_prometheus/files/dashboard_xpumanager.json create mode 100644 roles/prometheus_install/container_prometheus/files/grafana.ini create mode 100644 roles/prometheus_install/container_prometheus/files/node_web_config.yml create mode 100644 roles/prometheus_install/container_prometheus/files/prom_web_config.yml create mode 100644 roles/prometheus_install/container_prometheus/tasks/cleanup.yml create mode 100644 roles/prometheus_install/container_prometheus/tasks/main.yml create mode 100644 roles/prometheus_install/container_prometheus/templates/datasource.yml.j2 create mode 100644 roles/prometheus_install/container_prometheus/templates/prometheus.yml.j2 create mode 100644 roles/prometheus_install/defaults/main.yml rename roles/{ => prometheus_install}/kube_prometheus/defaults/main.yml (92%) rename roles/{ => prometheus_install}/kube_prometheus/files/dashboards/collectd-dashboard.yml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/dashboards/kubernetes-mixin-dashboards.yml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/dashboards/telegraf-dashboard.yml (100%) create mode 100644 roles/prometheus_install/kube_prometheus/files/dashboards/xpumanager-dashboard.yml rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/crds/0alertmanagerConfigCustomResourceDefinition.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/crds/0alertmanagerCustomResourceDefinition.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/crds/0podmonitorCustomResourceDefinition.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/crds/0probeCustomResourceDefinition.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/crds/0prometheusCustomResourceDefinition.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/crds/0prometheusruleCustomResourceDefinition.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/crds/0servicemonitorCustomResourceDefinition.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/crds/0thanosrulerCustomResourceDefinition.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/kube-prometheus-storage-class.yml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/kubePrometheus-prometheusRule.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/kubeStateMetrics-clusterRole.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/kubeStateMetrics-clusterRoleBinding.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/kubeStateMetrics-prometheusRule.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/kubeStateMetrics-service.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/kubeStateMetrics-serviceAccount.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/kubeStateMetrics-serviceMonitor.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/kubernetesControlPlane-prometheusRule.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/kubernetesControlPlane-serviceMonitorApiserver.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/kubernetesControlPlane-serviceMonitorCoreDNS.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/kubernetesControlPlane-serviceMonitorKubeControllerManager.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/kubernetesControlPlane-serviceMonitorKubeScheduler.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/kubernetesControlPlane-serviceMonitorKubelet.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/nodeExporter-prometheusRule.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheus-clusterRole.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheus-clusterRoleBinding.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheus-prometheusRule.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheus-pvc.yml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheus-roleBindingConfig.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheus-roleBindingSpecificNamespaces.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheus-roleConfig.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheus-roleSpecificNamespaces.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheus-service.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheus-serviceAccount.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheus-serviceMonitor.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheusAdapter-clusterRole.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheusAdapter-clusterRoleBinding.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheusAdapter-clusterRoleBindingDelegator.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheusAdapter-configMap.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheusAdapter-deployment.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheusAdapter-roleBindingAuthReader.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheusAdapter-service.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheusAdapter-serviceAccount.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheusAdapter-serviceMonitor.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheusOperator-clusterRole.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheusOperator-clusterRoleBinding.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheusOperator-prometheusRule.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheusOperator-service.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheusOperator-serviceAccount.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/files/kube-prometheus-stack/prometheusOperator-serviceMonitor.yaml (100%) rename roles/{ => prometheus_install}/kube_prometheus/tasks/cleanup.yml (100%) rename roles/{ => prometheus_install}/kube_prometheus/tasks/create-tas-demo-policy.yml (100%) rename roles/{ => prometheus_install}/kube_prometheus/tasks/kube-prometheus.yml (96%) rename roles/{ => prometheus_install}/kube_prometheus/tasks/main.yml (100%) rename roles/{ => prometheus_install}/kube_prometheus/templates/grafana-config.yaml.j2 (100%) rename roles/{ => prometheus_install}/kube_prometheus/templates/grafana-dashboardDatasources.yaml.j2 (100%) rename roles/{ => prometheus_install}/kube_prometheus/templates/grafana-dashboardSources.yaml.j2 (100%) rename roles/{ => prometheus_install}/kube_prometheus/templates/grafana-deployment.yml.j2 (95%) rename roles/{ => prometheus_install}/kube_prometheus/templates/grafana-prometheusRule.yaml.j2 (100%) rename roles/{ => prometheus_install}/kube_prometheus/templates/grafana-pv.yml.j2 (100%) rename roles/{ => prometheus_install}/kube_prometheus/templates/grafana-pvc.yml.j2 (100%) rename roles/{ => prometheus_install}/kube_prometheus/templates/grafana-service.yaml.j2 (100%) rename roles/{ => prometheus_install}/kube_prometheus/templates/grafana-serviceAccount.yaml.j2 (100%) rename roles/{ => prometheus_install}/kube_prometheus/templates/grafana-serviceMonitor.yaml.j2 (100%) rename roles/{ => prometheus_install}/kube_prometheus/templates/kube-prometheus-stack-certs.yml.j2 (100%) rename roles/{ => prometheus_install}/kube_prometheus/templates/kubeStateMetrics-deployment.yaml.j2 (96%) rename roles/{ => prometheus_install}/kube_prometheus/templates/node-exporter-config.yml.j2 (100%) rename roles/{ => prometheus_install}/kube_prometheus/templates/nodeExporter-clusterRole.yaml.j2 (100%) rename roles/{ => prometheus_install}/kube_prometheus/templates/nodeExporter-clusterRoleBinding.yaml.j2 (100%) rename roles/{ => prometheus_install}/kube_prometheus/templates/nodeExporter-daemonset.yml.j2 (100%) rename roles/{ => prometheus_install}/kube_prometheus/templates/nodeExporter-service.yaml.j2 (100%) rename roles/{ => prometheus_install}/kube_prometheus/templates/nodeExporter-serviceAccount.yaml.j2 (100%) rename roles/{ => prometheus_install}/kube_prometheus/templates/nodeExporter-serviceMonitor.yaml.j2 (100%) rename roles/{ => prometheus_install}/kube_prometheus/templates/prometheus-prometheus.yml.j2 (100%) rename roles/{ => prometheus_install}/kube_prometheus/templates/prometheus-pv.yml.j2 (100%) rename roles/{ => prometheus_install}/kube_prometheus/templates/prometheusAdapter-apiService.yml.j2 (100%) rename roles/{ => prometheus_install}/kube_prometheus/templates/prometheusOperator-deployment.yaml.j2 (100%) rename roles/{ => prometheus_install}/kube_prometheus/templates/prometheus_srv/prometheus-srv-nginx-configmap.yaml.j2 (100%) rename roles/{ => prometheus_install}/kube_prometheus/templates/prometheus_srv/prometheus-srv-secret.yaml.j2 (64%) rename roles/{ => prometheus_install}/kube_prometheus/templates/prometheus_srv/prometheus-srv-service.yaml.j2 (76%) rename roles/{ => prometheus_install}/kube_prometheus/vars/main.yml (100%) create mode 100644 roles/prometheus_install/meta/main.yml create mode 100644 roles/redeploy_cleanup/tasks/main_k8s.yml create mode 100644 roles/redeploy_cleanup/tasks/main_no_k8s.yml create mode 100644 roles/redeploy_cleanup/tasks/tdx_cleanup.yml create mode 100644 roles/rook_install/templates/block-storageclass.yaml.j2 create mode 100644 roles/rook_install/templates/cephfs-storageclass.yaml.j2 create mode 100644 roles/rook_install/templates/object-storageclass.yaml.j2 rename roles/{bootstrap/install_gpu_driver/tasks/rhel.yml => vm/install_bm_libvirt/defaults/main.yml} (88%) rename roles/{minio_install/tasks/create_blockdevicefiles.yml => vm/install_bm_libvirt/tasks/main.yml} (54%) create mode 100644 roles/vm/install_bm_libvirt/vars/main.yml create mode 100644 roles/vm/manage_bridges/tasks/vxlan_unicast.yml rename roles/{configure_dpdk/defaults => vm/manage_imgs/vars}/main.yml (82%) create mode 100644 roles/vm/manage_vms/defaults/main.yml create mode 100644 roles/vm/prepare_bm_host_config_vxlan/tasks/main.yml create mode 100644 validation/verification-manual/intel_media_transport_lib/README.md diff --git a/Pipfile b/Pipfile index 43b67861..e9e73ce7 100644 --- a/Pipfile +++ b/Pipfile @@ -4,22 +4,22 @@ verify_ssl = true name = "pypi" [packages] -ansible = "~=5.7.1" -"ansible-core" = "~=2.12" +ansible = "~=7.7.0" +"ansible-core" = "~=2.14" cryptography = "~=41.0" jinja2 = "~=3.1" -netaddr = "~=0.7.19" -pbr = "~=5.4" -jmespath = "~=0.9.5" -"ruamel.yaml" = "~=0.17.21" -"ruamel.yaml.clib" = "~=0.2.6" +netaddr = "~=0.8.0" +pbr = "~=5.11" +jmespath = "~=1.0.1" +"ruamel.yaml" = "~=0.17.32" +"ruamel.yaml.clib" = "~=0.2.7" MarkupSafe = "~=2.1" ipaddr = "*" [dev-packages] -ansible-lint = "~=6.12.1" -pylint = "~=2.15.4" -bandit = "~=1.7.4" +ansible-lint = "~=6.12.2" +pylint = "~=2.17.5" +bandit = "~=1.7.5" [requires] python_version = "3" diff --git a/README.md b/README.md index 5ea04bc8..78239b91 100644 --- a/README.md +++ b/README.md @@ -66,6 +66,8 @@ The software provided here is for reference only and not intended for production 2. Install python dependencies using one of the following methods + > **_NOTE:_** Ensure that at least python3.9 is installed on ansible host + a) Non-invasive virtual environment using pipenv ```bash @@ -102,7 +104,7 @@ The software provided here is for reference only and not intended for production > **_NOTE:_** It is **highly recommended** to read [this](docs/generate_profiles.md) file before profiles generation. ```bash - make examples ARCH= NIC= + make examples ARCH= NIC= ``` 5. Copy example inventory file to the project root dir. diff --git a/action_plugins/cpupin.py b/action_plugins/cpupin.py index b9712aec..fb42b07b 100644 --- a/action_plugins/cpupin.py +++ b/action_plugins/cpupin.py @@ -29,7 +29,7 @@ from ansible.errors import AnsibleActionFail # Minimum required vCPUs for the VM -MINIMUM_VCPUS = 4 +MINIMUM_VCPUS = 2 # Number of vCPUs (CPUs + threads) allocated for host OS HOST_OS_VCPUS = 16 # Minimum required vCPUs for host OS diff --git a/ansible.cfg b/ansible.cfg index 3334e39d..264fbc6b 100644 --- a/ansible.cfg +++ b/ansible.cfg @@ -16,7 +16,7 @@ fact_caching_timeout = 7200 action_plugins = ./action_plugins:~/.ansible/plugins/action:/usr/share/ansible/plugins/action library = ./library -roles_path = roles +roles_path = roles:collections/share/roles collections_path = ./collections log_path = ./.ansible_last_run.log diff --git a/cloud/README.md b/cloud/README.md index e40d1ffc..1452144e 100644 --- a/cloud/README.md +++ b/cloud/README.md @@ -14,11 +14,11 @@ Cloud RA allows for deploying Intel Container Experience Kits on managed Kuberne - Python 3.8+ -- Azure CLI 2.50.0+ ([Install Guide](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=apt)) +- Azure CLI 2.53.0+ ([Install Guide](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=apt)) -- AWS CLI 2.12.7+ ([Install Guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)) +- AWS CLI 2.13.21+ ([Install Guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)) -- Terraform 1.5.2+ +- Terraform 1.5.7+ - Docker 20.10.17+ @@ -81,7 +81,7 @@ azureConfig: sg_whitelist_cidr_blocks: [] enable_proximity_placement: true aks: - kubernetes_version: "1.26" + kubernetes_version: "1.27" cni: "kubenet" # Possible values are: kubenet, cilium enable_sgx: false # Requires DCsv series instances in one of node pools default_node_pool: @@ -120,7 +120,7 @@ awsConfig: sg_whitelist_cidr_blocks: [] ecr_repositories: [] eks: - kubernetes_version: "1.26" + kubernetes_version: "1.27" subnets: ["subnet_a", "subnet_b"] custom_ami: "ubuntu" # Comment out this line to use Amazon Linux 2 OS node_groups: diff --git a/cloud/cwdf_example_aws.yaml b/cloud/cwdf_example_aws.yaml index 8bbe089e..097f700b 100644 --- a/cloud/cwdf_example_aws.yaml +++ b/cloud/cwdf_example_aws.yaml @@ -15,7 +15,7 @@ awsConfig: sg_whitelist_cidr_blocks: [] ecr_repositories: [] eks: - kubernetes_version: "1.26" + kubernetes_version: "1.27" subnets: ["subnet_a", "subnet_b"] custom_ami: "ubuntu" # Comment out this line to use Amazon Linux 2 OS node_groups: diff --git a/cloud/cwdf_example_azure.yaml b/cloud/cwdf_example_azure.yaml index 01fd81b2..9e10301c 100644 --- a/cloud/cwdf_example_azure.yaml +++ b/cloud/cwdf_example_azure.yaml @@ -14,7 +14,7 @@ azureConfig: sg_whitelist_cidr_blocks: [] enable_proximity_placement: true aks: - kubernetes_version: "1.26" + kubernetes_version: "1.27" cni: "kubenet" # Possible values are: kubenet, cilium, cilium-ebpf enable_sgx: false # Requires DCsv series instances in one of node pools default_node_pool: diff --git a/cloud/cwdf_util/config.py b/cloud/cwdf_util/config.py index d9254534..8f59b21f 100644 --- a/cloud/cwdf_util/config.py +++ b/cloud/cwdf_util/config.py @@ -24,7 +24,7 @@ Optional("root_volume_type", default='gp2'): str }], Optional("eks"): { - Optional("kubernetes_version", default='1.26'): Or("1.24", "1.25", "1.26"), + Optional("kubernetes_version", default='1.27'): Or("1.25", "1.26", "1.27"), "subnets": [str], Optional("install_ebs_csi_driver", default=True): bool, Optional("custom_ami", default=None): str, @@ -48,7 +48,7 @@ Optional("enable_proximity_placement", default=False): bool, Optional("ansible_instance_size", default="Standard_B2s"): str, Optional("aks"): { - Optional("kubernetes_version", default='1.26'): Or("1.25", "1.26"), + Optional("kubernetes_version", default='1.27'): Or("1.26", "1.27"), Optional("cni", default="cilium"): Or("cilium", "kubenet"), Optional("enable_sgx", default=False): bool, "default_node_pool": { diff --git a/cloud/cwdf_util/main.py b/cloud/cwdf_util/main.py index bc4635e7..e42763b0 100644 --- a/cloud/cwdf_util/main.py +++ b/cloud/cwdf_util/main.py @@ -78,7 +78,7 @@ def compose_cloudcli( # version = cloud_config["eks"]["kubernetes_version"] # region = cloud_config["region"] file_loader = FileSystemLoader(provider_template_path) - env = Environment(loader=file_loader) + env = Environment(loader=file_loader, autoescape=True) shutil.copy2(os.path.join(provider_template_path, 'aws_cloudcli_cleanup.sh'), deployment_dir) cleanup_file = os.path.join(deployment_dir, 'aws_cloudcli_cleanup.sh') print(f"Cleanup file path: {cleanup_file}") @@ -87,7 +87,7 @@ def compose_cloudcli( script_template = env.get_template("aws_cloudcli_deploy.sh.j2") elif cloud_provider == "azure": file_loader = FileSystemLoader(provider_template_path) - env = Environment(loader=file_loader) + env = Environment(loader=file_loader, autoescape=True) shutil.copy2(os.path.join(provider_template_path, 'azure_cloudcli_cleanup.sh'), deployment_dir) cleanup_file = os.path.join(deployment_dir, 'azure_cloudcli_cleanup.sh') st = os.stat(cleanup_file) diff --git a/cloud/cwdf_util/templates/cloudcli/aws/aws_cloudcli_deploy.sh.j2 b/cloud/cwdf_util/templates/cloudcli/aws/aws_cloudcli_deploy.sh.j2 index 9eb681a4..168efbae 100644 --- a/cloud/cwdf_util/templates/cloudcli/aws/aws_cloudcli_deploy.sh.j2 +++ b/cloud/cwdf_util/templates/cloudcli/aws/aws_cloudcli_deploy.sh.j2 @@ -47,9 +47,21 @@ ANSIBLE_INSTANCE_IMAGE=$(aws ec2 describe-images \ ANSIBLE_INSTANCE_TYPE="t3.medium" ANSIBLE_INSTANCE_NAME="cwdf-infra-{{ cloud_config.job_id }}-ansible-instance" +# Generate Ansible instance SSH Host Key +if [ ! -f ./ansible_host ] +then + ssh-keygen -q -N "" -t rsa -f ./ansible_host +fi +ANSIBLE_INSTANCE_HOST_PRIVKEY=$(cat ./ansible_host) +ANSIBLE_INSTANCE_HOST_PUBKEY=$(cat ./ansible_host.pub) + # Ansible instance entrypoint script ANSIBLE_INSTANCE_ENTRYPOINT="$(cat <<- "EOM" #!/usr/bin/env bash +echo $ANSIBLE_INSTANCE_HOST_PRIVKEY > /etc/ssh/ssh_host_rsa_key +rm /etc/ssh/ssh_host_dsa_key +rm /etc/ssh/ssh_host_ed25519_key +rm /etc/ssh/ssh_host_ecdsa_key apt-get -qq -y update apt-get -qq -y upgrade apt-get -qq -y install python3-pip python3-venv @@ -359,10 +371,13 @@ JSON_OUTPUT=$(jq -n \ --arg cloud_provider "aws" \ --arg cr_url $ECR_URL \ --arg k8s_worker_username "ubuntu" \ + --arg host_key "$ANSIBLE_INSTANCE_HOST_PUBKEY" \ '{ansible_host_public_ip: {value: $ansible_host_ip}, cloud_provider: {value: $cloud_provider}, + ansible_ssh_host_key: + {value: $host_key}, cr_url: {value: $cr_url}, k8s_worker_username: diff --git a/cloud/cwdf_util/templates/cloudcli/azure/azure_cloudcli_deploy.sh.j2 b/cloud/cwdf_util/templates/cloudcli/azure/azure_cloudcli_deploy.sh.j2 index 3d732746..eafa795d 100644 --- a/cloud/cwdf_util/templates/cloudcli/azure/azure_cloudcli_deploy.sh.j2 +++ b/cloud/cwdf_util/templates/cloudcli/azure/azure_cloudcli_deploy.sh.j2 @@ -30,9 +30,21 @@ NIC_NAME="cwdf-infra-{{ cloud_config.job_id }}-ansible-instance-nic" ANSIBLE_INSTANCE_NAME="cwdf-infra-{{ cloud_config.job_id }}-ansible-instance" ANSIBLE_INSTANCE_IMAGE="Canonical:0001-com-ubuntu-server-jammy:22_04-lts-gen2:latest" +# Generate Ansible instance SSH Host Key +if [ ! -f ${SCRIPT_DIR}/ansible_host ] +then + ssh-keygen -q -N "" -t rsa -f ${SCRIPT_DIR}/ansible_host +fi +ANSIBLE_INSTANCE_HOST_PRIVKEY=$(cat ${SCRIPT_DIR}/ansible_host) +ANSIBLE_INSTANCE_HOST_PUBKEY=$(cat ${SCRIPT_DIR}/ansible_host.pub) + # Ansible instance entrypoint script ANSIBLE_INSTANCE_ENTRYPOINT="$(cat <<- "EOM" #!/usr/bin/env bash +echo $ANSIBLE_INSTANCE_HOST_PRIVKEY > /etc/ssh/ssh_host_rsa_key +rm /etc/ssh/ssh_host_dsa_key +rm /etc/ssh/ssh_host_ed25519_key +rm /etc/ssh/ssh_host_ecdsa_key mkdir -p /etc/apt/keyrings curl -sLS https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor | @@ -479,6 +491,7 @@ JSON_OUTPUT=$(jq -n \ --arg k8s_worker_username "azureuser" \ --arg rg_name "$AZ_GROUP_NAME" \ --arg sub_id "$SUBSCRIPTION_ID" \ + --arg host_key "$ANSIBLE_INSTANCE_HOST_PUBKEY" \ '{aks_cluster_name: {value: $aks_name}, aks_scale_sets_rg: @@ -487,6 +500,8 @@ JSON_OUTPUT=$(jq -n \ {value: $ansible_host_ip}, cloud_provider: {value: $cloud_provider}, + ansible_ssh_host_key: + {value: $host_key}, cr_url: {value: $cr_url}, k8s_worker_username: diff --git a/cloud/cwdf_util/templates/terraform/aws/ansible_host.tf.jinja b/cloud/cwdf_util/templates/terraform/aws/ansible_host.tf.jinja index a7303d7c..a33333d4 100644 --- a/cloud/cwdf_util/templates/terraform/aws/ansible_host.tf.jinja +++ b/cloud/cwdf_util/templates/terraform/aws/ansible_host.tf.jinja @@ -143,6 +143,11 @@ resource "aws_eip" "ansible" { domain = "vpc" } +resource "tls_private_key" "ansible_ssh_rsa_host_key" { + algorithm = "RSA" + rsa_bits = 4096 +} + resource "aws_instance" "ansible" { ami = data.aws_ami.ubuntu2204.id instance_type = "{{ ansible_instance_type }}" @@ -165,6 +170,10 @@ resource "aws_instance" "ansible" { user_data = < /etc/ssh/ssh_host_rsa_key +rm /etc/ssh/ssh_host_dsa_key +rm /etc/ssh/ssh_host_ed25519_key +rm /etc/ssh/ssh_host_ecdsa_key apt-get -qq -y update apt-get -qq -y upgrade apt-get -qq -y install python3-pip python3-venv @@ -207,3 +216,7 @@ EOF output "ansible_host_public_ip" { value = aws_eip.ansible.public_ip } + +output "ansible_host_ssh_host_key" { + value = tls_private_key.ansible_ssh_rsa_host_key.public_key_openssh +} diff --git a/cloud/cwdf_util/templates/terraform/aws/provider.tf.jinja b/cloud/cwdf_util/templates/terraform/aws/provider.tf.jinja index 3f86505d..780fd121 100644 --- a/cloud/cwdf_util/templates/terraform/aws/provider.tf.jinja +++ b/cloud/cwdf_util/templates/terraform/aws/provider.tf.jinja @@ -2,15 +2,15 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = "5.7.0" + version = "5.17.0" } kubernetes = { source = "hashicorp/kubernetes" - version = "2.21.1" + version = "2.23.0" } helm = { source = "hashicorp/helm" - version = "2.10.1" + version = "2.11.0" } } } diff --git a/cloud/cwdf_util/templates/terraform/azure/aks.tf.jinja b/cloud/cwdf_util/templates/terraform/azure/aks.tf.jinja index ccfffaf9..f36c2c9f 100644 --- a/cloud/cwdf_util/templates/terraform/azure/aks.tf.jinja +++ b/cloud/cwdf_util/templates/terraform/azure/aks.tf.jinja @@ -42,7 +42,7 @@ resource "azurerm_kubernetes_cluster" "default" { {% endif %} {% if aks.cni == "cilium" %} ebpf_data_plane = "cilium" - network_plugin_mode = "Overlay" + network_plugin_mode = "overlay" {% endif %} } diff --git a/cloud/cwdf_util/templates/terraform/azure/ansible_host.tf.jinja b/cloud/cwdf_util/templates/terraform/azure/ansible_host.tf.jinja index 2f70ec32..c2a413c3 100644 --- a/cloud/cwdf_util/templates/terraform/azure/ansible_host.tf.jinja +++ b/cloud/cwdf_util/templates/terraform/azure/ansible_host.tf.jinja @@ -39,6 +39,11 @@ resource "azurerm_network_interface_security_group_association" "ansible_instanc network_security_group_id = azurerm_network_security_group.default.id } +resource "tls_private_key" "ansible_ssh_rsa_host_key" { + algorithm = "RSA" + rsa_bits = 4096 +} + resource "azurerm_linux_virtual_machine" "ansible_instance" { name = "cwdf-infra-{{ job_id }}-ansible-instance" resource_group_name = azurerm_resource_group.default.name @@ -79,6 +84,10 @@ resource "azurerm_linux_virtual_machine" "ansible_instance" { custom_data = base64encode(< /etc/ssh/ssh_host_rsa_key +rm /etc/ssh/ssh_host_dsa_key +rm /etc/ssh/ssh_host_ed25519_key +rm /etc/ssh/ssh_host_ecdsa_key mkdir -p /etc/apt/keyrings curl -sLS https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor | @@ -254,3 +263,7 @@ resource "azurerm_monitor_data_collection_rule_association" "ansible_instance" { output "ansible_host_public_ip" { value = azurerm_public_ip.ansible_instance.ip_address } + +output "ansible_host_ssh_host_key" { + value = tls_private_key.ansible_ssh_rsa_host_key.public_key_openssh +} diff --git a/cloud/cwdf_util/templates/terraform/azure/provider.tf.jinja b/cloud/cwdf_util/templates/terraform/azure/provider.tf.jinja index f0e75bef..c6d54c29 100644 --- a/cloud/cwdf_util/templates/terraform/azure/provider.tf.jinja +++ b/cloud/cwdf_util/templates/terraform/azure/provider.tf.jinja @@ -2,11 +2,11 @@ terraform { required_providers { azurerm = { source = "hashicorp/azurerm" - version = "3.64.0" + version = "3.74.0" } helm = { source = "hashicorp/helm" - version = "2.10.1" + version = "2.11.0" } } } diff --git a/cloud/deployer.py b/cloud/deployer.py index f874dbad..f1f9f23b 100644 --- a/cloud/deployer.py +++ b/cloud/deployer.py @@ -16,7 +16,7 @@ import cwdf_util import sw_deployment.sw_deployment_tool as sw_deployment -from ssh_connector import SSHConnector +from ssh_connector import SSHConnector, SSHHostKey def subprocess_run(*args, **kwargs): @@ -107,8 +107,10 @@ def deploy(deployment_dir, provisioner_tool): provisioning_output = None if provisioner_tool == "terraform": provisioning_output = terrafrom_provisioning(deployment_dir, cwdf_user_config, job_id, ssh_public_key) - if provisioner_tool == "cloudcli": + elif provisioner_tool == "cloudcli": provisioning_output = cloudcli_provisioning(deployment_dir, cwdf_user_config, job_id, public_key_path) + else: + return ansible_host_ip = provisioning_output["ansible_host_public_ip"]["value"] click.echo("Ansible Host is accessible on: " + ansible_host_ip) @@ -154,9 +156,12 @@ def deploy(deployment_dir, provisioner_tool): click.echo("Public ip: " + worker["public_ip"]) click.echo("-------------------") ssh_username = provisioning_output["k8s_worker_username"]["value"] + ssh_host_key_raw = provisioning_output["ansible_host_ssh_host_key"]["value"][8:] + ssh_host_key = SSHHostKey("ssh-rsa", ssh_host_key_raw) click.echo("Opening SSH connection to Ansible host...") ssh = SSHConnector(ip_address=ansible_host_ip, username='ubuntu', + host_keys=[ssh_host_key], priv_key=private_key_path, try_loop=True) click.echo("Opened SSH connection.") @@ -213,6 +218,7 @@ def deploy(deployment_dir, provisioner_tool): with open(file=sw_config_path, mode='r', encoding='utf-8') as file: sw_configuration = yaml.load(file, Loader=yaml.FullLoader) sw_configuration['ansible_host_ip'] = ansible_host_ip + sw_configuration['ansible_ssh_host_key'] = ssh_host_key_raw sw_configuration['worker_ips'] = workers_ip sw_configuration['ssh_user'] = ssh_username sw_configuration['ssh_key'] = os.path.join('..', private_key_path) diff --git a/cloud/discovery/profiles.yml b/cloud/discovery/profiles.yml index f6a73901..546fb97d 100644 --- a/cloud/discovery/profiles.yml +++ b/cloud/discovery/profiles.yml @@ -14,7 +14,6 @@ # - cpusets # - native_cpu_manager # - bond_cni -# - topology_manager # - sriov_operator # - sriov_network_dp # - nic_drivers @@ -32,7 +31,7 @@ # - openssl # - tas # - gas -# - ddp +# - ddp_legacy # - network_userspace # - dpdk # - ovs_dpdk @@ -62,7 +61,7 @@ # - intel_ethernet_operator # enabled # flow_config -# ddp +# ddp_update # fw_update # - intel_sriov_fec_operator @@ -76,7 +75,6 @@ access: isolcpu: optional cpusets: optional native_cpu_manager: on - topology_manager: on sriov_operator: on sriov_network_dp: optional nic_drivers: on @@ -100,7 +98,7 @@ access: tac: off tas: off gas: off - ddp: off + ddp_legacy: off network_userspace: off dpdk: on ovs_dpdk: off @@ -132,7 +130,7 @@ access: intel_ethernet_operator: enabled: optional flow_config: optional - ddp: optional + ddp_update: optional fw_update: optional intel_sriov_fec_operator: on @@ -144,7 +142,6 @@ basic: kube_dashboard: on isolcpu: optional cpusets: optional - topology_manager: on sriov_operator: optional sriov_network_dp: optional nic_drivers: on @@ -178,7 +175,6 @@ full_nfv: isolcpu: optional cpusets: optional native_cpu_manager: on - topology_manager: on sriov_operator: on sriov_network_dp: optional nic_drivers: on @@ -198,7 +194,7 @@ full_nfv: tac: on tas: on gas: optional - ddp: on + ddp_legacy: on network_userspace: on dpdk: on ovs_dpdk: on @@ -229,7 +225,7 @@ full_nfv: intel_ethernet_operator: enabled: optional flow_config: optional - ddp: optional + ddp_update: optional fw_update: optional intel_sriov_fec_operator: optional @@ -242,7 +238,6 @@ on_prem: isolcpu: optional cpusets: optional native_cpu_manager: on - topology_manager: on sriov_operator: on sriov_network_dp: optional nic_drivers: on @@ -296,7 +291,6 @@ regional_dc: kube_dashboard: on isolcpu: optional cpusets: optional - topology_manager: on sriov_operator: optional sriov_network_dp: optional nic_drivers: on @@ -339,7 +333,6 @@ remote_fp: isolcpu: optional cpusets: optional native_cpu_manager: on - topology_manager: on sriov_operator: on sriov_network_dp: optional nic_drivers: on @@ -355,7 +348,7 @@ remote_fp: qat_dp: optional openssl: on tas: on - ddp: on + ddp_legacy: on bond_cni: optional network_userspace: optional dpdk: on @@ -385,7 +378,7 @@ remote_fp: intel_ethernet_operator: enabled: optional flow_config: optional - ddp: optional + ddp_update: optional fw_update: optional storage: @@ -395,14 +388,13 @@ storage: nfd: on kube_dashboard: on native_cpu_manager: on - topology_manager: on sriov_operator: on sriov_network_dp: optional nic_drivers: on qat: optional qat_dp: optional tas: on - ddp: optional + ddp_legacy: optional dpdk: on cstate: optional ufs: optional @@ -421,7 +413,7 @@ storage: intel_ethernet_operator: enabled: optional flow_config: optional - ddp: optional + ddp_update: optional fw_update: optional build_your_own: @@ -433,7 +425,6 @@ build_your_own: isolcpu: optional cpusets: optional native_cpu_manager: optional - topology_manager: optional sriov_operator: optional sriov_network_dp: optional nic_drivers: optional @@ -453,7 +444,7 @@ build_your_own: tac: optional tas: optional gas: optional - ddp: optional + ddp_legacy: optional network_userspace: optional dpdk: optional ovs_dpdk: optional @@ -484,6 +475,6 @@ build_your_own: intel_ethernet_operator: enabled: optional flow_config: optional - ddp: optional + ddp_update: optional fw_update: optional intel_sriov_fec_operator: optional diff --git a/cloud/ssh_connector.py b/cloud/ssh_connector.py index 7bb96bb7..89f57b92 100644 --- a/cloud/ssh_connector.py +++ b/cloud/ssh_connector.py @@ -3,12 +3,26 @@ import os import sys import time +from base64 import decodebytes import click -from paramiko import SSHClient, SSHConfig, ProxyCommand, AutoAddPolicy, SSHException, AuthenticationException +from paramiko import SSHClient, SSHConfig, ProxyCommand, RejectPolicy, ECDSAKey, RSAKey, Ed25519Key, SSHException, AuthenticationException from scp import SCPClient, SCPException +class SSHHostKey: + def __init__(self, key_type, key_string): + self.key_type = key_type + self.key_string = key_string + + def get_paramiko_pkey(self): + if self.key_type == 'ssh-rsa': + return RSAKey(data=decodebytes(self.key_string.encode())) + elif self.key_type == 'ssh-ed25519': + return Ed25519Key(data=decodebytes(self.key_string.encode())) + if "ecdsa" in self.key_type: + return ECDSAKey(data=decodebytes(self.key_string.encode())) + return None class SSHConnector: """ @@ -16,7 +30,7 @@ class SSHConnector: Class supports proxy jump connection for cloud instances without public access. """ - def __init__(self, ip_address, username, port=22, priv_key=None, gateway=None, try_loop=False): + def __init__(self, ip_address, username, port=22, host_keys=None, priv_key=None, gateway=None, try_loop=False): """ Initialize the class and connect to the client. The method supports gateway proxy hopping. @@ -26,6 +40,7 @@ def __init__(self, ip_address, username, port=22, priv_key=None, gateway=None, t ip_address (string): IP address of the remote instance username (string): User name for authentication in remote instance port (int): SSH port + host_keys ([SSHHostKey]): list of acceptable host keys priv_key (string): Path to private RSA key for authentication in remote instance gateway (SSHConnector obj): [optional] SSHConnector object with active SSH connection to gateway for create proxy jump @@ -36,7 +51,12 @@ def __init__(self, ip_address, username, port=22, priv_key=None, gateway=None, t """ self.client = SSHClient() - self.client.set_missing_host_key_policy(AutoAddPolicy()) + self.client.set_missing_host_key_policy(RejectPolicy()) + if host_keys is not None: + for key in host_keys: + paramiko_pkey = key.get_paramiko_pkey() + if paramiko_pkey is not None: + self.client.get_host_keys().add(hostname=ip_address, keytype=key.key_type, key=paramiko_pkey) sock = None if gateway: diff --git a/cloud/sw_deployment/docker_management.py b/cloud/sw_deployment/docker_management.py index 6e52c5c8..e3f095fd 100644 --- a/cloud/sw_deployment/docker_management.py +++ b/cloud/sw_deployment/docker_management.py @@ -116,7 +116,7 @@ def initialize_ecr(self): aws_secret_access_key=self.aws_access_secret_key, region_name=self.aws_region) - ecr_credentials = (ecr_client.get_authorization_token()['authorizationData'][0]) + ecr_credentials = ecr_client.get_authorization_token()['authorizationData'][0] self.cr_username = "AWS" self.cr_password = (base64.b64decode(ecr_credentials['authorizationToken']) .replace(b'AWS:', b'').decode('utf-8')) diff --git a/cloud/sw_deployment/sw_deployment_tool.py b/cloud/sw_deployment/sw_deployment_tool.py index 27783067..ebfa894c 100644 --- a/cloud/sw_deployment/sw_deployment_tool.py +++ b/cloud/sw_deployment/sw_deployment_tool.py @@ -9,7 +9,7 @@ import yaml sys.path.append(os.path.abspath(os.path.join(os.getcwd(), os.pardir))) -from ssh_connector import SSHConnector # pylint:disable=C0413,E0401 +from ssh_connector import SSHConnector, SSHHostKey # pylint:disable=C0413,E0401 from docker_management import DockerManagement # pylint:disable=C0413,E0401 configuration = { @@ -18,6 +18,7 @@ 'region': None }, 'ansible_host_ip': None, + 'ansible_ssh_host_key': None, 'controller_ips': [], 'worker_ips': [], 'ssh_key': None, @@ -46,7 +47,7 @@ DEFAULT_CONFIG = os.path.join(ROOT_DIR, '../deployment/sw.yaml') nodes_list = [] - +node_host_keys = {} @click.command() @click.option('-c', '--config', @@ -180,9 +181,23 @@ def _remove_ssh_banner(ssh_client, node_ips_array, user): """ for node_ip in node_ips_array: ssh_client.exec_command(f"ssh-keyscan -H {node_ip} >> /home/ubuntu/.ssh/known_hosts") + node_keyscan = ssh_client.exec_command(f"ssh-keyscan {node_ip}", return_parsed_output=True) + node_keyscan = [line for line in node_keyscan.splitlines() if not line.startswith('#')] + + host_keys = [] + for node_key in node_keyscan: + node_key = node_key.split() + host_keys.append(SSHHostKey(node_key[1], node_key[2])) + node_host_keys[node_ip] = host_keys + if node_ip != "127.0.0.1": click.echo(f"{node_ip}, {user}") - ssh_node = SSHConnector(node_ip, user, 22, configuration['ssh_key'], ssh_client.client) + ssh_node = SSHConnector(ip_address=node_ip, + username=user, + port=22, + host_keys=node_host_keys[node_ip], + priv_key=configuration['ssh_key'], + gateway=ssh_client.client) ssh_node.exec_command('sudo rm /root/.ssh/authorized_keys') ssh_node.exec_command(f"sudo cp /home/{user}/.ssh/authorized_keys /root/.ssh/") if configuration["cloud_settings"]["provider"] == "azure": @@ -212,13 +227,14 @@ def _install_dependencies_on_nodes(ssh_client, node_ips_array): ssh_node = SSHConnector(ip_address=node_ip, username="root", port=22, + host_keys=node_host_keys[node_ip], priv_key=configuration['ssh_key'], gateway=ssh_client.client) - ssh_node.exec_command(command='yum makecache && yum -y install pciutils.x86_64 golang', + ssh_node.exec_command(command='sudo apt-get update -y && sudo apt-get install -y pciutils golang', print_output=True) ssh_node.close_connection() else: - ssh_client.exec_command(command='yum makecache && yum -y install pciutils.x86_64 golang', + ssh_client.exec_command(command='sudo apt-get update -y && sudo apt-get install -y pciutils golang', print_output=True) @@ -237,7 +253,12 @@ def _discovery_nodes(ssh_client, root_user, node_ips, node_type): for node_ip in node_ips: ssh_client.exec_command(f"ssh-keyscan -H {node_ip} >> /home/ubuntu/.ssh/known_hosts") if node_ip != "127.0.0.1": - ssh_node = SSHConnector(node_ip, root_user, 22, configuration['ssh_key'], ssh_client.client) + ssh_node = SSHConnector(ip_address=node_ip, + username=root_user, + port=22, + host_keys=node_host_keys[node_ip], + priv_key=configuration['ssh_key'], + gateway=ssh_client.client) node_hostname = ssh_node.exec_command(command='sudo cat /etc/hostname', return_parsed_output=True) ssh_node.close_connection() else: @@ -268,7 +289,7 @@ def _create_inventory_file(ssh_client, nodes): """ template_loader = jinja2.FileSystemLoader(searchpath=DATA_DIR) - environment = jinja2.Environment(loader=template_loader) + environment = jinja2.Environment(loader=template_loader, autoescape=True) template = environment.get_template("inventory.ini.j2") with open(INVENTORY_FILE, mode="w", encoding="utf-8") as inventory: inventory.write(template.render(hosts=nodes)) @@ -311,7 +332,8 @@ def _docker_login(node_ips, ssh_client, user, registry, registry_username, passw """ for node_ip in node_ips: - ssh_node = SSHConnector(node_ip, user, 22, configuration['ssh_key'], ssh_client.client) + ssh_host_key = SSHHostKey("ssh-rsa", configuration['ansible_ssh_host_key']) + ssh_node = SSHConnector(node_ip, user, 22, [ssh_host_key], configuration['ssh_key'], ssh_client.client) ssh_node.exec_command(command=f"docker login {registry} --username {registry_username} --password {password}", print_output=True) ssh_node.close_connection() @@ -334,7 +356,8 @@ def cleanup(config): _parse_configuration_file(config=config) - client = SSHConnector(ip_address=configuration['ansible_host_ip'], username='ubuntu', priv_key=configuration['ssh_key']) + ssh_host_key = SSHHostKey("ssh-rsa", configuration['ansible_ssh_host_key']) + client = SSHConnector(ip_address=configuration['ansible_host_ip'], username='ubuntu', host_keys=[ssh_host_key], priv_key=configuration['ssh_key']) for image in configuration['exec_containers']: image_name = image.replace('/', '-') @@ -361,7 +384,8 @@ def _deploy(provider, ansible_host_ip, ssh_key, ssh_user, custom_ami): """ click.echo("-------------------") click.secho(f"Connecting to Ansible instance with IP: {configuration['ansible_host_ip']}", fg="yellow") - client = SSHConnector(ip_address=ansible_host_ip, username='ubuntu', priv_key=ssh_key) + ssh_host_key = SSHHostKey("ssh-rsa", configuration['ansible_ssh_host_key']) + client = SSHConnector(ip_address=ansible_host_ip, username='ubuntu', host_keys=[ssh_host_key], priv_key=ssh_key) click.echo("-------------------") click.secho("Copy private SSH key to Ansible instance", fg="yellow") @@ -491,7 +515,8 @@ def _deploy(provider, ansible_host_ip, ssh_key, ssh_user, custom_ami): configuration['exec_containers']): click.echo("-------------------") click.secho("Copy Docker images to cloud registry") - ssh_client = SSHConnector(ip_address=ansible_host_ip, username='ubuntu', priv_key=ssh_key) + ssh_host_key = SSHHostKey("ssh-rsa", configuration['ansible_ssh_host_key']) + ssh_client = SSHConnector(ip_address=ansible_host_ip, username='ubuntu', host_keys=[ssh_host_key], priv_key=ssh_key) click.echo(configuration['exec_containers']) click.echo(f"From registry: {configuration['replicate_from_container_registry']}") docker_mgmt = DockerManagement(from_registry=configuration['replicate_from_container_registry'], diff --git a/collections/requirements.yml b/collections/requirements.yml index 3db125cb..ea5eefe0 100644 --- a/collections/requirements.yml +++ b/collections/requirements.yml @@ -1,4 +1,4 @@ collections: - name: https://github.com/kubernetes-sigs/kubespray type: git - version: f9f5143c93f583541ccb6650eb008f7ef3d1bc3c + version: d646053c0e7fdc0dc95661d1b5ab72dd61ab0576 diff --git a/collections/share/README.md b/collections/share/README.md new file mode 100644 index 00000000..fd6023cb --- /dev/null +++ b/collections/share/README.md @@ -0,0 +1,118 @@ +# Ansible Collection - cek.share + + +# Motivation +This Ansible Collection is to share CEK roles or modules to other consumer who just wants to use part of CEK. You can use one of below ways to import CEK collection to your Ansible environemnt, then add its role folder to your ansible role path. +* Import it by direct cmdline + ``` + $ ansible-galaxy collection install git+https://github.com/intel/container-experience-kits.git#/collections/share -p ./ + or + $ ansible-galaxy collection install git+https://github.com/intel/container-experience-kits.git#/collections/share,{branch_or_tag} -p ./ + ``` +* Import it by requirements file + Edit a requirements.yml file like below : + ``` + collections: + - source: https://github.com/intel/container-experience-kits.git#/collections/share + type: git + ``` + or + + ``` + collections: + - source: https://github.com/intel/container-experience-kits.git#/collections/share + type: git + version: {branch_or_tag} + ``` + + Then import it by below shell script : + ``` + #!/bin/bash + export ANSIBLE_COLLECTIONS_PATH=$PWD; + ansible-galaxy collection install -r requirements.yml + ``` +# Supported Roles and Modules +Currently, below roles are shared in this collection : +* ## configure_dpdk
+ + User can set condition and variable values to execute different task blocks in this role : + - Set "dyna_config_dpdk_bind" to true, the role task block will bind dpdk with E810 NIC devices, generate /etc/network_env.conf, and add a systemd service to rebind after reboot. + - Set "dyna_config_dpdk_link" to true, set "dpdk_link_node1" and "dpdk_link_node2" to the two servers which have E810 NIC linked with 100Gbps cable. The role task block will update /etc/network_env.conf to exchange src/dst mac addresses. + - Set "dyna_config_dpdk_unbind" to true, the role task will recover system to status before calling bind. + Below is an example for how CEK playbook uses this role: + ``` + # bind and unbind operation no limit to node count + - hosts: "{{node | default('kube_node')}}" + roles: + - role: configure_dpdk + tags: + - dyna_config_dpdk + when: + - dyna_config_dpdk_bind | default(false) | bool or + dyna_config_dpdk_unbind | default(false) | bool + + # link operation only can work on two nodes + - hosts: "{{node | default('kube_node')}}" + roles: + - role: configure_dpdk + dpdk_link_node1: "{{ groups.kube_node[0] }}" + dpdk_link_node2: "{{ groups.kube_node[1] }}" + dpdk_link_pre: true + tags: + - dyna_config_dpdk + when: + - dyna_config_dpdk_link | default(false) | bool + + - hosts: localhost + roles: + - role: configure_dpdk + dpdk_link_node1: "{{ groups.kube_node[0] }}" + dpdk_link_node2: "{{ groups.kube_node[1] }}" + tags: + - dyna_config_dpdk + when: + - dyna_config_dpdk_link | default(false) | bool + + - hosts: "{{node | default('kube_node')}}" + roles: + - role: configure_dpdk + dpdk_link_node1: "{{ groups.kube_node[0] }}" + dpdk_link_node2: "{{ groups.kube_node[1] }}" + dpdk_link_post: true + tags: + - dyna_config_dpdk + when: + - dyna_config_dpdk_link | default(false) | bool + ``` + Below is another example for how CEK playbook uses this role in system cleanup + ``` + - name: cleanup dyna config dpdk + include_role: + name: configure_dpdk + tasks_from: cleanup + tags: + - dyna_config_dpdk + ``` + +* ## install_gpu_driver
+ User can use this role to install Intel GPU software stack, including : + * kernel mode driver : + * i915 + * firmwares + * User mode driver and runtimes + * Graphics: OpenGL, OpenGL ES, Vulkan + * Media: VAAPI, VPL, MSDK + * Compute: OpenCL, Level Zero + * Tools : + * vainfo + * glxinfo + * clinfo + * vulkaninfo + * intel_gpu_top + * xpu-smi + Below is a example to use this role. + + ``` + - role: install_gpu_driver + ``` + diff --git a/collections/share/changelogs/changelog.yaml b/collections/share/changelogs/changelog.yaml new file mode 100644 index 00000000..6b3b0f91 --- /dev/null +++ b/collections/share/changelogs/changelog.yaml @@ -0,0 +1,6 @@ +releases: + 23.10.0: + changes: + release_summary: This is the initial release of CEK share collection + major_changes: + - Added two roles configure_dpdk and install_gpu_driver diff --git a/collections/share/galaxy.yml b/collections/share/galaxy.yml new file mode 100644 index 00000000..85f25910 --- /dev/null +++ b/collections/share/galaxy.yml @@ -0,0 +1,60 @@ +### REQUIRED +# The namespace of the collection. This can be a company/brand/organization or product namespace under which all +# content lives. May only contain alphanumeric lowercase characters and underscores. Namespaces cannot start with +# underscores or numbers and cannot contain consecutive underscores +namespace: cek + +# The name of the collection. Has the same character restrictions as 'namespace' +name: share + +# The version of the collection. Must be compatible with semantic versioning +version: 23.10.0 + +# The path to the Markdown (.md) readme file. This path is relative to the root of the collection +readme: README.md + +# A list of the collection's content authors. Can be just the name or in the format 'Full Name (url) +# @nicks:irc/im.site#channel' +authors: + - zhifang.long@intel.com + +### OPTIONAL but strongly recommended +# A short summary description of the collection +description: CEK ansible role or module collection to be used by other consumer + +# Either a single license or a list of licenses for content inside of a collection. Ansible Galaxy currently only +# accepts L(SPDX,https://spdx.org/licenses/) licenses. This key is mutually exclusive with 'license_file' +license: + - Apache-2.0 + +# The path to the license file for the collection. This path is relative to the root of the collection. This key is +# mutually exclusive with 'license' +license_file: '' + +# A list of tags you want to associate with the collection for indexing/searching. A tag name has the same character +# requirements as 'namespace' and 'name' +tags: ["infrastructure"] + +# Collections that this collection requires to be installed for it to be usable. The key of the dict is the +# collection label 'namespace.name'. The value is a version range +# L(specifiers,https://python-semanticversion.readthedocs.io/en/latest/#requirement-specification). Multiple version +# range specifiers can be set and are separated by ',' +dependencies: {} + +# The URL of the originating SCM repository +repository: https://github.com/intel/container-experience-kits/tree/master/collections/share + +# The URL to any online docs +documentation: https://github.com/intel/container-experience-kits/tree/master/collections/share/README.md + +# The URL to the homepage of the collection/project +# homepage: http://example.com + +# The URL to the collection issue tracker +# issues: http://example.com/issue/tracker + +# A list of file glob-like patterns used to filter any files or directories that should not be included in the build +# artifact. A pattern is matched from the relative path of the file or directory of the collection directory. This +# uses 'fnmatch' to match the files or directories. Some directories and files like 'galaxy.yml', '*.pyc', '*.retry', +# and '.git' are always filtered +build_ignore: [] diff --git a/collections/share/roles/configure_dpdk/defaults/main.yml b/collections/share/roles/configure_dpdk/defaults/main.yml new file mode 100644 index 00000000..3cc7def7 --- /dev/null +++ b/collections/share/roles/configure_dpdk/defaults/main.yml @@ -0,0 +1,6 @@ +--- + +config_dpdk_bind_nic_type: "E810" +config_dpdk_bind_drv_type: "vfio-pci" +config_dpdk_bind_port_offset: 0 +config_dpdk_bind_port_count: 2 diff --git a/roles/configure_dpdk/files/cek_config_dpdk.service b/collections/share/roles/configure_dpdk/files/cek_config_dpdk.service similarity index 100% rename from roles/configure_dpdk/files/cek_config_dpdk.service rename to collections/share/roles/configure_dpdk/files/cek_config_dpdk.service diff --git a/roles/configure_dpdk/files/cek_config_dpdk.sh b/collections/share/roles/configure_dpdk/files/cek_config_dpdk.sh similarity index 64% rename from roles/configure_dpdk/files/cek_config_dpdk.sh rename to collections/share/roles/configure_dpdk/files/cek_config_dpdk.sh index 7cb7384a..3e93675d 100644 --- a/roles/configure_dpdk/files/cek_config_dpdk.sh +++ b/collections/share/roles/configure_dpdk/files/cek_config_dpdk.sh @@ -1,5 +1,5 @@ #!/bin/bash echo "cek config dpdk service starts" -python /usr/local/bin/cek_config_dpdk_rebind.py +python3 /usr/local/bin/cek_config_dpdk_rebind.py echo "cek config dpdk service exits" diff --git a/roles/configure_dpdk/files/cek_config_dpdk_bind.py b/collections/share/roles/configure_dpdk/files/cek_config_dpdk_bind.py similarity index 94% rename from roles/configure_dpdk/files/cek_config_dpdk_bind.py rename to collections/share/roles/configure_dpdk/files/cek_config_dpdk_bind.py index e3d579f4..988200f2 100644 --- a/roles/configure_dpdk/files/cek_config_dpdk_bind.py +++ b/collections/share/roles/configure_dpdk/files/cek_config_dpdk_bind.py @@ -35,13 +35,13 @@ def dpdk_bind_port(nic_type, new_drv, port_offset, port_count) : cmd = 'dpdk-devbind.py --status-dev net | grep ' + nic_type print('execute cmd : ' + cmd) result = os.popen(cmd) - info_list = result.read() if result is not None : + info_list = result.read() + if info_list is not None: + nic_lines = info_list.splitlines() + idx1 = max(0, offset) + idx2 = min(offset + count, len(nic_lines)) result.close() - - nic_lines = info_list.splitlines() - idx1 = max(0, offset) - idx2 = min(offset + count, len(nic_lines)) print('bind range ' + str(idx1) + " : " + str(idx2) ) else : print ("invalid input parameter, do nothing") @@ -181,13 +181,17 @@ def dpdk_bind_port(nic_type, new_drv, port_offset, port_count) : if dev_ready : # get mac address + mac = 'unknown' if util.validate_dev(dev) : cmd = 'ifconfig ' + dev + ' | grep ether' print('execute cmd : ' + cmd) result2 = os.popen(cmd) - info_list2 = result2.read().split() - mac = '0x' + info_list2[1].replace(':', ',0x') if result2 is not None : + info_list2 = result2.read() + if info_list2 is not None: + info_list2_items = info_list2.split() + if len(info_list2_items) >= 2: + mac = '0x' + info_list2_items[1].replace(':', ',0x') result2.close() # down dev before dpdk bind @@ -196,7 +200,6 @@ def dpdk_bind_port(nic_type, new_drv, port_offset, port_count) : ret = os.system(cmd) else : dev = '' - mac = 'unknown' else : dev = '' mac = 'unknown' diff --git a/roles/configure_dpdk/files/cek_config_dpdk_link.py b/collections/share/roles/configure_dpdk/files/cek_config_dpdk_link.py similarity index 100% rename from roles/configure_dpdk/files/cek_config_dpdk_link.py rename to collections/share/roles/configure_dpdk/files/cek_config_dpdk_link.py diff --git a/roles/configure_dpdk/files/cek_config_dpdk_rebind.py b/collections/share/roles/configure_dpdk/files/cek_config_dpdk_rebind.py similarity index 100% rename from roles/configure_dpdk/files/cek_config_dpdk_rebind.py rename to collections/share/roles/configure_dpdk/files/cek_config_dpdk_rebind.py diff --git a/roles/configure_dpdk/files/cek_config_dpdk_unbind.py b/collections/share/roles/configure_dpdk/files/cek_config_dpdk_unbind.py similarity index 100% rename from roles/configure_dpdk/files/cek_config_dpdk_unbind.py rename to collections/share/roles/configure_dpdk/files/cek_config_dpdk_unbind.py diff --git a/roles/configure_dpdk/files/cek_config_dpdk_util.py b/collections/share/roles/configure_dpdk/files/cek_config_dpdk_util.py similarity index 100% rename from roles/configure_dpdk/files/cek_config_dpdk_util.py rename to collections/share/roles/configure_dpdk/files/cek_config_dpdk_util.py diff --git a/roles/configure_dpdk/tasks/cleanup.yml b/collections/share/roles/configure_dpdk/tasks/cleanup.yml similarity index 91% rename from roles/configure_dpdk/tasks/cleanup.yml rename to collections/share/roles/configure_dpdk/tasks/cleanup.yml index 4edd7304..7b3c32b4 100644 --- a/roles/configure_dpdk/tasks/cleanup.yml +++ b/collections/share/roles/configure_dpdk/tasks/cleanup.yml @@ -1,5 +1,5 @@ ## -## Copyright (c) 2020-2023 Intel Corporation. +## Copyright (c) 2020 Intel Corporation. ## ## Licensed under the Apache License, Version 2.0 (the "License"); ## you may not use this file except in compliance with the License. @@ -36,7 +36,7 @@ - block: - name: execute dpdk unbind script - command: "python /usr/local/bin/cek_config_dpdk_unbind.py" + command: "python3 /usr/local/bin/cek_config_dpdk_unbind.py" register: dpdk_unbind_result changed_when: false @@ -52,7 +52,6 @@ with_items: - '/usr/local/bin/cek_config_dpdk_bind.py' - '/usr/local/bin/cek_config_dpdk_rebind.py' - - '/usr/local/bin/cek_config_dpdk_link.py' - '/usr/local/bin/cek_config_dpdk_unbind.py' - '/usr/local/bin/cek_config_dpdk_util.py' - '/usr/local/bin/cek_config_dpdk.sh' diff --git a/roles/configure_dpdk/tasks/main.yml b/collections/share/roles/configure_dpdk/tasks/main.yml similarity index 63% rename from roles/configure_dpdk/tasks/main.yml rename to collections/share/roles/configure_dpdk/tasks/main.yml index 430424d2..22c79b46 100644 --- a/roles/configure_dpdk/tasks/main.yml +++ b/collections/share/roles/configure_dpdk/tasks/main.yml @@ -1,5 +1,5 @@ ## -## Copyright (c) 2020-2023 Intel Corporation. +## Copyright (c) 2020 Intel Corporation. ## ## Licensed under the Apache License, Version 2.0 (the "License"); ## you may not use this file except in compliance with the License. @@ -21,21 +21,20 @@ copy: src: "{{ item }}" dest: "/usr/local/bin/" - mode: 0700 + mode: '0700' owner: root group: root force: true with_items: - 'cek_config_dpdk_bind.py' - 'cek_config_dpdk_rebind.py' - - 'cek_config_dpdk_link.py' - 'cek_config_dpdk_unbind.py' - 'cek_config_dpdk_util.py' - 'cek_config_dpdk.sh' become: true - name: execute dpdk bind script - command: "python /usr/local/bin/cek_config_dpdk_bind.py + command: "python3 /usr/local/bin/cek_config_dpdk_bind.py {{ config_dpdk_bind_nic_type }} {{ config_dpdk_bind_drv_type }} {{ config_dpdk_bind_port_offset }} @@ -68,18 +67,30 @@ - dyna_config_dpdk_bind | default(false) | bool -# dpdk dev link block, to match the real network connection +# dpdk dev link blocks - block: - - name: push network_env.conf from node2 to node1 + - name: pull network_env.conf from node to localhost synchronize: src: /etc/network_env.conf - dest: /etc/network_dst.conf - mode: push - delegate_to: "{{ dpdk_link_node2 }}" + dest: "/tmp/network_env_{{ inventory_hostname }}.conf" + mode: pull + when: + - dyna_config_dpdk_link | default(false) | bool + - dpdk_link_pre | default(false) | bool + +- block: + - name: copy dpdk link script to localhost + copy: + src: "{{ item }}" + dest: "/tmp/" + mode: '0644' + with_items: + - 'cek_config_dpdk_link.py' + - 'cek_config_dpdk_util.py' - - name: execute dpdk like script on node1 - command: "python /usr/local/bin/cek_config_dpdk_link.py - /etc/network_env.conf /etc/network_dst.conf" + - name: execute dpdk link script on localhost + command: "python3 /tmp/cek_config_dpdk_link.py + /tmp/network_env_{{ dpdk_link_node1 }}.conf /tmp/network_env_{{ dpdk_link_node2 }}.conf" register: dpdk_link_result changed_when: false @@ -87,21 +98,33 @@ debug: msg: "{{ dpdk_link_result.stdout_lines }}" - - name: pull network_dst.conf from node1 to node2 + - name: remove tmp files on localhost + file: + path: "/tmp/{{ item }}" + state: absent + with_items: + - 'cek_config_dpdk_link.py' + - 'cek_config_dpdk_util.py' + when: + - dyna_config_dpdk_link | default(false) | bool + - not (dpdk_link_pre | default(false) | bool) + - not (dpdk_link_post | default(false) | bool) + +- block: + - name: push network_env.conf from localhost to node synchronize: - src: /etc/network_dst.conf + src: /tmp/network_env_{{ inventory_hostname }}.conf dest: /etc/network_env.conf - mode: pull - delegate_to: "{{ dpdk_link_node2 }}" + mode: push - - name: remove network_dst.conf on node1 + - name: remove tmp files on localhost file: - path: /etc/network_dst.conf + path: "/tmp/network_env_{{ inventory_hostname }}.conf" state: absent - + delegate_to: localhost when: - dyna_config_dpdk_link | default(false) | bool - - inventory_hostname == dpdk_link_node1 + - dpdk_link_post | default(false) | bool # dpdk dev unbind block - block: diff --git a/collections/share/roles/install_gpu_driver/defaults/main.yml b/collections/share/roles/install_gpu_driver/defaults/main.yml new file mode 100644 index 00000000..f4928339 --- /dev/null +++ b/collections/share/roles/install_gpu_driver/defaults/main.yml @@ -0,0 +1,113 @@ +--- +gpu_repo_key_url: https://repositories.intel.com/graphics/intel-graphics.key +gpu_key_text_path: /tmp/intel-graphic-key.txt +gpu_usr_key_path: /usr/share/keyrings/intel-graphics.gpg + + +gpu_repo_list_filename: intel-gpu-jammy + +# repo spec for intel gpu release after 20230912 +gpu_repo_ubuntu_url: https://repositories.intel.com/gpu/ubuntu +gpu_repo_spec_u2204_server: "jammy/production/2328 unified" +gpu_repo_spec_u2204_client: "jammy client" + +kernel_dkms_packages: + - gawk + - dkms + - libc6-dev + +# intel server gpu packages +gpu_kmd_packages_u2204_20230912_server: + - intel-i915-dkms=1.23.6.29.230425.38+i53-1 + - intel-fw-gpu=2023.30.2-233~22.04 + +gpu_umd_rt_packages_u2204_20230912_server: + - intel-opencl-icd=23.22.26516.29-682~22.04 + - intel-level-zero-gpu=1.3.26516.29-682~22.04 + - level-zero=1.11.0-649~22.04 + - intel-media-va-driver-non-free=23.2.4-678~22.04 + - libmfx1=23.2.2-678~22.04 + - libmfxgen1=23.2.4-678~22.04 + - libvpl2=2023.3.0.0-678~22.04 + - libegl-mesa0=23.2.0.20230712.1-2073~22.04 + - libegl1-mesa=23.2.0.20230712.1-2073~22.04 + - libegl1-mesa-dev=23.2.0.20230712.1-2073~22.04 + - libgbm1=23.2.0.20230712.1-2073~22.04 + - libgl1-mesa-dev=23.2.0.20230712.1-2073~22.04 + - libgl1-mesa-dri=23.2.0.20230712.1-2073~22.04 + - libglapi-mesa=23.2.0.20230712.1-2073~22.04 + - libgles2-mesa-dev=23.2.0.20230712.1-2073~22.04 + - libglx-mesa0=23.2.0.20230712.1-2073~22.04 + - libigdgmm12=22.3.7-678~22.04 + - libxatracker2=23.2.0.20230712.1-2073~22.04 + - mesa-va-drivers=23.2.0.20230712.1-2073~22.04 + - mesa-vdpau-drivers=23.2.0.20230712.1-2073~22.04 + - mesa-vulkan-drivers=23.2.0.20230712.1-2073~22.04 + - va-driver-all=2.19.0.2-64~u22.04 + +gpu_dev_packages_u2204_20230912_server: + - libigc1=1.0.14062.15-682~22.04 + - libigc-dev=1.0.14062.15-682~22.04 + - intel-igc-cm=1.0.202-682~22.04 + - libigdfcl1=1.0.14062.15-682~22.04 + - libigdfcl-dev=1.0.14062.15-682~22.04 + - libigfxcmrt7=23.2.4-678~22.04 + - libigfxcmrt-dev=23.2.4-678~22.04 + - level-zero-dev=1.11.0-649~22.04 + - libvpl-dev=2023.3.0.0-678~22.04 + +gpu_tool_packages_u2204_20230912_server: + - xpu-smi=1.2.16-27~22.04 + + +# intel client gpu packages +gpu_kmd_packages_u2204_20230912_client: + - intel-i915-dkms=1.23.7.17.230608.24+i37-1 + - intel-fw-gpu=2023.35.5-247~22.04 + +gpu_umd_rt_packages_u2204_20230912_client: + - intel-opencl-icd=23.26.26690.36-704~22.04 + - intel-level-zero-gpu=1.3.26690.36-704~22.04 + - level-zero=1.12.0-693~22.04 + - intel-media-va-driver-non-free=23.3.1-704~22.04 + - libmfx1=23.2.2-704~22.04 + - libmfxgen1=23.3.1-704~22.04 + - libvpl2=2023.3.1.0-704~22.04 + - libegl-mesa0=23.2.0.20230712.1-2073~22.04 + - libegl1-mesa=23.2.0.20230712.1-2073~22.04 + - libegl1-mesa-dev=23.2.0.20230712.1-2073~22.04 + - libgbm1=23.2.0.20230712.1-2073~22.04 + - libgl1-mesa-dev=23.2.0.20230712.1-2073~22.04 + - libgl1-mesa-dri=23.2.0.20230712.1-2073~22.04 + - libglapi-mesa=23.2.0.20230712.1-2073~22.04 + - libgles2-mesa-dev=23.2.0.20230712.1-2073~22.04 + - libglx-mesa0=23.2.0.20230712.1-2073~22.04 + - libigdgmm12=22.3.8-687~22.04 + - libxatracker2=23.2.0.20230712.1-2073~22.04 + - mesa-va-drivers=23.2.0.20230712.1-2073~22.04 + - mesa-vdpau-drivers=23.2.0.20230712.1-2073~22.04 + - mesa-vulkan-drivers=23.2.0.20230712.1-2073~22.04 + - va-driver-all=2.19.0.2-66~u22.04 + +gpu_dev_packages_u2204_20230912_client: + - libigc1=1.0.14508.23-704~22.04 + - libigc-dev=1.0.14508.23-704~22.04 + - intel-igc-cm=1.0.206-704~22.04 + - libigdfcl1=1.0.14508.23-704~22.04 + - libigdfcl-dev=1.0.14508.23-704~22.04 + - libigfxcmrt7=23.3.1-704~22.04 + - libigfxcmrt-dev=23.3.1-704~22.04 + - level-zero-dev=1.12.0-693~22.04 + - libvpl-dev=2023.3.1.0-704~22.04 + +gpu_tool_packages_u2204_20230912_client: + - xpu-smi=1.2.16-27~22.04 + +# intel gpu release independent test packages +gpu_test_packages: + - hwinfo + - vainfo + - clinfo + - mesa-utils + - vulkan-tools + - intel-gpu-tools diff --git a/roles/bootstrap/install_gpu_driver/files/cek_detect_gpu_type.py b/collections/share/roles/install_gpu_driver/files/cek_detect_gpu_type.py similarity index 85% rename from roles/bootstrap/install_gpu_driver/files/cek_detect_gpu_type.py rename to collections/share/roles/install_gpu_driver/files/cek_detect_gpu_type.py index a3a409c4..6ca557f3 100644 --- a/roles/bootstrap/install_gpu_driver/files/cek_detect_gpu_type.py +++ b/collections/share/roles/install_gpu_driver/files/cek_detect_gpu_type.py @@ -41,14 +41,15 @@ def detect_gpu_type(): idx1 = line.find("Device") + len("Device ") idx2 = line.find(" ", idx1+1) chip_id = line[idx1 : idx2].lower() + + if chip_id in intel_dgpu_types : + gpu_type = intel_dgpu_types[chip_id] + else : + gpu_type = "iGPU" else: + gpu_type = "Unknown" chip_id = "Unknown" - if chip_id in intel_dgpu_types : - gpu_type = intel_dgpu_types[chip_id] - else : - gpu_type = "iGPU" - print(gpu_type) print(chip_id) diff --git a/roles/bootstrap/install_gpu_driver/files/cek_get_latest_gpu_pkgs.sh b/collections/share/roles/install_gpu_driver/files/cek_get_latest_gpu_pkgs.sh similarity index 94% rename from roles/bootstrap/install_gpu_driver/files/cek_get_latest_gpu_pkgs.sh rename to collections/share/roles/install_gpu_driver/files/cek_get_latest_gpu_pkgs.sh index 816e32bc..caf4aa94 100755 --- a/roles/bootstrap/install_gpu_driver/files/cek_get_latest_gpu_pkgs.sh +++ b/collections/share/roles/install_gpu_driver/files/cek_get_latest_gpu_pkgs.sh @@ -8,8 +8,8 @@ echo "Get latest gpu package versions" echo "gpu_kmd packages :" -apt-cache madison intel-platform-vsec-dkms | head -n 1 -apt-cache madison intel-platform-cse-dkms | head -n 1 +#apt-cache madison intel-platform-vsec-dkms | head -n 1 +#apt-cache madison intel-platform-cse-dkms | head -n 1 apt-cache madison intel-i915-dkms | head -n 1 apt-cache madison intel-fw-gpu | head -n 1 diff --git a/roles/bootstrap/install_gpu_driver/tasks/debian.yml b/collections/share/roles/install_gpu_driver/tasks/debian.yml similarity index 68% rename from roles/bootstrap/install_gpu_driver/tasks/debian.yml rename to collections/share/roles/install_gpu_driver/tasks/debian.yml index 9f569b5f..0d42f756 100644 --- a/roles/bootstrap/install_gpu_driver/tasks/debian.yml +++ b/collections/share/roles/install_gpu_driver/tasks/debian.yml @@ -1,18 +1,3 @@ -## -## Copyright (c) 2020-2023 Intel Corporation. -## -## Licensed under the Apache License, Version 2.0 (the "License"); -## you may not use this file except in compliance with the License. -## You may obtain a copy of the License at -## -## http://www.apache.org/licenses/LICENSE-2.0 -## -## Unless required by applicable law or agreed to in writing, software -## distributed under the License is distributed on an "AS IS" BASIS, -## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -## See the License for the specific language governing permissions and -## limitations under the License. -## --- # The installation steps based https://dgpu-docs.intel.com @@ -21,7 +6,7 @@ url: "{{ gpu_repo_key_url }}" dest: "{{ gpu_key_text_path }}" force: true - mode: 0644 + mode: '0644' # TODO: This file will block the gpg command if not removed. - name: Remove the key file @@ -35,19 +20,11 @@ - name: Add Intel graphic driver repo ansible.builtin.apt_repository: - filename: "{{ gpu_repo_list_path }}" + filename: "{{ gpu_repo_list_filename }}" repo: "deb [arch=amd64 signed-by={{ gpu_usr_key_path }}] {{ gpu_repo_ubuntu_url }} {{ gpu_repo_spec }}" state: present update_cache: true -- name: Run apt update before kernel installation - ansible.builtin.apt: - update_cache: true - register: update_cache_results - retries: "{{ number_of_retries | default(5) }}" - until: update_cache_results is success - delay: "{{ retry_delay | default(3) }}" - - name: Set fact for kernel version ansible.builtin.set_fact: kernel_ver: "{{ ansible_kernel }}" @@ -76,78 +53,79 @@ failed_when: kernel_second_entry.rc > 1 changed_when: false - - name: Set OEM kernel(2-level entries) as default boot kernel + - name: Set current kernel(2-level entries) as default boot kernel ansible.builtin.lineinfile: path: /etc/default/grub regexp: "^GRUB_DEFAULT" line: GRUB_DEFAULT="{{ kernel_first_entry.stdout }}>{{ kernel_second_entry.stdout }}" when: kernel_first_entry.stdout != "" + register: boot_entry_2_level - - name: Set OEM kernel(1-level entry) as default boot kernel + - name: Set current kernel(1-level entry) as default boot kernel ansible.builtin.lineinfile: path: /etc/default/grub regexp: "^GRUB_DEFAULT" line: GRUB_DEFAULT="{{ kernel_second_entry.stdout }}" when: kernel_first_entry.stdout == "" + register: boot_entry_1_level - name: Add support for multi-gpu system ansible.builtin.lineinfile: path: /etc/default/grub line: GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} pci=realloc=off" + register: boot_entry_cmdline - name: Update boot configure ansible.builtin.command: "update-grub" changed_when: false + when: + - boot_entry_1_level.changed or + boot_entry_2_level.changed or + boot_entry_cmdline.changed - name: Install kernel dkms packages ansible.builtin.apt: - name: "{{ item }}" - with_items: "{{ kernel_dkms_packages }}" - -- name: Detect kernel version - command: uname -r - register: kernel_version_result - changed_when: false + name: "{{ kernel_dkms_packages }}" # AXG 20230714 out of tree driver can't pass compilation on kernel 6.2, # use kernel in tree driver as workaround when kernel version > 6.2. - name: Install gpu kernel mode driver packages ansible.builtin.apt: - name: "{{ item.pkg }}={{ item.ver }}" + name: "{{ gpu_kmd_packages }}" allow_downgrade: true - with_items: "{{ gpu_kmd_packages }}" + register: gpu_kmd when: - - kernel_version_result.stdout is version('6.2', '<') + - ansible_kernel is version('6.2', '<') - name: Install gpu user mode driver and runtime packages ansible.builtin.apt: - name: "{{ item.pkg }}={{ item.ver }}" + name: "{{ gpu_umd_rt_packages }}" allow_downgrade: true - with_items: "{{ gpu_umd_rt_packages }}" - name: Install gpu dev packages ansible.builtin.apt: - name: "{{ item.pkg }}={{ item.ver }}" + name: "{{ gpu_dev_packages }}" allow_downgrade: true - with_items: "{{ gpu_dev_packages }}" - name: Install gpu tool packages ansible.builtin.apt: - name: "{{ item.pkg }}={{ item.ver }}" + name: "{{ gpu_tool_packages }}" allow_downgrade: true - with_items: "{{ gpu_tool_packages }}" - name: Install gpu test packages ansible.builtin.apt: - name: "{{ item }}" + name: "{{ gpu_test_packages }}" allow_downgrade: true - with_items: "{{ gpu_test_packages }}" - name: Reboot the system for these changes to take effect ansible.builtin.reboot: reboot_timeout: 1200 when: - inventory_hostname != 'localhost' + - boot_entry_1_level.changed or + boot_entry_2_level.changed or + boot_entry_cmdline.changed or + gpu_kmd.changed - name: Create render group if it doesn't exist group: diff --git a/collections/share/roles/install_gpu_driver/tasks/main.yml b/collections/share/roles/install_gpu_driver/tasks/main.yml new file mode 100644 index 00000000..26c4d4df --- /dev/null +++ b/collections/share/roles/install_gpu_driver/tasks/main.yml @@ -0,0 +1,59 @@ +--- +- name: Install gpu type detection script to /usr/local/bin + copy: + src: "{{ item }}" + dest: "/usr/local/bin/" + mode: '0700' + owner: root + group: root + force: true + with_items: + - 'cek_detect_gpu_type.py' + become: true + +- name: Detect gpu type + command: "python3 /usr/local/bin/cek_detect_gpu_type.py" + register: gpu_type_result + changed_when: false + +- name: Output gpu type detection result + debug: + msg: "{{ gpu_type_result.stdout_lines }}" + +- name: Set gpu type fact according to detection result + set_fact: + gpu_type: "{{ gpu_type_result.stdout_lines[0] }}" + +- name: Set repo and packages for server GPU installation on Ubuntu 22.04 + set_fact: + gpu_repo_spec: "{{ gpu_repo_spec_u2204_server }}" + gpu_kmd_packages: "{{ gpu_kmd_packages_u2204_20230912_server }}" + gpu_umd_rt_packages: "{{ gpu_umd_rt_packages_u2204_20230912_server }}" + gpu_dev_packages: "{{ gpu_dev_packages_u2204_20230912_server }}" + gpu_tool_packages: "{{ gpu_tool_packages_u2204_20230912_server }}" + when: + - (ansible_distribution == "Ubuntu" and ansible_distribution_version == "22.04") + - gpu_type == "Flex" + +- name: Set repo and packages for client GPU installation on Ubuntu 22.04 + set_fact: + gpu_repo_spec: "{{ gpu_repo_spec_u2204_client }}" + gpu_kmd_packages: "{{ gpu_kmd_packages_u2204_20230912_client }}" + gpu_umd_rt_packages: "{{ gpu_umd_rt_packages_u2204_20230912_client }}" + gpu_dev_packages: "{{ gpu_dev_packages_u2204_20230912_client }}" + gpu_tool_packages: "{{ gpu_tool_packages_u2204_20230912_client }}" + when: + - (ansible_distribution == "Ubuntu" and ansible_distribution_version == "22.04") + - gpu_type == "Arc" or gpu_type == "iGPU" + +- name: Install GPU drivers on Ubuntu + include_tasks: debian.yml + when: + - (ansible_distribution == "Ubuntu" and ansible_distribution_version == "22.04") + - gpu_type != "Unknown" + +- name: Install GPU drivers on RHEL 8.x + include_tasks: rhel.yml + when: + - (ansible_os_family == "RedHat" and ansible_distribution_version >= "8.6") + - gpu_type != "Unknown" diff --git a/collections/share/roles/install_gpu_driver/tasks/rhel.yml b/collections/share/roles/install_gpu_driver/tasks/rhel.yml new file mode 100644 index 00000000..20dbf79f --- /dev/null +++ b/collections/share/roles/install_gpu_driver/tasks/rhel.yml @@ -0,0 +1,3 @@ +--- +- name: GPU driver installation on RedHat + debug: msg="RedHat installation TBD" diff --git a/docs/calico_vpp.md b/docs/calico_vpp.md new file mode 100644 index 00000000..6593edfc --- /dev/null +++ b/docs/calico_vpp.md @@ -0,0 +1,132 @@ +# Calico VPP Dataplane Configuration Guide + +The Calico VPP dataplane is in beta and should not be used in production clusters. It has had lots of testing and is pretty stable. However, chances are that some bugs are still lurking around. In addition, it still does not support all the features of Calico. More detailed information: + +Calico VPP dataplane has only been tested on **Ubuntu 22.04 LTS**. Due to the Kubespray does not support Calico VPP dataplane yet, so we choose to install a very basic setup without any actual mesh capable CNI in kubespray stage, then install calico and calico vpp dataplane with operator based installations in `roles/calico_vpp_install`. Besides due to the Calico VPP network changes, compatilibity with other feautes in the Reference Architecture can not be fully guaranteed. + +This configuration guide assumes that the **"basic"** profile is being used as a starting point. It also assumes that the deployment is on single node currently (will support 1 controller and 1 worker later). + +## Prepare target servers +Calico VPP requires a dedicated interface from an Intel® Ethernet 800 Series Network Adapter. + +Start by configuring the inteface on machine based on the following rules: +* The interfaces must be from an Intel® Ethernet 800 Series Network Adapter +* The interfaces must have persistent IP addresses, e.g. 12.1.152.169/8 +* The servers must be reachable by the Ansible host from a separate interface and IP, e.g. 10.166.31.141/23 +``` ++-------------------------+ +| Master Node | +| 10.166.31.141/23 | +| | +| | +| +---------------------+ +| |E810 | +| |12.1.152.169/8 | +| +---------------------+ +| | ++-------------------------+ +``` + +## Ansible Configuration +The `inventory.ini` file must be updated with the persistent IP addresses assigned to each server. An example of this configuration can be seen below: +``` +[all] + ansible_host=10.166.31.141 ip=12.1.152.169 ansible_user=USER ansible_password=XXXX +localhost ansible_connection=local ansible_python_interpreter=/usr/bin/python3 +``` + +The following variables in `group_vars/all.yml` must be changed to support Calico VPP: +``` +kube_network_plugin: cni +calico_network_backend: vxlan +kube_network_plugin_multus: false +hugepages_enabled: true +number_of_hugepages_1G: 16 +calico_vpp: + enabled: true +``` + +## Post Deployment +Once the deployment has completed, check the status of the Calico VPP deployment. + +Check node status: +``` +# kubectl get nodes -A -o wide +NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME + Ready control-plane 12h v1.27.1 12.1.152.169 Ubuntu 22.04.2 LTS 5.15.0-72-generic docker://20.10.20 + +# kubectl describe node | grep projectcalico +projectcalico.org/IPv4Address: 12.1.152.169/8 +projectcalico.org/IPv4VXLANTunnelAddr: 10.244.66.64 +``` +Check pod status: +``` +# kubectl get pods -n calico-vpp-dataplane +NAME READY STATUS RESTARTS AGE +calico-vpp-node-48pnm 2/2 Running 0 12h +``` +Check E810 NIC interfaces for VPP: +``` +# ethtool -i xxx +driver: tun +version: 1.6 +... +``` +Check the configured network subnet for containers: +``` +# calicoctl get ippool -o wide +NAME CIDR NAT IPIPMODE VXLANMODE DISABLED DISABLEBGPEXPORT SELECTOR +default-ipv4-ippool 10.244.0.0/16 true Never CrossSubnet false false all() +``` +Check calico vpp status: +``` +# calivppctl vppctl + _______ _ _ _____ ___ + __/ __/ _ \ (_)__ | | / / _ \/ _ \ + _/ _// // / / / _ \ | |/ / ___/ ___/ + /_/ /____(_)_/\___/ |___/_/ /_/ + +# show hardware-interfaces +Name Idx Link Hardware +TwentyFiveGigabitEthernet43/0/3 1 up TwentyFiveGigabitEthernet43/0/3 +... + +# show int addr +TwentyFiveGigabitEthernet43/0/3 (up): + L3 12.1.152.169/8 +... +tap0 (up): + L3 12.1.152.169/32 ip4 table-id 1013904223 fib-idx 3 +``` +Create 2 test pods for simple check: +``` +# kubectl run test --image=busybox --command -- tail -f /dev/null +pod/test created +# kubectl exec test -- ip a show dev eth0 +2: eth0: mtu 1450 qdisc mq qlen 500 + link/[65534] + inet 10.244.66.100/32 scope global eth0 + valid_lft forever preferred_lft forever + +# kubectl run test1 --image=busybox --command -- tail -f /dev/null +pod/test1 created +# kubectl exec test1 -- ip a show dev eth0 +2: eth0: mtu 1450 qdisc mq qlen 500 + link/[65534] + inet 10.244.66.101/32 scope global eth0 + valid_lft forever preferred_lft forever +# kubectl exec test1 -- ping 10.244.66.100 +PING 10.244.66.100 (10.244.66.100): 56 data bytes +64 bytes from 10.244.66.100: seq=0 ttl=63 time=0.327 ms +64 bytes from 10.244.66.100: seq=1 ttl=63 time=0.823 ms +64 bytes from 10.244.66.100: seq=2 ttl=63 time=0.417 ms + +# calicoctl get workloadEndpoint -o wide +NAME WORKLOAD NODE NETWORKS INTERFACE PROFILES NATS +xxx-k8s-test-eth0 test 10.244.66.100/32 cali1037a54e65e kns.default,ksa.default.default +xxx-k8s-test1-eth0 test1 10.244.66.101/32 cali99c376db89a kns.default,ksa.default.default +``` + + + + diff --git a/docs/eci_guide.md b/docs/eci_guide.md new file mode 100644 index 00000000..7b83afb4 --- /dev/null +++ b/docs/eci_guide.md @@ -0,0 +1,80 @@ +# README for eci users + +## Deployment on VM + +At first, follow the README.md until step 8. This document will guide you how to update the configuration files in step 8. + +### group_vars/all.yml +``` +# Add your CPU model to this field. If you are not sure the model of your CPU, you can just leave this field empty and then follow the error message during deployment to config this field. +# Example +unconfirmed_cpu_models: ["8665U"] + +# Change field, use your proxy here. +http_proxy: "xxxx" +https_proxy: "xxxx" + +# Change field from true => false +kubernetes: false + +# Please contact eci-support@intel.com on how to access this repo. +# Also refer to ESH (https://www.intel.com/content/www/us/en/edge-computing/edge-software-hub.html) +intel_eci_repo: "xxxxx" +``` + +### host_vars/host-for-vms-1.yml + +``` +# Set hashed password for root user inside VMs. Current value is just placeholder. +# To create hashed password use e.g.: openssl passwd -6 -salt SaltSalt +vm_hashed_passwd: 'xxxxxxxx' + +# Fill your physical network mask to this field. +vxlan_physical_network: "xxxxxx" + +# Fill your CAT config to these two fields +# Example, refers to the vm configuration below +cat_define: "llc:0=0x0f;llc:1=0xf0" +cat_affinity: "llc:0=0;llc:1=1,3" + +# Update "vms" list with generic VM config used for eci. Use example bellow as a guide and follow the comments to change the resouces. +# Example assumes that there is only one numa on the host machine. +vms: +- type: "vm" + name: "vm-1" + cpus: '2,3' # CPUs allocated to VM + numa: 0 + cpu_total: 2 # The number of CPUs, consistent with "cpus" field + memory: 4096 # Memory allocated to VM + vxlan: 120 + pci: + - "03:00.0" # BDF of PF device to be passthrough to VM for ethercat. + # Mac address of this PF device needs to be configured in "ethercat_mac" variable inside host_vars file for this VM. + # Other PCI devices (BDFs) can be configured here as well if needed. + +### host_vars/vm-ctrl-1.yml + +Rename this file to "vm-1.yml" + +``` +# # If you want to configure ethercat in this vm, fill in "ethercat_mac" field with the mac address of physical NIC. +# Only one mac address can be provided. This NIC (PF) must be passed through to this VM and will be used as ethercat master NIC. +ethercat_mac: "xxx" + +``` + +### Run playbook + +You can specify one eci_package throught ansible-playbook -e. +For example: + +``` +ansible-playbook -i inventory.ini playbooks/vm.yml -e "eci_package=eci-process-automation" +``` + +Now, eci_package supports three packages: +- eci-process-automation +- eci-manufacturing-equipment +- eci-discrete-manufacturing + +intel_eci field in vm-1.yml can also specify the eci packages installed. But the package specified by "-e eci_package=" will overwrite the intel_eci field in host-for-vms.yml diff --git a/docs/emr.md b/docs/emr.md index 148be4c4..c2e2be1e 100644 --- a/docs/emr.md +++ b/docs/emr.md @@ -10,8 +10,138 @@ Download the EMR QAT driver package and put it in the folder ``/tmp/emr_qat/`` f ### DPDK driver To align with EMR BKC ingredient version, on the EMR platform we will use ```DPDK 22.11.1``` lts version. -## VMRA configuration -Not supported yet, to be done. +### TDX driver + +***Note: Only Ubuntu 22.04 is enabled for Intel TDX in RA*** + +Intel TDX(Trusted Domain Extensions) can deploy hardaware-isolated virtual machines called trusted domains(TDs). Detailed infor can be found [here](https://www.intel.com/content/www/us/en/developer/articles/technical/intel-trust-domain-extensions.html). + + +To install the TDX, you should enabled the ``configure_tdx`` in the ``host_vars/.yml`` for BMRA. Then follow the [readme](https://github.com/intel-innersource/containers.orchestrators.kubernetes.container-experience-kits#readme) to deploy the RA cluster. + +After the installation, then you need to configure the bios following the guide below to enabled the tdx and install the msr tools via command. + +***Note: TDX BIOS configuration can only be opened w/ TDX kernel, otherwise it will cause boot failure.*** + +``` +apt install msr-tools +``` +![Alt text](images/tdx-bios-configure.png). + +After the bios is correctly configured, then you can run /opt/cek/tdx-tools/utils/check-tdx-host.sh on the host. Then the following output should be get: + +![Alt text](images/tdx-host-check.png) + +## TDVM configuration +### Host environemt preparation + +For TDVM enabling, TDX kernel should be installed first on the host to make sure that the TDX can be successfully configured. You can use the RA existing role ``bootstrap/install_tdx_drivers`` to install the kernel first following the below guide. + +git clone https://github.com/intel-innersource/containers.orchestrators.kubernetes.container-experience-kits#readme cek + +``` +cd +make vm-profile PROFILE=$profile ARCH=$arch + + + +ansible-playbook -i inventory.ini playbooks/intel/tdx.yml + + + +``` + + +After the installation is done, follow the guide in the ``TDX driver`` session above to configure the bios correctly. After it is correctly configured, you can obeserve below info via dmesg: +``` +dmesg |grep -i tdx + + +[ 0.000000] Linux version 6.2.16-mvp30v3+7-generic(gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #mvp30v3+tdx SMP PREEMPT_DYNAMIC Wed Sep 6 11:29:15 CEST 2023 +[ 1.126416] tdx: BIOS enabled: private KeyID range [64, 128) +[ 14.295018] KVM-debug: PASS: single step TDX module emulated CPUID 0 +[ 14.295018] KVM-debug: PASS: single step TDX module emulated RDMSR 0x1a0 +[ 38.271537] tdx: SEAMCALL failed: leaf 254, error 0xc000050500000000. +[ 38.271929] tdx: TDDEBUGCONFIG isn't supported. +[ 38.296005] tdx: TDX module: atributes 0x0, vendor_id 0x8086, major_version 1, minor_version 5, build_date 20230420, build_num 507 +[ 38.296014] tdx: TDX module: features0: fbf +[ 39.228648] tdx: 262659 pages allocated for PAMT. +[ 39.228656] tdx: TDX module initialized. +[ 39.228667] kvm_intel: tdx: max servtds supported per user TD is 1 +[ 39.228676] kvm_intel: tdx: live migration supported +[ 39.228677] kvm_intel: TDX is supported. +``` + +### VMRA deployment with TDVM + +Follow the [readme](https://github.com/intel-innersource/containers.orchestrators.kubernetes.container-experience-kits#readme) to prepare the installation envrionmemt. Then run the command +``` +make vm-profile PROFILE=on_prem ARCH=emr + +``` + +Then change the ``configure_tdx`` to ``true`` int the ``host_vars/.yml`` and ``host_vars/.yml``, then run the command to deploy the vmra cluster + +For tdx 1.5/1.0 version, it doest not allow pci passthrough into vms due to security consideration, so DO NOT enable the pci passthrough in the host_vars/.yml and dataplane interfaces/qat_devices in host_vars/.yml. + +Below shows the snippet code for vm_host and vm configuration for tdvm. + +Snippet code for pci passthrough configuration in ``host_vars/.yml``: +``` +vms: + (...) +# pci: +# - "18:02.2" # 18:xx.x are example VFs for networking +# - "18:02.3" +# - "18:02.4" +# - "18:02.5" +# - "3d:01.1" # 3x:xx.x are example VFs for QAT +# - "3f:01.1" +``` + +Snippet code for dataplane_interfaces and qat_devices in ``host_vars/.yml``: + +``` +# dataplane interface configuration list +dataplane_interfaces: [] +#dataplane_interfaces: +# - bus_info: "06:00.0" # PCI bus info +# pf_driver: iavf # Driver inside VM +# sriov_numvfs: 0 +# default_vf_driver: "igb_uio" +# - bus_info: "07:00.0" +# pf_driver: iavf +# sriov_numvfs: 0 +# default_vf_driver: "iavf" +# - bus_info: "08:00.0" +# pf_driver: iavf +# sriov_numvfs: 0 +# default_vf_driver: "iavf" +# - bus_info: "09:00.0" +# pf_driver: iavf +# sriov_numvfs: 0 +# default_vf_driver: "igb_uio" + +(...) + +# QAT interface configuration list +qat_devices: [] +#qat_devices: +# - qat_id: "0000:0a:00.0" +# qat_sriov_numvfs: 0 # Has to be set to 0 here to not create any VFs inside VM. + +# - qat_id: "0000:0b:00.0" +# qat_sriov_numvfs: 0 # Has to be set to 0 here to not create any VFs inside VM. +``` + +``` +ansible-playbook -i inventory.ini playbook/vm.yml +``` + + + +## Generic VMRA configuration +No special configuration for EMR, please refer the [VMRA guide](https://networkbuilders.intel.com/solutionslibrary/network-and-edge-virtual-machine-reference-system-architecture-user-guide) for deployment. ## Cloud RA configuration Not supported yet, to be done. diff --git a/docs/flexran_guide.md b/docs/flexran_guide.md index 930718b5..ae208525 100644 --- a/docs/flexran_guide.md +++ b/docs/flexran_guide.md @@ -1,4 +1,201 @@ # Intel(R) FlexRAN(TM) Readme +This document is about setting up and deploying the Intel® FlexRAN™ software2 as either containers in a POD or Bare Metal to be used as part of a 5G end-to-end setup, or in a stand-alone manner in Timer Mode and xRAN Mode using the Container Bare Metal Reference Architecture (BMRA) on a single 3th/4th Gen Intel® Xeon® Scalable processor-based platform with Intel® vRAN Boost. A formal PDF Quick Start Guide for Intel(R) FlexRAN(TM) deployment using the RA playbooks is published at this URL: + +## Hardware BOM +for ICX +- 1x 3th Gen Intel® Xeon® Scalable processors +- 1x Intel® Mount Bryce Card (acc100) +- Intel® Ethernet Network Adapter E810-CQDA2 or Intel® Ethernet Controller XL710 + +for SPR +- 1x 4th Gen Intel® Xeon® Scalable processors with embedded Intel® vRAN Boost(acc200) +- Intel® Ethernet Network Adapter E810-CQDA2 or Intel® Ethernet Controller XL710 + +## Software BOM +- OS: + - Ubuntu22.04 5.15.0.1036-realtime kernel (for host and pod type) + - Rhel9.2 5.14.0-284.11.1.rt14.296.el9_2.x86_64 realtime kernel (for host) +- Kubernetes: 1.27.1 +- Container runtime: containerd 1.7.2 +- FlexRAN: 23.07 +- Intel® oneAPI Base Tookit: 2023.1.0 +- DPDK: 22.11.1 + +## Deploy in Bare Metal (host type) +- Download the following files from the Intel® Developer Zone portal. + - FlexRAN-23.07-L1.tar.gz_part0：https://cdrdv2.intel.com/v1/dl/getContent/784883 + - FlexRAN-23.07-L1.tar.gz_part1： https://cdrdv2.intel.com/v1/dl/getContent/784884 + - dpdk_patch-23.07.patch： https://cdrdv2.intel.com/v1/dl/getContent/784885 +- Copy the FlexRAN™ software packages and merge them into one final package on Bare Metal. + ``` + # mkdir -p /opt/cek/intel-flexran/ + # cat FlexRAN-23.07-L1.tar.gz_part0 FlexRAN-23.07-L1.tar.gz_part1 > FlexRAN-23.07.tar.gz + # cd /opt/cek/intel-flexran/ + # tar -xvf FlexRAN-23.07.tar.gz + # cat ReadMe.txt + # ./extract.sh + ``` +- Copy the DPDK patch to Ansible Host. + ``` + # mkdir -p /opt/patches/flexran/dpdk-22.11/ + # cp dpdk_patch-23.07.patch /opt/patches/flexran/dpdk-22.11/ + ``` +- Ansible Host + - Generate the configuration files for Access Profile. + ``` + # export PROFILE=access + # make k8s-profile PROFILE=${PROFILE} ARCH=spr ### ARCH can be icx or spr + ``` + - Update the `inventory.ini` file to match the target server’s hostname. + Note: For xRAN test mode on Bare Metal deployment, a separate oRU node is required. + - Update the `host_vars/` filenames with the target machine's hostnames. + ``` + # cp host_vars/node1.yml host_vars/.yml + # cp host_vars/node1.yml host_vars/.yml #Only in case of xRAN test mode in BM + ``` + - Update `host_vars/.yml` with PCI device information specific to the target servers. You need two PFs and a minimum of four VFs per PF. + ``` + dataplane_interfaces: + - bus_info: "18:00.0" + pf_driver: "ice" ### E810 PF driver + default_vf_driver: "vfio-pci" + sriov_numvfs: 4 + - bus_info: "18:00.1" + pf_driver: "ice" + default_vf_driver: "vfio-pci" + sriov_numvfs: 4 + ``` + - Set the FlexRAN™ test mode in `group_vars/all.yml` as per your testing need + ``` + intel_flexran_enabled: true # if true, deploy FlexRAN + intel_flexran_type: "host" + intel_flexran_mode: "timer" # supported values are "timer" and "xran" + ``` + - Deploy + ``` + # ansible-playbook -i inventory.ini playbooks/k8s/patch_kubespray.yml + # ansible-playbook -i inventory.ini playbooks/access.yml + ``` + - Validate FlexRAN™ software in Timer Mode + - Terminal 1: run the FlexRAN™ software L1 app + ``` + # cd /opt/cek/intel-flexran/ + # source set_env_var.sh -d + # cd bin/nr5g/gnb/l1 + # ./l1.sh -e + ``` + - Terminal 2: run the Test MAC app + ``` + # cd /opt/cek/intel-flexran/ + # source set_env_var.sh -d + # cd bin/nr5g/gnb/testmac + # ./l2.sh --testfile=spr-sp-eec/sprsp_eec_mu0_10mhz_4x4_hton.cfg + ``` + - Valite FlexRAN™ software in XRAN Mode + - Terminal 1 on BBU server: run the FlexRAN™ software L1 app + ``` + # cd /opt/cek/intel-flexran/ + # source set_env_var.sh -d + # cd bin/nr5g/gnb/l1/orancfg/sub3_mu0_10mhz_4x4/gnb + # ./l1.sh -oru + ``` + - Terminal 2 on BBU server: run the Test MAC app + ``` + # cd /opt/cek/intel-flexran/ + # source set_env_var.sh -d + # cd bin/nr5g/gnb/testmac + # ./l2.sh -- testfile=../l1/orancfg/sub3_mu0_10mhz_4x4/gnb/testmac_clxsp_mu0_10mhz_hton_oru.cfg + ``` + - Terminal 3 on ORU server: + ``` + # cd /opt/cek/intel-flexran/bin/nr5g/gnb/l1/orancfg/sub3_mu0_10mhz_4x4/oru + # ./run_o_ru.sh + ``` + +## Deploy in Container (pod type) +- Ansible Host + - Generate the configuration files for Access Profile. + ``` + # export PROFILE=access + # make k8s-profile PROFILE=${PROFILE} ARCH=spr ### ARCH can be icx or spr + ``` + - Update the `inventory.ini` file to match the target server’s hostname. + - Update the `host_vars/` filenames with the target machine's hostnames. + ``` + # cp host_vars/node1.yml host_vars/.yml + ``` + - Update `host_vars/.yml` with PCI device information specific to the target servers. You need two PFs and a minimum of four VFs per PF. + ``` + dataplane_interfaces: + - bus_info: "18:00.0" + pf_driver: "ice" ### E810 PF driver + default_vf_driver: "vfio-pci" + sriov_numvfs: 4 + - bus_info: "18:00.1" + pf_driver: "ice" + default_vf_driver: "vfio-pci" + sriov_numvfs: 4 + ``` + - Set the FlexRAN™ test mode in `group_vars/all.yml` as per your testing need. + ``` + intel_flexran_enabled: true # if true, deploy FlexRAN + intel_flexran_type: "pod" + intel_flexran_mode: "timer" # supported values are "timer" and "xran" + ``` + - Set container runtime in `group_vars/all.yml`. + ``` + #Note: Since FlexRAN 23.07, the default runtime for running FlexRAN is containerd to support non-root feature. + container_runtime: containerd + ``` + - We disable fec operator as we use sriov device plugin for enabling FEC device in this release. + ``` + intel_sriov_fec_operator_enabled: false + ``` + - Deploy + ``` + # ansible-playbook -i inventory.ini playbooks/k8s/patch_kubespray.yml + # ansible-playbook -i inventory.ini playbooks/access.yml + ``` + - Validate FlexRAN™ software in Timer Mode + - You can find the FlexRAN™ POD name using the below command: + ``` + # kubectl get pods -A | grep flexran + default flexran-dockerimage-release 2/2 Running 0 10m + ``` + - The status of the L1 app can be checked using the below command: + ``` + # kubectl logs -f flexran-dockerimage-release -c flexran-l1app + ``` + - The status of the L2 TestMAC app can be checked using the below command: + ``` + # kubectl logs -f flexran-dockerimage-release -c flexran-testmac + ``` + - Validate FlexRAN™ software in XRAN Mode + - You can find the FlexRAN™ POD name using the below command: + ``` + # kubectl get pods -A | grep flexran + default flexran-vdu 1/1 Running 0 91m + default flexran-vru 1/1 Running 0 91m + ``` + - Terminal 1: run the FlexRAN™ software L1 app + ``` + # kubectl exec -it flexran-vdu -- bash + # cd flexran/bin/nr5g/gnb/l1/spree_oran_tests/sub3_mu0_20mhz_4x4/gnb/ + # ./l1.sh -oru + ``` + - Terminal 2: run the Test MAC app + ``` + # kubectl exec -it flexran-vdu -- bash + # cd flexran/bin/nr5g/gnb/testmac + # ./l2.sh --testfile=../l1/spree_oran_tests/sub3_mu0_20mhz_4x4/gnb/testmac_spree_mu0_20mhz_hton_oru.cfg + ``` + - Terminal 3: run ORU + ``` + # kubectl exec -it flexran-vru -- bash + # cd flexran/bin/nr5g/gnb/l1/spree_oran_tests/sub3_mu0_20mhz_4x4/oru/ + Note: update the file run_o_ru.cfg for the port BDF corresponding with your server port BDF, example <--vf_addr_o_xu_a "0000:ca:11.0,0000:ca:11.1"> + # ./run_o_ru.sh + ``` diff --git a/docs/generate_profiles.md b/docs/generate_profiles.md index 4c5157a3..2d84f18c 100644 --- a/docs/generate_profiles.md +++ b/docs/generate_profiles.md @@ -65,6 +65,7 @@ These three directories represent available modes of the CEK project. At the moment, Container Experience Kits supports the following machine architectures: +* `emr` - Emerald Rapids - '5th Generation Intel(R) Xeon(R) Scalable Processor' * `spr` - Sapphire Rapids - '4th Generation Intel(R) Xeon(R) Scalable Processor' * `icx` - IceLake (default) - '3rd Generation Intel(R) Xeon(R) Scalable Processor' * `clx` - CascadeLake - '2nd Generation Intel(R) Xeon(R) Scalable Processor' diff --git a/docs/images/tdx-bios-configure.png b/docs/images/tdx-bios-configure.png new file mode 100644 index 0000000000000000000000000000000000000000..ce39ec8030ccc610720d4a02baf014edd2ba2262 GIT binary patch literal 53341 zcmdSAWmr_f`^T-KbhrYtw1j|?3rH_WcZo{pN=bv{5+a=o(k)V=bT`7nBCvF`uyigR zOT)wOUoW0F&+F&9=FGX~%sDgX#mwCI+~3bcXlW>t5YiAnc<_KkSxN5Qg9kW(2M->; z#>c&X@>zbK>HhZ6?VX~`gR()ot^2}bYiV`q2M;P?MAzm|?#l$uO8RaO9z1{fpX;IW zJJy2-4~CAE<)q(xne4RUef==yzIUg--zuE?2>Evyy5Gw3o?CU#McL?k3)RMr>9Zc? z)$_*g<%rueUvXxKG#S^kqF^1o6acP_Kc|JI;n87-CXcIzwyW<5E|(6)L8S2%$lfm^pPDNJf!+nJHWd=8(fO z;&6-E-WJ?7F7kc16G)z!Wvo8Gs9NdtI(%O4v8KTsexjhhBq9}+aFJ3MG5EUjcpwN$QtVqq@ys=sl$ZR8>v1^%d5Cfuovf0#lP@@sgms*s=OPy zU{O1h*0q;^PBgFQzG~-wZH{u6XY5olwt4*6^q0!r%T8J)(l2UGa(3*_WzT*&RR5>% zJat;C9ozf9H~ql+q~ms3nEHxRRgi;8gY6YvkfUJj^CM$dSi!m??hU2-H^5e z!2b;{C_Pp#WdLd`UlZ5Z0jA>>0XS89Lv*0dERo673qUn>#K&b9Bb6lI<+^Cqa~r zWrc)S!tG)6jBOEPVP~mBN%#b?Z(&6#?s=L;f2zqhVs{nYAmHmD*+o`?>-o&dWHv$J zaritOZf~8|5)^85xMoi`i+zaoJK^Ceq<@q22Zsp&a&~DvjR2%)&{SB%QPcHX#^;xK z%bmq=e8QN9HB1BLENXUG%g~{3KOz<@w8MYK^+JTgM6yk{yh!+fubycX^}fm^@wff> z(a}+b-4u^k>c+~-ireX}Tgge^^NZu;#a2;28!`{S1Ke+L<|Rf7=AMmezukJx-i?Y) z5zM5YvT(ZEjX3lRqet7FSnU4Hx1*n&_{1c_L&N*YIp)^ZVMYttF;4KAi6@OzgGx8n zer~R9P_kn5_?ue={k;?NFfFS-8i2sxqcvm16bb%mu(7q%uAoEjV3v3R1McI;O5jz2 zrPLeCf#eJ3ohR0De!FRRh@_IJaAhEu!voCd;F&4B7)Klc5PynpyPYh(itTLS@;&d% zL8>;}m^y<)U;?8iF< z`Si0ffwG|;7ophD(;vBYxIATLHnxtPf!vs-?d?!mlvHA2@a`ddty`LP>`IK(HL?Yo z$L)|d!!wn*-%6Dof9Bg@Tdm)INo0?iUcX4KE9bTS{JC>|AVXVEFZ}P{zjp3}_-(U= z7B%i7SY(Hzqhok1fuatC==!<^@N|>?n|bHBc43+EA4c7y$dS3Z&)eoioh;u9${57G z6lT2x0!)b+!K;wdDZn8XKR8Esn*G*s@A;VmrytJl?k)m2%N;sCH(~nsq}y(IaxOZw z>9cJKS3mnnn?97C<53^sv9{p{^GSVdrQ@fWSTNIMJ8!1gH!_o376T;gpF(ev(T)H! zVjg%^oh%ic3t?A(Iw& z142T=gPgmIobx@;_9!Y-U%<@F%zD9>EBL^np8Eu)1u~z0&aw+4U@$-w6F@}^FGNUi zx>jF7^sV)WIPX>+ByFYoE%G)NN(&ts=W7X42D1cDKLOTDz2~lR4Pt&x!>}KJv!ZUN zd_2nOk7qAzBnopx!v8gRb-gTj;iy&B;>FGT%?>qNSAV0*6@ZHMz7YZsjo3y98e>d* z8dvaE{S~?p{TKgYP|y0CVCC%}8i>v?`iz%8OZ}F2w>RgL^=%;m7byW9bq**kV+Y%) z9THSs3-Z{a#&l3ErRw>{#^Pui`6!Q|utDLM6l=4x?vGX%Tv6`Eu%Y!Jd)YG7`t}Bf ziWYolRivOUUtF{DVeO^D-+Gpn6Od8w;M@3DoSNn{cYBj5=&x%H`mLJ-1v51Sv_kfO z>)US}uFrdY1E-uW@T2Hl*ITA||K zC6DiCtVyXg7-J~!!c^CO#UF`0lUropsN3HdZGLLEOCt{lh zYEV>Tu@S>dK7Es$?p{|H+ zY~9DLn~i>#?am~8LK0z0xUtXF4E}KHP@}WL>$4x@AfSXrNqER{4>qqQU&}{RK`k?o zMO^9`OC*?5kPeSn1*Gt)0BFap9;2j0&9u;Zbv_vGskJ-OYQ{A*PejvzAak+4$nsQ$%U{JOWzGUIuK6IuH#kdyZ%`O4DP@mACsg@3 zdFed6^g!t6-|lY8!`|ZcOU)lO9E<=EPW&gDl^es!sGd)gRbe29k=L1G6*wzaJa1dQ zw?3s)3Ac(=M)U&3tbWe1n@Gv){{d+=8fRYYX^h+JxKj`Mo;*{D*29#3Myw>^kwCatOYD zexXmbcI_KT29%(9mB*%3Zf3ss@>qg%tyIb%A(Qyr1d-8x7of!FZqZg;zg8IgBup`_ ziJC_yjTiHy0$*@)!eV{2cD)J4^2$ANd|x2!jAm(z$kS63q;w%3#hCedTdBHuD(q7s zQd`F=s$z#jg-v;#Y-(qUxID^Uucu*Tgw0f2Lq@8S4jB2Vjk6zmo?JE3 z4vSx+0r$;#-OJS8dj>6#4saGF$(p==ZW+r3S&_ldX^#R%NB@|IM1jSR+MRyPwOGOesTnu2I%xK7Dj#VqUqxrRR40R3-C_HUB<0|!!|LbC83>_Rhpckb2ca2F2o2$_ zm*{Hw+1I?hPQ`Zirh{}doV#8|qfgcN+$DKeB#w}vt@*`DX5^LUKEktWWvGsw$vgOc zZcEgLS_TaS4PBo%@4`pP`yd{Dli{-%Rdcgv%$#l4k*Ke;36vW{hsh%rzZH*}pe4f@ zzt1*j!^%kC8;b-(#D&8>f0Wz2*{{ICBVFtaAsQWg$;PH4UT=T#tc&zeuAXS)rL(<( z;cSJ&*h4wFP%|?bVw2l`pX=Q_pH3rKcZLG#o@hk+wht=__1E1D()0y)&gVQiu{+WI zHa0ehJWZbFLARw}!I#}EuWeGNiCTNXubA~dVfiYwEo3|5UeJ6Qqbda<45|f{a0bZI zc(t0|^l0S3RIN4pqU?Wws?L9^s!`k0Bpp91l!L?~vYp<(xL=eB=jOnx@{^-;HgfQl z(UtH1M^7jHZ}z)wR$ZjLw$2WO5iF*iLQ~KCCnjbB$*<}%aQA@<8tEZ@-aUzD-=}BfccL5Xc4L#S? zb;jK>z3*w5=2D31EiK!SVJs61qW3U%56zk@`e7ZdCd3>B22pVKGu#z=iTU5HFq*BA zFDRsgh#|`fi%!uKJ11kKc01eaY->fHJ+1*X2Y+?j*-4If-?@qswO@y^yI{v%ytn3f z>VA83;sT+14#N&<{9iaeN?_lp864(IOc@3;FHi3cEQ4JA6f;S6?KL$ zXJJeesD2lDD^M`{+6vRKfHs!()zX{an*^Vr)iYc18C%Yw(R{eqUM!?iJ~zD2th%ga zm-wcEVS-20-w*2V4rzD1&ORWG>OOX=PqrUKj~liH&%lMfZwUdlC7hOp{$qMS&h@b& zPK%{dH*>&H<G@Qq>K_~#t_EUdZ*Hrl5%2D+8a;F=l+NLd|pVwVk+!chY5kZTfdTv zBTttuV3S~?ghXr1h@>8|m?RDsTNSHyuI*JdUgTYsu-zRrn*)eOxAizGY>5 z&tCuYh%!M>m|m1!<3=7;7mytBDG-UxzvG8~Kcu zhDsT*4~2#biJK#Ul|6$3lZ^#1zxf*d^2qvzK71Vy$qHkL&Cz*Ulp3uF4d(V*>Rr$I z-pW;m!*LqCd5MvZ4Gm?0E(Mu>7egEjKGI>wBcYAa!ShrxR?Xs5-)52riF3y8&$I1U z6kZx!q_}UDOI=YR`SwK_SyZr2K129%5l5t-TCMTk*>+ftY3R}}iOYxuOj~L4Ic(lr zvNT&NMnOv@L|PtH9!Uz?U%x4AW1|i16e<(m_X`7zgiT7p7Vg>^bE~RGo9dhzDhEL& zwjG^~=SQ9&Us#!imw0}1qyX@0%U!Qfl}F;6kju+CsDld!Y3%G%h*`$dL4Jde?dK~N z!o%4p0m8#Q@tK|kIuSE0VlIR4yf?WKKMI$Uv$EN8-%9!i0);#xXG&CkrZyRPIs1At zlVCK5d@{sz{wWzQA5A_94pJBC8^p1uhAcF7Bi$$bkC#HH_77!WPByS*9}w?3BRdal{^sAm3Y29&d*sX^I^)6L;$%)g%- zeaQlU8MH#pgVU+-=CLJVf@Tf#;mQK}=$)DK*g2`T%%f_aR4V&GiLumR>7&d|^XW@$ z)Vd`qc=(?U%e%s~0LUlWNh0hsH(E)mntK$kg0xSn|xGNc6_;Y3K?Q*73#>1SCCUv zXaETGVC|E3z0ig+HN-?@aH_Tg5vzezN8_=xgYG8lAqF~8S2>@}X-)|zv;@+yhQeP~ zArCciJHQpg9odz4??A(7P%lOuw{c~afs$R;j9@bwwe8!_l-WzXe+&kOfi zd?Jr`6%b}V*7w=3P*SrR#0j)Qj#V`SNtW{OifpG3z0m8zBjl6a{nQ%;SwB<0eG6S; zv6m&{weIMtaVS|iL5y(z^HCj9*6v#%I(EI$Y~b3_3f zgIPiK_4Ox&)q~tdEsq(LgND2DM*1uKjt)uUhI20sqCB(c%|#Wo4!jRU59hl)t=o!a z>x9A@^_GkIPj_8+y=|x7DQLzCMGYnGMzrwWFOO{-uhr%)#JkU!S`CrSn=>PeJ3GuK zdZF4!eo3DmKanlMhi7kvK)N2Gtk>%f!GqZESx|c^>ejz!tP5dOqp{&OW<7-R{CUqG z#dzu8jqb0tQa_~$j8{0X$V8^(d-aL<$;84v%7vdic6qb^trCJ*gf3i}jYgR0E1?!~p?*8xhSq=Qa`&b20dQ4pkS;Ej5Ju?X?3E;<^oK?kqip zO0BMLo(a2b8nl+)`)T`gU}Ter=U(B-Vm1Xpt_$KwXZA$iDG+!sSj zwlol-eEq_Q`Sro(gmXP(i7;Lt{~rDWG&=htAm%9xGf--4-SF4zBzEWkp9?k?5J$k36#K3bZ+&?ay9rP%Spr!=U zmL2Hom^}1u%0887=`I6Ojr4su#+Tm{%B#r$Gi;y^{IYj09ZcPU9DasvXl(LUDhFYl zWQ}8h!@h^3va)21=56Y>(rt+$-@c-W+MKE1{lyghD5~sO;DqCN!Sr#HFD_mEg|y?f zQ!h@-;oVyrc3T@^;NMagy_&3SS-cZToVpg7Ag`%+|od*RagAtY^ev%c6hRcnm4lT&Ybxcjt!jnL;d&A0eICc;VsrI(T~Z zK;}Md+G*sYZy#TVu%ldV`}wPkKhf7QYNYa85+)zTeC z>F@KTz$E;s$toLt*eTS3TRzlXL$YI^o8lK|i+7QoH{7~DUtMGEA1IyLdb+R#!dPQLMie3RySXGc%yLa!C#=;^0wnU1f!>NfBsj|bJrKQ7N&Zg# zYRY^&yp}{K80z|k&3bHe7+B;o(4gu=3k>_0{ay&@yGRWAUCfb^@^pn&mU;8{kR0|x zpzvW-TpWQx&kns62R>A+ffp(Zg!LYf#~O9+PY(}M_Gg<4`Py@Ee|!0!*cqGs=FMa; zC?PZ^5p5!*MimooBg{Lm?3Vm5DlI9=n^2JE0z6d-`n1nLt0-hf97e0JUTIdXhq2541pRj#ROO6l)OT565{BxUV=7NtJqrP;Ns5 zkmxyA9M$ST9pnNZ8qj`}RBpD8ObZVWkAr$Q=LvxHW2TU)hBQduVy(C7gfG!$;i+B` zBt_=PgK{<&EnC)+R9F6uJkCsBKAN{XA1tXkBv~NjpV}|CRsQJqc8^CCjM5^VIYO<5 zQ&)&$L5;rX6>ILqY_ZNu#F3VSjs(TJRRsH|d~e&4uMoe!p*D`zj%)hpRFQ!eiL5UC z{7rM?6{4hXK_*YILdTX<<64~VnO7A>X`%`(L=%*j&%RLtX+k+ucvNnmK$lhva`xnzr_p0;CG@;Oe>`3vmG5_eKwM&=c&(l5j|s7 zDG=1|g7&1zrRcgY9Sd^581)AHB6!8p9saYU%SSO+EnbzSuXjdy_QfPLRFA(VrWLu9 z2^6&rZ6q5!LT(kWftU6J3F>{@kU^Ey{uC@c>rU&U6_PGXZKdmH%exrPU8W~)ZMIPt z;^r5&UhlqEwN=Lu!6M+P#nl6Cfv!Bz4xz!Gn|1fR`sXIRZcM3Hr^B*j#-(oaR(?*f z!B0V!W=@hJZ%e$VhVzCt`hDQmU1uD!B55iA2H#AI61FiFa&z$Q;(*IW4uhVGzK|B% z+svw1QyzxXW^`#g8-1d!E3%hz<;h;MGu+My(5k2%6zsqFInbQ`C=?;pkMMXeW}xP0 z)vTED)r7lkh(m8is9JsHq3B!r)MCxKlnnx5_pF79H2$jFDnF}??rY&&hH0079+?gT zZfu(lRv80l-cIn5v`&dEHB`^S)Br}&iL0e(C*Jfva5>=-o5$2ye3cE%ia4P0^=%z+ z>XCgcKlhh|SS>s`B;e}OJ1Xwr4T`U07WZt$^ER+m{O8(yez}5m6T2I;y%GId$F-ul zIgx0*`?`3idtHW2#0F$PW=+w&`HR03W`b!7Ti{|QTIgVkO+M+WYBTha@X3zCtfqn| zj~kA_cS1e<0spMAdezd*b77W{AzDb#tGH>Xxb5CE(6Bw~-yu@;SK`e1EVg}bwsfP) zz8~-$>9_3e7GRfVKjl6TUOG)}k0_P!{FYNmr4aIH&-1CIlICMYLox3oR{zVfoaJ8j zoEm>OH%Bh^9D8#dM-(~Jlv5mZDkysh(*(6|h1Efod?bf3*aWBp_*NYY$?nIxmxoIa z#*yDkbkl?VmyS@ANnW|=gWA!c5ju37#PL6$gu+f_QlkYmxPV=%0 z=Xf)*SG@y?eqOBohxR>dpeAIamr3^VXRy%O+qxx-NyENReKy-K8n^redX~jdl};h} zPA^1<@k{MsXr4K*c7kqrqssI=xj7Uo>P?TGrw&_v0w^7+dA z7@eU$LasWRH+mk6$368T88;=biHaM&i-oRd^Y|`N(YZqEp?blYfUkjPVwgUPuVL4F&3k@cHHdiP*R9!Y|x*{pYT<;5N zOqSn&-IpM58LJDk1DA`Ax?deb{gaQ=9m=v%nJ^Tpt-iG0IAyoRXyMpoc2Zo-^G~Mh z%_MXzA$ut5&ZMR%X?296xJ71O1PDOEHb5 zyL4SvawfX$t4@)_8p_O{{D=e*)Oh5JAcC_5e#83K!v=}a-4<{^*OU#I>h9&7X>9kt zz2&CMO|qMjIJUJb=BzuuA?@pnx(&z`YP3wRzIDk*a!XWydwu}*o`&wkK#oM|=}>pf zi9{jE9`saCx4-LHR8>L(n!n|8Gwrk}nZMCm$L>oiEJ@VwR&zV)u%JJcCROHVosFr* zc9hiDk<>NW8geZdqR6Vt9kR@B+w_eqcAz(v4n6GPv-sT-4EILep7m|pWphKX|H=87 ztdQ5aNQ3wEE8Zfq%DFLyGj!yx|Et(QZxuhR+McbZ+#Nz}mPp(rU;FBuY-(gnRkJIm z;@v_7dnZ~+lP<@dQEK0=)mJ!xN&I!}azq35eOAF)nA@(;dzj@~TQnP3k;1-gH4P_s zX@{Skuv1U927g(g@>=+@VAlWv(I zbUIY?Er(|)P|3p!ZEwTX>O0HI;Thk|r9!X%mV*#;c5@wTm6`%mV0UljOh$ zmk%fbD&^*$fDs!xx!SerQ7M-%0GGDy!O+T&dnD6CJwh7t!K1q2;rRT_GI?PXcz^ww zJGn>-b1_lME)-XP5j^89Mo{8<-+o^cfHaNz{HV#|ZA=R1(@jU3D)iYbT1as7x8YZz z-9*e(aH>$O+p@?b-wfSUv8gimvW3nj(IC>?i2`d`{aN3c@C*FU ze`o`8=Lqi`g61*HY^l;|%hk9G{%ql)s#Wh+cv~YqpmJrFRyj*9E$M=PNc%uH1??Fx zw)jW)Kh#oUGXY8SURfE8J12}brhCuP4GAk1*r+Yi_O8a%>3Hw>F#1H4?P;(usK~$l z@}&;jZ`Y!`xL;nHM=%kiJ3MSaK){Q!{BMd$MXIp2k)T=1m%RPzjB%0b?h13hzb_S7 zba&e9g=LqLyN{b;K2D9HCqrlZT)!yW^z{_~DZ7`6v)^28MC6*78pdq+V|k}fE=`B$ zreB~0C#THkP5onZb8dOZr{4EAwXv-rT_mm+?xGjcF@Ea{ZDCT^JyOeyb!5oXT7&vc za5~NQu$C^%SyA0@I@DAv*BzFfgT`QXdw8ZGLL~N4w~?V?-gq-w;lC-Z6UZD`0%VT0 zkhsiZwAhKhz`u8Pe)Ia_k=+hRy62Ce4mu`&$_I=DJD?i z{2p4cvHhHQJ4nhR_PbA2%l9Q!3U5JPULL(}6)&?C=22f1kz8UN?GWT@ZY;{hot9R& zzOil)dog>7X|L`$>PC=t;+S4ZGF&y^1u3@Ow1a&5>>HXI=PTM{VC9%buS6|QZGvtSwzmVLC)}~eiF4HRn{iLs=m|%(e|UxuxM??;z4YT2rC%ta4+C0F0I+`TrQFriFoU^6@h$jyB-o#gGe7TnOl@> zSY|8QcRqbjVH#-qmT!XrXO;DE)5zn4Odwat+QDZNbYzys-A>*I?|iJr>2xy;`-(ly&&< zgxLc3XHyk%5qAHr&8O(ZgzWF>eXalv3epy;dt`=#8`-l*;?#TFd!QWy(RJ{9bqLJynZux~K@TpDN#b{#!2VV{+609!XAX~PC9PxXH zNJilL9vm^o`AH7ULCMaagf2!m77qW?bIbffCwR8B8c1)|%y$0dNq;S)@Y&tTQmc~i z6z%7(ESTxn#CrDs;zUl=(L2^Mhgv8-E_1=imBn2EzEpVyY$B&Q#^=9a!8OScCe?e~ zh#j$;YI_{#xh274T|f)s1K#`6e7@I$UOrbZhJf3|1Lo&r*<=ec5Wn3CjW4u=JUsV$ zcwphB4X=pG=G**O) z^A9ofOxBzJ7x@u9oWUV5r3L9iYy~xg{_Zf+F@)TQR)D#2R>U`8{HRFyy{y0d;T3Na z^S7U^t?GmOOKX;o;XH!lQ$Qv+;sJ#diVaH^@f+gh1RX$1D%YB&{`b&WnRnYZ#MGiD z!95q`crGu+;hA;`b0~8NJ8is+N^sg*I=`i{n`{snMAm_l%#k-9|F>z=)Ds#i`^RE;rdcnrg zQAhzX-_C=`s}4&#t=^CcidG<4LC(0@IhNTEIr)N=%IlV+F@2|5dX}Pz8wB-2jj0XIyTa!mo}+z*EOg*(nQ^45Pm=-u?^; zr&a7+Lot?X{w2-N;0r6_$^BTv^*2>5T;fZ9ttlAhoZb1~ZY`d~t+|1j5}zeHf3B?F zUOWl;yTG*s!a)k1atsd1n>hm|T3fWOi0ur~@R-Ms!^{r;RUX~EfV{01&N|S)*P=U6 zLuAsq@k;qx_=KSu&EoHn-eIvB_FZu2uHgm-0qP*dX!(&fu0s>8E@B2zQaHO>&>sMv z;La^RJDvW=XqgItd z;1}ddOdWvU;caREI9H+uOUK5GIllJ2!RcoC&hPUb|7X5=p1jtlEL8#bUk`EcNRs^W z=hh|QNcRKJe+!nW0IQR_VOg0#OIsWa@=Zg8lJXnWRqDzyx~ZXodBjNZ=of)&_T}tv zO{8Ka-qGMzfybWX*z9QN?>>X~AIAT^GycGdaina+qoqw>ONWl4(dKfHDDO5;N4FDU z!AxZrB}ewm;-)5p(BRX9gSc))X|RBC+n0wK3qB@uJv*g+t1X^P_2gS@2*(SFCWX|#Ji00 zl<`>n&6;aB?!>jpoFk!Q%8cO6XSAtd~~gh?H) zv~CCbCBiPZUwe7^K<3@rK?v_!AYYl|p2gpQ^HE0Ut!rga9M;hnFRb3_>B?)CT@IC_ zCiXcK~=nVeU=60g(EX8jPv1MvwKYwyn5nGKivq z`+Uq*G(n-TWCzKn}mp&yy-LU?%h3JBf88k(gX7Fl@+SOVliU=&STA4rF8S zJ!hg5mLQkekcwsUSNa`nRKO_3(YfTR(9~Gx2&((juR0*w6A?eK@OF7PzR=G_U}~Rr zNx5TPFRy4Yo{#mN>&vO}4Ey65ovx<=yd?o7g>8HbknzMBX1Q0AxYR6dB^;(LM3f3G z750)3A2R!QiCb(F?Nj$Vum3aA8O*gfr&_i8`FC8{n#y|Bw25O1dBE+%k}BM(^_sNZ zdvDV}KRV4OTbfiX=Bas^lKv9qVPOT!*Y=Qx{7=0s#lkXvo2kl2&3h58ooVUm=?Ax0 z`>I*TXJ_rRgUTX02`j+@?j9aX=S$V+2lH7CUXrvMgGMqR5the}CvWw<;v}wye%9na zgVr(1k3OyQ=9#R0yB=b6&%%;|T;dCF#grsw*oSqO)c)h3<`;#h3RsiA)6waUYjoM- zL&zE!(A@4W+>&^Edn14zieGI{W|}94_fFE7fBtI$QkRSS9nx&sTFCm(0(A&v6;p9X z<-b31Xa(#j+u1EWY%b7^_>xK`{m0g5WEf9gAr8inNIql2Hp9u`g!OO$Z^hOb6ruJ)Ur-!&5`({cyi=Om$u zd>UHL_x@97W_N0nSwYDk4jws2guWnk5GR+RS)NYJu5U>x$T?>y8h#nCgA8&sOTK*64;Ue%{ zNdg?uFU0X(p9jmTv&kpz(f5t=o;|W5?zA`@Mw%RkCkCC~3}eY&z7Kiy>s%!^c(r1$ za_sAAR_inv7Qp_vvF_)eg2Tf5@L+OZpGwO?Cy^I+#yUPLi~B6WJr*}WznW-P@?}rb z&%m?zvpC$>B!@3_*dveKeFvZK#F>LFn>)nR%5gEEm?Qi#$HSRwKN3WNkKxv2#lW@$Dx%yPy&qu>N#K zKePDZqA~SM+77OcQE=k^@Zp2k?Zw97{eI@5sHo_2{EyUw1Ht{-N&YYZpad_4$3)0| znCLh~4Soi5TRz5EV5tbSVIlNyq#)O7jPF05strpfn-+c zLSZyqsOs3J^o<0!Ekb_Yb@S;1>7y2d^a~{SX-1C=>!pg^{eCl2)*{1xZ?PlDYp1e@ z&uv`|Bh1Lo!h)kIbt6baL&Il3&Bvu*_4v=n(oiQOeibq9*4!Mq%ZQ*+H$ljZfpQ(B ze56n~8(#C$E-ojHR2O0vOuG`n+e=gw8@Xsk#+&f0IXXAscKRVG*7xQS$REq>eF18W zMG(d$ny(c0_J42vC{Y*SjE~UB;pJ7rHt{EhD_n2Ba&vykj&sjvs`4Ix&Hn@%{>`aU zK$XTT7mCYDN^6Tng(O_nEseC_SbJ+9vm>*4+q!1#I$68ueBL7$1YgYj@$9r8(&8qJ zd!d8qJx*rY{j0In%i!22GH=8SMQ_HYXv=%}2d@|6ZoSFR{mY<4Lms_vnfJx4|Ef?T zFTZGXeqN`pAh%CW1#x}U-RE}P^y>Zj_srTIrbb=ji~|y4{;3LM;>ZUOiRf%GL!UD0 z40U+unwp@DV+n}WcGdq`>Dq-z#jWTkYb(8}6L8D^y}At#A);DE%PA^CE>E`f>b4&S z!!k*z+T;G<4rFy&S1qkcrQZW8PsqTpaLGv8NXoB!foi=9pXz92eVe=rC6GeVRMzK+ zJ+cM9p@0V9hhX>2);5YNVHuq3nW9?)BF4+_T^;0+gg%eYX^ud+fR$0$JCfqVE0imP6M35Sv$n445ewUZ;2F-l@|j~bmdH@4eR`;w zXX_oVRk^QR@F=@9tUn{*y0N&2lh1XOxVn&bLv21!FRgt&%3?=*@#yv-d%*Z6HF#?0 zdXXfel2L@?$$!AI_NgJ&!^45;!w$n|+atO)z<959B(*L3g*YrCB3`T>n;W(yH9lur z{@~c7z47oxxo`E{rft$N^)Y2m(5|>+{~*Z)vb~YFT{z4t%WB@U&bPwe35CV|B#ybf z-&hMspEqtSbd7eH5#TIW5}Y+QmFjbCV1yj{<+m#IZo8c#nH=GDe&nZ%jOD~-Gi;yYlGW@VNV9OiyLx)F+C9QPct~uM0r7v>K!+a>Tel=@KS4$~V($C=W<8me;p^w{hzb82Pyxu=G zlAvQjMkw5rl`3i(8%Gf&D5hTR*=NUd%D}(H2Q=buXqw8%`Xw}as;7!rvWCV0g9VEC zorSn@9QVqA;!GccM>2T*f=5=ue*XGFu@ujg3J!S2li9B@8$qO3>ya(`5yU_0wt7)O zQwj22eHwP8@jnz|f#fOkf6R3-Z_j?f-PLr9cXe>Mm$$cm`-R93z?_!=nt(rFQd|tL zyi{H2{Plz^;5R@eFU4GXXU3w$L0&l?G(Fq(^6oaOJ%#F-BDHAE4G!q@G;OyQAyxwr ziW{BIuWB^36LNfT@IAOQ{NdrRH=%sSn~V+hn5y;b{A&zZ>^({*IKK&(kv}e=5-}>w~VdyAN`tORC?eG_}9X?ud^7 zPrjgZ?qQYVCp$)W>EVpn;l0~M_2^6qRiFH+d#(=W5)$ z-ZAlKe0_vRAeX&owt}X2^O_*r!DrMRFD1Pj03X~g{F7GYLjiomW+yeG7J9DX@Ox0T z7wo?=)ZQ7aAElj`!_BKjlc>hU)%cwqJhR_^E{-&6`q)W6ZHPRsvOY=R=ogEUy6quz zJ~?2*s7^}`U;Pa^c+zjOv{k)sc1zV3wCW-t>FIRF9)TbFY$OVyZG3($%aT`cmYt*P zdx+wXl_K#p{6`A;XU(M#9jmCB$Y z*r!x8tf8q+kz#Ex9||dU!$V(!voB@>ew39BHdQ&9HCx*qbFh933IahU?YVAd$pBLJ zNOh87XkfO&S!t)Ao+r>sPV;d9k<*&D>?wA;g+w!fmR=}UKGQ^#pXJ2DoySl^re4w&`N=vvu-L%^yb3n(D z#G|0=O0PAwRVzwhpbL<5YEa(W`%Nr^NQ`284IQ3G&Raj@yh@hO19kqXLUwc}ez!PW z29h{{_}nT@mCyJ1GFaYzn7qDTYZ>#W8@0r7f7Wf!4v08G3Jfj& zi)Wd9lSAyr41}R4C)p0MJ4|VB?4B3$`aLx-9xG$R&J&0F@5~(30h zhYnow-mCQR%g3Kxs_;iRk z^rNOpTeV{e+c1wPmwya8KJ371^^wk$vA_Y0ybYgKDmt>j^gL8B^t#b1Q!?@%VhyP^ zy?ZFbY_a-MLD9QpA}H6waF2dH*M8?K+7ljL;9BZ*vqhC!=UiYfzJQp;#AS+W_{+S%#ZxLB@ zU)dsl8C})xtLlwUdYU9Vm%fL@`9I@Pr{oZ=#-l8ltlZgsVEUU@m}ce6l}B%{vW5mC z{f%9PS+DDh*OUVd$wh8OzUkq5hh6nu$KBQ)Vk+gs_s6E><@lVw9xhQ>B*(sIzYksi z(1|X_z`G^2_ngDsw+d zm|r^_q>WbB>VF$j5p4ZlhCQr$Zj_hBa(zk_a6T|AgS_qjUM1+}36%`4e_& zK&5pL$XjYCQ&{?OU5rl8SbU0c_ypO2;s;9-|EuE^KF<*fBasxGy9bOEt0{d~4oi{L z({{0k1?~Dli*XO^-~7xU43AE2z|WL&BNzbzL~8aaLSxjPWx!EAoG^1dHgdbj%(b{X#u8M_nspji77gIdA_c1?vSYmTaI$s4>bv?C3VKyq2@ zYayFNw>cM@Q`pU}lqzsIxjs8(m?89$v6!EXE1QUntjDFTvev528@eaKkOan=i+JF$ z1SwQ350_QxCx^Vew{lnJig| z`=_th-4Hgm}Jg*0#&2r0O(gHTKUG||5xOhP;8GVJAwkRp@8i%F7u=vW7&*i_$xvakQj~hAkqeAbZtG*i|)6r=ZJ+EMs^xPz# zk3Fqa5+a(nqEJ@KT??S>ML$NSpi-$ z^OS@R-P;Vu{%mM3fPd`=RUkTeF3S2~Soki5RnhG|eRf!nD+oc|{d)k_;nO$#X1!U> zu*Kkew>Q_V3ECU0%gk&=&EyCGP{o)9(7Q$dGUMfi-|`ev=XFmp67<659|I!8>u=AA z)c9r0R^_6?MW8!hGs7$rcOQjNf}`$<{?K4kkauQp8w>j&g%f@EIWtT6&*kNTTTjR9 z7}6)-z@?0r;gOMq5~t%jqgZa^ruVZg(m}Y&DA%ev{$E|KxFI-r^&(cY)IM_h0n>-B z6K{oy%3&rt{Tx17-^mm}e;t62dg*W?6+Dm?=vmiZB{Q3bRf7>|U&~`3X{nsAHDvaf zcWN0_x4ooTlfeBI#+yw=kuRH1lerDq++$@r=i!vPuQhVpxlCpYUd`GTC+D3Y@1)SD>o9dc(J1^DJKr|QL-Ac;pG8&VVg5$y#eMpekM(sc< z2rBU{I^2x^^t=~d(8hNX(PKH-^<(v+!dtHhGjm{K&Rr>){V3`rDSWu=b2rlxEUA@V zZSJxpM9R$ZPB`FL_`FZIjhRLC;`|c*&f<-!rr?h>Ut3yTBRm$~{)CS-GvbHZ!6yz<`3KOqB2Bo8%)ejr%IAU*qH~P!uQtdvYwI6P zEsygG2RpPNW@D)qWUmq)Bq(Yo0zGMb)J`ze%>%>0HV#|CQLUdHQlp7s=B&ga%i%(F z4h7Y%#y-%r1nIm4UQ;#-kBmDG=1qC?@5SmR2SyS+tsK!Gte+JWdE-W3SpEW#a}&8$ zvo67wL(J9G=<*MiPPML`E!YVkp;>&iHVpYP4wZKr9bUPbO*h~wyzC^HD?`K!tK%j!`E9! zMb*D;-v%u;gbXc1BQ<~wNJ?csu+u;j@ z@q`h?FI8W#u*hyvYLbWnG9Yu`<6szA~D}mOhY(PWkd>Y0<_Ff>I5;$w9ig z&lIOmc8?rWO=l(s)3Ogev(FK&-4tr{cbvJYwyzx!b{XKZ<6$I=3wO*-+U@G=b~n;+ z_x4#ngH7Knlms*S%@o`a6`lWL^vn8=?W=XQG4ZXn`HYqS5o+sT!^ycwEG~h&dS%_I zAKaK(*o@6bhQVTau!#sKzwt=tP}VNF!kk#p%X5`4;5Tb-*{he{Sz*{q$rH z;Ts#x^_ssidfvkZf*@Y2({-|h$B*3&Yty$Y%dK3?oCES#ZI6q0-}x*H*`r)_6WZI} zx)ZH3l)&him^NxAk<&|rWS_~&(Y!0zcx@{BJufC}QmkF`w|mjIsV5vqtP4H=LfwlR zr{;IEO5feIQ(&&WQYY{)P8+DO9=Po~ou6)~y*!(+wOkF_-zUEMR^Sjx-hYg_Uj7B? z;^V=-6S9t?RpCbA-m)*!(Kk+NHGL*nVO`B%>9Ui4?a0F%#5l|JqO-4mqzoJ8eXo(R z|7!2_XdXj+04_;2&dE2;9WVJ8e;OgJu=Hwe!`k^HF&77hnWrftmMtLC4!&%?-@ux) zYk!B~{?i=nFe>QI!RHB;HhXWA=JB&%N6p8$)O+aZ^y!RMZAu}n+CA_6--gBvz=^=r z?acWNv8e2*CKL*&J9DFxANF)EhI2qJ9CbJ99RqQX@@E7}Uh5t&dh?#B(qU7Ec#+S? zMB?>uc1&#j2~JE5FF=z1A$94Bj81n>Uraf#ymTe0M*kao9o68*0F1Yim*C#*1-Y}x zdI5ljQTdorf4P&7?Oi4B`QvtabWHE@(GJoe} z?Zp%M(A-2{H@v6l_C!=97Z;ark?8Bc7C)2hH=cNqzEA4e6t6YL#|q*_ zCRY-K*Ic)@PuVkQ*3R2Io(2Tx7q23Rdp0)YWg7##0yAE{j&rbGX3F874X=IuI&P_C z(#YwDqF=M<`PPPfYF=FH9ap0-?O%HycCt(1hC4(AbwHE;@zj(?o0y$?K)5<`P!6+? zd)tJk@-ZgY5ykow%n(GvP%mF)6l+W=7Y`>Rc=d+oY3%#Gw~7Hm(q=WJ1P&WJkwtsB zfRE#=PkD10iGp2FZ9a*gr{Tgr;S^(w0&sb+=nFXwdJl{G*goXJhCMKSy+`W^< z$6t6+Jx^i9RF7r;NB+ob-=LeLYpjDc27Fz6Q(KWNaP9ymoUV}=S>q42t52VP8=>a@ zRIYEY_Nox@%v3rheVz5rfUEL*Y-ilxnAM%nd8pUA6w24RiZqe@s`X#=PUf4Gr}_<}q-(@j|WO5JE=H#s^o<6)kW- zWS`LfQ55~hq|+e!AIA#eWJIRiy)IF^@}M1qX^h#mWtg+p#A_xWf^CxR6yEhTv|}oyYWKgw@3*1(EH@wi zxZs~(?_cXt%N;QOU3^o!T~FrrMebBLdahB!3TV$#Qx-ccD~Z1E_c$)!)(`^rBq!I= zoLYVo9U_oU_;;1EVfM}$mOZ;OL_nFEX9fnQqtrj2#jBU8m{%8bh4AAj5oAx}9iHHQ zm{0q5<2Ca`vQX)xs^RRGJ724#kr!J{t&Q+CXQM!gNxk#E4bQ95<<`Xe&rh{PL@njY ztVV`M5jxa$zJ0dBW(~!0&R%`1a+~n|>fFPQeVcc*uSKmc<=DA z{FjD=Xhx&ok`Hh&B)r}c2Zd7vFB1M}ot{vvFyL?-pt@dq$v+81tn+?yb!2@>pC05B z_HfGJSN*PG22C2s;0p+|kXCZy*J5a5tx5F<J?2N*N**S*uyC}jky$faGp z>HNY6Uc8_qDq^=ey6qb<&^a$XjSOdYH1mh*?vGgvU(@ImN+~AIYf+EVCi=kC?^TIb zslMmuKbx+!MaIxdbDDG$v1(Th{&U+tEG$;PsI|d8S7M~`2?>pEr8Zxha}kkG`*MFj zZ=0GT^?v*Y@OVfX>97F-Ku+sL76~LH>GC)Zz1mT13Ko+_zmFkByaG&bpj z@P97nCOb|noGi|E$_TKeKl3@(=?@&~eC>*rY^In=CqhDLy(^Wv1KU<;L5e&|*A;fq zCR>?WA6ap~dEyt7;rw`N)1oRi&3EaB;?M~tP^y!kt3Op#v;;aiJENqT#E}k8;O2v1 zJ${q*{>N6ggJZ9b>3x{Ptj!*kgD_U+AMx7gIB!jCcQsd6?bnVJsMQ)w-92J%_E~T< z!04cyTVt{_-mQ zXBE7(D>r?mWij?wRWreLa+_24Q5Vu*3+2x&i3}Hw~dz6~R zl`V9}b0w`E-QukDfBcWtidkxg`yG`-N?K1jsP%xXJYhAdS^MYHlQZJw${vhXc_W#T zg$CnrZJYU-QGI1Lq0FK5_}64WxPh<6GR}%ngpOFD|3Ph2u+@@NB6;GDIpwh9_-GO~%Q`lw2l@)tX;ZN_<9YO=}%M=yMH^ zrBYF3FG!+VJ*oyg2X+BeSnuteO&jv9Hc!h|KsDZ;RHjL~aL}?xMJT$Y@_%_k9q^i| z9Nek*(TMOOaGb#~$Hza|Gy@S;z8#-AB#gBMAx7vBw-tCgKEVAj9Cgq-8-|U|3nFQl zSf4ya`~G5o*zI^<>5w;?ffm9fO1AYWVrbq5CAd~&8i#~L?OS*{g85iY|KOnX_x*}n z57hmywsFdF$$5sZACSi}>=-V0DDaz+V~wTo0k9DdK@I5WZM&$MqIY*v$icx$to0_8 zH2(GTW9kdYbCDb3UfsKBxLj(l&(PDjl`5m&YNj7pebt?9{BoSmL!SV3{+Y1%os|5g z{^*K^3?+;hfNo?r)cnPWbQ=ArX1K7BwMM4R`GT2Lg`_CfWAalQwxpO?HwJBXkb&J6 z7^JSFla^QbD|#I^H?vPFM~%NPt=qxhb^PPeaycUoIsL%rC&(Cc*ucHbn80@*z*zP0 zWZI+I+S;;PY(b4rNRZ{Y*&XI?YMvcYald%x!2TBZDdBfgZ*fJGN>(69d_Ye22au-2 zL4-fTg0sVfZebzunau-r{@F1_P_xw>Fw9H1#RyHd z3ST@5@M^XZ5>QbxBJCW{DL<6--Tjv_(UL2J^T8w6 zg3wUVfZ!RyB_qNqlqLYz$;6no8R5Lt2)wG(zR_ zjnig^wK8*B5%o1v+uT)BmX_pOTPYu8PDLFdMBk}1`GJS@W-{R+kg%Zv22tIXnwi_x z?k_Jd+Bv*^`%++GL!@9Zh9WJR`HOj0#%x9kdc-bjJu1(*g~SM4#MMgV!C0B+`_^y) z9o(9rixqSzfh!YzkOoYz3!BUSNf*tCP&v^s4Vc{l=i6}`TU&8!lWNZ&kNa*=$Xaj8a6Ju<)DitVWp!}@hw#209SuuinkVn9eIM#7JQ*DGGsKSD zaIB+LC3&*pnhuS=y{RWIod?<8#a5!|0C57oo;T4q1c3`%+DE9YqT=6pUE3%wCgyia zzG|bQtsOq0>w&Ymxru>?hn%T)JY4X{GaTRH@BD-#Dvo3%$Aw6QE1Q$1b%tt8OpP(m z^9%6n-9~ogSXp8_lqGtuo%6`2_O%QUvGdZVOI3sSVPN)&amP`}wp*{6`!hH;WL-D`C#8)vD#4yX{w`!rv9e@-*_vKacl#v4=TOVAWV))R_;LcRr`L$H4zc*5O;yUq zdAWM!b>FM0qGJHi!Bm$T7-*S6G(=0wne6qWSC4(#Q6vn#gG-hWuX}spRJ*=XZNOk^ z&Ou0M@9DntfSGY?e6{d4c=*g3v@fdH?6^BuPj1$q8pFlF$Y?b6+XQ~#`w^Uu0q29M zSN`GH-KCtYI?b!z*HN#rgJ|op4m3~Q^yE&Anfyf7P1x8ZtoK+m3tzx>BFsB`+1v8> zciKd}&JCu5#L^56*XLCD;5?M`cUxE}XcyV7g5oXva0=;`Da#kBUa=Zeg_iE$%-lBXbmz%gLKi*h5?oA>`UBV=AOoL3geHS{@a-K_oBnEJsNA3!mEq~X&j3T+D5 zI?DEx*Q&Z&>p0iRt}i4GcC{kPYVib-eUlIW!Tg2mQ_IK0K`p1GV`|U+V)$C)JfB7xPL5(+z+&Z`A@#ul`#^5sI6WH3?Gs8roL`LV9-9A(P@8XzvdYUdSa3H z>!tQEhZBD{8xRX9h4F%vKYR!{Tt>*Vq$<$7+!XoPt#XpV@3w?<%EInEx1(Zz@EG_!a?{^a^> zH_jcI?uNVAJAeHbUDdE#Mg(4anx?igl8Jy{-_{ch6Oj{|qErz2`QIEI{L+vas0+u@ ziUd;h+7a2YEOkd~E;kF)c%3!Be!Y&5R}1^BUhl->eLjIF$%m~F-YjQpMMwi8Scx*q z<%<4Fz_F;Ht)@25bV+!B*UAVIdC{VYbq)OX%JEu&=ikuQ3gRcUG9Y}-DiuX#eJfQx zy|S{h@H01E%+!!Q5$;Luc{$ukB&%nNLz>`7s}6+!k@qNOhB=a~iBX!lofB*rPdJ5R8=41INj zySLm6G%w?|KsQhVLVlSX^nb_P4u4;8Q}1vI2SxSw-0Hiw&7LYmVjFy#+AwWuT>@gq{O?8%U&4@kMW?x{Q7 z#ccq8n55DPOgrLT5&=fQd1?LM(NQ{oS2)<7*JVc?1r=38?0N0qAq^k0S?dW?-b&ze zwr|b%`H$liSeQpw>=yikhld}R-ej!my?UhdL~jKGk*nK8KnqEN3%%f__`c>58dc3L zwK%|G+wv#m=Wkn88=3@;ui~Ji~-+7;BAO?q$ zfycEdtRb@h+azIHY0dt@(NPP)+=9Xn^hs%u38P(9%>!g>riU0pLh#x1rO;qb;$TOqP&cmg&Oc{d? zR_TzftsVRUNnqcMUEAF_8@{k8eBwF!F~zx%jq6vuwV$B-J7a<5^(;AInC5zC^lop1 z7bZH%Z)Erw5q3yOh>WbPUq;4!xr8A@K;@SfWht|zWjk0)J!97x$lwhk6#I>Y zIlh?qlW#Z89T(IBpEz$k>m^vjW!|-Qj-p_Phc$`PvOsA4R+AOUGW?*lXw4H=NTY8I zNZQvdZ5RnIZ%fvMvzi4>=f?vTbAU|k`~OF#7;+ta0Np#yRX zF;lq~1&--hk#=p>L~w1~V)w-r0}wN|*U}+-x-iRm!1NlBO)=K?!|v)PEw081Y$+8= zk)$(D6%GO>J-fqE5I=6$#F&C5rPp)KVF%?)Ev|rg24zvZw;%6<~MuujQ}0p>vr7Nm@A)LVeo0KTV^_UbZway!%=z~FJQmJlV7iI<;mfcj~PxndZUT$OT_Z`$4WPb z5-WNVjyx}Z9`!~=yq^CK5H;SSldv>Y^V z!(?Q2ge@&~4CU@AhL>*jMzgs7W#OR1 zlcVN%)R!hU=;S-M)KG)3h-YU(NeIA_ZEX7YQPgi0b3gHDQc*XKE{LPc`gpv2IyZc zje&(sA~j6I;QFLL*frT!!{x~Q^c^W8&vezJ&>MG+_Wc@$bh&0Cmvto=Qn(?I-kzEk<&96$uF`2B zB8u-nqk_>@Sox}3y4nYp0`+B7s2_&wtH=MPsvelk%^2+Pd*z>73PyOzL~Aj%Z!rz-jXhn)YOF*uI-XKT2aq|Z&60`*;iD)1IEsJ=Xt zBFx{#i|BbaCK^`{eE+&rk}$rQqsFL^pDbi@kp#5olc#_{dAQ&237yLC;a*|8x|7?& zC(ntno%>AkELfgHhsI_t(i`r~qVD1QT7SuvX4OA;jDOOYnEo#rSH2edRdH`R()I4h zvhFD5zQs}GTCMI9si%MYtWn%qGTAwQ+rrjT()uI+YXMuTuVO}?fZAq)W^VMB?9KbA z^YnYRGp|4W2+V#>Y#tb|{&h`a>EW2!ka0yNdLs7-;ABx;1_*!zwj$Rr>#iqq?awxZ zo&KEvu9ez*O9^O55_|{VlvAZ3obp%WI9KR8QW89s?TYRPc}{IeGv$AU%1W(f4m7cfH7+$dU(<IMH(42Q6>slEb>)Ms0tLC_~OE*7Ua1S}~ycgQ%p7QvknieW`qNAZ5oW@`PfmwDXr9Jy0Yj+}TB1*SV-UmvZ0AhTCyGo3Se zGF#6#P)bKqG{-Zml-MnbZNox`VIMwzT>PS@3q@07q;gvUwY9aWH!5FYeNAhBmtdgl zv9z*+0?4GaV6g{Xxqg|@lugXs7&|pJO(`ic5D=zYfBFkZVk?cXsCoZGfl&HE!cW2?%)0G-5&v$ws5g{{0Q#H_L#}5zf(Kv^4lZ zFL?A&1pshcL|!)V53v9Fy6Z1SYri-O`j)Z`kglz}?YqDmvwU?%w19Z?xhdKXetAU$ zJj9sw0Lj(P-u~5gYc%jAU{4kFlUr8^HMg+%#5qC8CUl#Nw+}Umdt!dwe6#)O%#wH2?f$#S3q~Ovs`NPw+%Tdc;^(9< z&#$Aa@qi>K@!?9@v)^m2eZ`l}98f zF|^ws{7WxKZVk*z3H?)z;gIp(2zz07TkjQ`%L@Q$N+Tr{u{zn-j0m9)0Cp&(t<&gk zBm<$%T6$W6v($AoS+BD--GN&w9EcGrsp$mPQ`f(&-z*Ig=d%lkvjUPbCh>PkI~ZgY zNQcKbE@A_x#6`ou_J`3#di~7_m3pED{Kdh!Bo8iHwcqTrh|$k&#t%;)z*YH1*R%#M zs6agzm{)~jWQTkplbT!G@V2bs%NDJ?)xhvLQ{0&jBJsJTy}4u*Y1?kR|GA^3{NA{& z#a*JKBdm%~UqZG5#31cr@K!PUrXaOO<~>kpg>m8H(4#7|+Zo=F2{|R5Cd+ zg|j}29Y8TeBTmq7` z=;U+677vON|6VQq{BbUgSg6s%ockcjs*nLV=Sqi@_$e7mk$2(iM@vtPWYELb{b>05 zI$rd@YRS&3&`bb?XG4j3Y0du(@p z(w5i7y?WD?(?RPk!y=`S9%!(fyCi!iNsHkCA|i_f#hDrc<1vICtM5`{jfNIU>)i4I zn>WbS4*m_iK@x$H4bILDIG6`A7>ghSk!G^R0ed%SJQn79^wPfP~YseJ3{i{tT31>Hq?X~<|xe2tBYc$ zrZzhEfrgbuwM{Gl!~!Iq-&4(rxC^Ow0J3CA@X~J(4ZCRu5D+%HXtMyq&<2q@Y}tJ= zRf(}J8sq6byAB*W`9cr21>k{tj=#FPI{C+f5Wx2NGguNFr!85BuSHhyriyqgwFF@~ zaJ5~Wp5Z)_c4hAD=gD3QJ!w)*ZWMDLsQ0>AsZDuf1iuK_gzwn;FiADwklhbq`bYQ* zQpByXOKKbb4mhl;s%ajXmJYX4GYY>(4@QvqwtgIrK0d`0GqHdI^GW+u&h>X!VTkGH zROCpHR!fRtT^-%J(Lk_mNHaAl2_9oUqaQMJQCBrS>UT>I5MWP?W=u#oS>jLIV1aV7 z<_}?{hUG>r&Ky4F`*SNks+u6jM^H)YD>bO3ekkBh^`!R_tRQQPB7~KTw=3mpmK(t4 zc72ugBb(?!hN|W)qlXymEUwb>Rn0T_3Y#lIQOYGkE1*o(_Ix`h$%}@y={HvgxASLB41_A7~!udy;S7JofbRf*N4VPZiKDX*< z$cL6}Qoi6hsIJeyoUy>xzq984#a9E8W?QPb>Gk6G1L2&c2R)PK`fY8-D|U^uz9{S< z=&0QzBiW(E>!T&*@QEa=)#$+1W%Zc5aUF^oh-Dx%xQ92_cEs2ah)?Teb&v#$bZI*t zE)(+~8SCrQlybEGAulwr+TuuRl#E>9x1Ml=U!#49e{Oj96)B0Jn61|+^6%Q3i9GLd zz*%#j)-(|*TE4Aw7h1FJ@hY*g90%R?9Uop<4`eaUrtLBbl}R5-Y;D_8y&osHBj{4e zRiF_NAh%gdZ*Olmcu04ZyQ2~WU$=JkL62Ze?5@@Vel>s5`~xLOg|YT+q7GYBi!y2~SJun;>c>LooQ6cf3F8fSOc zOaCpc>hl;LTho`Gb{SX(^Ec_pcT!uO&1((nTiLYuTO?6=nFc@hPHmcEfRS|*9XQ$U zM7)z&Sz8N@85)WHuqR9*Hv0U%n0R8gb$j>6r1LhB7sgGQ*IReT!b%b!5)<`L;`t2` z=g*12z@r_skLgIq)cIh0-v=N7=T~0)oY^abSL+A08Fw-*p;rW7!+7IjCV+hu!tL4G zLcUKe0raW3aeaLjV)D0v2=1OhU}v7v@wA(SGzBJ^)v0R{2s~Qa7l{6Q$nXMrPg33!FG71U+ z8`w`WY>1K@ZI?aa+tM}W<`m4(Fa0q&1t@e;g@XNy8&!p1$Uq=88T_ez#lqbL(JN<= z$++yYXf;<&GSH1sQhrJ(iSNy#L%G4k7u*|n4b-$)=}cW8Xp_Ti+5mmM_}IHI4MuM8 zG;L^g2kiEQ2vi&4J3liMYQc?6GFyV2$1h6fM(T;WteFTNY>3g4eLASsYY4rR`?1(^ zX?1~qu^_8k>XwkEUSjpy1zdW#GBNQ&5Q0Hn--vE4#RQ~ZLSPVHb^`RX^ftth_<~_% zpPSkmTY6~I3lU4^ov8ouHjIy})}8&e1jB4nAGS=b(XrIPz6+?wX#vE4dQYp3!K{PR zW8_aPvr%rcbwd3-`Jq)SOmxE{e>MulC0h7w6jd~aBDJ2z1Yu;y2R)U*=uUIY=F;Xi z{h%EneoxRc61a85m7LKWYbi?Tzo^)cK!hbKJ?U`Kx}<+cR8fx*GQf%bES=i9uvgHhCkfT?|nE~9P4T1IX{F8 zbVJL`tzrK{1do2>S)S@Vs z)S&=2job?;%jM$yq>ZIOB`3XnXjgx{c5PMz^om>Zut zp)mW+-_7D;98aW{3Dl2$r~i`2>(?wYI58Ua=;F$qR*-9Ql)Rr{E^sj?D6_Df(%BgY z5fUbPlgTH63}|f~X}uZOR(`MU`=;wD5KD>*&VEQkA3%`?4JblHn5G#e2m000k$!+8 z66(;C$M~=`Q$!);ocO=iyggnX@2V3D{vfBXm6u%(nRiz)%*Fm4+xdzj^f`>B>t1J} z(sr9Wm_VtEis^UV+ttP|b8ci3sZ-U}rEdgrc(VDkrd!Y(1J3m>DE(R>B@4}rZ7-X3 zeIjFxdn7Y8nvd*<|E&Amaa=5&-=W4lC6L!hYX*h#`GKN(rt(NTMl7(eq*RK|#-C3` ziM6)&Xa%4PQS!|B^}Zgq!nTc!Y~!-$Qwt*T0{XI6jsaEFlsTCB8ff=O^x(F<6z2+H zZGR`HLI`Yo>*!z{UOU?qVd#MCNBzEq+kqolF-FkziNfBrFR(v&dvr-qMk(0_NP^MP zg4*5sF5K%?cXN)28Nid)gS`;9UW|{|0mx~@Z(2|`i*H9%I+}K$H=q9H7HR&CY_y0> z{8UjHFl{%QH%mU3-~heZipyk#(BosAR}^iCySv+4Ik}Y6Hr{+|g+mbE?@9|MdbP%d zCmDo~<17+R2kuqf zH(+&F1>r7jZ#N@Td|<%sU3WcWYu_Kg>0nv|!uDy4JS~1(+}JK|_hwCdfzEN05m!gZ z=Z(z|?8dD%o8)<4DUlu{=v^AyYRu7N0N}+nAhA|WxTL(`^;VYD&AFwfV%#)U=(l1W zSetK@P1Fk@ffZ5Vo;!Vo>p9o!glhQdtF%BHYy5-d8 zkyt%_?L(zAdcsCM+?9f^JqLwCJpX zQBfJc<>)U!RiqRX4nRRdfyg$RQSb$@9yWC9NhPt_eKscvb{n+6lG#Z)otj?tBI~Iv(5JhvM()p(*?XL=@Ucpj`(WP%QLtXns?cnH6uS z(`qN3)Y*>IRypkjtibGiNJyp;ih@z})Hw|mSd3>jaao%65$4fH8m%#Ki95MPd{O~? zfX8ERevQUYMat{K#crnafcXQxc;2h~{>Qe5g`19chJL`HU%6sudFQF@6kD0DEqpZ! zz6!f%?(rD7m2x157uJ!?%n^{S(uwy*Ci9YTCyHR;+1hT~+wf(aR`G%f+Dq8`v-`B; zJTP^`a#M=kaO+{PUB4B z8q@2dxBgCI)r$6b+Ym*TM7O$r_)xXD^(&v#b<8MMqQ8VrK}LGHn>YVePrre_hIbI{ zR3@WTw|RdbMaZwC_uDZ1n6{Y0BymIk>B{myo#t{vguVRPDn?$EfSlO<8gqc9mt?Wo zK=1Ohcd@=I+C`$S3iZc!p!8DzOpLXSCSm>eh`W0mh5TR;q%UI^!hM>0ZWV1Hk0 z4i3JLOgKX$^_AN+N(FbUVA-CFUck`+IN{%KVf7DNvHy%1p%~Y#AjW}@k8xNaMUX{R zf}mCS6F<(F`kvRL8xaPIFnr-pZ~307g%+SKIZweJ)Dsk$-dpH zN+j_AxkpMP`bW*N-+lLK@x(wZ$&DpqlPc6QxWLb`^B2YP5&_t3C|SK6mMTQOaQ zEKsl4%LPi}|7&*-TPfwMpkKcLBU1OvVYvqQp&<$y8rjn_ zjYM7bqvPYlT#;MM;)W|0pm25N4jmO0V4M+t@9Q_;roaACSWaqz?m`M3dIU(8fES=` zRgMHee+l!zc=Q2-2B2-0pG?AdGt0`JvKQHy+gcLZKCT4dT8w6+))t>6fQ3w|PHNv0 zWV#~Bi%HgerMjx7yCO{r{3mI7sOOC$B5W#w_DOkpMGIiJPfe-J{)P{@(%ME*U-ngN znl$FGyV|#}_B!p%->?H_cYH7bLu7!dqc^*ifX+M8zUI{$b@n~vvHgq}79myHSo~OD zD!nO3AdUGwo}S%Yy)Mrzt5&E}hUHV&mwr%H z4gLH;d9puTlH{*!Y!s=h6C4^;We^X0{X~G5TI_0y)#wJe1^!?qlQ>2^`XmK6#&9UZ z>Mh%nFjpTm>)+B=Q;G7bjihCoqy|Ec0E0|*&BSW9iuz{`TFUx7jId@FGu5eo;N=c? z@$qqXTpu%#QUP?Roe1mS)!EsQaB`Rez^4%?QM&*J+Pu6x&W<{AG+YWqGPl`Mb1=2< z#KO-Fr{hu^H&=^-;`VEf8(e^lwP5%c%IbDv`QUd4j{x58q{_8tfY9A?xs0x?8jq)U z{{mq8NQd9}?O^jH-Oh$LQ>E~sn6#_Wp?q5TmJH^#2?bw06#N>AQ;N}MLoN2W;1UIra6#K>X~Dqy@x+ z&j~)U^hlOmT%aPNA6EBajaN`KT+&cf%M4LxNm4*FjCW8`x1f zpH#KdImg=C-Y&HCpmwKXqGHl_@-x$%zZz?8WE1!nuOU( zH?!aSGmP)gIT0Y&Jhi3ffP-7-kptxPlB-8^*f`ImL^F%w;O>YGI|mmriEp45sQ{}h zJ8G$LD#O_UoR}elDmXiNitwbr?fvbtLqq8APnR${R;4y6Tk_pX**rbLY`>ro{JnSo z{S9gbfxuuSzzkeJPMEJQ7{$iMW=J<=o)rz|)PJN}q#3ap<%RbcWIxWf&klj;-Uykp z>x7K}5Dq>JY)n8N&xy!tKm{!BVjY`Cz%hnMEXPpkqh@$O{33*lT~8JX%wNXg=dGQB z9t4i6!hYSqA+)?$oc808Z^_G&ZSAxS!fzvqDV}5E(aNv`<+vI>)LcFujT81~)n=FtNr+oWe-+uBkjuhO90@uciT7{&r1K22eUTkI1 z5it*1sikH_0AYZIP!K=SV}I#LLSK96E(U;u3jpz6FOcI09CDK^^)3pTHh${$Sg^l2 z6uAS%t#csry2|W4_j!%XuV{q%iKWBUNktrEK;YH_*WJD0sqxSpa@~#$pHsc1+wGU; zEEN?Mx;RD-&d!qN=1glGNnyywQjDReekKoM93t`=vMS`KowFk`uz$->0U3>dN&H9y zfwW=hxTf+a8?;3e#s<@eF31SjKVBc^YEywa>?mQ&y=%MP-{Evl)MW|UEjL{H6SG#i z7FtMtwm8I5*tAJVqWnQAbiBf)4HwbmM?X?@tT{>eaO|GpdN42;5rW3X#{MbzZEKTw zkcxx&LD(kUot#0%(3KwwX`!zy$Mri3rkETeC5+7W!MMC2=mf^23V!PPJF{N(-@x4E z3UpC3^y@%^c!jVJnP~@2DnbwuB&BcOZ~~CQZyACCK<{+PQi!b_$Nl0iIj7T%@dGTl z!*ldAiV?zA?2Q6i_7Xw(;$K6F9)I9A?a11t&YtzO`IOh1X^>+gLIxL-Y^!O!!IM%< z3v0^)id+HO=0kqsT39n!<0Vy)kA8`joB=-2Nlj^O4HoodbpcS1+=+#fN1cZdZCop8 zb}J(Lr3OiK{Oi9M-=UM`ch1!nANB6E1dna!kvqhA25oK0Yd2;Rf-WMf?LC7*zhh;wWvLLKMMY>iAtjbQG_+j5lmYl@dT4nRYuk7f zuKwsGQFw;nlg;(tB_7it`859L*DjQ;D;7D2L)RX3GjH4KerD)h9AVMXwQI^5>OD+` zfco*TX8eaV$7;65If>6c`yc(#<8~zDj_B2^R}iX)E~Z6#1@#|%G)3|54a|-vTbQ+0E#g@a!%)f_sOSJ%J^mX5(1vM? zV+ViT@j6f+c6W{UpkRPuMvs>Wg7rtV2Hq-kOj;bvU+PMaHc5mhaq1i~eofhezOcw! z(W1j4XKwMO6@-rd;^ZI?BuGEy3*|5iLu8s6oTJ-@BGfNeDkJbOQS0iwIK8^aE>Ka7 z&Vy}bf1lQ&H<(LwCFwr1w135GAo~J;&st|lx}@lK!snJmA(hV;06!Xb#nM0cB~xmj z6@5q*^9uV3$9*Iz0snxYPwlEB8A$DpM%fPR`LW!T9)}3U6@P#9h)A3L%|{cqB5igl z*-zPY`iebf6E9jqG0Q3a0rKL+Nt6BJ?5wj?6B){)Z?vCzKcTWNtCHl0t{1_r8{-tkbEe?s6^8H#u$vM*XnM+J_eAn#>qgB`Ake!W{Im8y8WW| zbA^oF=%J*JcNCE;S%;l61JIU%XS>#vzjYi{^5Fz%ZPHx`ITQMi!`{_Ze6;pT6zuE7 zA)T4(o!yKi_!b7je2js81QQ#hw?_Y^$+&sMvuX6T{iXtKFHN%RmsX@E=YMrD$JN`H5>(=Ws zJw*6bo8U9DErZ2o2q#5g!;!?v-q1fIm$ES9&i=PIB0x4LKyvoB>X_X~iFa(GtEBjZ z0U(WSC7_^WJ<2+#a~r+TCC2YIBLo(WG~$GG5*CBezDLRFsAN|+DIo*c{IV2+kHp_E zN$M-9OCW=cN-iltq>yPhcehTZ5`bSJ!;3VULy1~m-L)7GY;DKDp?cXx5v==6ngHMH zzg(9TWMsZTePul zFJ?t=^nq=Iyg5f#75ax#RvnyJ0d)QTsQ_8(Nl9gElGT6yCNM;jlNIb0&DrhI=6t-g z0`h~4Yb`WE9(t3TDr)~3Gs(|@&s$JaYoEQTWkW_*-mwE-K%#V}8`b!idc=Q?qpKBJ zhyy4DGAaV{PG4O5V&`l>(T*DyoR9aUE~mFlhEvRsU#Pwq;)GCphy&plht?0#=;x-m~R(>tvE+yN>#-}fhUmB28U8pwf>Az1tDg^w$ zgE#X3@O2hIQT_kBm+n|VI+v7?l;@KcFc`kIYT{tT2Z{tivfLJ=Fjl|L~ld-7rZ_OHPPOc+9-OJ5+6Q=5m^ z6hH}!7pIw|BdByA<2Dkue>H3RHJFRL7pBf_m=L-YwjqLkn z7}Wn*@&PAMTZ+qOqOT7)i3Td-m#r9m5y8Q9CH~tRLmJ`mZ zHQxGnv#)RUG~zu8cn|(RuI-4!@z4+|AL4)S41HkjbTH#MKqDeTF*i4NaC{t4P{2~V z7`Vhh%e8lS;6$2-Bulq*wVR#uxyqGA`ZaMw}86^=w&+d5QHFv6c|QgzS!UFM1D7fmOb(L&bYAr2HZhMCnlE1 zw2V3y+d{rJUS3@(rZ1=AR;b8+aZUP;9f%Q9*h{)G%$$T@4<}z4e7{7Z@ubVqMO0O3hMBo?KACGUh)o%gE9heQeKNO>> zjM;g4XamB>n6gpiZ~q#A*4CCD3_yTq-5Es@{t&LXIs6qyMjY)}L3wM^X_dl|PO$o-v@=k~5`C6Is^4yLn&h^?6&cV=|lzP4WKG0<3v>YVG{@={qSc?c*bAVrsg$x%u}XaHQpx zmlH%rMw$>@USFHG!ys-)N=CA+y%qLn*yXnF`%;jH_1y>e0xGHJTf|PzE(@bA26#Vt z?D~inmX?@lltJtjAlE|Nlh-*tw)8#Z9zHvyQ6%3ns_VgJBqv0tNT;Sd$m z+N?&1a|PDYTT8y_4y*|OjHYPNc`7n5zL9>Tsm#=)Sa8rHBviW9QnSW4&Po~jeZ!ty z%!@ELDt9@S{pqu=eOY&%$3WMK3n2;f6Q4KSKW(D;U&Zdq>gp+~v$_WYsZ)eXZW49< zL;wm-XV1$5&ebftHg?vt&p6Yi4lGdrOI%Oh#vouZ`Ej{*J~%fXb~+#y0AR-ta;OQQv90k?$|KNM7!(t zgfw32gg?U>;4{e~%vCdN?ybRJolp4o%E`qUO5KeUp3vv#>$i;h@_)5vI~4buYQD@3 z@rwK6b_2yerz~g|VbRfv*LTebIMmJ;sQOl?a#Ei6xVS52(uhy`2U>!)K6k9RD{-0( zE0fRGpn=F6S{idikiF72xp_>N_UliFOl4D|gDjE1uF{ZZ0^_caQA-+Z{{HS9&}g|I z(4!7Z(!ytxrb)cs=z=qfgQK>G>XT$)uyj4_oCQTR*$CBy86+GW*LIw+j+%1iFBDCF zT}`!kpH{~%e(%qgiA+ajeyNre#)}=w(!OifN3-y_`OP8?vv#vf)KC`>yxhA< zR)Ue?2A$g3$yvI^qz%l@V$ENMbV2ZmjM;*2|3ZrXV~T83J)Oh|g%4!f8R(S;P8u+C9G}B~u(#U|6ZOw}@sQ;jlb#;= z(E0+YAu#-g#5wZ)SIgkXmKGYkn|6u6xMT&1N>4Z7cV?vs(o!t7K4m?8Dtuc;#A!Qh zkLHjf=*X%{E)vb+-jVdo*B=Ix%g~RSgo}0C?<+hwK9{l7rtbd?pne@4#{$|v-y;h* z2zL}bgp$t9Md7#p(Sk|>-VeB?KUmjNKj(0Z9CZWk%nSBbFMJP=wf^*Q8mZvs-@lb2 zOi5L5jh()I<`|%%gObKNpy7RN-#$4x`cPFYCfwkSOH3Y|1q7Brae8oYL>LP(N9^#E zkc#YnF9$Mw;H9L1N)(Pg!&8jC)i8&s$OYlW$TSWip*lgr) zfgt{pYyM$X{pQ8nT!*O}CC(N|3KQ2-oC&y9ngohJeJ+P3+y9lNyT4b?)s@I!3IUgb zj3$SBi2GsYtI+?M(?WPkNKo27f+8agt?d+L zn@wJmoR^jBcQ!V=mYf;M5woL{{!=L){6a#s9Nk7|4W&_2asQ*}uG!}Q*fKmlv!h!R zh5T`Ke!HpZa{YSU=ai7IZXyc#_xkpF0NkO*NAbR(91aauHxG9hHT8*Npj>db;GL41 zq`iD53`tE5Col+whDTWY3=16_x3u|`0F!`}01wmIq(87KI=5{;aZd6drqILcpepD; zMapWF{d<$6pB^6MNFFk2tH~##|JfhJPj~*hS>M|-k9Y+PMSz^eSDkDzOw_8HQTum` z5%zwXgS}&dh2XQT&ry|$(FJC`2t4qCP*aA^$!1jl=XYSR15^o1alD%b=)&R_0w0lD zncK#2Zg?)_edEKbD#`aF4Avt5!xtqec1SI)LlgKrztr$$c5Uf6|1ZxbO#5&04%)z zDcb`=#}weiVY$vh*^`h{Fh*v) zmh<<5JZXc|3b5`b1n8c%I*((6Pv!;&Y{1CDdhP6*^ED4bv70dPrQ$0IFu;tfCD=^$ z)?Q6n>FB1Wni&|iIr5HADCn??rbM^>bai&t71QCocJP+#S`SCt-q|Uiu3%9#dSFt4 zYITnm@EaaBw+}9q!{PfIV&A)cUtZsPrJQ@cR2DPT<+NL{Fs`?ug^4U{} zRRMEEl4VC|!?gWTv#Jv*ZR*luiGb==sFYR^Qk78sG3~>ar{pfkq6F-bzcW3F-@n33 zFr78JLqVx*v@1%KT=J-mh!HH`MIcx))1A<^d^X{Bgjd#QguurqQG(KtKh?%LI7Xzg^E!F9PrgrX<($~qL;GEt3ntXFOMe(TVR0`r1`umFg91Bed4)VjLUTCzg( z3Vpucy~_vn_1KhZ8hg+FyVP&=tF)Vs0vxi}J@HSptL?K+$z(Rk^7Hc#dRU6(iMwc{ zh5eI~h`0F^jU>wIs?jX#Xe8HV(;^&p%_Gc!O9ujnd7a$o7J0eD9=r?oMIc=Z`yrv1 z@G*v(ol%-lEtP_l9MtMHo?tKU2?sE~bO!!W!Ud2ZAhmyQy{G3SFug;B(&R@ijiW5B z?Ud$~XpVU;H7(j66c^HQfQ^bKWv-CrhRCE$d`Ccx7gZim7cqE<2lxJPRD<)37skm%`yG-!Ac1T zQ8epIIuEgcr$^NwB%)>HH-@3Cw1J=}-Y4@8{!&+=ehRTiT6}y!8MLqy+ptDDEbXSw zvpfHj64+%V98DWg(?!VUI^Ai7XCmznPR?4uBndSXCR-#h0njmh0?Im(erV{C;=s$2 zCgdW%#{%APX}QcAD%CW$gBJ^Cj#;1mqF0|&1lt}PX=$Md)=_Ja zV|32N(5J^>h)!AoI9Vl!6=J4$2+^j^nOWbNo;Dl#T-~>@bD$bsfXKjw5_wY2^{Y^`%BJfuSh(MUI>1-K#}ToH)P)ea_#QCa^a*~hxKMmbd3z?N;Qn?vjWhT* z)_{t}c}7!{gcFdlqJiP|xAk_Qv+w0IypYJ&lG3=24GY`uh_CN7UfGj86#u5b5cn;n zv5L!D(h`yH+UBq3EqSKBFNXIcCp{hk|2qWoAeF1Jor|t1pp=^X>IA0t0U?ctI6xWC z1qH%Dy>j$x-lDc@9gIl%L&C*2XRV)H_J6F!eAb}){fdbwM?<4$CSf3@qzeOe8lw_4Q{KmMH@tWmvk^UiyBcZXo-2G{VFeM2X-*nB>>{ z`RH4x2iTNSf}s*3m67j*fKT$h^`JVXh%@%M&n=4E(!MX9Vb}^#NfTPJ@bjn#nrxFhKO9t~|Ljrj>8t@d+irrB`s+7ode5;%?LsiFD7UdOJ_bf>nR zOWTGgp9h6q3CEAU$*Ksb;cqa?_;)(;{xNC0)_X3tMY(Cdh3@`n4yRr-T3j!>(v(Ft zutp1$!SU6>bO|QlObe+&e~aR3dQ3K;pp6rkK*B|UOU@8{?kDjJ7C{1Q>9n+f;bS?a z2yAw-8H=D)`S@kuSYIF`Dje}T;#G6twEX&|(~is(33S*bo4VnKRrDPIb{v9BffjH0Tv zT9Vp=1?J6UJB*aSf!4>uA7%%)?fYegjxkzhLU%9pD(W z7IM2FZg0ntJY2Ot^lN9tAu5&u0hh24kY;v~>icI|&F<0fwu9{N)o!NUb@CT@Z9kTf zzjj-M;dnf8qv#!aO(`$k`{@+_YCQ|dn`vN20`Rn~_(a8^ z3GojyIDu79Xr;a<6(7Cj*HuM~9N+4MFRvWan_xQoMrLr}bQ(}>0lWF;iT}TAse#=! zXIX8>lz$#(6U z-0=~Yc{&A0INDt5omj)E*F^e+bz9lNjL3xJ{pSg(mHL@)U9OcsyG4Tw%zxVcZu&}I z>Fu%iuE*=tup}_!=Pjc_l~?6m-`91K8#;0}Mt_CT%T-2=c@v;`a=?_lU4lI3=^ZT| zD0-0*GAHs+v5Or}+`>m(ywkG2A*D~u&~9UW!ZV3>-#B!!dZ^#)Gt;zQYOYt-xDKA5 zqh48mrA_qiikE(`Ah<2Mu>RQ(KIJpSsqu)Qoc|?Q=Ji8)+y-sH?s~7*JJNySJY+6v zVDhuC?+*D*W7j<2oRmxr(V%X@Lpj;wjUx347!%oY=nPK1{@v4oQZC*Tc6R1cUS6&( zHY>uPJoVL)(Sd$DTRBEHiJ&)2jbvot*O(}V>bK5I$d(mzAEQ!te;*Lx%RT!Ubm~Q! zG}u>U_Y7{cU?tGyPLG)d;k=`0L7Tvd#nB`p8Y(tCd&}7IC^i&$$3%4L=0gl&&9NgouXJ@>*;``%07|Q-me3i~^Ygn<>c45*RGX4Vd+hTeCbrqV8m~AINLS#-CHFtGF8f0^tR9XA*|p1?pp-9WOR?3s|^EDs6(Anr=~}Xk74QS>jk;WGMY^Z)wO?k1_Jk_J1Bv@F}Ns zaf+ioLqQI5XUYRnB{GBU+dk3%+T%GpI&$BgZzCxil;gE^sU;45v?^dRGoy{_ zU)eKGR<_xmW{1*ok>)TAe2k4k?Sr6`MA^2}n6~9CFHn&`45B&fSJSc}sOaJG(&m(@ zlsKwCPq*wcTD9&bqWd^a&GoP`$cRC@#A`s5K*#DQX*u$6b6?!(k2WKoMrphfNe)nM*I}mR8`IM2u`mAgwF691sJ=bb5Ausqf07$_8GVVREIwa%NO35x`6;o- zz6|7(CLZtJ9jGMZ_|tN*-L~>cb2wD%-(6pv-^m1rG~ny^QebZ~$?3gpF~E;M(I`)yCt&(=cn=1-K}nyiy*-mQI!&;t2BL-+AhUF##>!iJugjEW;1K@se%|l>`nSj7=L_Vi z;13iqAgqw)*mc`~MshR;c(@F)O5}oy!R=oYP42(BuBL3%EOL2vv_U!CDC+G5HgnCP zLjCp>KwXSd>EkzMbXO-t2ua^9B^1oh1+jVqB_##QUF=v|S89@eBbY8|`uvPH#56JFc_w2KpubMeYEXKZX=*2?g_3dK7eX;)r)?~(r>tJaSu~F7* zrn)G_4u4Gd>PK<_$aDTX@piu=jwg8e`eLI62sLEhDg*^e*i5@UiuQgh&dF@wj1&rk@j;h%HkEsmxG6syQm!StLY>>Gfh(`4h8*s{w;v;pKWnPyeaHatuf!*3 z#>TDK_hy+Y=NIuWz~O5FVp0eM0>?-+G5y)Ph!G5g@O}@>d6UlXcz*P5XCBE|HXg+M2A0X40n@#{$H7}CZn?#aIkxZ2EU}1|Yn;8hXVfzy z4Rnit#>QUWHM(6fHQu$|69(KXjw3;6P7*jqVXI$Z#I{&pp&ge3A6Z53Kd?h-+^YlM zt-eX)mbI)Sk24G@1?jH6<8sCKk8Rg#h3i?l>ov&j-)sZgcx-Ar!R z{K0&|4}yi&1hBvX3k#^E#g9D$t9S3{YXn(Ym0Oc6Pd*AIm)0`wWaVaU6A{zNR$~Qk z0T37sov@;0SQ3UfZHCCPvf-O-PC429f;_zCR1VDU*;xbt3c*{Gf=>A*mKsm&+*JP}K43IE7fZ!*gxV!J=XO3=iU^ifM6l zaNyU})I_}gbFSshk8cWkdLz^(AC@9HW}fm0PsYz01j}Kgp5RNgWWAPZ^;g$1j2utk zRfu|Xo6YgMACbK@Yi{)TwQpb`a=+Fhq2PwOdaG%6vSaUEk6(0KgE zNezHaRR?3B5Y~LA42~)ph*>l!O3+RLYLy~c_3NZ8!V8;izec~K7*pR zSPzl?m&8-?!Y1U%bYExi;UzW5v2Mryj@94KHEnFM%G=fc%$BXS zEu?ir>{Ja~3JVLZ>KcyqZ|6bj|Au*7rGao)Kv_XTs!1My=87HEwn^(*>bg8_`Oc=Y zs6n8p$d))>5V>e?kLwA!MfEiK0E7za1>NznmoK}O$J1^KW>Et^Prs)ON%$tq!#DnW zX>~g!4@2vLqE7Neu>sxg-$) z`EmNJNhbf;E56^q?+J7U^>y4OGdYDES^6(oEmxE&!IYste;5``6z7)_*9%OttK`go z{u6QWAok6Hw%(nfKqR7DLcOO!0fPCBO+P55hbwKePUTHRzhFl2N0u7PAH}9b`~X-* zAm?J!nv{2pOGpu32r}yNXB^&G_4Ds8Q@xIw_%OcqCRyeR$P553u2R{Gh?ty#MxFiU zXr9Aki@eMys<(c3gFhbxKdSMhDf|`vg!l!ENX77n>Y3@oo8TdnH7o?O{7GI)hLOfF zmC01`KFhAm$FpJU;|kzmOj%xePo7^vl41>xD|Z5`<%0`%~W zt-X)QZJYhw_dBkRoI(D$@ur98Kb(=C65;gbHo4ro(ApKR()B zCh`LklnyX_<#aeRIpi2T=CJ6X{kY9x`im5TPyW@Jc24^J>5CFa{XiJ#lI5+Hr^r0U&{wC0d*9P&Om6e8 z$#dV6tcq4&z-*=eg$XT*V`9MzwYJzIkReB@k9EdcRnr}>O3fr}`COG(;& zED<{DnD8Uz6}%*_7ZwEKPmSuofTC_?UxJJ}q-P} zngVzjGz+|Mq-YjLSTnTc!~P2cmhrmrzrG+EiJ$xce}eh~jewhquA?{{6}3f6y(;J7 z-yXhNZ)u^D504GL0v%|;vsplqR}JJC1K+5H>m#C~npa7Jr&TOt}8oQk#~0>JbhuVh+o$N^iIU7{)74#w(QR z1$!M%uC2p82m`)KCLv9&eNvmray zNsAgaeiS8^;^D~qbEPwgteglj`QC>%b9}f(1a+oAB_r^4IYtc`7yDA+VgDWn4qJJC z{#lm@rsbChSWNX1M=ux<{yFyG2wC{oLqu08-MeXw`Oeang`s~Rdx{L3 z2QxoKzk8Q2w|dHAj~(4%k8^Uq)cKX0_-EvvB|%CahX zMS@iN+dfC$+511S`OGHi0RgI@?JJVG-eSLj5$!hcx z9wN}}<=5!Z;=h7MN_4JpDF9uVgF9a-d}w4Y?E9oNI8B%Y9SP*-Jsu-?i`W-zKN0jK z0Ov+ae$*3TcT_AzCq%>u6%_%?ip+}}g^FULbT(J}v%zW0x)&%a>Vplu=K;Srn=2d{ z)tz(XBWW;mdQe$~oly;JROSeM!}Lh{s$S}mCQh%K<-~5yxgT1vCQ(-YIYdE0DRtd9 z6h7|dzL}bw>~38;?g=V6(Q~+4e&^;mPeX`u~F3PcH2nK$|~PA`oMi; zY%B=GFX4cQ8CX=*+A{%{-Kn68ZN5xeCbWRhdG4a}Ylvtc=D`gr)O?$&;SVjmXny0v^|AFjhcZbNJi2~ z3ni-ISjenT=X<*hk_r9`4&O`Cq+UpmAK4prIafUl;2MdI6+)p1Km2S zQDfSE&DgtF#yr&+F6T*$_d0LwwmXM#`XzF4B-LXGrzwrX^TRv{44GP%jb;MX;~>vo zMs93wNqL!I{(SSzc#yZm+u_8~|ID$!I_}IzDK4I}6wdma3mU(L113yXg=6lC^hr^*Q z-2ip(Itc^0{f6z~P5%ph9Le%06ljpLF$>PpCM0lCXu-jS&?r9xQ5ln!He>)chD%P4 z8Tn>OOUYaN&`p?_l(cr;dvMbU5PUj|W<7!3#{xo`!CRDjlEFv^H9QZ4At7yFgx>!` zP?Qb~G5u3(gQ|IsBr+y;u*Gg11r4}!a2s`wQLlb~h3ewFT&G?A9swSqD}o9SV%Km` zOdb}W*o60+Pq=dqJ8?QFJ6cTAz&x3&-Ki&&yg7}Yg>%WFjM!~uXm;n2Sc9QIOhsri ziJ85&*;v8M>mAPrdBuUwrLk0Xo%cwIQ54d;{nlr^F)`YW#zI@nw#~y60qrNsqqDW4 zGe}BZ((U72rN$FlkVKwK$v?~Q7q&u*7r(Ci2F}eH+U>SWeOoHQvvCc=AZc=NaG0z& zr)|8bIF*+VUAP)RC)v%7GHhM?>S%vx&H1Lkf5DN!=t)}wMS(+IHa5lXEGpKs?BWUk zv>8l9vJ-xX3;6Jm#yPlgN2AHI*y|im<$^JQ{v&xd7OBX?Q_?(HZUa)U*uP--E0G*K z8T@|L-xb8&fr^3C(W|q$H9vuB`)RDzn%CHVun4gPqUS1{?K5$*#cjXTtk)GU&tcM* zu3d zBc-)k3=cw8e@3<;g=u}jre8{#?0&rX9&>ptSdFj zy4T{0_Vu6R8uk!x*fiG3d8XE>=jy02f%OzxZd(*Gy)5spyCR95F1hddvvxYVw!BUS z0aAxwd6qwFgSwi|_76wykM<8-=PxZlwIk!FwV}*FYsod!2o+`5HrgWVe-A!SioSiIJHO)mEfUCgKuodQ?9VYDt)VxhJrY4=X zxexS6j-RsjJJs&Gl#-q?#tm#cRmU4Q!SfYs8DHijb$2V94EQUcg?-w(NLa%r_t-bx zxtuPPSB_fsEIGNd(HAtzzA}OhFHP?_B-L@qp{09}-yR5K5FcsN<(+NwvUh2W%~z(s zQ`gXKqfCHv_TF!Dd>-jriAOWxF2pqzA?l1pR$0`n=-i;Y8z2%0fzY6~dV~5Y$!Y_K zk^W}5fTMJDY-}+^?1l!|QcK_Tk)AzTU0-h_hlG-O>uTq6++2;wwD14nH@fMT<&SUi zDX7M+cuYT8z0JZAob0eTc}>0-FW?FHlu`)RUatIOvL0VT@V3GJn>iV!l$=ry*V%8* zcSBrX-mEc8T~!UDgizc4Qk-PHC2x%vrX^dCr`~+ABuQo6T$^sRbii;OgS9F{WXzF* zoAEJfF^h)W^ZI3Pzfsnn4i0lMk;&;9VG>T! zAj;P6_dursL_*d{GlFDY_3`5JauDh>l7q=2XQmn#j9+<&-LIfaQ=i^DIiLK*r8f0m z-#gvdtyoCBzastY?tn{xEe0oQib*U=!)TsJ{h|!#Xb)Y3>L7!lttD%^`aY$TSTBWx! zR)}7@pa;orp@DX_882gKT*;qW?`Khzq63R+7HS3^SA9t2wN6`;=4Xa62OPu!v3z`~ z$87R+^k3Uc)je?;l%y?vG7lPw5RFqLeU%^VajmVA1Ox=HkJtL{=lMg(mrUY!W!LI z$wN?L-KdRA?Z=SmEc#dIwrC1u&>~cNwAG|aTDocE(DF#q;#DX^_Ra`<)n!G{P`0A+ zDk2q~=)UwXki>7XD7mkF8Qsw#hsu18dS?-BI@EJ2uRO3c5Z;>RAvDdeQ~P5>IK_$? zk{aJhqkcO}u|-}ey!@h&)YRI9*^`Iq#}oBXbVp5|5PkD?w%44`HJn1bNGTyQe4<9@ zW?5*M;{H!i9UxgZ>1waCaAkU2bxTc<7*Af@;Rg*OZI%-gTZN3=zWy~3KO!#cYNZRm zj-N!^Ht3X#CD@=!vg)pgy9K5;U51a)*Y!%@(1ZxN&(O+e#3JU71a;OD_8P^9ZzQ4e z_M&_$y*~_rLYc%-{A^$MoPd~aIoFi3i?hA1+anVc+-xpC5>FE>&#tgM`qLu(!X;yWHa?+bq1 zRlyjN*6E%FhdKi>M*ns=3MxF(o;ggsC*?-P?eVHay^9Ji*VLZG%4_n3nC4mo zIr=15sV=^cV@uJ5k(c&L=g!OhdRudytSpM+vr=HZiSA zv{e-jZ98sjr{FC9Nqfd!Wl&J+#ZCM>aY}ea7Y25JrCrB@<%mw)JHDZLi&bIdNInAC znZ_L=h14aU;Ip&t@Ah`L+So`Hj5I66E_wU13He_0cKo{&)>mdV@aQj2n*8^MSfl`rONR{!tpXvw>_t z?Ni$MHk`0PZZv;;uv`F*9FcUp=JH{+!{rK%IW8%`+>(!Elx2Dtjbg81?bhdm!Tam?&!ZVfVN=hwN2I%=uUX*W2t?^J(u>9CLYn==u$oes{{3ZY)MWL1WCR5fH-2=65RvsicK`LEbAG6VbmPNx8oXXa{^6xgf3I?{Vzvv&OG*c6Q za^oP2!_AhSm1ca2|DUQ7JbUHGW{Mx(s3mCLtd5F*4rzXb1&>gn_*^+kOa9+R`tNf} zQht~~{PXlcZ~s60{Qvp$e|r9X6Gu1%)&1awMB;Ms>d$L(5JU+#H&Dl-`uM?6=L3E8 zQ-6rh`YF3@4A2q#S#z@;trRA`iHN9PByl!%a%$J3Res7NlA91NuVF0s@Mt%WAdp71 z+v+(DX!bZ)CEEofnk==2ZWv61e^yGk={_@u=;=Af1X|p$BtO)lWQeV@pG1Rd=gpn} zfjjko{y$iNgam9dL-i?z1g{D{!7LXqZ-epz0s|zdKDZq#6OR?R(FJb$bsK;;Ud-6Z z3^8HOc2p}NyK)J+RyTD)1OAwubLlWMH|H@{)$autftM(8=gMyfty$zYF1BW9cNG``}>KwfHQMD zajINwn=ow7L$F)e$#Oloo4XIlAAWb>CTbwB?NN7dx9<})f^zS7x2H;wE)oo`-urW* zIR%9zD2A8xQT2yEN!NXzt<5hil?L=&RJrSUxokLof5P5^a_xf^;@z#sawCvqUu}I5 zQC&aXswrh+tL@F0gZG?48UNp>u;M_C)1KMqKMRN`jYsLeo#Tud68g`X;St~$oiNKL zvdT`ekqZ%sNe(j$^CU=4*tCDzvZSdpM(xjs9K9prG%GX7EKY5}x2+0qAakmhYpB9H zaZImhBBJPv+}O2RcO3C~=Sy`}D|q3SIP#%!4_h>2Y}j>wbE|}FN9=9^2Nj!hA=LLm zPWZIpzNquknYgWYCz6`RVd5fJOAtroYlElgR$RZFX^sz@h(Sr7mY`8Fp2mu|!{n~j z^pnm>BZ-fB-Z&8FRwv10grM^VYSa+|^l5&FB}8-#41kdI1F?Xf=U}$ffp#KLPlm4` z{%EGhzNo=DE^=3BtZEVgQXq}EE9AQ(qJMdT%3N$D<7hx1@y$NoKxDvK)!YvU2Rr-V z#F+Pp)^y2XcLvS!nJyu6*W6qxzMXlRf$vUl=Z)m86;{OH}9Rw~Ic{dtwtKnCp$WnH=$JTItzXU(i44TG zJTdS_xW=x-tG$+_zPwpo>J72W*_d(AoMn~#jz>=S2K-$-(gGIk@^aqMg`Gv$LCqJT zXCQb5Tf4;VlmT(`#57|7(PS7-qeb1MPmhz0ILM}&Oe#urzj6Ts@ofMEk^=oT)>Jfk5C|MXbT1u9B!5E`?rI5;EjYsW`+7x{Aa{05%R13^>Is(5TGb7Wg1 zw_UAm_dTY&2xLWudLP+crS_BfTbmyml5N3LM|d;!UdU4UpX+hLDgVHfpY5&HGPBN? zPzYsQ@n#quJqK5A2ExBXw+w_=1&?oUZ_is@7|0Yqels&cM@h+(Zq6^P<8YkAJ8^Yw zUL|%6+6a)swMGq+%`rzXAp7!S|5^T=AGLY+h)X7eI?0)YynOlx2J@sB-_?Wh=%vRZ zp=Y?GFXsSil$3Yho%F3h*&lpy?@s!!G|arQL9*Wk;KQl@L=+m`q%aRqi?}|;K#3Fh zD5J1Bb}T%u6P`dLwTpqpM++59=rSeSFdaZdqe^rnHkTp7jf;thtZVc?NfWUDUW$T< zOfC3SBQ#x;P$phebs`ssGXjl3LDgJ%tI^WhT3SnMB7Qa%jAI$``su)RHIQ3{%`DBJ z-VFk%OZi;(6fhV}+}^%~%L#!I*rqzg1>jEN8-K2#GivJ9*2Qq!agtpT*lHTQir5-9 z>1WhdFG+~pznV5rifh6PR>$`w>!y9p6GQ`Lz^*DX0(2stH|)AmdzEj_2v~j0Xoa8h zk>>ZryAdzo=8BtNFn+ZtUCj6#*OeXk#>gtat`BWGJ|pI%K-~2h(+88NFo_p^Hk7nz zal!fTMMdon(W^ylXP6s%x%ohphoe!RPMLjf<;Mnsl?K#jL5@WsDxti!mACz*2l+?Q~kdw(cr6h9C#=DO>y5M9nMhVCu2)*P8ct^2*E}wbvT?S6swWqYc@O zOjNS1$$^`JKk&hAs+n;j)a`0B_waV#cfL+833J2ZBmhUW*>ON51b7AjL77mf? z$!X@-tY8tOweS36jg&3Ov{h6ayRV=*_@OpPY6{FDoPPh**eoP*ZZ=0Z?1~`!KCe^0h4(d7<1a6p&l9G4dq&760pPcy#TR2j`G=8vzrRTO=ZzD2b_87Vc z?W^J8vS<~x@6Fbgnr-4C25JwQ7M@q8c}(dF9;V)(B3IfPd8_dx(JF%$xzP@CxP`Hw zy$~DJ-LJBYE0$e(g{?~72lP^!i#HgZuHdDKGYOp-$SrPBQ6qmvmi`}z-XJR>3fDDa{ zv|S21*^y!*6IRnfD|Zb)d`Fz;%v6xRxj?xOPSYV7bZbCmnRps(NELWKFB*D6qOfCZ zWw+VdFG9wzyt%dd+)bFaQ7v3a$ImL2EBI08_unF zsY8>4=nYR%(&McMIf6%?X2&Y73$6EPvF|q@#b$)J;yML*O(fA@rs%=DM92CAlL? z7G0{-<2V!dFF<91!Cvo&fPzi^^eOiCcwyUev{=;T@wzd8X4`sevfegd<Fnkx#Lm>rpP2yWFApB5<2MF-fTEn zYCKHoz_AScW%fLyhHL7YWk^JQ-A3Rv26izUzQ4Gi;q4!ngVqydX}8-Nxup1GwwxeE z9Lw-#OfRX98pZjgN>WC~Z?uv9Mt4jhSG(5`iB{9qEg3-LwiuAIox@ZeHvH*D$*foa zT)d%w9Dmy9mSTYsHiUw#W&H^0?;D(){;=5NG7%FH8cAT&o8*Y&au53W>IsU!;`{eU z+R9b=8RZ>oq6lWaT88Y>Y8FDTY%k$ITC7kD(5A*^24ErEN=>+&>)^9(xLDus$7D|h z5liMFM0|hmdQUZd_m)CJEqs3*BE^f*N^9!Ax+V6mcy7e^CLjHLVYiFuaP?1exh@;V z)L4NQEhD3a^*F~JtG8XLKyF)TL879=u41xB!>_M(1vrPHL&S#>_HziK6P0VKz1^xG zRH@p_II*;?48ueIoJ2W%nUa3n=+{hKB()*HZPWLoK z;;}BR|C@K&&&YL2^WE2S$r1+^Zem>@b|u#LJz@v&juQhJOEL>bM_NZFuBV@R++N(C zI+Y(Oj)2~9NhjXY`Cb0+XfKc?M1dVbUf9BO@2BwG_jAiijd!e`Ia5Eq?lFBSGU$Y3nL za3i;WNKm-eO{_0+3t=uYcn-hBqei6kqW{dL;&}8Yongv$zQa+&1lYd$OOLr03f>Ke z2ISvW3hqzWyOFOdllv^oV4j?N$gy5<3mV|Ai_EZLFJToqSm8sWR!?9y+%DD+)o1%tXCy52xSe_ya>>#03pE=&xQ`~R*Pc;W zC-{K!w0Zd~Gq>zh16g@Lq68(nuAXsA*eTOwIX*WLtAvy}$+`iZxmnKZWo%aMMnad= zPeecQLBH#T^0$t}5h%~*=}VCASXGy?IsZQ^+jjWU_1>MQ!#-zSw4IxqyxEME?Nd|P z77p39Mz;GCc(1*$5!&g+(BgjVLsQ|D=MGIxT&>G4Bo#>REDrfC18lahdOUmQJR9CC z@yVsE`f=Cmmdi~n?7Yw}0h}!BpW8prq4-6Y%e}gL8B8xV_s?M5=ghFB(dz4e!~SlC zA6B1UD{?e_5O7{?U2bCj=)t}CV#`m@yp!eHuBk0aJ9o%RRRGw1sNDMMgv2MOCyRme zonl;1G#AKpoL^!zs)2eZ9`Rh14iLx80)I7!V!Ubf@*Sz&1~BK-OHQv^FeOLmZ+=G%f2;Su+;pb;?gO~HUAQOziJitd~wDL+0F~E{_qR> zc$kkft9HW4-Kre>`c=ZXo8%QbukPzyuO23>H(mH}v)TJ;PLl6bIF8*5dXy1H>`zQKX&C>Mv z`#hlbQk1BpmF<@-d)2*FHhFPRXP*Q1PFMfh_=h7t#Yzom*>{6oYqq}FY`XgV&9(2s z4cV8qG59SpV{=z9vtf7J``*V=o8!MdsLv~}pmfN5CZCd1MZvPw9Sf|lyxex!=*qd^ zy;W{=lvLE_8Zrtl;&zVkm{Y}@{!x$PG9$Q#1n$s|&FJlr+gDa#|(iVPpkxi zCnjq6R%*@b01hRvoHbjTdVY?Ak2uVh?Gxi8?cHzICjH`U`mjOK&KQ_OIn^3^d^G<> t@43GjxG3N-s2KqrTRGNB0h{qdJ>x&O?87~~{5CTHfv2mV%Q~loCIG38s{8-| literal 0 HcmV?d00001 diff --git a/docs/images/tdx-host-check.png b/docs/images/tdx-host-check.png new file mode 100644 index 0000000000000000000000000000000000000000..063837706e5fad8d51c492b9d239bce41529fee7 GIT binary patch literal 216364 zcmeFZXH=72^EZm3Vn;8U)SD(z!hqdc5(P|#GMqN<3xbo}fr<@th# zs;L*HDaWtRDK)(ZJ5*FB2sK3o13#-x67;!+TGCFST&0DNp^&r%2JB;)*y)L!?3FIV zi^Z>Fn+!mVaw^_yCJr?Epu&Y*#*?LN+2lTv`x3;WTK3%jMsiVOfrX;iD>xKT2HZ+E3|;gL7#ut0Cg& zw0#p128-PiFYeE$^!RBh_U|rq`Yuu3>79c$=$m_Dk4;U3tkb`eMUr4duRO)t((a?G z51%t#>~wP7W}`1ruZ;!|*D7xHwsJR2-Q+>Wa9EFh^QfF+1xAt8KRh>(Wxe`^NX{8U$5|zb|p9Us^(ydn>m)ffmep>W?<7 z0W{CoC=>DTl~GGAo_+b>XDX^|Z{!*Ne13WMwEdsY1Gmn{|JkR!G25RMQ&Cxl{GSf~ zKj6WtB;@#KDX9TyZ)fAE&|zno%LTI^mYWuUhjWU5zejuX4xzX9KW%aUpI zS27wCQ>-pfL_7DXr)-kz(1dHbdtw@Y{@*1cG4V-v`PJ(7Hq6CM;kFUX?!ImMfeUJ{ z_w}Lv$pqp(a!o!F$hn$Z&trVmnhpJRsIb^5L5=s*tH>D>HdW=S&Sybi$yfjFk4oxX zh;X2bm%m;|N5|=?=1=8`uNR;87pXj7ruX`(zf{+pXJ*p42Y+bGPFv_EI zudg$~FErTY<#Ny1;jhxOi})C&XGv*=EIbmEEkR95U;EH4YV%kIR~ab^YYSM=IV5j*D&fc)X1z9bEo#G zl-%`owF}Z{eguEg`XzC0%>#Zx5;h9B`2Zj(=-hrjyI0e2EH2(5VaqA%-#5SNds*%f zFF7Vi>fG!#po&dfTLZsj7Iw}Nw&mC5}E=Eov2zn@KW7V^geLJriT)F z+zz8Sm~WXH6BEOr+f;b-0a8Vm8Nv3>kZ58S3y+z4MhlO*0M~U?UmLY`RKo%}f8J&E z`$3vzF?UZ)-E%(3=GpqJ05%!Dj&pzEt&~uh9f6?%um(V}`?|vHw}?eU+7TmHFYB(g ze2>u!Kn88j+1}ADJDe+8LK}KN3q14h5MCyRA`fE4MUp@m456dqZOW%;#Mu^Fyzd~%dT`5J@4)6t zWOsO>%|!eXZxN<7z^rjm4_C}jsIbm`c>%4_#X$^u;ceFOGWxFojg?DyiIk%AEZtq@ z9|d+_bGZa;iw^snUO8jzN0eAQ37$H4ZmlhfG8ybg5C?^}%F?r-DE9%zQe{KiiM?DQD@g-V3s|>&W3=ut3}>Na(P?OFFB_>;+C;vL)9_#2pkD%ysRW&F}^r zd`->pb~|i!Jb&*(qAB7fI5zXYrl+4WJ*II>-Q7ccciT;s-@n=8V?=ORD1&?Cn*c?q zspd}=M`wlN^e9Rts09m$%W-mT(A;~O6!SuQ|DBPj7f__o!F3nar;&W4;EoUnIyaT` z0h3H6_Xe2fmb)V-^Mi`UMgyRas28aFy>LzVls-4(0*Po+ECgHJ%I?X?QavZ7Je92C~Pvlyq9B|RB)4Z z5R_D6SKvJ;C-f6Y&Oa>sbe^-~y{XReUIFh;(t{;ay?`=EcN#EH;HB?>krYeK@KC?z zFQ6Fxb&X7i{7+>jm0SVvq@J$`Dd6XqpN52GlrHp-$Hs(vfyH9y*WdFAg09o?4wHi? zD+dKSwdz~Stdbl(hx&{}WcmH2i&Rf0>d)N1uWmVX+_~hAjIJZz#hvd-8M1)9uT1Ly zP&isJ`li(FPbC)VxpUgSFww(w_w*A&rvQgJSk=|NJpwguawE4YqS-d|8BgMm-Ox)p z6qVnM^A@u|%B5!+#LLrx{QNnxt0Q#2R%+Z!PF~lUd3=h+wYI_PkqA7;z>U18Kj}N~ z+EQ&{=riH3e8;^3Q6mZ;PcT!P(Q;bb`_tyE^3e$DfHy92-wXxTQZ{KE@05_}$X;JPoiSIME0QF$UI?h%;Nk-e)L>M#_1Qoa>*c-Hh z6Ai4e+iJxHQh6t%OWBb?zkPP$%vx3{3kZQMWNMOheyOvFUPLI-K+pS`)W@az#_IoC z^!opSo!I`>^X$S^NpZy#!(r5L^J8L6I7d~yW_4DvmbUsMEwEk3xq+K|!ne;qdUSj^ z3}q3idMUBHuTN?<1KV_8f^e5=q3ym$k@sT?_e}e2<58|*5j|an;Vc)`YReRGQBOUH zkR#)ky!5*HLJgWky}f)Se*tn-s>g5|rO`F?+2keR9ad$9yIecQ`xkou`+7yfe(!xa zjGzFAn|s5;vPRd7W8u+WRw=5wM#fpES;y?YC95eu2rPXg68^ZnL56$y=ryV?_(>H} zhdHBybDyI1_;2=VJeEst1xH6L_r~k(%JEllqvM8S%VuhO_3d<b{%Krzs{{IwwTUs0J)wy+- z9OsjmS09XMzpQI-S0K{83XQ1!4JB7^%63Mv&eZ^TLxrVqbVuK3^@Sfh@m(#G;vBxyat)nRF?sES+(+Fs^os^clZytE21UAwNWw3V&5{RQ zeMM_FmXZGW@?#HoBaFeD@$k;N+`;CVarkT_=Ssi8*u!z%?z!|heV>!^FdODFx2Gh< zL*bR=Kw%$jp@^wK{E|M@cY7uMjqd-b78D#x-x$-shAV z>*aT1P5cewa~jXAm04Gao+o5}_&lfMgG3|JnWVwDWLFP$xt)1+x!Ds1Vg7NY6?#Kj zx7kl|moR6?5+h@-OM0tgNY}Z>LYHcT0yLRt^jST6M^xv43{oVGWdUS#9KOJe@N(*d zw%ry%pTRx;&fiq(=v|Cw=$*T^ojX(#C>%)E^w+1IS?@gB7Y@{pGJ9F2+r82BHKD}0 z$`g~=do<~s1q6Q|NjYN+I+rIWW?w)KaK#7J^OeoUmCOZnPtCW_Pwo)|qf8rCNJ?|< ztU*??ck*eds=HUU1-f2I(Fb=_-?Qu;q4sXcDV6O{>s1r*HR>yLj!h@O4*0?2*XwIV zgLwQssnwMBHb9>rA}_$ywn_}+EkhBvkt}{p30Ky1p=$|6nJc5V730laUaDJtM#sTp zt+m??K@wJy3mOunVy&iX{M5lCSq&tSf!%^W)le2?ujepa?Io#FvCv#4QcedTZZ*xn!VZ+?%$+;SI&KR~ly)>6Zmv$XMD z*YP#XY}ZY|Cu%wmYkySl-9{M_3Xb_34Ar3^w^vc*;`5d5NmA$d;P>ZpI`jv%8?OEXBg|!d*zERL zeLSh5(v87v>_}JCGx#vB6J}ee>X3-std9B{j^QwcP65`7SQ+hT^2;?I!@%uFA4r*5xUr+h1 z-lER4+gf;caQ$U;55)tLMVy$b)i!>~uxq|Naz4nv+>zS8m3K}=yDQiKbFvx6`;N88 zI&~RkPPUFl4jfrPDghqL9Ww>;mIKqR_v+-%w}K{?-Rta}fq)UgMvdO#m>xW?^{u`? z4%?xfW-_2G%^X+5f-cf%p0LV zqIYaL*>66>I?Dy79%3xzLrBK1jeh`p-$<)y~q(CRpjdtaJ6TfbaW zv(Fj#gPbu;sGB%a(;*j(Yd0BaDVqMQIxX4~GvB8a0Jpjwt|-INp_-#aqP~dm*y8tT>Kbi_ z7F!%!b(MEy#qBz|oZ#kTg<5^+y&q=wG;?(jjvo?m@Vdjof?=W-NzHD-r zOJ=UWmcs080GU6#n!i!3oxJ)(dEXO611UHTT=I1)bu!Ir*JXY_(<50Hx*k>aazpz@ zNfGT5mDk9C`^j&wpxQz|gregaw$VEZBK99ITh!k>#9iiX?@ST@i+(<@%uJ5stv+jf zsNGiYmO^yIcDrl$r1X~-Q;t%2Qs>>Y!TTe5j_ z@}l61ZDQdge{=$A!D5P*j>oUFpK~?;|O_2-i^FZ(&Z(eDjXH^#PQM27(#_?*XB%G+`wbHCr z0O-yEK~r*!SQ|(0HAoX1n@=F$g#r!4YJKfj59k_X+!C#StBLBddoeUN9eU?-?8OmR z?H)+T9Jg$D+_`b7xkDQyS$-YpxE6X!64!3gUdVH z;+iZVW6ja9Pb_?CqCmq>FO=-%4H5dD&$iyqIBis(!%oam5GIlW?6eY+fgG>}!)&>n zfad#&p!Zfj+$2H)!W3#|m8QAnjNZ_GQym3!_OH#BYx3Yv*Fr;`)QK~WB=f*2U}NW@ zf+PNhm!;@Keb!HG9|O}xh1g@Rlsr7>kE^lm`;LyUx_;}PO6T7cVeTpMJD%d^A* zN_|GtwbrzyN{%hjdBc0#{^e_D8xfh5d8ZznbE<BTtcBaVa1O5=Qz^_V(n4#i0oR{msl1+tE$-HmeT@m)0oo(t$4}lG zm`!da%uVL~jiab){6mDvo0lEWp7}X^LgN6<&#Sg*YXW$-0>{kV*63K zSY@BWd@uMqX}Qe~h1@?}tP(59hU0e&Z%r6DPh$%KTS*19KAg(qh&7ALSH~J6$DoX( zYEmhHry4R}06}Y{CX!*AhF?~%FiXwN4Ebftjh zpeyQaYds4SQv!EQ@OH;D(3;lhCC`O1gNFdAU^!M>b@H^NRz^PTFc9B)z)GuVlQM{OP?%( zFof^VF^LZBK039JiK-lic?K=%7q1kqF4%7`mKMlN4>SQTZ2O8Fx7Fm6T7HbyP2sq% zLNl}k;W}>SPIGyC2*ON!Zj`YGKKKBl;ZHpxdQN}1G(CJHstiKXXp*aUawe%&0mA(u zYt?(c0<4`ReqbI{LmzupmSo077ANJOT`Fs8BP_TwHN23)p=Vy z9T1(zza>=%b|GHZZKdx+IIkp!m-rT%M8szzUF1aB=0v-12#Yj->h15o0-auevCxrX zp)*aZzt#^D0kzRuoL^GOD^s`n68bHDDtIq%N1I3Dsix$K_@-sG`5~Bt{x0jCXnmBE8IB30*%>P zaP9Q?cx+AWtO4Xc1_*Gf1&tqE*a!vC**f5_76G(sR?on?|B-}S>iclpw= zw9${o;>h76agIYEF)%Jk&PwQry4!44q0V5#fezPcTnEXupy(rtTAEG&N!0Q0X@-c; z9}np+l{F-45A8dulag2Pc~)%B=ukGP5~wpLKnM*LTz41|_En4DE2WO`F8^_bG;j)3 zsBL_72tLPDKhVm6*DC{XdXzK=d!;zJH5D9Bcc3SJ ze;N{xqDrG+M|WY~epxkMqddpyFuTBocNVF^hDLXbv#U#n+$@4M!E^2@w|_?^#=MSy zz6fT#Pemn9j96YG&u}-06T$oK+Rv$5TzXSdV)+|<$SxZWdP7y+egK(hni7ndUUY_> zANs!KHQZ}Ig&fU+qV%ysm4M|e8Ux|KyIpk(VfeMJKJ~Sa$7?;bpq9TRH0PZC|x=VK>XV@JaV(^olw+uaV_G~Lf}>(>>2_dxY)TfU9u5t@D7-o8~OMXLGyufMeZdZpn}1{4z; zSOrL!aRo1|T8;e9QD|$I&-ksf|3h=*X`tY5Rq^YK4Bl%GssHnz|Mllp>Hqjah%nWF zp0u#{QGrF;R0Z;n)>2A;N4tq5r4b+qPad(**EX=pcq_mn-*}NKFr5{VSS^r-Ys1Nz7d$l1m*F{15M3EV4KRiRd`2|AK>Rdi4S39uw5g# z72)a^iWtylP5-Hy2|^ZONPU^f#~}5~Cf!Anq4OY!*`*Zw2LBt%%KJ(Th@cK|*sgC4 zT*9Kw{k9hX@*c?h{4oLiZ zZ?c~FKB#qwH$iP_vGg*MWYO1*(<2a$?@K?JW-ROC`fz1p~&%vOSQ zI#UcTOzIWa=HiaX(T;{SaheC<8;P0^kihPY&^1U@^D-KPVBh9i%841E>>*IFAwqmD zt01`bxTN`>?A|DsN)h_2DM|S1(&#d;E}0nxs3-L+C9Emo?j~q^1sVF68o*k%7k}!* zrAd0V+pV0eqiQQJ3IG*HOKaP<)z0(oCYS~3%O7hq9pg;1j7B|=xcm2xu_CJwsd<6Qg*pfC4;e2bep-}*W^~JB?^m%hUuAOWJbtjw8u>m&Xj*hSuz;XU zYzmmuz8!j`7B_30Fjur))aaZpx_($zfHQWLm_rOh&kcZ-)D2GP(x40R4kf|2MnQ^) zSJYn<3M6ajIkW-*A6=L}TDbMkA?vq)iQ#{)sRgS3hgG~^7Wy^X+VRhhDm@qNe!?x% zhC`n$Dag8qe;Dq!+w$~>U$Qms00i;SYj2-9XI~8)63-b16q!wRGs-lWGz9CG&Bc6J z56&Job^*2ctj<$>ks90d{DG4v*igWE#M*(jeRE$AHCJbETUuY|+M$z6ou$pnT+6X6 zzyLaI??l7VyBmCe+&LldQXV0$RT0%>2bGTSMoPCo@#UN?SR%ktdD*JuC@VS`73-U)k1d<9vL^a%I!AXD9ihm?VjUX0 z@18_5kHx1YSsmftlIsHIv|34@nGQdGxh10X+Y-EL@p*toCWnklyP+tW<8)G&hRhaU=Yd)BDI zfstd{@hB9R@5W^T*Jiyw=$B>_r{n;St@Y8XCg<3E$aINFh=SY;SaFmN?C+CUf=tPv==-}-O0O!@brMY zD_o8z-)SzAw<>x=3C2j7I~o&K(w-GJR|{CYe3Ty>YKw1;lnr}J9vZKuZO`m4Jajx6 zuqPQF6VeZklZ{Rit0B1#qdqqWqBbk036_puteDjrLQmvh#^h*q*orTBOtt#}(|jy^S|sVhfX)X911x zX(X|_u4?!X_%E!poa=t5HtlW^zW&9h?6{9Sp*n+wQGoM-=(oP8z>}T&`IhXxy75}? z@|EJN7t6IDL>D8~w}W7OfL#tOYhwx0v*=_C%R5W%l?4>&jT?YmH&}fZ!yq{t-13F+ z$&7!7Y9{4wSI!Y&7rbjIedWaKsNCi*84%2V9QYt1m&v+{U4J|s^=(*sXl3>X>%DlZ z#UH35F!v4jhErHX05&qE^{1l1c$fB4o1x7u|4HA%y=^P985&jfI*_Dh{GVvWMxL|3 zSVFoS;MD@+a%#cVtc~-xmf{TZM#8E$9EQl5nYWq4eg-TRA|oP#gAN7<2QEKQtFJBd zN!?y|%Gxxyn&a7 zflyxRKC^E%GRfUhbwS+$WqejRn%(qnzrZ|}=gHN?xcX^9=CXGOUjW6BK0_d3g}SFE z7!lm+)f-g=CWJ{??9XXHq%x7}O~IoV9~9PL`&_qM_Kqx5(_x;3d9StMb{WR7Hxj=? zvs6<0``HNQ5|!3t)a9JN&ea@CL&MU#mkM^ik9NmN3(MgQa!-edlI0`Q^q}?ee#Tmn zfH+!hl`B1)T_Z~l!Ymh+Clj8XXsuqPMau9^Lm;PpBhr~5oL+f2`MrBb>%h1$aOy9_ zNT1&Q?SrYfeV}Ug^rut<7g!EE6dS>an&?BO>=;&fBTj|o|`{QEPgvq8x;xMju z6qrQOa?|VS_Y%bSzhYE%P8K{$-P;$;W70o7^`3Iq2q%U;&}xlFkEc_j*V@we+ju^b zd9eGbUV*-&@5z}gEO~W>`~ag8-|1f%WpiIj?>zUS>o^bDuWMIe+n#wg)Px-YuU-NWz%T7SP4vh zPy1+WhX|LCGs^<|sm}jE=}LR`Yvl(WlV%(HN(@YR6V zp>78jtnaZG8R8I0v1s77xc%bRtz-!jO`Yg%)O%P8=Ihh`a}Lz)r4g8i~*9}1~mAkh8h zU8AbA%5`ZyYpLVI)1kl(m8I-ScF#H+Yx`ebiB+)X?NIhWTG~KI@dLI5?Y!h*%iB;_ zDD5D%EqEI9D$<=8?UL^bt-j&c7f`=f*kgq?+yf=U|Yj+35ZqH-V7DT)8ABhys=d$>r=S7;aLFa-2vao&!@jmac=Q< z48|O{??63GlHPwAv_MyJ9O>flbkeV64~p^M9c~cn*A@m3_1?bqY3?*9;oY%%+2pz` zx%#m>J;M9u_^IuMv_)@Ka+6Rbma?t44OlBPSeMRl0l3NOZ-5UkWqy~#F_gEhMvrsqB9!}exqVfHhbwI@dJ(JWis1% zS#syhx0RulD?F^&W4JpN+t4qMGK=t7Bo0=KJbm(ef>*W9Pzo1@Y9NZg@EhJID)`By z+yh46DAZ=Fspkc%b=}~p(ed-gzr9pz)?XRb^$ll#M*{xSkwn`k-9Pg4|DR>MD_m5z z13RiEf|WP7f0Og;9$2>Wb9bQ$zd~}fQ#SH4jiKaZX$zopgB$tS>42An#{~7dqPmZh zf(cYz5gJ`T8-%(GgZFQK(5v@X!rD>_AUyBD@4&n!iw|gq>D^GMJu}4yQCmf@kykAL zqCH>2Y2(h9%drs&J{GKGN4SL>)~Y&^pO$E@Q(+RFO!0Jrx=^^9Shc3CA)_q^EEYCl z8v{3qeUtxDMYjwtkmjWhz<-nGuIrRPQfv-GLPa;8f}bSb?)cg)!=qV#d?}Vyzq2FI z>Jqu+VbO4$jTH;>IIe1&e<@gu$P*Iu9KsTM#LEaby%p^D+MX51^jQOxY#3bPyCXa5 zTGt~Qe4oK^Tc99d1^H%*TsQ?Ws42?=Ns}W9nGZq5eB!BjFAwV=saHL*b12s#-0_{d zf3xK>=&&aY_G3nNHTh&LYaEb{Vc$X?Jg?`d6THNB<#bd}*Aszj53+HGK(V@OLUA7$ zM$jM6o;xL)3gP3r#Gig-uesdWF?1<*kUBn-c@9Y+R>hi|#7cy<6*pN#N9V~U4I;nuCB(4#&Knnj#}cL&wFauJ7xP-G zF}49_dqro17mmp>47H(O&UR~0lXRl`QZ+eM`*rmC_9tFq&LpTJ?fNwh&bY2l(w%nf z9I4>>M4ixUIOYZVh7YA>tgdd5G2N|e4oJAYYlgc!et>n0n)G-8q-J2as(kyv!M+|$ zr99|N->G%xMnp5aDKV(JV=0PMU}9oO(#+VtT+<0J#IGh{K3;t(RCRyNR9EAqjOf`! zDr|kwlR~>bYU{hU25n?x$y`@^IYPa2@CT?Ub3|#_p1yhCY>&Al&1%CM6af)dD>xZ#rDk)l{Ith z>0;w{{zF$6$F58bj8r5cZK$0i)gD2z7pf7m>-M?^&+O*H0q0rISMq#%0wbRyxmT-1 zz0WjtP{s^y&=rhN%?)xG1gE8pqv_5{r$}K3MO%uj%QKu*mn~m{-0DE?xrm;>EbeX? zYwoCTtiPC|OeA#xW-fg(g0Ag_`z<~14t;SZT0hpMx0ff$}XCxZ!C3KvY4rTJmBu|gS{uypc^O|PJ0$n-Y-hwVQEHtg4;=K@U=_O; z-~-gJ*X!ojZ-tvBaEVkrxYSU55Xc^xNZ?2|&u91^({lk{pW`s}Q2m*x;L&lRIGr+d zKIW;2HOR}_eV@@k=fsd`gL$UkDx{t7$)M)l_A_~&-Ty$9lAE$l3RPlU`}!9jf5hbE z^~aWEAZK}&8dkcs_wMnIDZ28hc-eU|F@cI-u*}Ran z3R>?~U2p}BGB9&5^v-8a?GUU!=DMTy7C%L3Cc=t^(LEZqr80AHA=eJ_dQE4=S-Q>f zi`=z`E!XR?1dCx6#&Sk(mVXpL zbW$kY!)zCUxyNWVyH2|vMmoW+-qnW;?nf6&>ZSN?ZFyQK1q8h=d?Yg8osx{b*x-0r zMYkFpWEEFaV-HVHkFOXRd%frIkXz7KH*J>P0sPD~DFz;X-4VOU1BSmp@+Z|hzV^P` zRTz5k&>anxa)+&zhg?udMhiPe4Ng?g!YIau*k90Tw4DMI}By| z`1XUmLp21N=A6Tq%2-|mEdYOC?4eJfhKqipd!g_1dY1&T`d45riyhDE1Ckfj+7qb^^$JotL;uTU6J3}d*nKN)82au z+9xCRjJu`htK_f`Now?!GeMapcws0}*^V<0)8bL(8p2e9nX9f|IsP5Hrh292nQ_`) zk&5h19FyMokgt*2}*xZ zikRe0ft9LenZ!rd!y%L!Bdo@i;Q+BPLuqqGsFCL-9F6I47OYK8-#pSi?!DAc`v75^ zw-Mj@?heONvf^2bfOOu8G`iCldm66?Z5kFQ0s}R`nlJf)Clh|NQYn%zdWhn8HY*$4 zrrRp$YS1I+=2toF?)HDbo_!oz#XcHb4ZQ%Xo(4`~TU-x{Ae^S5Fb{6$jC_|;?Rvm&D|KXIo4hWk%GS#9foBU<3StBt~jKZ%T zmVEm9Yfo2L=GNukcX#{ma!uyD>W?jBE7mFIo4LZxVC4}oZ=~o814G}sly3YJ59|Zz z5rP~56o<<*1#ezePf_6y9J5Mt*>*)Zcw9IWq9OiX-b-`)`zI@NZ5qGTR>ebh{84#? zM#_Gc-J-hmX(PRvIiIzk`|SVlIcY2*MA%%({Q!OW@s@P54X6JqtV@kWnCok@H2au_ z*wT^%-oaFHdakoBv3E~Tbn*v@&Sc=K*muP84R!dl-FZ8MWQJjML3|?8tQ-g$1(`i} zwW1NNdEU({JFUD%+K^4mncdZ)DQd}SyL5UGc&A`%^jyfu71tZ{QlrQSXLB*}uu z($QB^Hd*7|=#Y5EgNHfI9yzP46PtXmE+75Q5T#(syAKyWh!2=Z?|jytc5?>5|IpR; zs>fq&ruSqu*(5Zcyva+WOVRP}#U=6J1O|tqrps3x1L7+L=lz{J0PxcD@X6rkzI{>eo!=)t{iA(| zyH9xA0p3@TdGTN7$%}8GiWp=v-$1EmlU@^<)_l;d`(i z{v&K1{*7OzuoYQFo%*<4=GN(ruMw20x(F@Ug{u!7?CfmRJOk{iY@o??2(IezJ6*1{ z;|q*>NxRu3k$}fWX+JWSH@PiQO#6`TDfy5}&mH-DtX6oG%w8@(1g& zqb@E_;?3iZn*F!Q6&8+aS5mp$Q>vq3M*~tqq}b85+bCio4AV6kvS_nJ7V8CZswRvN zMfCK1>QwX}k+UrA6t}jUiO)0!dnYbT@+PCjU86@D-i0_DVh{`FyCzBb49`8-@@=;a z8+d%y7z~|4CMsTu3ei2bM;uLL_dPu6UE=jw&h&)da@l?pbReu3kQn0uwE@o;tg}u{ zuo9{qa1Q9Bp(yzG3Ux^UJlx9(owQEp*)I233Q)QhI*$VsLR%_%IE^|4VQsrDrtGU6 zAreQ9VCbxYK;drXfXZz&fm+hgsCv)M0$wVXFgPER~8#Ee_pq&Bk^U0*Z*j7|^eCl}jQ;a_|v zmt2{NbcX{Tl_kIg*B&@m+GMfWMc`mhUU!z-Ei#L|1O2=&ETOm$+me*1`c-3^8^o^vy`53Q6l3XZA$-#Px+s{;eT2T-1;94{&ywT zSHmf?=4fbSOsW2SLb{y@H`ycCKGr>==M}jJB-DrwpQ%Mkv9se%*?-n?a3g~h!JA^v za^TPAo(^-;<^5nv2KKQ!j5(;C7>A{G%6aoK@7Mu23>P|EO1 zH{M2faUxv)b&T%==kzF}=;Mh`5lXVl<1Tmq_%T&Rg?xsF?!`A*k?{p4ECrE0<)0+r zuLDZRbsz%G0dkNR@f8(nrug*d zW6~m((dZ=W`%zPe6=zLLBEf3Q{|oGz20yML&BT8F=;c%*7XMF-g+hZo-RBTtsfiz8 z1r^kUDQKPHIvtmOd{(6QL+3PPSDs4^u-|;5f1Ju+qc!o)h$s$^ zv|nRoENl5IHuhh@&(p`VZURhho^Q{s1(B4jBx{S=Pey`pq7I~~Ui%8aB zi@f?;a`gqvN9qtgJwcs+c6KR!WB`@Up?;nc6ou)aE`OlsVkZgLmh%)9k!=+^HurIf zx6P8rMVfnnz@c&8{8cbdZez<`p!byQL}Z&3YK8TEB+0Z zV6iByzFzTIS7ac2R18+uRY=d9y{B*HV?iTiR1 zG2w`_43y&}%)Z&^vdlK$=XFl5bWEQHP(qxdT$4|%I+n*bz3hJ-I*~u|V}89_gOU{^ zi#@|GQWT4Re{;|FhVBfr{OB$mD4fX8x*r_atOB+(r1yF8>%0o{5wVRr*ulhM!hgzL zv0r}V@v1^xtVRSi$Ld<9KsjFkVWNd9H9D+?_7wYO(Pv*ZwC$U`o|K_e&&`W=W`=4G zOs?dReB}0sb7mUInF9X2xYEO{Jw1J6yvppM|AXLRU;7UgVaL3Tpbxkb0xchkG4#@LK1+`#^f||ip5xx@9iD5*iMB5TT34Z9n zEYlZneE@Y3ochRx^K!4_Q#Nq+@+ z`Byfl7rjtkuW+}2`z=E4_5;@ZxgM603t~&jDug-68tYElJFyIM=}GVb1)e7}sgBD- zjc_k;ccYVgzH&%shcE;&Ww++Gm0_I>vE9xrz0|UDz$Z-hN|jEI zNL&pN6q)BoRyH=W!2~S#SS}S6RW|07zDYiR`uy@YnJm<>Tw_2+YoJ_IwYDPjKI@q` zf6KJ$YFppKs)h#6{W#m$!`5&8C`q^Los>Ou_Is$T1(!+PLt8GVLqRxQXEXG}x3M0s z(CSZc`to#pyNNy@JxYLY&n#njW+uL7$scY3)~RuHm1G@cq3ixtn=BcVY~0NDy1QF# zI}@7L%I)d)OwCEFaKvcVE9gk(w<$<1_&cZ`j1o9IP`3R&CBBd`nX<@4VYj$TQ3s2j ze#$h#<5&bGNM6#f6aRT#)6~@JQqF#zJG3M-nl)}f^CLVW;&K$_&=ZfNu&kSx#FyTy zo5*(LU>ufX?Rq~o?ylx)h}G*0{|aC6czqjPd+Y1%pv-A^oYESIY!>-uqu^#`8dK9t zDEp|Cu=2H@r@uH=;i=Gcy76?K_D?31gB{rL;NAhs&-n>+l~AB@la$p_w?_llwHfmJ zo-PjDhwj)?T4TFbqlS(`zy&{QOIw6+#0~Fek-}1tUT0kjb3%GD&&jPftUUeJlcUdh z)zEWjZF^IQh*guURQQ~gUJ2m;V(&bon#{VsuY<}s791UA5Fsim3IYl$0z$+>FVdwY zHb6j#fV4mol^IbHP^#3R5Re+Bg_0toAT>yp1PBTtKnRgQ64Kv`duHzEe(n$N_jj%L z`T%Q%#TCLi*FO9F&))wZGHW+yrfR1DohIKrot}3b9KzSXj>3GZAh(FygLO7jW8f?Y zjCYTvD+ssS0qX?Lt+;#2Ns>>Q~E$|`cn?5C}Qcf%)t z&vJJpGu_)W*xpR`Lfdse2O@sCvjTpEUHi_=jep* zNS=21wxuSB9mVVF$>=T`XgHU++CD;%o%}u|JCfvfe47w$D4Ik{G87H`I=`gR?i=!W z{10XG@jp@RmzE!U##kNACu%OWi{09Z+thI+3!JG@#FFe zOCYl1RSQ+)eg$p<{Q4}$mb2+-O0OPKMt$k9KA~fQJA1Szgg?V7klTFnOhJS7)14SSv&OmuTg_1kJZ1($4; zhB^l3MUswZ_HHC~z>>mZEDNP~c7Muon)$jptXtxfF^K93&{Z;U2+UMjPW6>Ne`E63 z5PMRlHmB=ZjC4fZ@#15b<018w*`feuLgwqKNgRW0fU$r>k4AcSNkajiKHRx_^2&2 z6wF~)zk?0|3kaEB9@8{rPQ_im5d$l8Wx3F-gh3o5;Glul)4; z?0@5YlLi2eE#XS#!Yuu)AFJ)~P-J7kquq*unsn3tKZk&Gt$(S=N>9dmB64ZFazUDd zTT(0~;)&kn7l3T;gy;1WrDu|^8!unH^*jr7-uF&b`OO!IcImKL={{(t5dD^bv-!w) zQN+4T&UoFQ73?YC+uWyxX3_HK;1$8U)8!j!jU%sBQ@z@Nssgn<}`{mn%Jf zDNgLKf?;^$IybjSL-%a#7&%7Ub!N5IsBCoCM40VBT#>W@GmPGiC_x2ZAKH?wjvr9>RN<(gfgE!y0W4sXkxjQ!2}JM-`e$FjskeGzTM zTK#*;)anP6QQu+t7f8Z0bNcub1DHkmMqORqlWNx1{z0@)#T7Z8DumKHTS$iUgzv2z zMRN|Vj?sIzFkk6}GsASy9~9@dM_%gL@lC;S18Hk8(1E9@FSGolC38eP{!FGQ&UA4J z&x0j%z~Ng>cgh3|2Id~u%i8rl-R#}bbj*tgM3SaIBBC6&$RcR5PJVAV^;*7s_@Ue_ z(IumKDg}>nBNodU%>*jUCqC=CO4=-aIjEO@%4nC{bJf~fy^T?k->{U?$3kxZ?$a5< z(48a`btCZJ(J=acz=+HBxlI#;U$iqXd1q~VZP5^f0+(Dl{#j8|(=oQF*FKqeJGe2+ z)q=hn;VX%xQy{7?mnO^z#N!gO+8i4COA4+A?Q9b8iFh_I`Gyq*IhAQ2GhY*kV5Tu$ z{+go&Lrwdy6vNHrVZ-reNqNtwp_WOXyXPNHw^j29YNzTDkvUzdz5%TM%r=Pz%w1_9 zuz$o${(vg!oF=IOTltAT@Vfe0j%HadqWoJ&xuH z&v2i37B?F zWQ|`~<F_wRN05B>mjRgXENuEOI0BcYr((YgKSNX?W;390-D)jC_^ z^6siP4tIw8>|petr#;f@k4mdALJfy;DMp-cfGIk@>%1EX-&V*;tx zA2H0R+r(S>XEya1tv=Aj_$YKnRtufd6*)7(A3wL>lF)a|ci80Cu-%`F?_Jv#BRLWF zfW6tcj#ordC3OPzwhzX@XC?+Gi=g@KD-^wx_W}eTdr%Z^dwQUtq;i0hdIp6$oRf|` z_Be8?5?6bf1n=xU;=m7P6VhuN`g>W^j2ps2rLsIj6y_DvXElG!LKWtbKVC!o2;I>lM z+0670Jw75_+$*Pfly|sEGB9}zx$tJK@A=95YMJmtCll=7dkFdQ2MTFQ(e0wAS08q% zt`1n6SEA`lORIup#3~5-AUa29158yAK`nOt=!f3m>>CxRO!;A*5mVXR;koQta z32=Vven3btjFw&Vee&h$#+tzwko?!D5-#`6RCk+KWWGBCM$dCjyz`a2{l3m5QyBk} z(Z2sL4<(!&MRGIq(C}5*7BbuNQxIwV$dEJuFwRSEyJf21MDOUkPuPA*S=sZ~tj0s5 z=7Q|5p@q5(FZ8|hlH{G-=#QDMF{6rTbH{IYO+xz?ncM7t;buwyuH`$i{Z4_+5xs+Y z*5x;d>RO+BCTFQY4A|0s$1cNnMN;P@4Bw?^eVS;)&2*p3T!>l?H2y6F*{AJ zn(P+5o7$tbheRm);OX0;q4n;!mmH|1u5M~OkSaq^=?y0BcXXamW4c+m8e`uh8Mh8( zrg{axIviPoGYe#*s-*{j5*>pulP`Loy2&xt$Nc?|mpGFjn@luK+`4euG>@HeN9N>! zvT`&e8Rdi2Dto$W-SQ>65v96_PDZ486{9dtjV^V45=OwC0c%AYl+1fHs?4@OE<;;e zZ<*K8r(2bnbsSxk*!K%T%TaarM|uNeRV|wdwad#E42w_`E0Uyb;0_8RiwcLy{%+K{ z#Qe(xG8)|veXMQ4V#g#@8ydSKGKGMBA$`U1*q4=B%8Q$u=flf?EtI@k$=dAh@q5Bp z#H%aM@3<0(4H^U7Vg`>^a`pK`CH?2$6*a5zu62Ms5J1|o{bvT)FcI5esN(UH+6zOG zR~r)k5028FJF0$oJz*4#2p)tu zf4_^crc6u)L&H9kt|w*dztouobL>EuSY{(Nw7pj#i(o$OblnD z#-?_AR+H7VS(s|15Ua)V^D(yVhq`D4g~71cfU>CMkBq_wt|=yr2zlylVms&u&6~oU zcA{B%M8y6TJ?<02ulc2Yk>mU+uC+-X`rT`gNwR17)SY6+IjnhM5wutSy!JfOFS2Fq zom0si&`XQrH;^JfVi_m)RP3QAkY0)K}@=3a}*XqPlrE1zB_xkB8 z(j(rQT|`50{c#ut#`4sPh_Q?##JRC<-ky2cZ=;u7MAy{$G)_A!NzW)U8M&?FvKEw* zn-P_^tdKCl)48d^h~-MkZOz(RsS!^>jc)YCt)w-M9=JT;zyt}~DQD!8^$Q=lF$$R% zdiE=6*~P47^}WfbgVv1OmW6k;PXwAO_p+15dd=l6vL%)3z1;k|#v@(J^N1trgm!Uk zdEBl3?{Gx_B}sA&K6`L>*`kbv**!wlGA0#OWGy?)v-FeptT47ExsCA1e=XGK|ByQY z7~K|l2TWWaJLnJlpM#0>kw$Oz{?{SSI@A9?)cNmLx4ysRFulL-{nyvK*G~R#C*7u> zfCjk#xMqI=ZhLY;dhKe2a6yN=^2z2u4=E^B{AAf`vP@YNrl_3ZhvKrQKapZJ;z6-D z(Q>h^95D@&rl_9O`Azl%glRYNWEJ68vC=>RkGvN{W#0xiw6!1xw00Mlb!}?(b&iAV zOV1nG-wK|^Kjw0xua?nEZebi&WIa6{jW`sKV-373RZvAC93V)93q&MnLR1+Tfr;8= zqE3Gd17RTK^E!ktYtL&gu;|h`$$8`5rV#f2~oddp=$ze}>VQS;Lv9Toifo ze{{X}$cCbZ2X=i`99e0z z994?8Cn91jAQV41l5_wo9%qEEvXvJi(Hi7$pr>N#Na(E~LWD}+TrVUokNpx#V9`7( z1kpF$hARC~RU9WPl2$yC=p-}NsC^CgmkLoRZ+3Xj4ArO|HxSPATTSJBZwGZQvoo2K z(kAIBl#9|Kvu{tXgV?_wSXiFg&4*Ldr?}Bxc@c{k889jqAZwlIVfri$PmcenVYqIt z>70Ps#W@)34T%dBO3t$e&^bCLNrRH-zv+*gK6i~Vc+4$H-P#q#UEgeeW;z|Q^r~rK z{$h<_6h?=Z4sh&|*Z*-9(eg>Dy z^PX6=au#%SI(zv;fg~CXozSPRZ69tZI*gwQlIdX0gQ2KK>9mc3g&9?1hhD%V9P*`6 zUrh!qN>oqH)zOSrh0X{2tL7wuIuh{H-09$1uOo}~x-byROYFi$nB`79 z7JufGKMKF+N;ibuMI;yx@ayqEgcI8ggnCZ$+_+}yz{ME(x3Z6fi3oZWf=K?d_GyZe z&np!i;jVu+dh71~1>C6i_G}46nwNp%$lXiYz?I2^PeHR$2=$4T+0*@O@#u3tjGT%7 zBg1`OAleb#_YiDd0E;KaNbXXj=x zT3E6UWsD9h*>#$cj&*_0rRIU(*$-brh5pD(8z-jn{H}pSlA8u;!j~#6AJn&~^?m!Q zaNdm&^qTO$ZDk)ADPFh1GbiB(j8vnA6)zON3`{8-5wBj1UeZ~>Qo!Rc+|-I|wpKa_ zlczWf3;wXD<_LM_D9MUeE0!EsNse0)=$Hw_EpCjG!yP`rn5c!z1yx zYq8aY+t83Gx_flxQZjp}st|DKx-J#jb@ooil(9A@VJS%RNTf4ZHC*b*-nA@N{;eG1 zQHpDS9@_ZHsJ?DsWC3-GD(^z?$|;l7ejagiQF?k##7y8+yMlfAJoII-LXv!N@f+wxB9h3iAV%rCp!s93Vh&zLz30bI%n4xf+ zk1{p8}e*q})qI2v_t9*Gw4X#vQ8*k{qbUpL9ZLcMgy!aiy*Vou9Pffn0ggdEPu@bYA>Y z=q25Gk?JC<4)e9kHycC+F`fzfDp*9xt8=0d0Y8(5jU2Ig);vIcE=wv`2x2NIr=v>v z3YF;b{r)ImgKF=|lw+0G(MRDJ&I*dpsp{mAX>ZBS7F_Y`Dy`n$jvAb15>^|{X0pt6 zc2CDj!=J-rDt>F&Qp*c>I{$2?8RaNiC$PRN>1#oN(@$YIOj!M?`<`x%y6H0_N(eZ z1?&mMZGpU{gOb_Bcrh)xNb00s=BiTrMRXSfIHPI>N$~atQwzeX7@*E?p1$t71HRog zO)9ewZujJ9m1XVEDi-3*b`I*dGanockGthK4vlgm3Mqa7sL+9WFH zN|!*gCDcvSM7vZkTF@#>#K%g&mh3i^PP{Y~(Nu-XxJZ&XZU^@zmsIPCVYm2bVv3t; z>`fV@UPN*Rd97?@F3Ga?HHEoDngwinI##AWXRrjj#o&Td#{HB-S8=jCJ3Xa=+a)M) zW~=O`M5pc{TsjNRgvrtcZxORhPZz4~2Q68dm$a3^>1;yYu^<;{j|3TTY82?RvM$ub zLh83{VlH}|jXw}i#A{$7GWPUiyJfOE9W%jauUL~uTBf2LBcyykXGnMv!hnGDCV)_b zYQ_rEo72}|-#H5+@aJXO5Ff)pt&+MlxsU;UZL;hFM#PaFF%Kvf`Hf041JF>B$97q~ z?9Kdr3Sah})-fPs$pG7>C`Y&*@&_uT#7}DNvI+R49uDd&>)y2wJ94M(t<9 zpe0W1B}YPg7dsa0tbKqcp>-rop$-|#NU}(4O2#WrsxqH=Y9Q{Kxto;gI1w(16@XN><_^Ld>}dYz1k z(!}C9NjUq^&)~)D&1y2c{Bjbg%-x)tDzht9m}96{ga{%dWPL}0TL>9VDY8#S5E^y4 z2oE*mNWBtV&`#wFL${$ZRN~$;r+gYwfY7mQNNQSurHIR!Lns0EvUIT;FO&)Rp&#uT znJ=2ODF{q;Zz=ytv~=5(aH4gyu`5-gGm!}kClaNo&o9hG1%uY8H*PBCh{Q-S)S3)N zX~j#;Q{`1&au&Leu^af8(uYu-Wy5QoHDzdBD+#rYgsqB4f)pvls$I(S=OkDU8+5 zW;*Rec(=%Nscb&Z#EC6SEGw5eq_a`929@^RBHbnQZiglRBz0W~A8_Y_tX7WjJ z^K||QheHx#>3`0r;*CDZ(6Tt&mw#BxfL#>O1e7!)XXTSMEHio4D8(N@R4niy0^ys5 zt)*daC<+qyq7{>tCPtL;&7yHgf0=BKAAdkgb}r|T%;R6{LeV%gB?LlGepNbiuxmL`WPPPylE`{hHU;lojGtenmedlCs~xW6UHybgY*K|JhsFEH zt)OI$`@SIZ0u;&Q?hcSt*u?LPT;Rn9b=LT-+!Mp8b-ss}xRO%*s^SnXdr_L8|3MG# zQVPvCRv_wvC{~Eowau9Z7{y{RiWa_-_r`;q@kA%e%@4fYvgHfayxlP(+ZBb05ZiB6 z^Ys?Z(y!95f$b>&%;78a@S@A~^a@l$>pP+WM8WwfV!$>)#5}UZeb(>5!VFYl4X7-K8#>RPT~SyB zy@yDn1=d=GeE;0J^W1pM#H4)f9?KNjsmt5?%B;tbYG10-= zt(K=4oP`>vX%x_jpimJqaLkt^ULUM?dwiMeRt8CvCvYblqU~5Ec^*)Oq%JlorB(LZ z(Os~+V>T@w@ZRl7AgKo-8QidFTtaK+h966+ozHPQ(%FUUio1bnSJLAK8wmt+0%2EQ zrfT9xtjvraQURL0HmZXt%eemYZD5~O`LB_6kb1Q96izO)i=RlxUi~$)c+zD{93pGo zgHG`*+}U7vM5$PXZN=`=(zWg2K9fO^+Xu%^=uIl)UEtWgu_j=`d|ZCf=OG z^<*+Qc{rQ*iQXw?v5O~Kl3$BtPpbpE1lo^^SY4IidoXASj^-@;A!_IJA{MGVDtwZTS3^#$gnWESiJbHw^)~jg6yh3a#_qIvc%8A4n1|KxqAj#E#lh zQs{FY*Zwt|4{T1J|MHWM$E$mhey0iA6@0Al$_j^Ubgy#GP+*^Gz0RCWG3QY7w!tzs z^k~`>^0C4K!M&nzkanFb`N)(5=!&)6SVUhF|AFetTe2W-O16qwDMPHVuPzPKBqgum z!Oiz_@d_Lg>Uexy&BZqie9Mx&5V273gr5sXqXst{TBLG&G*iteS4$+~kCo)Axtrw_ z2C#=G$9F4QgfEWKmc@}Ch}-VtPu!&WNeiHT9QILn)S@ez8DNPhK&00eof~*jD`o}+ zfL6Sd!#E4916&BN)GA23@D%kXCJ(F^w=G2e=@bTalmM1lK)7*^(zdbdECqP&Lw%zW zAo2}SWmJlsZnK&n{+c4fob&Dc{RlD*`?gnN9*l^vMSoS8 zST0r@c4`~W=_tFo400+Nicfh-XU~0O1%q=dvm}WyS-qk#HL71CRYwXMh8eX?lpn;tDbw%u+7kLJ zG=_2$&NW~OeF5XLL${MpV#z>+yl?h%nq`*0F^7ijau7WT=d%=)?Nk~?ao0x$EtUqe zU#4skZ%2nCchH{fF8c16AL939X>mF|$UwY*@!DgR6`s5v-U!)KfOFeEZ=iQJ7`!^S zJ{abKFq(?fpZWtBkL~;BILD0rHg)$*S_ly5v?KShcS%>?l;H>>ug#G)ttze$e_9UIZhE2auld9e_Szrz*AMaKWcy`|E@ zL>V#?jQx|34Kjk~V5xuUCUuvRUxcgZMO{U_8o#iXlpy?vF%sS#*0Iz`bM3BFJ;fpU z^V|n8vLE`LTpfJV!*Abyht4-*XcFn=XhTOkoG9*CG_T$rWqj`KE0Y~mZcG<-Fi@9j znr4{WsmD8hvX$dS`}F;udfhQ)<6VF7zp>EWTX#-mnw3ou`asuM1$BY~>7#}18S?Et z&vqfI^==o_R!mj}WJw3QGf8y^$7JiK33Jqg8?=%mb!7Xz5hSTXb*1B=XEBKA+6|KE6k)C}5dWfi6^O>4XGpKl zp-Yz$=^sVPIx^|`7ZAfjFGXk1-%pN!6F-4vZGgqnn5uOr)T`Z-4Mqx8Tp|o*fT{nv zy11jk(a+vWF#CpE{?t80w&t2IkSQj7a3x^57E^_<5saI@0;q ze5o-Mn(l@o9yK3g_^{|%rAL<23sB2NgXl~O49|RE5l*tVXd=zsrg3c?y;q`#Ee%I* z1_|(!mg=%sQ@f|&x-22Bm)YTbs7z+rt2l>ikIH7DnILR08|K-)Bs_QM3m+YV8I)}( zqOJ)a_ptd>?a3t}l3`!r#erPG>&8uzZC3qoVQ24eL*-)9@=8_`L{CXfd6BhWyLNz~xtGJ74Me(TDcxy>Cb>u@kKXl|t>)8s;>ju}-4# zWfkuAAsq!5h;c+2 z9nTx0h|iX}zn!m+;FDW<4-NpL^-HEwu z!F?C5MdtA05D%c6(y6dAW5D_ica{rvn#%>8XV8lKsF};W)A3^E;zYtTr(xSC0~2b4 z2E;{fiJ9vBN)S?35r=;mL~zZEn4*w2ek>(h3YQJBPbed^H+PIHoT?*LiA|bXmO>iy zPf~^;6&b^_`)9f4PIEiGVTXd)A)k6k150Hu7}=7GySN4s((wj}{h2UhHmSZAozB&^ zlsT2w_UsvvH4N4RK3U(FFOLZ)pM&@u|JNoWD{2x^HL#eNKG==gnKVMJny-7%Df#6G z-@Dvp(KbEV)~OUC@&7u}(^ZisLIG=tHb##8YR;&35zsCoSG==kmcH9B&^%^d?IVWc zeq({c-7b(R>Y>wt-1Qd_>kJ*a^e?JE7QLK6Z7V8ULmG-?@=i0z!-Fk0g!xueZl*6Y zbj8ojw(+Z9Eg-wDM9!n$JcF*SJeImS&t4VEB1tz*_QSg{x91IvnXT3mpAuB&JHo{e zIgYqfQ*n((EjBfQ?1M?McAJB1TG9%o<7AJ*+5`S|nGIZXOw7>mll|?4Se2m4PquD-gHlk=MWyp_jKzMYF%otzcyX^6x~bU6r9BvMwF z)oZV^E)PJXK!QFi z3*R?Hg7ea+^(qW`Fw>CvuoU4;l<|(rVg6>@1(+2j!Z_90nu$7}D=a*`Xuq?b-|&5g z_T;AQd4$OJJr}M!B%}>9JJK2kWxlu6o1wd1Gz zy`~bFR%gwNTCmQ&?~Jux)1oJAFmRh4W(8X*Rm;SeCZ(K}SLZT~wk_5_&RK)zoOk^9 zF{p*p-5K)&YvZ&DH0n3C)x*Jtz|a4=`kwrEAHuc&4H><+@xMt;{~sCp{|YwsUKZdl z-*gzqDdh1bz9a@HCg|?nKIw+Z{e_jlg8Hv(N9A87_kKQC%u0|4LHHP}O`8^+JUv6% zHR#R~9O$#0Xz;d_*qyf%@Xjrs4yFR!P><-nxlM=lwO9ZA@}2 zul%UjNv74+x50d2ShE^+q(xUlAPNiZ%-~CFXv@i8J^E0tx)N)8<2AI^tJtQt$u~BV zX}RbTuyJW%8WfQoC^`LpJp(8!m|!aP?q=N!6_mX0$gvymgA(RobY` z*>yzccISPqc4a=S!+CZSO}=_{)RZ|BEI`I?9%ybixy_yck4KS4~9>yltzWvXLMC)E}0q7@Yl&CI-2 zb5kNw9~ac*yMhEO*@g`zCTCZ}g#~jl$1awjJI3gWq5UQ<)@Zw!r6=9qD2=Jao&OA} zmDK~?!f2zK&%?%9#Tadq z?PD^QIGSsS{at?gLn8ixj&Um=gZ{AEpnf6OaG=<0sc=)D%63Q=tKN ztf||O8Sx}$&*fthUM4A}@_%p`j=ai!uKePq^r`qka; z5ksg??gyrf8@6)JR+AA32*Ej-{}*=xcZL|=Gqs7^|~BCctNKDZ+pHYRPk z;!`p(yY2KWZBxk|ph#_vEq%;z7!@QJI6?7m@cPG8?)|3DSa$Qo?DIP=6Ob_rU#sBphW#bo zUFqu$PP+jNTLq2rmrnlHPj;tE0q)x4eQ)}x`Ken1re3*i%}r@cR_izKGt~`)%&hE` z^TVjQ<}^sG^}@Q}N77>^cCE*OckAsUQOtK8R^$7IRW^%c`lRR3pwX6;7EnjCoF}Io zDsGdWv{C3CyAu5~NE;Cgi;BCPZZ*eZ&wMSb1#QspGppkE>;+I7F^N@vF+^w6Y?fxd zJSPFLz;WhgupPA)+(~4yCD_%C#*Fy-MDVi5nsJzH>xR0Q!s4zYUL?Hq zTbunMjq)7QP}j3jOxw!IGxp}8l`_@Lsr36X((%srCj(1}{Rv<~anhTeFHc$rUEZEqyB56h>e(99wQE_OY0;vBTRcI?Gewv1rVA zd)b+oi!<8JwTg8La`RENz7}&{){c1IcUP0m%TxOuWRw15q^y5nPh3A)ps%1pp0t`i zqH2%md)NfIs6GVJpBn)wYLzOG1eKBaa=F?srFQwT2iSRM9_S_~z>8o0KCD2cXGcui zvz-2U_9)xuOxGLNJnmYT%^v(S(kGCI#|vh1I5Prw<>_C;N6M6TsPx5MM8ZYcF%vj` zi|H;~4C#8N3Z>~t4I>E7J`F)|4FXXBO${}BW8b2z!dyS zZ;tmhrW+>#@HNU!N$#oSmaTTRGsO$WdS4w4t~seo&PNrM+IlfPN9DZF4`&3%-F~-1 z$;|052)I=A3>TXzWMVJuE`1`-ZQ9Qc?&-|&BU=S8)sGPgM;ph0<#lk!Cb~IW$X%mD zQQZ{A!CuXLPd4nj!Zh!OU63`{WSLh2R7b-;fgzS>r@;<_28+9~pyQpR{mMi4MUY@?j5{=%REHiWdz=w z1p6Ru*;S6Jv8G>2`xC5rWuVIs!ET;`x+ua1w=MWzC>cZ_+BvOmto24g;okFFp}xj> zWtB+)vnPu4B0-SdJ;s>_=3XS0V>`Y#*$(#|h{jjS#?z6-|G+};KlaSsbwTRM;_(9N}EGrd{rIK%$#|L!`&%iE(U*#(#l%a0>?ig*9ioUC2>n%%S zcY@{H3&Vj=udk6)d{_fc?6k+5 z6;=hB1ijvvJVPgLpLP`RJ9SQVrJg)%NLkq#q~e6i#dxApZhIph|I%k>&cw<7$#FcY zO3!D-ke>1B;)1MdvrQ}2N_=)aC~V@n&to*vb-%fBiOEsrTDU&iTn`;h18gy9E0>~u zIbd<=J%IxMcxkMmOl<~i!Jl}k>RX=x_E-P9Jebc_oloC1lIyR|xh1c=b6=8VBJ$?e zFB>6KC9vN<|5{r6TkHK?;Wdr44}tWPE-4gIOcw{#bl-GA@L$g#!~kr#tcD|rudgn0Sz zN)@`3TMvR47nQb24qqj)_kJL*6|cI^){2XYYZVGpNGKqv=4zmknvJy&p4KkX2fmA) zZ~n40jj?T)$?r`btu&}v+U*QcDSI=%Nlfl$5Nw{CmDh6k1V z)et47O9Mq1>=>}t{s{gHc^rC(7nZ3dVo z3~~R}5_*>dY)G?Tn;z|bG0;|7o#QZfePAFm6#_x2(u%2U>?DN!e= z&nqJjT-2z%!|!A-{LkuwKd*$Uf9xxNC2%QI?dM)I2YY|P!${P0>_`=Ete-_qsgc_& zr)Ux^;as*oQFGT~s2UXK1WM3H;=^r`^OcG&#JIDSQ~ef$oL?Qft_^wyf1uF{JmeBz z#6@ieBzLR0!a`lcGWO^MD0`gN(5w-gI zV#KJT4|Bpernm9H#MjsjBhnHTM&{X_40T^wRyXV;GeVxIImGVYhkBGAgQic{m-(4i z&JF-*DVBJ(pA8D`Dv@@FhDBJm#gHnLCij|`9@p$8z&B~IQ|1P9$C@y5mx1&?$OEU# znxexAU#f&tRawGoe!9<^R}>BZ_`iaq68-^4E&b7A3gBW>N`2e#5~J(f{Wi!`r_LIj zJ*(jqAbQo(-=9~bINerRdIfn;7;DnDb3HCB)TEA{FcvK!hLdV*9iw&xI3sWPx&Lfi zh~>3>6up^p3_NA95jmL_iT}yLEDkfeUn7IMS(Wy5CBo}m(W$E?a`*3W)@Z8mCH zVT!$4UjrO*X8KuIAt*L4pK)?%aB9LYL_(gP-qN(l(DSG5XKfVoLNv=8)+Hs_Ux4D31w`mPXLM&Kp05*xQn z9N)F=GTs&Z!Dq^Y_WE+q?|oSwHKfGxJupY5p?vMxvMbBMyff40Tqnhu;0^Va!ed5{ zN28vPpes8zY?@h~c_nAvjrL}unooZw4Lb`?eMiPx>-m-r8wDCQ_8;FffpnHH8FP}f zy#JHK3m4?rI`hH7KyI`vAKG1EHN+n9l1+U@9dX{923txo#k7d7xR*7R7(Tg*fL{!z z4*<5_Dzpfm0%pJOKg_8!kx=SIzAj zj&A`GnA|*mwmVMq!>s*6_nCPTTlyv>ag;qbuuHO zIX6WC|E1I;Qa)3DGgqaTCDec_+(3`S5q=H-)>9U@I~{@{L_m6xd&cgQw>oE*KO7BS z%;b$UtpvIde)c*3Fc49LpVaVX+%qmuf3xfG7}QOAzS=diPisj0)vzEr714H+$rwFb zIH-YwAXLsCF0spMZ&niNf2xz>yvN13g4n$w2E+$hXEfqASU^XC@knUtpSyms*p78!lhP>(5>aDD@3Gy3Gut{ovX7?URm(rcRRnE~s;ijrBaYeFiT&h|_^lTe;pt$<$19Xxaw!YoprYq}? zuJv;@5V!rh?!K{pw!Zb?cn+e*XFcTY8m)7;wofzknU$27A7Uq;<+5S4{$ajto9Nqv zbF**HuD@h?1+>Dsps)QHZW{RTQ2DH3(R{L`@|`4K_3r_{du7AFR9X3qJeT6Un$!cO zE$^$J9n<3TCLah%$gi|7Eh$6tHTXVb!SVE8iW)a%E$~r0R7YD)w?T4>x|Gaisyj;~2;fb>vN?*DNe`ezU(UsP%Ny>f3s>iLgaYu1kX4gSq z0s0if{m7P#n~12po_oRo1iJL^w^?@u^S7}0&kJkcC*R*N`!_dW?S*?A*HBphbM@{1 zZ#{ZVZuS7|$hE=gT~0=T(apcF^CfF8->X%UXf!2#smq>z56TU`1LJq**(ZG#wyKJ@ zdXRNwssQG*tOC&bJX@UrbY5T89WTHDeuR~JEDS+TowD~?T(K7AyhHF$3vT}jPyGnl zw`@kqDN)I4`kOM8K;Rh^?D2#;Lp?fjTa(JdULr2MA_Zs(%j0_f3;hH)%OVn08qe}C z6+hxfAK%fGw*3P9;0P9+zLd&4wjc*EH>H6aItu(}_VNBjwEFjFS(hDg|0f>+C^tGn zm+&R+fuZ583ln{x7$4561w=P+0!k;kCwxxpu8Vo)R+Z5Uod=+$i@{9y&yfRWSsb=xE8(;ar%iRC~`?i&A-1Moku zFm&f)4#2{;|9g9KWVYRcTPF_^LPJ;o#7E9tz9u?Ir4m9JscO~IEAl`7 z$O8xup#05uP@Z%~uY;6`m4%xRZ2IAdizZzR@MTffRE5ldu}(^!aN3zi<&H&`#d?$aJCM+&OKo`WezW z3F6tF9K2ki1Sp}B`*>SUv#kq z|A(>nj!Uxr`?$NVhHcwssdQy#W@=?-E~w1Ca^j{^GxydUKxSoWj+{9$x8hbb2T)m= zn&L(!rE;Jsk|Gcw@SI-P@A}=(eLw#^{NW#`FUNTt-{UyG>-fBRV(kCoTCYb+d>3~M zb$hsxwd>kA_yWu(F?TN{wKpi@cDZIbMIj)7oAuR6Q7P!|a?kHrv$>y(dG88dvMjjI zg1PS2MjGKUvSRWrN>rvspgGnrm>Wk!6Zbnsoz5%^T|Ru23oD2J5_xkaPf+oam=*qg z_O1B%Fy|WIIf2$}w7k)tiKoO;{o0Ia1;uJOv)PF1wcsCC$f3+~!P{#UNuxn(TE#sb zR{70fq^%3mi3*JM*>1C^W81rrf9ld&8;>l9jeyx%WnM-Dh&-Sppsl)M=9HH6soqhv zo~z4D@;dh5N*9npk%Q|vmm(e@tL01)DNIpqF;sdF&TO=?Le5FJOxGsNVH0g@=_MV5 zb*DcK+@-XS@XWaAXFrL;ClvAs01W0BbOT(suT^H+WE*;p`ilFrbx~|!?5Y3mO6om- z>`n`pE2C`k33y4ef#&47_9=O&JY9Dz>{q1n$@MRvM~Sn1HcShGUyx{7USvLq+wii_?aU+y9KWszIxQt5UOY{XY8(c9`I z)y!L~VeJ*ZkLaDX5N*zrVjXN_fba5On9=CpTJApz<2`rWH>*RW?2|lx=r6%4P>y%y z?DAw5G`5V8dVr-ik_0`Yd+ZPfCia3V90&?uS_Le)2N z$j@WIsGs2k+(aIYH{zQG3QQW2MnltHb z<+%MRgS{ulVMfnj4`Y%dZYwnzf9Z6#>o}~UKPfFAAJP3tD*U)afyE-!o7*EU_ELN8 z?Zb0NiW#@@_b+Kau#hrVJCU6&{J}Nn%=+fNFG+y*aQt428TnX>_B&xG|LEQFu^;(i zjos7Ou~)lu?PYJPpjbYD|Dt!L7Qao`KkI-KO3X{a{4o1>TZa44hHb@Ob=gl-envWc&l2gVA>s3fW+^$2KlDS&lq6e(HXbfHWe`$aZ zRFy0JPc7UyBV74O{OU9W2gW6LnHVERB}9m)e1BJraI8|f?w%3Y7S;Mw<<(;3Y;W!Q zmRHW6@63c&SXS0c(HwT-A(hT}$b+LQzZzGUTw|0X{C%~qy=&3?wnn!QJOg&R^eVw7 zk9kT^DKUFM@7v6heZB#sXkVV3sJfryROcyi84q59yg7V)6O9Zy`J$aav|5>pD;?(q z&_PnrjxZ|$YqEJiD|ht9AFLPKQkc-iJbKEUEUuWZ-NJP7)a3#F zOnB`=9Pz&W?;D*`5xH_(S6v2~iSf55q|)|kfzONj=1K_1rp2r1D@cXh z6;E1wDRbyF2<|$JCOS>kadw#yfSutat3T+8jDoPKXBZJ?7ymB^Sj2w+f|m`5;w$K{kB)Z7udQ%x#R(%R7zI>G3Z5d)5h?5 zGa`3b(Sz>i_RbFdHp5++vznn~M^FER7MudYUO)QCR-e>Ru`Vig5MU?lW}< zWdHSDuDg38nwpB3Uov4?(P;w(8Au9!K)F@(s8ok~c%a_rviu&aQ+1y~Uj{9yqM3*2 zJ>9GBYd;ptqq6{SZ^QVr_WN_~mzT-AoPPYsjCCUs%)TeCrgj4K#wIC!@EI7~V{M@7 zZ;$b(a_hrYgWeTW)j8llLH@pA1@5ejGL1BnUe~XGv1t9Rm%R25Mk#w8)7yBZyu$s) zIZ(&NE^cZr=*hapbVM!dloyp*7F@Owl+`Ywx#K>vtTfy4&_|3W0GVPIU5# zBMeeE6Ht8R;v^rk~V`on4{qyZ?5v?tdsf zzJc$yy4{kh8E>DA#qHaB>$OG`^N}ujKsl>jAkyj!3JN03uHSx`5i|5%j~vu`jZy|b z9aI)JTkqstXo$k0ZA;{QcrP5+cf8VRrc3Q2yKVfci~2eBqDsVdz2HS(ug=MDjjM50 zR}Ma~FdEq_YM|+;=IOpNu`4|Zk3R`bx*7rvDO`hIy&7R1o{Cav6u)@CNsFaHXEU)a1v?W?lR*vV~@-ab! zU*&!$#92#AAP=x>vciV%*9N{uR7I9<&jaUz^1AK=dvnTh1l_^%nfgN-jU`3Ums^Kc z06m?gc6lz^#t;kF8Tru~Pw?906Mytn0-|?o@}@yKA@C+7~o3sp9DKmsB`)R8cXG)hw=FFjXMp5litG|3l4s=fyVmMqgUHA8pmf& zYiF7_dWl}k#50$QN{x-eE#|_Z>BuR5bC+p4+)0sHCO&Qa=TYYAn?PtQ71N_mlbkWz zZ)6#8E!!)+`3$6T8gWmgE;fHuYQy*@NENQh1;UAeMD{~8Sy1idIU=+f5g~GI3}LS_ zEqL&6sU7=)!8-J7E3v67wD|=#{U&ns8N-Rnm*rb86;I=gPnn1FN~cG@g<5i6OgI}l z4rlrxM@hflhC2Vx5V1Fdwref*f4=+Uk|y(`!_WhO3|zfu+_3kTa%+uy;l#Dcp-1n6 z7iFGh?_B=F|N1{j;9_oUdt>eA+}8O2F;wh#M9zgRTNIzczRXofs@d5KOlOM`D}{EY z_h*Pa8GT)s+b6G=IEm*E-MHMI*h?Bj{kuiiiET!i#ol^<^VPnAX1O7Z*X_R@xg9s= zit-?geOqq*9Z_1L?%!V86-Zqx>Wz&C~Ua2=b2VkgYR?M@eRE-q> zxUN-c*3LZ<4KxS-^I6`ysTZr-^6p;~`TMcR@&7Ga@&Aj*{4RP^n8&E$it;^V*ZU8v z+uKBFk_%Ezfb?s*;BJVu%_4}kldpUL^v`|uad{6mga_x^148W2X385i4U$jF;ho)XuemTRz-Qj$kGh3KSOA?p`kmY%+mTE9Hpe{7 zWt&HHU9mGL-KWql4}fRo*6TK4$3qT`LU7W`=tA`G8w+%GL)~PO4KP2iOnpRb1&fJWR6_Zg~iK3=l$kv93P9b!C zA|Gz#x{iJ}%7HHKQH8p;+C0nlfN?Q1^m0(#+NYjC>--@_kXTdLEyD_2`deQlzKp#{ zT*STk{j};r?WgioCIqPWE#0*SS0HfOa3>4FCX6$0ObDb;zs;BQcJ(lDn1h~S@r_nl0 zKjCY~>@Tk=e_lSEY5e`EXSuv&XyUjLzmj!7Xir03+nFEtNNIC9+v#P3dZVMrJ(-mB zY=x^eRULs%?2$H~kJ*&SM32>wWZ_=JVNIqU^t}5h7KFO><4s@V(Y`x#R-=}q1V?$`Mj9(5 z0^oUgE>hn>Kyg8q!~%7@}uQ*m{PLKjWQmYW?eH7Hf^Md+0M|3*@nRLZ z2$eH5f$RzkxN%a(MCBp}U&3vXLty+egHeGCfAE*s&U=F+N)z?D7j9us(P$)kKp-H` zQ4P^J+qzo5P7B$R;kQ={BqG0z5XSBB2r7k=vO#% zXNY&ivy3%zY1!yf@S|6i87_O2e*`+@WOlTeGh6P+SSQR`0JPKctzQB0`*%mbe9=h3 z5W*V1UwX|dY--cLKOUM%fDIxd<({T})mW2JjJEAB*t-c?niDBlKrn>cyVgA->OI>AJ?+ypCHRuT1OTRtp(f#883{T|cX-OS!Z^}<vNnc44GXc4FiYjTLkp2hhA$>Ffk&ud`-m;qw>pYG+%zvXWARa* z+P9Nksuh?%jN7H4#48+lErS&kt&;kKxNz<199DGt+VSYiJ(GqyO%@fu;v4K;`ed-r z#!^%<+3*Vr-Cy-X^oq(34k2r^FX;mev6{;tj8481=d1EhW1BPCU|<#3eaD^+ROe~c zv7U1DCHLB1vf7RbI}#c8s}eqH)W=CEIT}aB#sl01*7P-L*~CS<27Xm=OlD(0>hLe>`;@*URN1RBZwDW@d-n~}&Ayvsv~?dKtu-!%4A09_WzL&aNQimY zrBNlsHYg*~mEd-%TC0jZ&tJWW+bHLcx*h3%CHT-SYzNJiyF2a1DcV9Ehk1^?*xy~M z=S)xfqlE@+Xh4-P>!6cr%%XMcg$i8x%BO;G(YzXKC-3{?rXtYaH<@uaavFR)owSu* zw)d7doZ7f-&0C=mLU_e&%vrdyJsB6(J=q@rmeN(6+l;aoN+d6Akhp^4`d=b1#)y-lmC5UcTyb8MdpPa92B$4=oWx_#R14Sr*8L3LwkUB`UHpbx zj5=6{%J(ReA;7ga-E#PLPKfdvK+w58*091#_uQ#6ZlZMXPRhS9!Z?24DRAqozcL}ANNQ2!6GKkXWPDb zJv5Dwt1DfkKOGLb^QW6@kX_B|UuVTnqLO!M@C*W&aku(DHXoEu3j&oj7J={PA^uMc1uv{l6fvLEG#)jN;S6 z^FaVQ4L$&MYr?q+kIzSRY&M&(|tujWp3D;o91q5qjRvO(`d<`Y|;KM zc3#$(tG7cg4eU>3l1D&Js-2M&xrY zZx}~4odMwDC(&;g2w&!$j!Yu1Jai760x$O3zXI2%kum}DZZeBA=)?0_Kph5hEDL6y zC(Fl?MIlVJ2;Shjn3_?^_XVXt#o#X(MTs{Q9nwbS4KiuQ?NKKRu*A%t%;Jo`MV1$0 ztT(!EL0p?pZaf->v2_#;MnoiWH^6FI3aJWhyQD38XsQUlaUhPT8|O?uP>a+TaL}WG zEx}6N=JLv0m>CW>AtiNEVA(T*4hbcp0T}jno$#Uuazkh!)QvDZ$u%PaWOlONAbt=p zGdGP}AV)>M!WoZF4PCF=MSWeamb`q{Ao4{m1BWzi!+Rh=Df670Mhph#7?MY;5Up78 zE2J|<1(RfxNt-C)z-NYE%Fp^do#dhnQ|I-GrCS5}E8!OD<{y_gf-O~Eza?UPj!yi~uPL-Te^1f3^xR#A!cm!o5jBCLWfIvSlNN-2MYFq72cM? z8p8=+7&$VnNt{=U}76n?+Y@EOq@v_2eoH-Ci_fC`ni--}pyN)|2 z>|oS_BKkg9({%qFgklxSX!6Ci1b`+P zJef(rV!JX}oJ-v-%OwHQ4rQ>JrT+7E>2n27OC}bN) z)U6f2SSP~q>Q<=-!BJ{>Vuo|X)O6WsNR&AZofWMQ3I1MuWJHDUx}NY7&>j~3I0*Fd zUzxK@|9U~UkM*S5|3CDy8T z9DXko^20{H@GIxx_A!05u^;aPYpuwBbi1BEUxhzM*)Sog>E+H^Lf_i;CkalE&C48- ze0@EV1eK_qu@MuEfN7IuF#f1iqk8189x}|*U$hqzz*nkFrP!344R{?>Bq$N{M-6F3 zg1StoSd+P!s>$%M1U2;=+9iN#w`j{>=fH}+v%Vkz zc_EEs@0A=X0lg0=szXIbtJL^mxfb$ort%cSDF}tP_^oNt*14P)r+)GDeB^T6N!M&{ z$8w5r8%pV(zN5f^YsPKiwkg2l4&@=8K34sGA}P!a0lUZvBROpA-Oj!5B9c6m50ouV zRA(p3|0*7K9dQhsv!`(9n?RC#8PRRJgqXSFUuC?dmhyI!1x6Hl( zq{d3bXKUrI&DWcYvg+{@%OxvsT{Ha{*3RWkXIq;X1@Wd|;w#B=SmRN7zh~7E$`_iu z{@9ado-mBE)(tzg?}W3}wjAM`S~uDA%r^~~fIyncz0A6`k$7)opmWebWhe7&!s#)f zMRLf>^xz`)UG7x|W4N}AGnQxk>kYkOTKehth~07zjCgSho+blW1*BOVU%M;~AIWEz zSz3ubFL`d&o;asJvtJ)M?y$#s3;hiyTv)Tsgq$_-p=7D9OdXAkSE4ao8faxih-#$3 z1PkeYE$4(?s)`fMQ8*aJd!KdUbE#Vd=F}&I*3m{>=Z8`QfShG88havoX&>NMUmBcV zGU7!vtTAqk2yXp|*kV6PBUU3yh2dug1*L1@#h=#~1Y%=HRli&b!6l{(=fSX#bM^uQ zpw6V)q&>v&bF5*a1l7m5ElmS>+V+zj2dnc7Fp)K6K!&Kc;PZy0 zh6Np2HTxx#ZUcc`&y7D1HX6MJ=0hlWqeUSDa>0v&(0=C2ocl>i6m}@!F8WPe!XP={ z_g+U&v#~CH2y%`;Z({jUMfMK^;Txo}B1ZGqyT0f>X)mO&jf(EDlC2WL&scxDD{^gp zSJ3E^7T0ar0$tQT_Hn&rdqs#;pq6u9XcT{ABxLzkd@p8~$$SZ8GBfeT%>rPSZ9-}amno_4OlH092q(SgcHd>)kk8co_2L%~! z@((vXAJEiUCjyLi4Mtz&w4AP95;PY@4UJTA%t8@rzbRqJU6FqGSFS%An$N~>Sm9sl zRB2_fSkmqf*M?M)2tOMrK!vi5jS#_+Cd!TApN=GoG;NiL!ySGY&Jhx(LGRwa%G;Gs z$uzXC=%LZrF2v7k#o-bCPcQk^5J2 zkIuet+XC5vO6>#Ni-UW0x!(rTWUco-ICSw)sg-Jom-k;~$saJ(>n+xetX**ESw}gL zb@mEoP?QnWUDvyrVx|-##B_ahS2`T zDVF-%ki**!rNxdj)n-{5yV?q6>=DMNnyEY+IFx?PGKcD6%A9#rp*gIGdj5bu5K-wD zB~>TzaRAGmNaK=g8B^CW?8!E7s#8#zo}aQp+s|1DjS*r`R@%|9A-zz+AHv%SS<3wK zfSy=EwKBKmjur;+(U-`GQhH|7WN(2$pm`2oo0$DxGUtU3PAyZb&yz@^b_Y{5&+j;K=6xHd6Pc2wW%0Nk?>F6Eb zmTm5`6GlCQ3PLC64olLs#QBOKu;|`UE@Br6F+5gFmyT(`@8?ndl(iyqJO(kha7wq1 z{^04@^3Nl=ErX_z$_4W20%Xu9KiIk-?-lNF*v9EaR{otx10KWqG`Z~bvY$FW_9aji zPWqC$Om;Zof_Pt;kLNOZodvnbnH72gJbxFOIMd64Bm2wxDDk0O48LxX-KoE$=gR)5 zLDfFTv4pb>GvZw6HudWt&k*J*vxSGluRdwudQM^D+M9+-=^tNb+BN!(nwYuHjm3@s zQL83*l`SO~7tt{yJ}h`AmOQ;41I9{_3y96X=JnEZRI|YQgGw+N{clw@s2?i@9*g5# z>9y6hp8oFakWb;YOS3B1U}pl|DVkmSd9NvkPKf~~%bhu|zEug|Zja9f*Ccll&kJ?I z!5k7V$4YoC5GhSWPN_~T&0QU$syLaPL5NJWs@N$UE3tLl_i{&KNvP#=yuvbqFb(*j z(y#rCWVm4YAg}p&JvcX2C+6L3m81?3_*(z4?s!ELXyZ7mgLSN%2C>((w)`Yq5DJF=Y#cXG0Aam!3g- zHu0=y<{4rc`&gE?_TmLA|M_&UtR`5rygeo#2fNu!Zu$fvKOG)s*Ivm&=+oJ`%&ndb z`hHze%98(2B)%?W1;w)ytou<{%F93xy9H7@!m)jP2fcd0NhBdk{MdXeh#wNUhym_T zEB!zq^?B@s1#%e>xGio-4PBUE=%DLJNxVn|1yBj*Kt|a7V}G%r%qdfcN=NGD9K#Vt zrYd}xb9#^r+A+6rG5{&UV!T8f@fUym4n#o1c)8O~HFf@j>cEMl=F%r|5q8H0)m%HA zIa$o+3pzwUZ!$aGS3n2)la{UJ!fgAyJPXFm_ZOTne6LD>-Dm=VTYD>(C=UPugc{3!v`%OPIPOi=e&&?T>2{3zeMe}refexnLn;)!fd;a2bS>bpkyy<=_= zVG_$9jlO{1-W^;)G*8Oxtd5KekqI9V$t%y*%!ze&N~*U!YceTSJFpnJ4Ei7eb0=pn zxLsI3%H5|mBCO+~ho067-*WfO0)s@k-}J?_!0vwcs47CuytQdhBI5F#4c6vTB6r&gx^+q z*mo{7f%(X;B$vH-8(GvsesG#dFm*dznRJm>rVaNcPAf#$DwCFfMx?q@I zH_~4?vy+K!G!jabY~(B@p|$bz1xRVE!hNCm%~?TTM}L`i^@Kl4ja*q450cSCWC_~iM}ZI2XDNIfaIqk5R{Hd<%BFCBNojZC7bDoD>t?V zPa>3H*3WFSt|7{dv5o3duHxsB14s8=KOgv2X=d;ry9gzu9%edsM{{E`gR+bYi+ZQ$0jda&_q7o)4W`t6NDl?Lm2 zO*2Q+oJ&SQTU1Sy=)=9MqqPb~pEB#WL}=a4t64pp(Wbs?wH~?sXo%7X&yI)cwgYuk zk#;d!z|SK(BuuXq9=nnUy?_ZpFT-_ed*S9GzoruF6}t?F5y6L4;Es0_=?bZzHG@gl zQ}_Bztqx zKv*^r|FQv|u-;voR0N{+9xTN=e8@{G1TatWO?=a?^A!R$3*m6GJ@Sxs95dX@&Ag{_ zFWThkLYm2UP8*0Ozk9MhT;c*y2Q~R~?(iX$$g?w8WBtz+l(AE_86G42Gq{c?zIeGM ztE;;((QnE4p?N-nxO%?+HL$}Nqbizdq&7L!?HFGtE(MVe;h$XefiFNJ4{{mC*Vg^= z2Jj(AOW=d}36>mx>=~aw0wo?no0hM?p$6$`HHl4`w($i69yo@lO!k;eJ8Io%CBI$(1Z)tcm&koj{8O!ewoCDC2p4G>V#iM zpLjSA{?o#H@t$rd(5o5~_fKnBovYcGOUxv{JimVmd+SgBB|h%BX|u-J6(oF;3wB6$ z)HORi)#Uvj?f84rK=bOBEmHSih%n0AE7SG}RGk0VyJu44qJKkBL$mz>Y+Rd3trOC~ zrkN`#qb=O>y`T^YbTf1&5|pnWvW@tMM0)4I$k^P_&%z7#Esu}vX7CQ2`KRH*zwZBt z*6aTHzsrhu&io(k`&ZWszFzMEz(6KrhT)9k*CUI@oEJdfb>M)aL^PwhmCfAGPw`_u7>idu+1v}?v=2N|RbnZWYxemRi%|6_ zJbmG|kxLZa+dKa;kVmgK;d_|LjH<6<#oqilRBEH+>BPH#vm)<(Uw*yGikxICO&sp3 zLISdt^QkxNm$Sq!er>qyc3-0+gW?A79rhUWY{1}7GnV#NvZ|CQ(vvP1Ce-X4wB#GY zyLX>{jY75fas@w2zf6p(cxY5uz!Lgb=Xn45`&;&w)&pu#^emos&xP&DgHZo|qu!@| zoD{1#wO0{3$0$U;vibtrS1CU|w;ikvh$)#&*pkMLy>}d~NrZ+^ z!&WXmY{CB}(^@VyZa2jIky6zpPrS{P-QiV${+@9+xnL&GI)q=Sj!RfB-|(*7P(6yP z)x^3HORGOBAmt%_ZsB3g@W_{Eq9P42f@^Lz3M8mXdsodMNF_6@W8=vb`87ZNSOYkc z0cyj3^rkOj^wOIx(sIGRCkvg+R@ylhwcaFDC9yg$-6hbhvJ&Q%nf@TxWo%yJ*f}hh z>em?Kz2LW1sXwy>IFa}xK^ZCKZsLvD zTVTBW!Tuy#;tgC=X+E(QxNkJlPnSa_c?XbOtmXmQiri!Olow;p_r^d9pDD2`DE6|T z0^#B8Yg(}urtv&7ur{K1x;aEsxgqO|vHDc+@G-$}T{_bH2NfyWC7-6qH;aWo7jUXC z1db`{wQrmqsaH)x9~~79Z$|xOzj{EENvsR``a|ur6G3mWh5UBza5Y2}*&$JnQz{Pp z7t_Mu#IywXOdVP}-EXWL6|XUC)X-dswDR59Sy84#I2zZ`c2ySZM9)fZjYnHtc!*3J zyPVR!NE0!`B3ck;D^tU&%T}+Vc?#;7 zjwx0ICEXVIBEGtmT_E;HAiL=%v#bmhT8Fm&z3^cX7=S66{#3hIa zOSGOEw7@#lh-H^(<=+k;j`5F3Mqii<)Ttb8>v8{OaeYE9@{Tdv>*>UkNo(V+=0XJ( zJG@mrDOx>0p))lzMIB*}4eXY@QZx0Cgy5mM$1Cx^)83M8`^xGi zvcX_2x_!ZMrrHqal5N+itR@2e;3o)3$1ADAkipP3ME#A4q&970BCT6p0ZimyA)A;; z;@F(U-0LG6Qn5|NK~W+A=zVmar=QD8k*9K9$b=(`h845nx$^om>my}!fD)OhUkFgz zUJN6o)68op*JH z|El%u5@VkJRHwprlM>k<#HyaFY>Q+yi(wZ>Sk*-L$ED6KnNMx)23t((KXVTa+a?w| z%k>t@lH=9Q#!T5a(v)b0-xr73rfsPwGrBCq<>Yyv^w@f z!SQpIoO-_ZaMF8os@YylNLE;j2jUkIdP;*(kwf=fTh4X7lKGJ;AD?%{RG_TDNs>-ruNG6(ppD*36_jin=eP`19q(irJ`RZITf}5c zBk6jPUyq&%tG)1)6K-cmPN}XAvBoh3=m9Oy&?C>>keNd#lDYH&c!oMSTkBy)x|&vd zG@{U+#fD-O?1*kvRQHMgC!f^D81hRB%2l`OF@wj?I8|E}H`t^MIXbUJw@)N7B409G zQcZ7Mu0WA_;ovwxxBf^CcD)wA&&80iap1V?{I=u?Pu0=-FL_s!xaZPmPL>F|yy-7? z{RSUu!7fZ#SgE&@teI=Cr_PGe@>0Hen8{E(*{`e!N*TFFLXcAaUN2}n;b7_vrDS04 za2X`YzvJFm%X$(3yxIKJL3B^uK6_oz#EhxHRSu~31^XD;Tuj8Kt`Xz$jBDk}` zr;;O3eUYB+t#aKxPbP5!o>uEp#rce;r6AjYrJ!-vp>{00NXy1exu?9mCq}C3d}b=Z zh)e<)k-ofs{kx-f{e|b}_RmFr*oL&#QwjSJQR5zwJR;sSa#?K<$`ymk+YA`vC^Z-F z!4snp>q`=6X3~c^T7}d7dIw1^*+ZU6atOp1*X(}f$`wL}p@qFeYGYTW#l%IorEL#c zPT5^y7pX|A08;$n@>Un*7IV<06?@*wzWFBRBJWECY=g&nP>c>Kr;`B#VvlPRI=8Q2`QF*rs&qB}3(?>(e>%5;#RxexA zb0NbW8!^l-@^v5VEsKz!`jJ>5emeJt2tDaNzXCG&v=8Zcx0hTk`UhvC%}w|g2O6q| z006l0-V2NTW|k%4a;nx}HXb8OLyjjrh?F{j|M7sS5D`f?irjlmacbdxr2>7+MDl_j zoMazR(?blt(`CrJf!(M&ayj%@S4VzDF){0H$9e&v9R99tH~!vz>7IoW`N<2+qXxgZ z67-D+y?H*{x~+O?k6nkf@LfljqK*`4mC97JhKK}FX63iZ@no)62%;?v-7Iz}&Zp1Q`f7EYcgY_{E$tMV8DE=Unkynrakhy$q~Z?H98ae)Jy` z)t}tCt*5Df;Z7fSps)V$A)#z{N7{MdkeK}%DYD?f8yp>@&?+!fpakXferj?HDvP9= zm;;HH`F)UYEt&qDGgcXQ87Kt*#n}jsY?TL$tpzb>23MUhvX{`fBokpwnxso=8GXK_ zcFFBN@e(vLcBreZ=u{lO2@eZ*p4o^Ruv~qr#bFSkf$?y8)doFJ5 z6HW^D&u|ok0l?nRPOO3e8MLx>IK~^6aQtBoy+h)r>_lthycXKRdA;gxtsl$rJMX4h zwHCdqt9H%$D&&;=?nbSBrs{Ly#C4;^RSoho+Y3q8$ z4opV#m-4c&-iIV>S<^6J=}pq$wxk*T-b~j~=b4)*+u3{VH?YgQ;O{n!AXNkhBe!Cr z&y^h}k&8KnW0YxbXPZ@z+bH^6UXP{0(uUM%Ia6p}G67qhyZv};WwUGNzFWOL-zIXc zREd4${m*Vx2qi;)gHvoG#YYGv_(4|GksElh-`*bff3%yv7ij#7zRVdOGyC|{iS;ha zS>0cfv+Wvm*&66CM9&LbN3xnD$rVvP)!vg^{j~zimikSewl(XxpJd_ukOwm*FYAvZ& zu)DN%GV6oRE|syI?;Nd~I$GFB@g&TrTA$;5!zQH20ma@ozI46m=3VXLVDmtBk549g z!tX^CSo{$ebY_9~T)LWd`3PCh;^2?Xk%0KwT9gyk3IELgBO0KJ#$R;M6k$@b?aw7A_Kv8<4Yvf6jvF#qsOFtnWpk~J153UquPEM;n99z2@>+;m|Q0 zK!>!y-|e4iyJpO(6=W~J)gI^XBf>t_5dI<=EwiKod3^7r|7LYfW~uA0XB>nxOi>`@ zm7k;>*0TKC6NgDl$@m;-m^aH@9b=wdLhqtRwcO{TO1U$JKt^Z-1)DQc4Vuh3ul&EydgnH1E1-qOtcd!+?z=NQG8rX#~5w!EcjEweB?IW1W($5zLA z)R5cYdyAp_hbMEpBqXWY>)d12$bl1c7U_X^b#hw*rrX(9kLC07eYOF-jf*syevVcT z$9xy~etyJHbR_&5USpFDxjN}`EpL+zS=ax}Pf6roD#N~8A-`n0c$f#oi^r)emQV&k}w#IlUzcM&m6Vypi7%i8U^u#otIYJgR~Nc zVJbl_t$0-Jq-R9ZIHjl#U_-`lgv$kU)^DREB@;8pjNm~s|7An|9drO}$kDaJ3&$q+ zDpHwbv0jS+cX-|Cl66Hnk%Ks}-cYk#15ko6mJTgI;;dj~w1-2ZT-91|c zYN!S>hpH$AyFuRICGE2bJ;_Sxaz(dx41NWx;sG?|T1hF$JoOyv%D2mIzEYLO2G+uJ zl2+nB$53etZbz#+czej|xaydcHn5fEoL`wy!?FXzr6+KB!xPmsK$F=wzuOs;gv)aI zmUVx{;HSLO(<$;rAcQOfw|FQiFco|wMFBFT8Lhq<(Y*qYA$tvcuIzd-CVbA<#s$bk zxD7&VahSryc3i65y3Y z$AyB`uMaKzDcl9q#Q{$hW=>az_)Xc^r0h>IFd8AqZsyFp(0Wt2q~b~El;0hB3mh3Y zY;KZ;AO7x{0aTcpvos0vpOnrY``^91yyX$XBl2m$&7o=VKl>E$_0eD`5cB;j;q&+7 zj{kkL{GI;)5i*s&`_Cf#pI-v}*#8F%sr!GxkjHlcx8na=hTk;|P0AL@lh~KFDZl#f zXMO6ol5kB)7&b51nUf-dXA@?Bkoo%i$*o@bTO};lRw39^qGl4WI*}|Fab-fYx}2zR z38+JH5_}h*Nz%4&c_f(Gh({FSB)uhIFE3Ley5daaTmKH=_dmLl@ShyrJ=wV*bcJ47 zRI5@FoZ_YyMb?ze=KfseiOmj1LT9__(^m-0`)qV=Z%moaXY%iC3C^qVK-69f=VwD= z$7cOt*G91$A4*R~bhl()q0Pb4MNv#MM$kf@O!&PeIO9R^HMdKZJUsasc0*(n8mM)N zvRsb>-0zLm&&k_bLcv1ykO4G|d8u5Wfr4m=GnS1;)|*osa**cWa8{b;Wx`ATMzbcX zjGWpDp*E29tiw&=2D21Ma#xFQbM033sjNK>JaOI|41Xz;gdmSwhAh4JLvOrYr1!pI zPcLjpnWQY3MF=+_jwrUS37wWVqp`4qr?FuW{(IE#OP84{^}9b@AYQ}8qbOm)0b3wu z_XhTIR(JElH7cK@2%RUIf|U3WKopm28_^wTnYi(u_&a6%rpPI*B1j-WuMG%WI}*wl zW1}a_$j!*#4?AWui|ZEmKL)DDH>w~pu7+S6SY6j_v*y(!uu73?@On1tuDtMm3;-q^ z&WRbru^jMgka84shA6B@0lFUW4YZZQTzdQGoI!z93)tIQ;ZLn&$5*;8mL< zXv5+R$45ntYp+b*5l@I!G~vopG>HhQLH@oe4X`13U(F}rtjAI7qjC&m)tDjFlWM=t zX@uTUGBU;-)!xW+_DlZy>j%&K;u_?HZix_bj(PWW67&m!WUjLjH4Rz^-gZl89Dq|y zRq8kgLMy>y1x>x9H|`FFW=}yAP0h@u%9UACnPa4*@>8$BWp;+0h(2XLtbxL=Gm`N2v!%$j1l4M#Ye~+w zb!(%B!escHK5~aun=--S#4x(19OMB<7V_@0=PJto1rNPdW9uuDuE(fglJR< zvI#PjhU%KOT%0Rk7F32eC~#9_=*7#DS{^G~wU;TNFKB#oF*$TQ@h5X(wPAmY$xoJC0j-ho>y*b7d=9;kaPLr2yH z8=8uH)~;_7B3srN{VlWlInmKrrLR3*c)m_L`}5q}*s72Rc-a{?Y2VrP+xhpgi}Nny zge-V+L6=Jh* zJjwls4utZp+(5VKEGumIFH*vs_-91TCJ<5(xr5lG8}59N;}aQSry)dHGJ7-;bQ86; zDORKS(@FXW19F`C#*eeQ;3$Y8LxOwoo5{@qH>}_xzg58-Zi)kMD}t*m>+yDXo+r!` zP9Qn=cr=ipn={$>-uiUF6wISFZK2n1MG>Dz5C#|NVc|Ofpv1FW!v)1F@Va1pl za!~Hyj7WdGGd99w%{Y}rBiU%)$s))uKK!qnyf_S8!ByaMzPiSx&9wYcYE@%7)K z!*;(zkF$R3(g+$OtIa5p6Cvd#32L?G^wIkM*aJ1lcsrCdn_~Ecu$9&WG%M}Gu2Uql z1fRNtk_)(EKwVsc#!9i_o8ckrafoT!SzkdXsCI!LqcF-ksXanS9fT0{e&bL-Dcuj4 zc#$_t$eH6|5Lhu;3Sa{@3Dh=L1B)5_OQkzeEk$t~9L>T9bBD3GPo8w$wavY5OVg`- z^ZT17ICFVi-dLE=l;6^x_j}aog0Iv-?(Z;|w?>}(#+nuE;Yg-2KSr;` z?p`QT8+i#J`I1U`SufFW5-;7jD6vB6M?37cD)}f>mPWySNBR;Xe0!jfn0znqVa>6s z2>y-J`>d9&ibQd8Tiw5rlVx#@BLM~PSHGlnj}wsh`AaISOpvXR z+l&-Ss#VpxT6e8Qm(`AKm5wkdT|vo_(kZ;L`WEWlGI5WuFR!jmi*CG6+D!d06>NCB z0XuW%#jU~5AcV>0;q3*}Gq{)dAf3dH)zX*vFgymqdjCBt9|_wJYsr*pB)#NU`HDoG zvTwW}PnFRxEfZAd9A_&0q(q;R+(bo^7w*CH>x+rbzri`rf)k1F#@s3OAyt#e{Fj!( zP9&K=OzDoU5H13JrI@arKxPi`)SToC5H1Q>lEF#g9854L z?EeP3C#=xq4nflNP+Tu@JRj2i27lEoY#kc7P)7gqLxanDwQS1ISmytXT^rA9pQrDO z6kdl141#Qv{vY<C?F_NkX`~RDqWg%qSB;BKtM_o6dNL- zv`7t7rAB&9f)z=C&=CSefzSgINFWJGzR6mTYd!CGviJ2K?4z9%y|^-G<{a%VW8D8A zUnXEC)`r^Ffjk|LUBiF-9!*IYHG|tTB}Y;|LAdgo6+fG&y7<%K0qbB5{yVBNbC!~1 z#>?Wmak|0>9csm|Rd198u&S2YTvQpf+LtDC0a$Q6a#}oKlE0`0TS`3V#lDJQ>a5tQ zuXY{^9cS?MBnLftb9q`l9A4_nm)L3TI2pG!B>xeiB(55kS65rIHqq;%7(CtjFbZ2i zT$w?h2JtcIlnOq}Y_h@>qpl$jLfYkrPxIyC>rB>cD%V2#bs(&)W4tO*iAFY)3*mdv zyOAt>@--Qq`{+dQhA3<13B}y)7n<-ked#+Vm%e-}_+~IatIoJ!asnBqM?)1?A^T;? zqx?^NJIM^*xzN>F{+B)%>nzK;2OHzGmM{qud)nzRZUtf=s}P@ydpCyi_uyz`hQ(i- z)0Tu!c|sf3BX~#|k$9y2S00|P^19Z|l69uj9+Ebz_g+ezk4+D8f`GWgU}AkSda?g_ zapw%^{;G%s+whePv~FdokB=>2FC`V??o3%CHz42M=7I^IND=m2s!1rFPp1%F;xlWR z27A?8M`2(-Y0`zUcG-9SE|L!T7XV{qJqXPIun20TQqGu7J5IA1y*DYE%-b$_pJh(q zIV%8vrBj^)r@=UL#g{`7^!DDH2apo6m3V(TMhC+Kl<4S%7=kE29vA|evl)sO=oqG? zcX{9({8APYekpUQ$1lE&$y>p+$!I{AxRl9dKTh_2L*!jYiECW3rF9Cb0v~3!m-{4h z6BYU6iRXNM{Ep;)sAg#9dty~Sy0U>wrXl<7MKX0d>)=i@ekCib6qK1M{y5qgdv{cl zr_E4b#;RJgf^jcuEkolZXc6UuOvDQRZoa?H(${&)$4yWP#2I;j$GKF^JOt--(_vNf zZy0s#L;Zr-f-!!1aS7xml>-TawcdWhix9^_qV16*%meI|Ncv#$vHQqUB4;V{CY76| z#m4Z(L*KDa%^NNwS0L$+{jbbY#(}dt+|bXYZH9rV!94A^b-ZSIcxgLRGJNUzLYBq! zjT#WU={ZMdtnu*jf13w`4USL7hWF2Sn^oQk8dKaHunrd>q-T5_dU4Y_K;n)tAgipC zT;>sM#P})+_+ldk6uytX5^Uv%1fDMRVmCo13?5hunqR|OAxPT+UwDCHcr_vGE0DgT z^Gwt?3hT<6MsxZ<3AEWF?%QQ;#NiIeJSjTpG6gxuj`<*(sK-xvB3;t{0W|hk~+R)t(mFgfy~42BjwMrPnNNn zPr!CIBQqsWBYM9Tuc_S`o9KpBH+V_!F-0K!%udf|syGdjrp{QdN#pPGulx1$ukxA( zkb%d_TuQk)e!hdiS8bn{a5+i*hb1ELS&>g2wdd=(m+LoL_m@E8Ezleb&|T;-uLiTP zU0D1fm6a9x<4_3layYc#U)ybsGq;{zSk3(l7y12TXe?VRLaF^wfm08ECY)ZsIB(Uo zzW&6GMW!G}8)xN%TgR%1N*BU35LjApRf1X^!c>xHIGd@jvcDV zUE;vmTI#?c0f3Ps8&}SvtLIi);K+r-_h{0@^@fOXE>3VWn%pNV`r$1kp0{J68>Wir z>zPk|n!48tX?JuDp=WeGW25lQ-AI)O#qDdf+3^0uTG~PHly}lO7<)^C&g!|#2hIkB zN-A4U;#aVpP}@%hmJ8effqI(We8n2#-JHHlmxD+gnN-BRXgNRs!Jowvq5cAEx!u_l zN49+5NJ{;%{AsCZ&0k=!Y;2Wcmy_@+@41)1ZCvZUjFfz;kUU*ns&^`lmtcKVZTs(I(bIJ=s-S?p)%XHEjj5&Qu# z_Z?({;#B!)?BeKC5T}!>!Zl2y&(b(9YCq@G!*yP)hwAQL3%``XHNoTbxmqq_*-XW| zXiUzmB{0Vy=y#);;s2(a2Go49>#uJdM?lkGf|kw4Q76PdB^m7t>wWVar3aR3t&VI#Hf z23@7}zIj$LI1NNi& zuti*WMk$yM;p^@V_jq=LWEV9NvdVd*Cpxdr&_Nc73cr$^Pq&sHU9B+HWC=N-a|=Yu z+|Xu(PZS+Deq`TI0vAr=XZJt*JZD-@uM>9V?>oWY|7D((+dmz5phyHL&N)dO;J~KHWMn5n-ecYe3uq_Y z>gWvO7aZ#*^Cy_w6vJ7EO95`AKK-^?B=dW`%A}C>*G3;m>aMgC$BjeBl0Hqzc>&x= z8<8dMhDm69s(Je!xfkbeH2`U{+(@k_OCl3Bm=Cj{r^Y!PFOie>9|d}zyjomq1vbFu zSl{FqmV~$)xDgXDNAqA9P{JcKy5}!$WK+-`G{kJtn`_34c>N$;_yYfBtYnml^=M<+>BO5*c$}Bo9aG1mu)?ngi2bR4T%r11gF>H| z{UA(T{g=Eec+}qGmE|{XvI18`U20z-qRF_AK-oLZ`vzOkVVy_QWqo`Uv0gfg&_F`t zoOL|m$d~q%W}0fTEMDdcnX)?P0Op8~{n{i)-s|CVWg=KtfL*si{)Y`MF|3wLKfh}_ z<`=tHvbO~ItE$5=*We}d2z^Bz=z*?14;9Z;6q!oNd9v9W2#Q@jRD_~!YVj>=(?}U zD46*Z!vwW&JX5{P`0!^??GwFu(8p=~s7Rox)6RTvq$F0S&;hV*p8&>sB>ld-{IX#< zGE?`(Wn`G$S8MFri$3?Bq+8*u0RllE75WbSg_#HVQu3>t6dON@kzT zaarkMUHuH+q<;S(nNzT$RzP3#A32_>^8#pc<}+Y1`%V)HN4}|DM++8AKC~Mi?+ULg zQz=B@AN<-eU*xL4&iiiC#~!$g@758y>yI#df2tnuzlSiLs;2~;*F^#+by|J#Cm?1T zJ;nUX%b}G~>kLTd$`@R9>%qB|cdK>&5cFocm$p6rb?mY5U+)=F!H+h5l!#Vm6f;qY zR%kp%#LIzj_w?Xft)S~nI8#jNMv2GqH)*?;bDgVK!_A=My z-{Ct|2$)NHn2Ed6^B%oCy0ErUds0Lx=1%ne(=|>iVZcNnlEDjNr%FMwR@m@&_?9$1 zZ%z!`!wpt>LCue(=5G}&Lz@M1Ux>~?JX?LJf7_)2lpcvQ1G>@9sm^Dza=&@=3;LZe z4WL0oQSLXntYS#!A=zL3He{adIPY6NW>}~Sf1A+SM)Jr)#Uo$?bxxURK`5gxWI6`x z#1{{;7(~K@tT5ZcQ)F#8B6oP_W(S|$J-G^d?cs3bJ@3RNUes{k*kSiFoR;lNo|Y-x zvFMYt=Mby)EBl4NatVvFR*hK2<)7#bA3MoiNm#M z2FJtp3)8qi-$aZc^_r3Bx{irAEoxaY9+>rm zIM}GH6`19iH&i7p@NBgy_4~&%&9>^pHqY5E=CiBWiOx5ruantB1qwArpvH*y3%dzs zUA(Y)x2mbIVaKda@6nCwq!K5v4$By&Ln?`9h0rYF*CCJ#1gKaB?}QhRTz#e&Uzg(1 z$J4sH36Z=5KqQlW5GTECHX)LgDHzXA%g0=XP$GYC&|7%A=Xi?SJZli#&MFt7nJaBV zB*QmE6da=VOXy!;>T7UsjF99{WDgd@*XLr~(GdA?vl5;)Bk|c-74ks}ny)KKX&jQ} zo3@tbD}1N+E^@BVSH#+9#nQq#e-sL*)ln?N?*kd|+jIDW9{okLj|k}V@3@mNXv`~Q z`si25Cuiof9(yir)i_3(VHBE3YOMO2Zek>xeX&lNanfG=sr>~yt+OP}X6d`osCd15 zkgi_dRqATDQ=C~6NvRf5b6BdIy|ll8cUR7-lqBv?4!6C%0-AkL(-uhPy?w?$ror#N z!qplgPKQp{EFx3}o7A{wN+Ib&gZ%e>wK;wK@6$2w87D3wa!_U>jG>=?n(HgJU`9v3 z3g7?AMze%|JH5e#Gw{ ztJbMZ+D}Xx{`3@FJ%VeZIt-a9eq2!oYnxaq0`HsK-8FFzB1zjePf%vK8ACtpyPC9$ z^g}gR3I(0-t!D>xv2%xjtZSxxaKjsq>I!tpD(pVbLJYTQh(tY9#M0w<6wXWTj&6Ge zeM{1;9Y}H(q?uXrvg6Q!fuzR8mm4QruO~&eTj@;59qxr;};X)2b zt40~)%wLspRmGw-{He6YygAb&Z;8{qr|BS=L=8O<%P?B~K0pi}U0W!$L_5qQulScU zL3tbD`Oct|yAsj1Rp(~d%E!5vCjpcsqX3?q4&cce+xa7i5xF}^$wIB(Qox@Z#8qRC zq!D{rrON=Y8swG!1=yNosRn)U1gG&naF&-M$}XpS=1^q%SgL;yyKbWW9H_F1^xY#S z0kKiQ|BEGwm8W011+XO5>qb^fmD%vsWOQd>d{*c8wqu!bc1BJ&&?|h~{?`Rzw+7WS z3k#T%4O$eGe-&mdHF$(1H|oCsrIZhRHyJTejd@QwJYI|GqFhT-V=cwpMZ=ulauUy< zL`be*nlM~z#ZAC46BO%Kb8d+TG#c{B>oc!I-WVQdNddmHb6|1pu_ROVG@PR)YLmmD zNEvgx@2tsO_vbBfc|!12HkWF^I;w)g&2#I z>zQ78`W=sadZ-rnB_)6lWa!T4?EUsq@rbmq)au&nxN);UmDCRfg^_T`WPDZb-8Tf8 zx&GPf1MgA%Y$|$VOH>paeU?y_UX2F}kFWhrlca3YB!67X7H0eXLi|OO+{3aRn+d4V z|9&4>H@A$R{Hu@w_@8cCvl8SFsX4e9f020N5%gohf3{*bzqTdnIUtYrU!RxTQ=I=> z`K;yot^cY2{c`*N)c-zz;Qwit_Tm5i3;+KO{=aF&nW2|S7$|CL8o)Rsz($^X1gJ+tsn6a7xty;>K2%D5b{aRGk>+wA+lR8FV zv&71K+_a1k!Y3nC2N}~=2d3O3epQj8hM@G_3S)ik!U|+NhB7kjUSaU}oeR8p8bmzC0gUeeL?{YnI7=Z<6m*z6b4RX_+j&c1Wd&XH`JS=L~Fi5+}LRihL zcV_Uxf;|kp_Apf(1C~@1>Z^O5KezmG5SlQZQGn5bG;?Vu;fO2d91;&9IkPdey-h=% z^ZG1=|4nOdoVlMhOuekgnY%{cFeeS3Z{=PN;5>_OTT}T;g|8%ibH2sq$k7C0$6hau z9$U%V@t=8feRzuwr(Hr+$*&%G=hb2u`^7LoI6w_gw&iN7|!zpxabiQ)mY~gU6(p_Ad-XGRl7ZEeAK&k%XGt(Jj)}m6!XF+xLl!4J#lfQ$C^>@wbDx*kACI5LOuDoD zGQ*t4Pp1?*UpZ96UkE`0cl0j)V)iXGXVh|y_f?b*h#wIGHBZ*L8FSqn%h$7P>WCA} zn6u9&JMs(g;M74$t}X5YS!?1YgqvU*4#ktCLJ_Qo#zWD+TgmDBL&Yh%K>0Tl#^w?7 z+>r(;)fXNDfkH32!P8FBeO~lSS++Fi$V@18y zpSJBd<3K$0-RRHzjtq~0tNB>Jp0w57N{cPGb|)j=MmUu>%~F!*2Z@TJKU;ZXjIe2J z&yiDa+K2VzH6h$+2JF~Cg|7%{s(hexUdygW65po?v znHgtFA}+?!n&hBVS;4x*HiJ~VyTiERZG+Z382G5Fk*By+yX4jW!NF!sxN5d@$>40G zXL5eSY3w9$T=+n@8O0)qxW_N;pi^=P!oG_4r>rZ&onV?eyUOIw-*j8iQm9j~`Lj&t zo$u_$#`dy^*+Iud@4@oWF_KEdE&9mRx*Cgtla6Z>8$BFGY@Nku_^4jui`7OPr2<7l z?*_>jqgzF4UkLGuhf5SrRaoP%FTHjTPu+Miv;J8^{!PVY@mak|=8@NdSxYKaHldzeVc>YL?*g1);1{5(B>WG28kKn@Xp{3?~8B=u-Y z-~g5j4=FfiBEoZY$hyfo4;1KrSfwj_pD$KTto%h#w$q_h!>!%J#7ew19(>U1jTts$ zRLZ)09&{>T_O?AAb{=1k&Jy}2oMrcLizc19M!o|5@{P1wX$qsD)%Y)xh*_<@(_b98 zC6!q0MEcJ*=deqFIE8b~Z4Z?)=ak%v6%6On(%u=T8{GaH zjNoA1POe8I?X-p}WU3;0ZAA_YX0736pbCPxYd<3WhiTMKH}$~9j0}T7uNokaFTl|P z>9i8T|Mn6lV+*io9?(E-Ka$55H9_h?9w%RiueX_%O^j}^ zgrnvs+n&+KSAw(K1%tFLjd!YwqytDxYT(qF|8$sc__Nh)-B#V$y#*wQjHc1_WU5Ej zYmd%z(OAP9UDhQ^7PSHm!LDQjm~)jcH*5)Xv91TIiRqEov~iowFu?dtXAItd9r4O| z-g;_q^?2-1;p$g2;95BCPA`G&O7}P5nf>>)DV6S#cw9)sO1> z?Y_mt7q?B8=YDACH&3NJ5wIv zZH68gQ?PphG!e%G+1R3(e%h|7K>^`&vGVxeC{TH1R=Zu!`5$Xjjnsi=GvTcQkl*i!5>d%3ZXA518{h*;13Y3d&jT)ywBZK@!gQsVH zV{Wob?yZE@RdLe}!rqwIX{t`ry$3KXTuJUZcUER*3B@OGu+RI&lrm0FG^8P`Bw(@K zZ5e=A7Cf-ih-Kt!CmP3;vg)9vLwmHgUKZ`m@i4U=c&Nqpnl^P&Q++Y;gWJ=iuAX@J zgIbh&7WG(rz-hMH13?e*D9tdj#IK;mcg^r#$>=%%_w#X%3Rx9%X)vEAO9_cn=42z0 z($vTTXo5}OL$54G>menG>G2e7`?E^Rs@kPJ13h6SB4oP#b6I7XdY|#;kYJRUl2p>1 z3~sddeN{zn@Ujd}u@BIW?1%2~({|;jBEq9*ib!=G@!GOk`@A`-BXpl8JRGP)*xfG~ zzW@DpXU*GWOqjGYQk=PRd6J#t$A_$Plbr*u3#JrpI8arwBb8^nCtvPurj?nAy!m1l;K9S#)fY}GgVv++#7ocK)x75xN{uoPWRW3) zldX1-&I#RKtHaPv`10^8|*(`Tq> zztj2j&Vy_8rwPHv**=AeSsN%6HsnTowCF(c!gwEfnx|k2a6=qZndCeBd<2J=Da_*E zaL%kj-TEDfotl6396J&vIcefk8gx>A?B~xy9nhWUohwIW;H3{Qg`9aRl#1~DfQtWY z_d$-So5(k$oYFoW>|YI4{FGKR9$?;CZ*=W}fF*mTbB&_%L4#RwKh9+(@}^mymR$rqg|JFWwua#L(iO8@-h{~2G=$oV`s0BR|ZBT=B z>iA<)&&!L>ax8UzGUOq*>nIdvLhT~W|5m+Q+C3yWkZT6G zrE@&OcAJZ}Z*@Lp^j0Mse`DEL@S6Kd%@{I`dK(+B7FoNu&2ar2Hs})p4qqo{RzSq0 z(8j83^r2ifa>xrgs~J!pmi53Cr>HIkt^Mk6FBs8V4oln( zPL(gKbDJGeM`|10_kDB!iZ2-3Xx>LYD*zVB%gaPp&qoo{$}V>^0>9e*dA1>PJ9d>I zrbO&jDjqwRTGJi2CuiP)^|%iMRQ4k9WTWB6npWGzzJ=po59WTUh2kNsN7J|itYRmivP4`p@2cnf?ljEBrx=qjZ>B8O&9g_BNcXhuCP0Xmi+C>V)s70C3wnaQjDV(Wp zziMkL=7YbPBp$Mn|EJmwwOhcrs0Mo0v;Wysak{TJAzw%xA2H@<heMZ|yWC`jXH%^qD-f@2n15S^8RY&Fcw`+XY8^H!L$g|t`30!aDT>&GXhcx#6B+ezJe4i%UE2ckqxcrFr9O zgSnPWuwz^GS!gfhSUS1JyHZR01iCKqi!15Z9T>u{T%CNg&U-M0+B}Ry8zL8rwU3|o zA47pMai>GYLlls`3yA=g5-dI0T<5rIGCe8@e^o+RRiIsva#PI|%InsDvE{RJN`#dy+F!|^xEO}o)f|Q|WVwAr1vPY4i z+jxWvs9ZfaE0ezDM}eAQpAAYF)B7{M185sSwba$cNA(YNOc>RHj0P!G*>xq&u2RQZ zn8v*v{pTB+j1xdrzXO3ao3C?)DzHNiyS+}?k~_t6vu+)!T&`|)7R<~|G1Hakg&Wj0 z!i4M~wA4NJD^*o45cFf;WGE7;=F9#x#qNq{DxTS0;DjQAhoU+~No|9f(^^}#_iUXO zUK`*)v@(|@AeU6a2+V(xlwXQC)3(aBcx#M z;RWq&ojumehvLt0luVxFMx`xG9CQREeX*rKPB;}b9@ET=A+?v^8G3D>9?)&1ozWXb z-pb(^w?@n!m$L?^F-KebB#?H+mFjCR4^kB4$rbJ=y$Ci(29_-`>2FeIJvmIrfsGEC zE4-R_fT=z+ErWnR2j8vDeb9cPpGMGfxxj)Xw!90irgyWJnWuvr%Y!Hf$`Q3w*RZYD zH5g0~!s!$&RmDxRacRF5$N~m7YnwJ{%F4>RCuoydt0OV;U62?Hb9k5=EvUEJLwS0z z2*-HK2Tkz^9ZltUjryfD+-#&`1s&G$S=C=BS#0``e>!K9HF`)HetDgJ{tPtKlwqrB z6V5+I`7Cs{H+UHSedP#q6&}QYHG=2XJKBX^QfVyBy${XD!l?#!Z2Pt9|F}_+PdCLh z=85~Z-r7Dhl(l-+Ofx_^dW-62TfNVj6qQadS!*T0+;BQiiG%mquw!Butfg8;_ijujrC>iKww|y-;dh+H|CpC zKL32A{@>)GBd^d1A4@=OCA*uZF8p)pmRmRek=F_TJ3ZJvFj|j9I*zRmOauUjx>tk~ zrjz(@nj2Oh`!vI{R0{soN!ne=oMa}OwNcZj+LH7EbY&H&jlL4!E=A_5qLWan?^OL4 zS}!aW;6sO#pA14$EbBlqYe!nQY~imOT+y>aVVopYNn2@e*g5PTcc%KnbB$-IG6&fu zuH8@xA9M0<+HZ1cWcR&nk0kZ5`!)(^k+QojFH7!zygxtcl}^$ilz-jbEUobT_ODd> zVAiSY0@_ui+n}PrH=)0MOJIY51Ss3Ip9l?{wFFXO>UAeRc1a=eNvEvySBr)eD@1oC!xdsyHJY{YnB;kHKY00sqCaJn&avo&bgR z5ql#}kw6lOs;I12v1=W@DW*tE(?!BZRh`$C*FdQJ2iMB*a;8 z#PaK7+H(hoPBb7n7gz(-$W=mH5fGa17SSdjZuqCa;1VqpApvs_BPMo`m`f!a&Hh&u zD655;_UktkGKLe$Hae$KNM_eU6e89TNfk;SGd*lfTPZ<*^Kdgptd~iZxgD^ZR%BB9 zC*^PQq>}I<9EzxQ!Qnn{D;?TI2_Z1IWV7o$R;A|G>o8|LI_tGucKl8g ziMl6(^e?2eXTyT=ZN#la@D~O2tMN^wB?bs!{U8cZPS!iEB2JdZkv_?s0abZ;pz|Dv zIDa^}-y4L=M~Q^T1%BT6JO_$<-h2x-I5sBLubS362ge5urK2yV!mU|Yy0UdhM^$Yn z*V~J04eAV;T}4Z-+C03|&wn#bIZqw(gX%L5e}G3F$VGU)9*#L6jZM7j2NH7hBMsr- zFa0fj1ww zMVvaEMPUdN?#55foXOc?fgnhn_c)Vv*`)eVX{vh^wKmWjOBpK;}sD5gbjyVfeLP{Q|Q(GMF2VP zCO3LKPv<6xxKe19+C=r7TTvZefF-fTYP2RM3k?GF6l>XnF7$^%mLh#Rpc=Fv_$nv% z)J^W7uZe`9Sk)hP<&L3_eiWTk*C6vttw0m2!Y!Tu#JNh=9+HMEv!7WWmaKO}TJ={w z+2O=*pRfcGW42W|Q>0ivgD(z#jLclJSc1%A8h1hYE?G#+dCHwBnbW}RawoCx&m|7P zEnB%yEZLp)&3dndf3G~-s9Sarv=XLK{F~0m(FVBe;ZOuox z8Cb)EI&KtTR{CPCdb=a_dPEkgjf|wr3}EMI_wU;{zhMAy%eE?LHHd;-d45)Xw%^wP zKNZ|--6{hjsAh%qsM4$MyE!hFXi4q?=$5A1{2My;^v4iz@iE+bsVW$;G$%{vk z`2)+ZcU<1&Taq=-pVD2Q5CEqapQzkd!o9k`gokF^;?_S* z&}&fWS444Ob)C9h1Cfg#OFeLMu+Ay8+hmDE@+foZ(tERU_X1ncTVQAMBEh$5#0<{7 zAP+Nw4Rgnfz_h^QZ?hg=A^p1DzY=+i)FrSGaaacOx(oW`03e8>&Mz3kfAL%@5<`JT z2Kcp$FDq4=pvWLMsWHvykjmH8ai_-}P%3+LZ;ND^uiI@bEt{;*o`@8_s6Fu7Gn3~qyUiI$(vjdO7)Kvk>wYTl#Gdf6-efNlGzCFgV4c^Awg-8 zIVd*(cK$Y!M%ba+WjUJC@$znQ))*o;Fmq8h0GYhVQV2twd_ji1v_&4m@mjTklTY|B zW$4)k7PfWa_=jn1m5lnj9>&&Xt&8gp485OB**t_RZ!CWUPXU26bKyodHh5Z14CrE^NPJ&&CK( zuld7aZMBeI&6YZ|94f?+Mb1g_uw9-rWA}_vI1r!h>W}yT=gqrKv0X>*B+BGl|(s4>fEz6My3-hLG;Xoz;%Y=e~~VUKVpTY?-=_5R|1U7 ziilj#qdXFPfUow-uy^PCp-rouu$#0*FrVMW--25`3C!l(VEsD)>QdVRGnzaN2G*?F zHnWu$Yaf$PF%3Xl$eCwWoX?`$l!|%AxB8 zHc63?38skC>HEh%ilYF5TE!_Qop>!m>DA}`HE_m!#Gtu|8~Op2I8{$%(27^Au8e=HCo%kZf3>2r^-Lz)>moUj>_gL3n;`P{0CLcN6`|u@uJsH$LN_bX z${j#g=Q|-cwSuWb4hSHIumP5!e#rJ6-4dswEI>Ww5P!}y^S__>Z`fr_8h?yl#pm{C z*I*v{E&+-1382U^b2u3+*>4s{6bXB?XV>oum^ZyekRYoH3h_t|?N+jwKD*{WXY{jR zHl%zfp8FKnNI{f+vb*_Kp3#_&0#ylG4DYnx`^BmD#0NJ$Q&SqD&&tBaG2Vc?EwJuO zma9$)U;7m3u@mXE?(b(u7=$C!w{<2QsL={+o-g_#*ZhdAy378I}#}0nRP? zQM8@6&KBbO*p61wt0Y0&K4vnLrQn6gnafA+l=9+*w~>zjg}V%Xc!gH0T~C8M;?@c} zfh|Jb$JPKV!Icr6v!C03ztZ_PVW4vocU|b6BD9AGDcLE3)TK?+_z3xHZnDiO)oK0u zpIQ%%0P|2Dmu-+8-XX|gr&%X(Xp zVC6>hCh!sx(b{<#W7>&0MLaC|?a6MCbZ}6klTQ333#RoenUo}lo-WJZ|21#y)2duI zk@B8U4r0FYg}H!arxo=*aAxeI2p^GzeNVWW~F&td}mlUkkbY zw)M0gFhdWp8bYt6jP;G8%j*olo&nc&>M=a^e7ZZN0&=I1fBxj4>e8|=f%L>uVi3ZylW+}wvd-y7!Q4LK2q8?e`sukgDJ`Zi=wiIz? zG4RQQ9loKghgwJig!oW%-G9}k;r&7$_eKhuk}T3rKX5K?uH=RF(GW6-u)}+w-6StP zo`X?D@ZU!e8xPote4Eh|O+@XMuAgSQRCZs^Hqg{l()zs2`?_B#kW`L|$lwC;*mW}- z>sC1=l}Yor&WfEVAzDJIaJ*%xMpWAPC<^a;v_AGGGR%KPkjYcV$Ack2Q`X>(;&&X| z7LGKts}f+G^a_WQ=~aW19581v>fTBY5w_?cBa{-kKu_Xh{zMd^rYy zELxqv&_LsF!j?swgoAz!S8=to9snmJtK2W@|YXBefk(O2Nl?I|OxCZYc1B|(?v`5Pm=69O{#OgJs=DiwI3t=RW) zMp%9x-q3W8M%;VK%ZE~~>RUHziMUjI7%0x(vxZa~W7O~c7k=3?b3_w>U!wkkU)KD3 zTa(vKlSdmW+>$?MDKZ(PE!shoiWb;jUsI?#Pbt)HE{WNXl`sSUuk@9ZXj*J;AmGeaEr5 zf5Dkt-E844$K9F&R{OTuIrAJsxj~%Th@UR%UVG#~VjHQjEyugE+Hvf+bG-|x&DB45 zZT!x`-MYA6cH5TggB`7IvxIt}Z{4$QEP@iqFv-{XZ|wqvwrH5chm&Q5{=H!5W~e)Y z+y3Y8skZsATFd_`n%;H}oL{4CCr|Bg`S)%|oLm6HCPiuOneAViwCz%uFsxiU>ZWT6 z@Nk>TwD#ZLT&oRpGoKpNj^1aFvwEE)x8c=rT;5@-4X*M$^r$Hy_~VdVFm%N&BpjTtC^6hm8scm8C(PYM251G=O7YDu3)NY9IyV*Gvkg!}J(oTA( z)S7Fb@<;6Nl*N`UEzdQ7DG^*1)z6fd3niY}B1CjOWg|% zG4dzTqG*0vDM6Ebq#aX6YV!P(wtb66v5{MMZ>=$>cn;#WE1>&u!|gwmO4AJ{ywf|X z+9vb!3i;-9$(3i*?#EDHgxS=v9J>$Cq+~YTb+0!Ml>?{w7L&(wlH{$mEB=WZfZu(% zmkEfyp0h)T2&@Du_ODbQZaLp(sFHps--hO$@AK?~tD@)tpax+^CSJ5ADlXZn6xG%! zpp1_@ELMx2NlZSYL^H0@0?&`RrFtn3*+VO8|81%*y03mKxt;~*8=40UlpgA+vi?r2 zKByRW^<}8t+RGB}zhs(R^PKIHSW^%cV)0tTgPQ@1Vc?*mn<{RflQ*@_`U9=Ry*n zG&*gQe81urw!PQK5P^W@X;Q&2>1kv-jz{0PD5PHSFBwGc1mHZRsR}Hn#6rVYQYG3k>uPZ4Uag+l)#ZQUj4hGQ5(~Ow z+S}UF_8z@4XWbwcuJWiksZ{T__NR|I2Y2n@)Y_&LJBvwMzDl_;5*|GKML%&MukL0- z+vH>^Y_*{x!x(FkqA!DOJU2IJ3mZQq1$q6BHgBi3IvS==$bxsVitHcmD*H<>ls>Ug zBB=|rTaokPZHA>q$HmAgBb9?IUPE;YNkWqE#=P3I5;7BGq_eM(n#ZTGj@Av`{U?t9 zkS=%B1N7J|-k?#*jB`W7_Ne4!R!b@@r|tUfnv=!D;SpauGCrbfpu5!caq7RUU10O$}I@_PP|c8Izujy z@afY}V$N^B9!k76IPIcdQs)bJtm0n5R!RAiM+|-+LIeg=_p$f7z*x}U#yX_EEm~Q} z0X65oVxhc$(#+w5N)WdCc}Qvop#SV+zh5*#lLjajx3OP1SLR9M4(VI5gA#QfJdJN0 zX=<7vdYItd-A3`QCX*XiTx&67E%rKJ*8XH`TOpc5?^~;ivu3L?<|`mo$kj?cMHLxa zDa&7-Pk(OT_ik2K|(gngHMOQs#qwQxX%%Iwr9 zPL`%RywG@cjWm#C+tMCc-HM3*j-ateY;P8uqsc;cSS0Q*Ev1$Qibx)6JdxP|61gd-=(Nt zlhEiieij`#*L4-Zq})Lnl%#1+JjITdwrAEvolNrp8#ZeRJ~>osakI=+F#3VVfI-h= z^~Y;@PRn*x38k2C%}p0y%RD$!pa&dRC^zB6TA*{2Wd427yxA6#Y~o`HrACKys(;+G z_gjhm@qA&#F2wyceLM%^G_<#TXAVjsRkh0XVBb8!-8XgW_~Uo1Xu=S$LUdO6*0l~K zj8$Zm?;{xa?om>V!CnGvRFT}#W`!^;jrQ(S>JSagbIRg)w{C|VOMK8|BN~1n;pKMI z;zU4ZzVw~ytd3=zQlNpNo3<>?zw2`u={8aH--qkDl5M=RGvH*yt6nQLB?T*0hK~yG zr@zNgLV4eO?M)S$?n`O&7189TV>kTdlI_mtOyN-*w#0M3zHKjrZV&)9tS*Siy=sqIR6a9ISq6(xm&iE2t2YAqE{MH@bn+v)QjQ2}9JdCBJvNi8rd6 zK3Bv8k5o^)kgYRq_RP@>kWNq`POT)6=GD{xeZ=e#DUj^MUiO^bp5fTf8cNS5UsT>C za0Gm+hML=x2d-$-0{)O#P0)(ehBjvN#V*wnMhzP`^V^Xu}4}~b2&)EX(0Sy zP`F)UPNo<%^y(mn`TdQF44?+-^d9hN+y~Gjv##!bp5e`g1sm!5XU*bEH3~|;Wl(}4U<@mf`@!rR&?$so%NOSDvJt%Z&8HO9DY~Q*a2n!etPbX z=`#Skr{vBTXDgz`Oi|>~jI~`OW}lA!f*qdeNkFLtDZXD+YxmZeJRxb8T4RkOj5l8_ zzB|n>W#-jnY6P9ubxI~R7ekOLa}n$rTN!essAr#xTEm26U^u z%)t|G5Y+hPQR#?cJlt5aZ+C=)O=xkkui)O3*V@dExr5SH^6H?tB;DgzfdDuGomL7h z{*ECX3QrG`(Zz;U`RaJhuTT~dHPDqpq2)Ne3z?2&7=PVF@;WdQf4b@YnK(tCmpRv{5cWdB7s zIWofbku1?p_WELZ4Q}N2Mb<1A5Di3R`b}$tZ2o;swetX?g7YQyTyE&iL$|i8S9P7| z{N8v1R0y_yd8w%DW7mdiPf+YPdE|OtgB}Qsb0ZaRNqIYNhTWgVVKIO6IR3t6RJ52| z~s>FGJs15uc2qn*MnDL@CgCpL^oJPUiW|fMdqMshFw{+|<5S#kImQIc!vG zpfAcR|7^^~e_e$w+g|<#d;I+q=MVgEVif^rXG3pq;L+1tf8|C*u6E%xawGm}hNnOU z@ir-WRzpH*nRTAUg}NGLz}*Z?3ZL>7|gI$KPRMyW^;s_W&dI5+cxal0D+4}0&~)@0XpjiM-s1-wK=Lks6f}dX#D?35uXnq}R}U@4dxF4Fu^zfT+{}0VxRvNb;U|srS8~{T|1D ze!vcXFbMHneU7=t9OLT8;sd0i{>Toks9gAc?A4%9hD&h}QcjNG%+5-!4?4Dl2w|b< zLbgF4gEs;7kYBG%*28uTD&A|^XG;%-1FE}W2A_l6cIz9j&TAMidx>yd?oKozNS2xA zwL0_;3k@$TpL6imSiSCqXkQq29?N+C&^z$sveTC6q5DM^6hP@dJ^IJKV^8wpKc^@x z%;$D9zTNtJt$0v!y(cbWxUfuhf?1C>DAFg?rMQ$;bnow~q70NCO~}b1hkc3-(!u{g zqs{G_B_;OY8;CY>h1(#RGRS##uh2t{PSx#Z#y9!KK~`6_^GDcXs41^pZ=j%O53r6gRIftxZi*1E3v9R79->3=c;7daIh07x z@V@52IOYjjQQg_vST0^%V=W0*IE{@XGY3DO2ix_*xvfaT>pqaYY$=P1?(h4}UKc01 zEQqfXYELbHF$s++Nyw|@b;=kG7+|iO10m!0M@m!sVTtKQnr+QOi6clEgU&LR>uAA< zS?oQXX%mk74n^#`G{bTY9m2Cwr*YEO?lV)5j;~Z9Bfp&lh}m4J6?Ao1k~dF!tI=Ga z_yN3_O@w}1fiy^ZZgSSqo_wY`G>dlH4}LeQZIrrmSm7inGtY?+|#GTYLQIE;l1Z z_t-XQH^OfXVVQ-NviFm-I_|O+|91XQ`sLkM9zAQA0mKfemPEGYXalcU3# z%i;_RcXZF~;i_7a1+Hkupy+!h4VZVgXsl-_y6cok(=VGww;B>dQk}580mdkf?j;xjSGEE6chd`xJWn3qeA5I6BBL{BebodMIh%u_{EQKjz#_JiQZ; z6LBEx|bncegFLN^(bDPg5~!j!hk5b*UPScDc~ zcjf|F*T7iD;$8%D#5O=p;vWM;B3dzzV90Um`^Ko$IU`R!%Rc1vCF2&2XCZIIlwKee zMA%13E2GHwlv{lyYuV^43SNeS>(h1h;F^LF_YP(2glj}YLIIOb-&y;-XhXrTw}~E{ zZ7hbb)M%_nDJ;O3?t`peDK|ey?2JPEfvDTBS5S~jvT7D1tS_G6n~X;0Xp2^9tBd;DQwFlIBX%@%m&u4W-2l<6H653#Xs$CyNmVxmq z%aCppebXMx>Z3Ky^mFOP)~(Pb#H~!4+?OT9h|HN)#I3jjz&GHH@n2Eilc#}0Cn|m3 zE+9Z{VLFdTF-WNQ%CQkV3`Q#YH62RBRoX}^ z#p_136H3H3_IH3(Q!lWzo9+@F$GRjhe3~sulz5uh`1+|w!U;$A)^iy(qUy$z5yJ&z zGyTrHh8fOlE==C2g!rZ!Qsxt{nyr%-X@j&?&~Ap?65{a1*VF>Doy~BpHA6w$pS60L z1vul&1Z4YtwYqd}gb1bP(YFqGosn(>TM5rsKd=!?eqnj?0p$GzyzZi6q5bqi?8SY2 zDQdok#vn!m#PrwVH|9Ik5K33EK|z!ot6dGr)-rZ=QPDsm^H%BRKNDl2$|fR1Zo`06 zKs%R&bNx~N*w~`|xGyC=o)`3(1)Zp`+ThIQR6JrxjTRdfx0_)vpyy#}iy0NZHut>WLQ8ghwr08RI@?(;PDo{@iDv=vIHPPYlw__!GC~M5rVm zijpY1?p(=-$Lr;_MN# z)2c0lOf(RBcAr0^jICzs*4^Xm$RP#nI9aHOQkhF)ba%jU4)c;1p$;E(;dx~eg2etFck;_bNuSXPJ{17@l^AW$e;f_NaB=+7t zJg-5s$)Z&a()hxug8EfhO%=a}VV3&=e`3&ZMPP>Cy6YaCU0PnQkq1PF#zHBop@2h` zKRXmK)`?-O&1U4}tT6Ms%a0sJCzOhstR70%QVlJ^_!W+qo&z$68Y9N8l$bA7l1I&m z5qBn}Fb@#DMU-n;?|N|M_}OE%Ug!&(PB1G8daKE?-s~P{OaqF|xKk~j8~he%H)_;& z`U0$haYmDWPiFD0lbmV(%D!x`{vyh}4n5w*;V^=gva2x{?QYAgz1|pvXm73;E(x|A z-+67y-5fXYjvIHaTGGd4u|arVS={EFDBO=FFQW^6%1M9O_`Z{WBDD(xqC`@qfbm{J z#GD0;Wa_75lL!lTVfS069!Tm>Ij_;DaPF7Q>rberXpTUjG^$DNc#IJz{ZC9S;| zeJp+)R$$q0m?D(3aH8P~&|19v5E(tkmnhZ1x&RGDBDNmbSr@YzuQl|2^z%{b`of&* zbWzw3G+o;!>D6z_=Be~R)DsceNaS@PpIqJ`^p+2Vg^4ULoeBi>>~kqMt%00S6y@Rh zZO}Go0eYOG`vGVqYeUwy#IX`0})a7AiclXcdC_jynyr3$jU-h{v{(3J2TrGCme3qMQ>iFZ*R+{9eSKe`?F{t zqORrq9ma!p{%>3f|Hm&pmsAvXCa^Oy{t_hp)q5&FM|v|M zWCJzf_|RG|2nKNa-Sx{eh`D)t5Qw0i_hzt@Q-P-#>?LEa`yfx5P-`R`7Upuu&z{lh z1%gaIeZ1B1YIJnsfmZpjOM5xn1 zQ_Yb}JRz-dCSft?`4PfPn7GjP9}pe*{#NEmUy~LyHs?ah7bd0)NaVE~-cgGUhHU4R zErZYW6U+65jE7$JJzPME%PW8X!Zh#?IwK(!c4=b{av~DEyqmMA6DndONZ?~w@hjmEx&)OrIT-dcXG>p z@&c?#qwlGA?%YgUv%IaL+mk!i+=WiYTt$!K<6Y|W4wE92wet~D6@0a*T3pthJ?*jg zLIYw=c~bqQlE<_edrCEUb0D`%at#()tPy|mKIgrU^zt0~jRP#V)LXnpD-9~pfc@fa z8E}lWbJ1B33yW9-<*ev77n4m7ZEQtpkGaq~^T?OfdGqc5OIL^X^dAu0%dM#J8eA+dAar2?pMMYO$+;7rYPh1X3mSz@vL3jkD8q+VtU>CbDICw^-4Q` z8ry2TF}~UCKvZ?rEEu$No7Y6y8=IJ#Dy>7%7(hN)PFV-uR`KSeS@zY{J`qO()Uy^m z3H=BFy25#mI;sUet=441rw7VHgrGy?m>dR_x)&BxHgkIY%lODT@3qY+7j&r#H>v=l zOJT+W(<|RLO7mDV$Gy2(w%fTo>NTJqo%=eBfWeU@Y9@;Xr~#(_p|J$U{c|zW137a&sAx>Z49NB&Kk|m3C-<;<`R>8Lj67@?^IS_%uGBJb@9tCJb2H@=s`< z9nG7`Kh`!hg}dnhOal9yF?dcKWQ5{1jO_BBD5-@INTb=~yEQqxyhL3Ozqe=%B1;J5 zo9JO+nRd!8QgC5Jd1*YUOPtx{Kg*3u)U;&?{SLp_9X&h6U z4|`twEi0kJsv8}k*k-YWyvGSiax3ks+GC4pH^N9bV{ze`@gj?rpZePMW2UwT2&W?@ z>dJMPkQu!LNx6}%M{Y>$-e584f(-CwGv<{|0C!9qm-0iB$VCZdp39ZO96ekm-Pyja z4(!!)I!Jb+0k`|no@qnC_~H*9dfU}{)}OIYR4se!AsDmfZvPw~(8LFMlkMl;fPEIQ zyvIpTkX5Y(2XK}H)v2NbfL}dtKqvdpuj-l9MsyJj>tU$_%G?6iF9!luuiEC&5n24aHzlTz_C)^dVY@*p9DJWR`ug5T z!G-PRX$xPUcYe)JUqEfVnj8zYtiJcbf^qnAQO|B~YpKiVz6}UKKe$>+$5Ue`MQf(C z0Y||}#>8G)Xb(z3efo;CK=hPZQz5g_Rw-{e&=bfaWAEj$9~s3AK4paQRWG<@+vuxe zmyly;CMFBN1eG6MUuA?P*D>5#F_>5iNSZ!Vc}Kqnz6vZY+^5fQv5#NiALAf%l(cI# zY)5k?upR4(4iq^7L0`vEWiR8S-u`w5!>;ajN;lPunkTvuuj-L-#2NjOS+>MFidcgp zb`8qC7h4;={GUkzWUr#mLsUjNY8_7o9t9KDPk!>13#x2mc?sbMna0%Bg&GN`Tj;aM zosxkxA;Yah-XnKz03{fF0F)`~dLQ(v>{sQdY8O_-B;eRTNsph}Kls7$BI%I&{qMb` zRi-LUp;n_ES%yidqyy^I?O`|&%M|m*_SFBv&S;t7qKY4{@ng&oYXm=OR8%T^3CMGeqkE?2w}BP zez9;!YLUFnps{ICT&htORl{I=I9ZYTa)H0RhA0iAb=lZawmjJ!p0a$unI0nTb2n&e zLXOyOmaO2+B55K!zUOP_hakzy1J7#UEb-g{)l-lAG0)+Rn9W}=93Z0Vm7x-+Y`cbp zHweZ;2h~*^7(CG~lSHhRL}`*wiz>zA0sT zQPWNPSjNGnPyTb@(&y~);7+l@I@Y0Yak_P@Jw98m%1!UTm9e$;$GmBgK z%(OE*oTbE|2~=&}^V&+Z8-k4FOdNy57cD5Y1|oS5}UQ%2bL^Ytsik5wtAjy0uv zo(d$F6UA%6R(yOEN<=6Lc?38Le$?WY0KeeR?(%P2xgr)~`!-Va)(fck&v3u&$@#I~ ze6Gg0C}8^+kK!+C=Uv_i6>*|)#Qle&6Z8UDi%#ftQ?R6cWs7y&gnJ%vRis(IWKqux zWF4K}UzmqZNQ!V8=4q=T$OHe?5~W9b(>A5OyLS0P-%m3 z3$Abyr}Z3#n1C?ceXKX7cBV_j@77y9Vtv}x5j8lUS8Ea@XZ%g6DX~>*%PVr@nwd1# zZ3bSFQVLxjFH$Uv&o!!s`aY}zO6p4PR2$UqWXW-tl&A@Paelk4|8I3&-V9pa$h=LM zDFSDas`{l|(`nlSk&_KXU^((U1lii|y_LR~!zt}H?Y06NcE zV@}VlYGF0jBp`4rytu>T1Y@XuKl(V!8TXKB0-nQF@vK-Fm3-@Og-yviD{{D>Y&5d1`(kogZBa zYq^(<&|wA-@HbZL`9;F)iaak=7MG!0HEgthd@BUP)CW<3as5*T3z6yvjzH1cu9q)D zQm$vdkMNR=)P240?BwUmE!>{=SuK}zJ6^x*vc~OR@*6Ps_Y)~%d4d+4%Z?WIdWreYUIfoFwLag}Q6x+f<;XZ*w}gLYXQ#oaJ0cR(_fjDrQ_}j=-G=0& z{QPR6!+g<};l?%#Q1%uO#BaG50h~0AcDon1J_kf*1|fl`$--FZ0P5|By*+utvFr>! zA%EI@vhFwchwjO{`8*>bfJp}!o9VjcpSah~7ynqNc>WeGh&Pa4VxHu8+IG>_wT`Da z@C?)AKXZug=4;?~=H2UVjrLr|iFm%mpbg2I-Z>1$E6tce%6vp3sJku z`{#)RNq{Z;yF&$llSE$B;LeJH7&siX_^Ai95Bk-61 zZ<(`izpVDmJS0v5c6((&j0?R)uY&qa%giW*?g>o~{Ha$4L@gbEo5MYimN3puD$!`F zylk=16auubOoP|v(gmWg)takyOhj4ANS_DcS2vfNCw$OYQK&UoH){qm>a*G_@oLFn z=#Gy5a1QU!pXMXi%S~kMYywK~=aI9Hm+Fp}YfkK^_N9xx_Dk9(8FQKIdlec3@pp~8 zOk!-fY`{bKwMqcMo;B7k`sM9To;~NZw+&h;jQ~JxO~0M5L`^p;+IItFQc}u)ljK|KzzOdE$j_t}xOH0xHzBtKvDTL|5n*2CTI) z;ueQcX#!u>=U{-nlV7MYu?2U9KwhU@a{U=u0jNzqpa~g9g|+M7lRE7JXjNHJ*F1g` z?T+dFKZ$nwtyK~S=IkMy@dcaN6ZwGYyGE12C@Y!{51{7|bJqaFy1T?C+k`-kKT{jt)j3fb0!iaOroZ(mM(`sht@De_6JP_?_v^~MnsO_srtSr4s{N%}$rEzL^8l|-9Bow6GqxLAcZ$b& z8}u$}GR9)JwxjevGwTfpz*ha4-UQnImqw{xXBkB*xs#NqCh6N54RcJs$_tJOO&80x zj_x3P$sYUQYlY9Adz<`fvQ0ag#W&}G;kKOYI$2_Rj{*m63n}PEs`ma87UTE_KHx8&*6`;4{FZx3s%H2qCsABQRZ+ zv6!y$qASn$oUhLVcGJU(vEls>)z<_-x zu68m}7Q}hx$0gMOqa{G(9oD&qYwcT%X$%nkGpcm^%p>6dVe*)%)v^L<_CdYdn>Iz= zyD5ton=DX*J8O8PZzgrdu2dALA^|$^y**T1GDhUAotLZ5B={Np*1|SVowUP)1ox)P zqfTvJA@Mf>Y933YEWmVb~Lpkl*+pEwBN)Q(D2oiL(7QO*^;gAAdP<= z(%VUPZ>-YsZ!B?_;}9*oV1T11@Xb@`hm!D&*M{ie?#C~_MHvdq{yEg1xqMQd_}@Od zc!Sa5E{Bwj>Z@cG7Yy4sg8k1Wl25;s!mS4Mdkg#X_{&DV5HOJqpJ>}*hUrR)R$-~% zt#D|NtN5pAg+EAsL1B`VCbB>fnbY=-H(`0b(ilfFeg8^mr7+bs<&{_psVKN+ov^GQ zzB{aFbFWQ!bQzF}@X(yJzjC-?Upv5)8!4YFIdbDlc&`mU+t$r0CL%4x`cUuKwf}hnPr^7@xt+t_*8v={UyFrrQSYs@xPI%Ef>XBL zXG*Y({~>n1e41$=R|VX#pkGSf=0!}Nl=NjVuKUHo^FXMtdA2xzEvcS(8fE^bg|cja zrA?#7ZK%HQToq0y39*+~FTMqAXv;!}YPB#aLfMVx(V^*OJXu1Am+eE5T%!!4J8S0j zpQQaXI3rU^E0*{{ia{UDU)j{x>Hp(=H|T|rOn7T@R)KhY#E}rUk7F5yK(Wy-layZw z^U{x#56A0Ga$N-1wI{Kn$4vvxo12z3a)oW$rtqkY*$ zgI(`iN6vIi!--=&R*U5on-c0-Uz-keYBo8YDb|r{EWQI+@&hAsp;58P*Eh^5LgfNA zwfDB%4RpS0RsL&weQxvT7{JZTNNuzsVFG>Hgk>pEF3m5ytab2CZEhpDMZ@ZQf^}{i zr531=5WO-;8=1==eXu zemQ5oCSgkYRbD4bI8wRH=GUWo)1pYEJv9lU6x2RixX{}~#}W^)uqO->vDHixk}l{F|M@L;M~7ITsi9PveWp*2r=?xYi)#Gf&L zcTG;^xg=)w;zmMM4?veSFHs-7P*Bpl;9GEwku@$>A#G?=$?B$*-@p951lP}d*GKKdp34JLU|RU#|SrzS3b)e>;nQm8{3$6MU966T~Z$La&K!+OXsp{Hj-bKx1W zaFH-f+pY~THK#->C&t<8NQ(~)8WX1bEQdr%{~l!8jqT)R#1-y0brSd9kqFd&9IkVn zW$zi)-Cp!RH0_?|p+z~Q&2S^P z0{{T`8>n) zz-~rv?idXqm0T^BnRgo=o)O`6Vrl!n4h2ZPSX}N3vVkhQ30Uye0R)!kGbUT4vKW_T zC~!IsDeQYkwihY25SUanWU`24|tk2luHhL^`j!)~jK`_njcQU}DnjI->1z7;qPmS;jq<1BxwqgDA`~&^@B?Qv5P`1ca#K8I_EL@@LEkez| z!)E!jzu`BRGEt3*$*!-}E;<>_fFHaI1!f8}6YcI5+q(HOz$#r7SY}D>K-#7OD%&Jq z&*oJsLwML~O>7M)p$^ivwGwTDG({!gO5h=9By%KQZ!vK#J6l`?_ z(LlTZcp=V0U|=C@>r(Jor(1km#gbRkD?E!~5vadfGA}4^nQbDHsrAp~r(@3k4d)nd zxAU!O)F!_lQwRMP2akfFbIDPC zF1II9Y=1xNvcdzYUI%-V2%iy2<9f>m@KMvdF3algw8?*^Zs^*@I3j7WF>g=69pYLWk4ZbT0c$b$i z!#mo`&UvZ4HV~8IOud3kWX(>*ng0{|7c%n5F<#TVTU3Vb?QM2>dPpX!vKheSCvE9p zL!EB}WSgS#ej9D2LX+NV^~E7QYz#4;&8$HEZ(Ght|6e6Vm=^%^;{ST704d4;di%dX ziT@iBlK=X||F`Gy|NjAW|0^Yz|JjM}x&g}4jh!c-pSb*;ZgifK{|+-aIW9T40T%k>H7ZCyngqAsj`i4 z;4I?1(7=NGj|AddOC_JX;uZ=+Wr}RhSb75vi~amnY76ck-P_J}6zJR!5UmOpRo({L zWv?zd?%{%GvDvjQf#`Va-%>SzlHdsQI9sbHDZ)-RPuBVMo3@{7OP+p(r=5dTYu^Gc= zx=HxP5_3Ow&2%Q-2iGQk;2J0c8=yR-}@u$h(nKZ0k z-=z&!4%z@!kC}0L_XbkyqD^9XBl#j~_r%uflN9nUELo4ZB<)vx`J#%CI4b|`kC&-} z&8&2vO*3LyXdC(~Ge0d50;n6{Xs}5GcsF*()>5fqFBliYl2J#UdOjJ#6a!n;Jr?tO zh6mOH!qoN&M@3G5`)^@t`!{*d-G2#FS4RC7re@>G{v}LZegBs*_433&agILA@%PES z9;8|ktR`fxdgmeD#;eJ})#m)_>T35V5AajSY%;l8IGE5d@49tfJ#Kiw71iM)N&m<| zKyWyTG#}EM*J5WMe~4~-xQ2s#hqjjkiUse^43vI7@4BBxOH|w#16%BXVEMzPl0fsu z;poW_<`~!}*WMqNy~<{(N^(zyqTraV*`>?xnY^-FBo(Z9s+A_SjWH+3GG_%=TR((nJ zaKGz{t0A}-FkF-eVca!$?d{oyYpN`_+A(u=Z{oSvNyaDxH>_Mq^|0f1Na zn~*{@Z}E4HJ&`})W_VtQ3l+Td=Cd(xxlYdrZ!b(S;x10ZN=^fIj$ayM*>;>^d5&V1LbT zQ)F}M)7w@G<>#S$j5;K_3CcJvcs1=>B0~?&3B$u?>_OJH4d~8*&n+Y_E5( zaYBNE4BP!pyc0*A`)-R+$a1is_fU?Pc2pSK_E&KFKnd2?%lw9ltu0GUI7!_4SRZ-~ zL~L)D^#-vkzjN_5+c70k-fUHHr|MIE*(dByoF!;qJaib&aP^I&6UhwlEd*rR2y_ zs}9f=JLf2$d-ZAC3)0vzhkT#UW`_8Rv_=Za zm9^#ZTM--fIUs-0Yk6S>CqLIU@`in zqjAAfB6{QVHET;eio^=ehsWsN!FMCwmn61P83|Tky_)Wih{udQOFYPrh`WDfpSFX% zp4zxzEADTEaCo(Q+!jaKN4V|UGgkKbn{A&bVn(4dL(a?6YFig@nKkz-f^0EqO``3I zOXFuHGz+TMY9TR>(b6;$It}gK$$9O@b=UsBno@dLvzWG5j>Az-F|o8*sUmhQpRqGS z!xB9r1KQmfEN=aodR0!nHSNqA&N`Nif>?Qi_aHK(Yl8hBViaFdfl4x=;ln8tok_#7?oVXrGAS=n3*PI> zI=eTYv;LR-bei4!^nbcUpAYSMUKv~p$ zPx$)wSjvjcB1h66b?bH9MZn7>bEn~O|6#QOT_aWU(-*HN8NR)Z{k}Rb&fnqyqp|`Pdw*OR!?Ost>*JBoCIyD#-yKVxj8ApkwXs_2aES|_~6eD zeBFHKkzh8F5cQP)CK*)?dH9ra5YEa?DAD3prL`B=onR3LTYaRM+<{jQ#Cwq3f&u2_ z`(ao}zdCu^x!NDzJV?3Wyv(ZLG>iyYXJN(Y#-{HseqL*=!F{3p%oT05WhA?x#$a~= ziADzC;JH%M8q*e+KU*>1jRYCes9Nd!v3p}uEnrMfkax!7A1Lsq(mw8K!ntI1$B5&Q zJ*=jxlvffqx$`RoN}h-=(e(Zis+F2(YLJ1S7kmtnT|2{VbvQmX`P!}iPfzZ(Lv=q! zO;%}rjL0shSDQ$@nG6`&tff88ArP9k{f06tfjFQ(NN0_Dr;|3=jHpk^W3Cgxe%#bZ z+aF1fw|D7@ja9p_oF-x6FA# z8RvQ-yYF)3-agYgxsog47xc!J-<_rloZDyQX!srg5{rCmru#GPryVSpMhT;VXSZ}~ zd}g9`DeAaKOai~#tBz#5>TwY11a&+1hkh6By3XaE!s*)L19(IQ(1(YkyU8+-M8Yt%aZc`f#29SwdYfp2G z!veZYQz$Qu`<@+O( z+%0&(L?7TZ>JM7$J36zef0QuivH_oEQ(EzZot~w z8{{?Bw~C_C0zW#CP+z+#)-Bfio}R?V050Y9!{!&=L>dPLum^pVAD+ zlMC}6{e9V$HEn!qxSu~>YOQ!Ke`$olOt}Q5jat+2*P9!MYna7LlDQ*T84mK%s9@P{6tyyW+TyWiKTwYxyDo4Bf*XH}qz* z0Gbkr#M{vxLeCgYth#{X^E{aPWU?Xe%$kMUzhAdQEWCL817H^!r8N2|%cwcaL&_ zb6J;nS`Oc6((}yb`feF>Gx|xK#fuvH9XUP!RAiFv$!duub8bu-?r-3%$1Umj^-@XnPtE zmHN*sF#6q)=6~J-$r+|2z5o6GA>aS}1(nP`m0nFpSM4zk*w%R!zgl*>o9Bnt2IRn= zcgcKW$o>>RM%-0`$iJTw_)YfTL>)ri&d$x%_&CSb=fd$7cj#7(i_2)PeLCl*UlkY! zwX)TA>CWAlvL2^l=J5kAl(N5{{ISiv(6b(wN8MEqhx*YK%3J^9&?K=CHZX~~*(X{K z=n%x0llF_G809VMf__PH+zAoMC}T~$E+Qs=e+sC$o#&fvjbF|w8P%S6fZH4iDx@^Y z26)K%(M%3%i??%nJPEqh2bI22T7ACP&r%R3b#EqymvyY%TuZ~`U9J)z9 zoxc0JUD<_}pyj7q(C*rES1pa^4!O5@{`W*V@5Fdj?=JYMS=0;abMW*zcCK6M$?0m^LNE@N<_b_kXX6OzIWMPhLpF&)OPtPnD}lfFD1?! z5tG*PrBS%Ob*bI-(Nq22_~Q1{$|@t&RyWs0pnPJSe#iLzOiBfuzd(s^11^CVZXC$C)?>2a`P?(V#OLCi9jKDOJh2b@F`I#m*%Q<_$~l`&1zvh8sq(%OnmZKFkFh@r+V)np1? zZAwSi_X=ykReZsZIP-Yr(3MA%TM9lJl5v;BSaf`^p!vs=FFS@^|&0fJhV@~rmp@$;(3XufG_?W(*wW4vj67Cb#nP(Vi-lfVA^S1Ku zLZHXilDzR86>>8#c2gaDK8+@4ZVt-w!3L@cXU-!Z`N3VchMU`#2Y2V8h|;$c zq-U<}EQM_^sZ>_{?)h(f@y7dH50h!n7foC}(#I`lbLMm#*JIO~;YGFb5f+7`d!9i4 zI3vSnhI2naNNax(rnwrNCf8RU5Vq($sii=kwJxA9NBr*j&vtU zhTA@u`xg1E+PYCKod_$z7iDqb87YaDGG(AKh|Dei2#fBQ1L0NX2F(HxrZ zGKtyn)FzjnyfCPFy_V7xG3nu|JusL3R^JCBr&%4B`z#LzeeATK6nn2ZE!3mv^<8>j zgXSGWA-0Ob)S@MIigt>yN0aCM;|1&`Z^dNJEso28%CIG6dZ?Jz8M6THC;Vi}y_!kH z%b(aBMrI_xmxfxCfyHytEtEbiUD+X>p|XeDhsFfY2)9I$iq{jC1F|PWZdq@VUz`o& zXEP;^Uf%f0=JQ4=?2uU2WJ6D?XM?gs>SeK*3j z6w;3I3z?K{WXNjT+vW{HnuWty7l^BI$;GYSZM4$9(U}{*UDG>pptMQU7Bwp~@Aj1A z@Z!Su?2DVI;I4d0E5D+sL?Q5$p}fFiCN^txxTU7WD5`ZmiAW7oWwc2HUB;30mg;W8wn1?|t4>e!Nns=bW z6>i>OYLNry_h~c4yP{aW)GpSK-+R?hysWWb?*uU(q<=LkSS*>AQOMP1|3JTb|Jq1j7*he?C)a#e&W2nld!2F+%#%5tZJ|S4t z?z$PyJnqie+zdKIZ~l7+&kWmsuaZn>PYyHqR@-M>wt*7$V4q5+8)C%b@#gUzO-vA6 z2^E|#b}DiW9+~9byClKyN^z8`e^|qGBnGt27NvW)1gzF{y7`pGRX8e;pOnJnMG0N;oh)pP z?}!Ge{-@jgw&ztipyV*#wmuayv*y-kpN_6ux~|G%Cr;ca;5v&fpF-C?6Vjj9nCVq% zqn|B-M>(BqQ`J34WW&}A7CUO0nH>DB9D@5n?DB^2)!+zFaU~fn&T?qundXp7$nSpW z-7Hd&>+d^^vBDpf^6QAQ?#+5{;H-aBs_A6WrntM{T-ko}7IXr~%NZ-}vy5~vaI>T9 z8h!V+Gj|30IF&!qXg@gbr+gA;)lF1A0rrLDR5Bug-DmP^o!_tBRsO6b&x%5r#0&YT zBq*C6Ld&aRXab83N@-+ejv{#Qg`^ml{kJ35Pa{7Vd?EEWx4)8ra`pTv$Xz-Ua81wt z3wdw?!A`4e$!aRtVb~A6XmU3{t*C!^p@zP;4&hf4#=16i98-Z$pU>oBWY98WxpqpTY*I8u1T?(-hSrd)hJ#+Vy`%DqUTQwLac?shMQ&o91A?w z7KltO*ab>&TK-HLwWGDYrTTr2+cfou!E9F_k?)%HnU0j(?0X>@4ZQlP<-A~K)vQVVtV|yVsGifX5?D#X4D(VyqBC951 z&Tv)xc354qxhj@Z6n@e3aK$nP=pvl z2r0J@@#83EZOq!RU$x8%qTS zBFMe*@Y?E4*U0xi`>xe5!UtyL)2hJ69h94`%9iByI2gCOF;7z^FfU@Gxc|Q0Pq(+U z*9rE@?DHD3BHM5#2R7|=;8ei0)Vq$|A=NF8T*Y@k8A^Evkz)Ebq@>aMCv~PA^wS2sBH!C)u)@Y5QEHKP!?oGN^NBjj1NbNb(XSIoI56#r~yj zve2BEf_iCBL^PB_KxIcvvX0aX;-~t@zVL~&sM(2w00}O%$+fWTq~#jn;aM>ay5hyb zt(HA!#oMD*I{sQAcq;Wl2d21$|V`=O8d+AYG zK|eVmd|<3B8PWQ#Qa?7 zXLoK9^7JGZDbg9K)IS2Vo*tJwYj*vb3)aWe@lde0_ok-a8r!*VlDrc0`2O^hvdt#- zV13Dkz?k9~-O6dO!WWyG<@r{^S@FW9txDtNl~g5_QK@HUhl9gFx9mG*az9%um~~xk zmLgnFYFxjmOcMmJ;VGlYv|#DlFStY9Y=(ZbVd+am+{DLywVbhyjk1pOx>CGEg$MB< zCuC5YN71G6gj9aw+}RA1^;Zux2@c59KOFer8IWRpDnq%j&kpFCee@DL-cG;$y?l7f zynjIxY`l)_$`W3Sem+{>?d_r$7YhPOc8oP(w6i~ZKrkHg$;@!7V|@_^L)Pq6y?+4{%I82+Oc{Soh4e)xP~i7j-i+o*RnlQ~lY?4)tk8$_eB>lCM9 zzC#%@xOwHyzgH&T@5k;5spw^R&0q|g^Cr^kw`iS;@U{28zxMPlr*K1wsW#gli!K|@ zrOw+1Xe=lL@vt#}z*<4P@YVF__f8i&ru1@u>wMjdVFRxj+vX8Avb%fImewN9)q&D= zu6P%_wL`yYpluy8bz+WP7gF=KxoBHzCfzY`_Eu8kmU<{`(PA{z`O^!Kaf8psB+&Mf zu_ffM-^HJlkmOGYEP24U;Veq-49h&+Mfxuc2Qt`^#h_DPwzS-eoSHk-gxPF3x0ZaU zG5U$O#Wy9wb>rzc9qr5yEZcmKDCz!{Iq&-3fi;_x86!RTuMMZ_B|in&!@1w^2(9+S z@M6nzPWS(fgI>cl8m9lq)FRH;GWGY#E^oCA^N5L{Y7d9t(e@V=aJ^$Vjw-^7HVLRF z`9_y#>1QumNY2R9VFz1bkNXWdnwe9DiF2+W6_ZxI4Q@bvwcu82+0s_ayQ_yresyNM zzu^DiN58y_+L6YI!@mRA{k`4l-RFo`_*BnOZ(URIJ8m1z=#}iOMB~ki?b=4QKf~9H zztw&F0e*XRDAxVHva-6$?m-D-W|5llvcG3<-khy)SLmZG+tV?Az!3+JCI?S5RR_ED z-J-vM1bsqB{wW#x?Yg3tj_=|`U;670#(C7B(2?{zd0CHiJ#7zJ94?43a&M3^>@HBN zw%0@+_#z>dpKfBMcrsxc>Db%$>@zP zd44IgIXSVLi{`xma@LYP*oj0>$*&R_BJc<+>Gi3zL^wxX;+)g!TG~_o6kHX>F)t5o zO=A7F<04}|at{`!D3<=q#U}3gHeTb^&Y5c8>S{;hyXBkLyOSsw6(S|O_qE_70`t<1 zvPftsld%YJ1`?zpsb#FyF_4y*lMa@dp3&uhELw$9M-j6A?$aqz zB4x$Q@o;}bX)wkkk1nRF%A#*;Iibn$`i>bI^`-CU4@Q(P^D_=B-EA4u=b3b0c7G0C zYkD8 z2{0AA80oYgL)!zr$pzdJs#}hFpj)Kc4Eh~3vS~E_cDe3h!3xe!2{h$yBYkX4St!^g zeDhQC_LQCUL}Kyi2G*{=!veTe(5sKNB%)wk7;z<n*1#dLVq}i1-Z0ZNo#|r&Q@()iFzLJf%)%VS&9)&-+6SR(@$;eXYigIN(0XYfKGVV$>Q2Lx6WH@^Xdn}D@1Wu*9D9{$F{F*tDUv-xgM=Zf`(wmEwBpm8X?t=A45TjnxBXi#Nk=c8nJF~-l(m*8?67s4T( zxcM?7mt_@I$tK|vX=iLW0ZfkjXc+WcaxDJ!M5mTTcm5YP`jEm*;+eJBUdGPm?d$rJ zKt+oZg>ZfeF(A@W$^th(-D)vn$ZJsm-S`uaGn3T~~wG5Qz8rIWV z+7IWXwsvJ>6j7bR^d|DI`l1i0h>ehDJZxdnj#yO=b%Nw2f=fQ20sr{65_}sC&fp~Q zVrsJr2=)M~JoG1NJ|!2EXX*JhU(DmHJ?nMew+UrsBhi>}Gku;A!$`;fN;odReRdQO zsCUhf-?zw1Vci{tG2KIl{1T0GsGaMO)1*A+tnDDRLgr;g6naReo+h+#TPiB-tu9KM z(}u(U{q52lG8xlX^QnTLWFjxvU+tmTluKcLk|>F?^mkMF)x6wDo2g3}!qk>Cy>5>e z@vbdXbi!`D7+R%6Vzu8?h%8<;wI*CIi&>SaR0M?J2*%9qa<@vi=fr@R3VG54w6$?a zH@Tk`F}dNqY%`!a^sYiCFC$r~8(Rk(#Z#h62;!fpJH1QL+qbL=sHR}c3M}dPKU@Jvb7g%uyLzXp`3d6?~5+~) zj8EX^H3PePeHHXkkBEe8Yq1@S2TN43uJ(NSVXw)IHeW9fG6^y%=-8!0i%hX2+sD{z z>nbwDhK`}*$^A9Dd0nSTIwru`n=WW&BHTBVtVrEz%k+mnPes)U3^Hq>`wD;rioZuj zP|6d>M^sl=5eK+zpKH6|a8n@cUketR*d~vP-autzqsC@Zjj#;HLVq#OL>SGN9N)6{ z46%lvCF(bvCI&8bc2?`N%v01AX~0 zt=h#Z)>oEfpmAGbyn*X}5tQ*Qcnr9XZ_prsrhX{K9!GKf8J(T}TB;*!T7NZf+hRD| z?CSY=m^0$Rek2+~dPjk|g!5FV!x=9H3a?1@`xa?Z=K@en#&QXy^TuGfU;@f!JnQ?n#$~SwN&SDYwub&CO0isTk$?I5EF&@s#<13$|KHmRLueo!f z&EBTTmtO-MIX4CF{v25D2AeDL<5LN_dvS^Ux#Wd;{tBf#zm?HVTxf*L%E<6y1U}8Q z&ZTitIyruxH}mpeRMl0&lYms#d;Ie2QHuGtezOoRXflVw7XtV(~ zo-aOlNvvX`ag)&fZgqpXBp8l)$yOgHHC)!S1PBQuKf};A#2_(_ts6-|Du}B-{%-zNh*@j{TX-7yWuDK*rgyf4w<%s(Y0wNG%6$EndetBP5sXH1-h1=8mt8FU z3HGULSL88)W29n#_QA?llWO4AvwXux6t~LN{6oBmwT+sCq)z!4l=4hEZi7W4c@0s@ zD`jIa6Y-!Iw0}D?-7nB!RZ3jiyf8Oiy%U1|vL)jR;rWFz0>>_?B?*Q}%45nYO6Ip# z7N1e&4fy+*4S^HXonhYq=;h{l2(EoUzhajP5-x=0Jf?_5)?SzCWMm#yl{|1BG-?F4 z3VM$@rQ_G+FH@R^Sv^}!yA<`_7cy0sNYIyygvRfLnrOB(pB&cY3b8*JVSwY6UU?1W z)99Ib(qIWNdL-H$>KRXH1Z~4Mcat4DQ zBkSvob$^+U9WsK(36n)YNCEKCGb3dF6FR8~I_D+H7hsEAkiO8781UmLJSiV#qnaTw z^Oxf%F^{?WeTE!7)TeIJ40(>k&s3<1>`Hbj<78iVe5U3uWAR%HM|7O$^gIG{z*oj^ z{IiUPAi+cxpo?G6C%f|20uZ+{kn$wp}{|^2OlG>x!IKI9E!K_Rz_a1Q9 z8H5FH3skX0(#iT^!a0_-!8)!nE7IpM{L@CD;se_1aveoJTo4{lwAhQe1g;=XR+ocG z8~X%%K5jE+`0~KaOls24YTQ)p1GS(tGq1sc?G-!G@bu_S5*}?JW0JO==B|Fom98OW zul1Bp6~+D7X7BH_oO|5g(fa1O};YptyvWH zkusrf`ARupF=pdCj%^EVPy|$6ShpG5+-|M}ezFVyNW$hT5=EYGD;xL{*1AKi-=#lw zU$wW^IKO9dJC5do2dTHQC=KuYv6kG%cVvq_#Cf`G7O4M`GCx0j~dl zmdyz@L0>%>Duddx=W-I}s{sD>Aw90e^eM15avHCR4&fl`bF+?5%G_pjefF9J$Rh!( zy=MFHVxvtVll9jj@w$*=0BT;vc6NGK8OVrR^Ekr#wV0qc>WQz^Pml>`m5&b(Ta0b^ z<>*(FT{A_R=2sR;Md}V={+ttGPS7*Ev-n8?elmV@$sd~Z>Xis_h^LQM5w;u%!u_eM zaDEHND%Jy$UO?;IiqMN&wD|i3fnvD$HjOMKb34q~L`fOf=)OLkrIb&E+mNLAMmsh zU<)IK9K)0Sl&6D_@x~8oT0@~JUScBi9f5><0F?mFMdBOAUv%>1szvB*amr=7O*?%vhb+?Li2UN%wRf+iF)= zE9riRvXvT>C2e?*^PY<1nDOua=ki$d(g0ZF=Y<^kN@K19;%Un0PulFO(Q`LG&xT9e zt$#NEU_tnDaC93f5-MA4RQMcO@bzZ7AQutwi|2|MQv2Z40s~|eJ zw15~9V6`qYw#GT)s<*?#W<7^aQ)ExduP&O8cvftvZz`S7-?-S}4e|2zIQ@+7H6hz1{^A4s=@fQTjrK*R5j|K> z*)=s8t7}&amLhQN%T}A~&ElPSiXK)7e$)dJ=(n6^bn|(b(FUn4$$;${!-zuEiXEbg z43`RWvfMi0%|}~66R5esb0-!TVJ1gwFhlW2u~3xj9QAe0H>=p4DE|eEms{#S?5{=W zPlMkD&tj0^z*jHv43oT)YGlxSaVZkMvZ+fxTlOxBH}`kwp}%QE9y> zE7EN_kRxPsAINbewr5 z5g2w7QKw?14vy8$Gg<-KG&aiu&{_CGp_+{MWgNiCFk$BSbj*}L)n3+BTIum~`H3mY z)S&*&&oVcP*QH>_!Zz&b6x%52*Fbqk>2LGUic2w6-K=thM1H&^7 z`vP=(g79qZ$~Hx4Dh!4biY=Qkx2+Q^iMdcNwEvMW+^4q5wft8+cPbZsmH0H?y-L-8 z!OQ5Alk|iFUw}2(q^+)?1afm2Rwb_5hxYT}vMqPliXRu?f0h^y6vCThi@c&6qg{l< z*AM~?=xirskKPg3DDVZ)MNpL5_fB=MbG^OI)q>Q2V~p{JbLN=3*Xi6<1!+OWGiMXN zzSwC0CtYkdcGPnru4#S*G>HnMb@=rQ>su|W628n|&ourEg}Ut(7pIn!Ufga9^skMbn6xKXR-+V26@9CoPZi@QDS*_*x@btYDH zIX|Z5vb2`RM}iKa%}VAxvtPE zRet81$(X6@RJPm?X~fCTO;KC~(&(gwSLj046wLuuG!9{kOjjb(&M>pwq0jbN_{UJI z!jk!YtwOElpSHPc8CqisHx@X%Mq!3NGMC{pH=33?*>!`8{E7QX^UY|@Qlb5iz2Og+ z7^N-#5v_H8e8tPI7L|B=kqHrtF@d)Xd_k+ZGN&ZwpTXZpDn`;)y~A28U!EVR|Yq7CLt0|YkNsgel#(Cy(LrJ^ZiJHsf0e=2eoKqnVSjLcWg)*<}p z$Ym(BlZFo+tY?1t_q)g_0upUuDkwl>`t?t*np=m?EJQzpYFfG=0AV?6928Y{yKFZ^Z$bAE$#04-xxm3hl{`ZjQ?}d<{!fU3&Zz+ zESvti{g3}ky9L#>|8e3X_}2fs5G(Y**wTH6|3BaH|Nja*1d{r{S%tpL7cchH>{WG7 zGM^tN-QhihVruDIjlXusjvW(PZ@mb4dua7)Cyh>HPorD!U3>Dhw@5IK6S^w7u~;Y> z-0r8F3I}_N#Z~kGRU1t`=TDoe6z$JyiEuTKV9plP7F+8Vdesr#(0Rd!FuV*N4}OU6 z0ix0-&KV_{b&t9NPCWJ5&@bM=xW52DT!oQ24TJic#YwFn17H(vsCV+S(&DJ`UMemN^^|K2Y9vdb}s`Z5=8dDMJ@ z!X8xKb+>W_6o~k=jQvfh`yMaVKZv>3H}3#l#8fa`+Ekq(RdcN|X_ z5?Mg=Uk9cnY~`RbEqJptX;|I_x#!8DBeK6hxN(BV3p%M_to7ULyy9)~U|x zP4nvHq2jVBnQU?X7~(}&pslA%mQKJ{sWi(O>GYyjEZ1EsXuhYvn+!;PHLsCemb9dy zcl)_$aWpf0K{5D?pV*!>(l9Y@Y}`C(2nY@?GeBkJC!R9^5WN>~9x)^yL}6*#I)S4FXrfp0`Yq zAD-9ZeJc_z^Is*z7_Up+a}Bt>)-IYlPWAfFgQoih4MHA06+7@3ALZ+mIxdqx?`cd2 zOlX)Yg=I)0WVPn^?v~(j)bZFC+Co>;GbD6YoEFKS3)MIF4DOFC0n)plwQN5GtaW$y zSnAA!90WhQONYJmhp(@%8t3D4(zs(%v;pa3chp866tg_1i~V4?;G$(Wz4mb2ec!Yk z)LzTCM?q)@Rrj1MY6%W!xiJ$d{ErgH^7pv1;kbDUFN4OACurr^E`|gL&YI%&H;-q@ zS2a5QGMv;9;qC;%H_%S&y6T|s=5Dot4?et}dic6MS3>8_fa?YC>#VL*iHKUuWqF!F z?{&EBVCo&$q?@7ZQkE2kzG9qt+lzF-U5~X`6o79yE4t6pZn~1dl>|i)vTO~XSZgrf zpYGQm&=b*>(P~cHjD-xp+k-u>svDH>4BuTVW``VYGU#JUWnJ(nyL20rs^u~4{+8h^ zFQb#n$nbLEN#;I!v0q26bRcszGsji6eR%18;<5g)elWI$b%3*%xY_D6Z^4fPav=Px zBgtj*r&PbNJQ&P&IAmbzko!Wt`~1*(uwxp}0()HYv~fye{)!%NvlxbJIS~6owIssa zt}~=82pISIO~h#8t8m3<)>~YXLq==#MYjNZa0@AOv%M$2$jelR;vXVIx3z7x06XQ@ z^#P9#d0C`yUjfUU8e)Vy2OD;a;&unj@80pOt(`nI?jGL4-%SQ%+hpKNdvlqU@*Y#; zR;z*CTz;FRfbA^tA3fS7SbZQ3s($3wa4<7rsK6Nrzr43G(EbUj&o6f8*`_oqr^3X^ zyrN`N-jl@X=wjhN>~Ezw=)4Yhr6X}Y>A`^BZ!&KzWEfKE1gfyDWC0t_7?@uNT7&)J{F8G&Z$ zy&kYTpt#Z^H9(hB4WqnlDcP!Ay)!I1$qMHA`WMd}uo4Ia%K}ut8wDZ=Up{{`9xo}s zij|XutEsHY=TQrXZCF>Zs;h|PKg7(L-RB)pg30d|$W8grq*)ZK4Hy_Jy}5ygfjejt zHJ1k2qtuoghY6oc(;mF55X|l2t_7hbXNA8tk}8XUHG30l&x4m4q|*03J9F7_(X zi|&+G?HWGWA;V(b1SXkk6rQaHtYN_PhRSGUYy7 zK<*`II8X<27!}&BVe$o$Jx@&Fyhp^K#&C1bq@2x_pvfqU-BhvEQdHsBmi6B?I@2cy zY(NMxGfa_Loogg5cd+Yh(?Q$L;PaH8o`ar6NXhjxl16}ho}cnD&o`?7lFc}4Gtk7Mg#gA zVs-5M325cYo(xh0hs{{QHD-wAjw<*KI`s)nMnlK8_jig~5Vk$dN@2hE{#_F^=kOxS zV4R2I$NBL^_>RJXtCgr1MsU!rq6{Z&>duVt?=mItbzAv*QG(MuYij(CMH&j@OD~+} ze(fQ6T*Tg6y7l4KF-h>bSl7vy9IB``l_LdR08PI*ywJZ3aG&7$L zW)QCp{ZK;ja+Y+gWMs-4Ry}aX!5t|Td|4vzGN0}3fIj+2apSv8W32n;@tie~=YKfQ zp$|hs+}srdjn`9`3hQZHaZ*P{V`>}WQ?UTx|Ffd6SX&{Lu;U|z&oZ!g(?a}@QfKO@OoA@oPC%mG?({;w7qZP7{HE6}2z%NF6%wJk_oxs301QRCeQcZd zZ7D7hKSdTItJuzs+5lr+fnaK{Bkh|CBOpvZPaqz`E2rqWnym-#j|SLF0E=*Nxhs+g z_^HqsW4q1mE3Pu9Mj6^R!nc0gq1j&S;X;sg6Yihn8)1pJynnGmFXm1hJyWw0(6GL_ zAJTnPVg%zn%q;a zSV_;@N`Dmxs+;$T>Q9S;=u)$%g5A|r53ANysrsmH2tgwM>^E&|@K!BjEPANj8%p<& z+#R_O0TPqcpDW1$sS>8G*hgCKrf!S2u+?eyM+63|IH`+8${zReloY#pWcw>(5gq@) zR_{hp+lTBFO{;aKb9gm#-c7-vs6w#>@|0b>=Vdh)bw&f;r*R>l(5jiw53 zO~RrK#PoG#zdZlp+j0*M zM|`jt3sDpJzN{=y1J5KGbd|jN`q$j!k!N4#duF~S53@acCY%$61wUHM!`(`S<@M?A zcEgz|U1_iRp-Ygyt~C_yE9hoLUZrP|v9ek1yA~^TnTDDH(Vp$Ol9*#Kbjv~-LjPUF zd&%;3QM(gY{Rz#5drVoZBej^$eCiIf9rqLYXJYZ&Ll(IFV&tCTE&a2}n^-uNU+#?d zb!k40VwLh@|I3^7#$U|(5lAg>n)%rM9SDuTa-vx0tg09FDY@?Mw`dZ+^rmEw#o+TT5pTu z;>HiILmi>@v-bsHQ-^L`0y8s=il6Ov(rFCMXH`TkJ6Tbh)1z=wUk*O;XI1v}d{O(Z zs`BMS;);Vq%TG48pMZ_sjnj%D?Z#s(FF37x&npI9m-G})*#mbSPPz9n6Q1nPDv%Sm zP@+#|dyih;^Wpa$kKS=(1gu_*Jbo?>J;Ore1&d%fY`$zPKJylF#{EBZ@}=LsUL{Ow zpITQ?d+NR=2syWnZX_gpdBHKr^7HyVQH%EZwL{}ekd?+oH}{2SW@X&+UU5>EuK5?{ z^E6xP(k3(dnOuenX`w;u*{stp0nOSCtL^J46k!3|N5@CQc_j&3ccgUG&d%uG*0qco zOkbkC7+oG}EWJVY-gwi~)b=H6&qE1IZ#)6t{p}lbZ%^Ie{N-G5U`m$nb$si%yK8O} zEk}!7&x(es-&~SIDtN%JA_1x~%C_=0t;mX_oB~{IR_U80fnh0A)>%RA&!3$y$&GUa zr+Rw2k8ighcF)hf^siaTKX^U!lI|wE8n&{VB$uL@6uAtSBl1SSdZcv6I+}ApF(}*l zIeup=+nK_*P-Ks48V=DwdS_RSdWCyYJR1)?2`NG(QGRT`*L65Dt3FsNs2hv^RJsAz zf5kyOu22=v7;kjlcBF?D>8LM}`|KZjxyB!bU)!R$z5$llhz0L~fAC>Un|aI%-+E$~ zXQ{A|U|A5DwAuRtqcfo8m7|~FD+fuzg@9ABFlGoz;`WaWSJ1x80syB(fCY?f(f=~C z(%*0~tOYw3|bnn`HxIvMO05d;A#l3QcSJDIea;LB^+Y!f&q zm#Cu?{YF6=iBW+HY2l;Wj}7k90c#wPkH4)mOE}E!d)M5W7`!f-QJA0m;7?Mh=c~so ziXYSoQ*>nnBxEiJ$LBbhjM#a~+H3Ic+}yhyx-yyW1_dMK2TPmk1PR}zKQF1vhHAm3 zM}R7IfFs?D99FtW)>&m?)5?m9q1>4(0bXIfOa(}feFm2n%=>gAETC1a{PoIOoI}Rv z=Tl&5e5(Tpm*~%`+Dxe6H^Wbg#Y!WQV?OvQyf zxZB0nx}Z4*?tCRdfBw_Cx52i;M>BQ@-gvH`x#3mHV2NP95Zv!AlFgg;=xuit1PyQ% z>6{yDUy-P;x{&D|xl|KJal!|GNYt_J*b9!z#sw>1v~r)^JiKg@Q;5UnUy2g1!)!xq zLWW(vvbFeH3klY7QxD!STiAUg9TD&%MY&oaqx=a#01&3HbAFkd&4b7wZvo-ivS#2oZ&Af)imdRuP z74TfQkVh{5V}7T{;$(b9W6VPpy1U0>!-bqPr_xPzbRwe-@s-<k% zXf%?~YODGVvUOsghUTqA`=#!UCox^3C?dT$QZwsVv(RVf6;2*Ng)o{YV!c(4d#GFA zi9$>^FG9uj-d13$(s33p-|LpyJ&6qW?=l=;i3a~X-+Fje9XC!?STL|={qe^AL!)T- zuaj>?dGn|&@+H_}^vJF9ol1v?F=0)D$2CH_RT6ZG!@a_8UpF1JGB?h@yvZ|&7V=ma z$iZLybk1~PN;_monQiMap&S@SnmH(thmLyOaJ2amb9VW(R`j*KQo%B+1wFnd1>5b_ zRTNl}ij&lLLAL+#Eh`~p4Ls@_A(ZjHc6em%&*hv`F(LnGoWSp_O=vI^LCt3YypO<9 zL1!=bP88RPnW*K|d)m6~zqId)2YOLkSCJ7r@2f|pdf0XhXFT%*0scHrD9+e0yzP=PBK&4Bb5UKI`pGq$MGXQyOyX~32^Oo1u zq=nuAM}HOUZ=15}XU(TBe%xGb<=)WhktfZu7Bjut3TMaU6lAR z@yyIM8ayqCEYxX_>o!!Z!LMW*al~bh*u2{XW(-N4lV|%)H%X5U?fy^smK!g?#5#=# zwvx8#r&sVP?Ng1D5{;kynNyU{ayn_aE{Tbs+-$)mo9}-7c)yp}Uva~a+uT=v{=esP z|6i4RLkknkRF%Gb4E+og={rLIV@KVS0%Azj;ocXZAk#vtMRX+^mGrac=tYX;SdACq z3A5R#&^=l}7rC-nYN?R1fPXw#{k}z^+|!Bjfz zd9tPF@F90#o#ZikGX9UVpPC5AacYR(jVtPZ5BATl{(iL*R;r5hfzK^w&N`2bmKHB- z>gJ>|_o*L#wtIzVl;#!;dXwLc z`uT$)xijm7Fz$+q<{C78)eYjR`sB$#Y7Ppq+~cXWZcMU3yJ9C&jw0Of+RcUg{A0$p zl=E15W>BZh!iP<~!elx@=O+ZaZ?LZKqC`xmhg~&G1V(?E-EG>xj6Rn7jq`rf(cV&B z0KU(d9|D=Ut(qjp6Gd*Ox1ggL3w~-&naejrrOT#*>Xw_^Tm83co`hNrtQq@oY)Dnx z92;g$wY7Ds>fiLl?>qg7wA30@F@qIJRuhr)uY(M%h=-Z6l8YH8H^8%Rm#36m6YUh5 zTDcGcWRf|pM;^6Z_zh`Boy63vP9A(q7m~^CG>kT5qr26yNk|pLlLGE}h}yfc$Pd?EZgKF)RMA2sT&%U42%TkQpnk73)(+he8~J2F&GE3O zVd%{!H_)7R)_Ueqg!TA|uqunrJc>&Qtl0!eFDKJEbW3D;JWcZ#_i1BQlX-zc7>sNo1+o#<1w{p@JKI zgKf>$A1#u-d*oi5>oCcO^eFNT4hbGpo$(SY$61D!XFWkL>>Ub%6eIhPf% zWeP9ZIQT&KbFe6lmE`VmMv+D$STc|9LmX@dfo%OQ4X)UeicyhRHilZ}_AHxk2lZBt zj#+kR?rK-bd$yLGX~Rq?kQK}`_ma%6CCkwrOM%*Qa#5%eF1a2#dNhP~t=^g{POxo^ zG2#PcmnUAR;gxpAua7}AZ5~x(r+Nefm2FZAfB)<`mIo$s-Upx~-@}+hnR2%}J=YwH zkdNwXEdOegzTq~^LmV!0a}oZv(UzD?))?JB!#udob}z01D^!{b=BLp0kucXqG8xGb zhJ*h)2oIe;5GA0#*Z9uN37BYu=Ddc{<@>8YVM{wgvy=sN;7oKA#C`9cuD3O9eaJs) zI>z>@8Yr4mz#e$UIWyI*PPc0M#t7G$6UCzn0SDQb5)A@F)7hG-nig@^ZA)q)4%?bR$wy z#YK8?3XgD%%an;)XI6Gu&}ET1fm0`rWE$`g^XCbzEza6o@1fKBtE|*e=DQ4cpo_J> zVyxk&m8d^FAb-o*X;8hf#BB%}f=^YFJL~E_o%8Z2XmR01K=GWj!khm4dFkt18c{1dGPmc69n9KZi`6Y3#UZpT8e%si;%p4#_g8GmoZxLOJ!tnd zMp{``4cQ&XaRP1fip8>vclvE~PQf3}$e$WTx2`l7MxsjUUZ)_UF~>XZ7o%g!3v&=k55vPrv%%jeZ<1?yF+yFEnCkm^qXy@YJBW{AJ^K zTVD>jH&w!+bTg}}d+rfb58zRxfOesP6hmsY5~qV0dL)bDElLag%-!XEkcrJk9d5&M zeBkW6QnOK4g$0l9!;JK#E}fhVo2<-wAQagWeiL~ZeoLwh+ZBw<{af`J);r$*3(YqNHL^B5 zQUdOBq1Z$cDMFfSjcfG&+aayb4qz)M9Bel~+@{BsnUUpvM6$=p{F{wHd(=$vu#T8# z8c1_S9uUnn@sf*?R~?VrmF47|Ng^490S~5DvhEjP#e4<(AF>d>v~|vzeW(&cOs_w( zQQX40_2Iv4%##8l?dkgtyf-%5${_+#sNyF7fSkU+*%ltzWakkS@Nt|`#y@_)YK zAlE?(t|-Ta1WzoFmrAsxL@?jYU#C(&gX4;Cb1WbC@^6P|MP39#8Ok}Qe(>8%Bbq9r zb)|@|^2M6zhxF7hnnNa+$Mf%;$(Hsyjo3V|lObJ}Q!AIEkk&*~MF~{QIk2<3A+5Wh zFn3sBR#?d^_`#&%ncqE%=t_GHx<(U^B-+2E`62}v^|GU$M7Tc@z+PAbv*mR$Qy+Wf$yRQ=XizUUNUh zKC-RH^?U&$`0m>w$G`DrDp$%{o=@(ADq`0;PFs*KpWS|Hce!lV78r0ZHI6zDij_($nL9Z)4kEHoMaiGs5A=$1+;B-nnaHh)3G zb}{`eyf7AqW#HGsMY59EQ?>S!`EUQOi>axi%;KDkHbnkaTdt4)mf6)Skm$s zO}?l_R1&@+pyS8zIx7}$YUxQkNmV^pu9dK;&mcGoNaq5#Z0Q=`uC;1nN?(I&1fE8` z@!M)B8CEQ!4J_0`Tzy2Y(Vb&TQ&XvK_%jkZQmYBT#P|BxI;Mn1jJ2MiwS?~Uv8DHi`WlL; zf>3*PIwR{BEylLfR6uopG?4eyAq(ts-NyJ!2)W6?K``6aSDCQv=2us6^=XA`5Tf&l zpu3C0DG}s{RlvolX3Wb6rdJ%Y7zECZmxzkN89iBtR@rMOFb=VfNP&Ay$i|GW+FNxJ zi6m&v9Jo@c6(L=D^!I^ECTf$>RUmkY3a|?_9cqkv1IGGQ6({0yJqkha)Lfi38zFk+uF55WcLJrF&-9Aeq9X-nV>RRqZ zlfMg1d*vaTssgfnoQSnFa&(5uypMj`S)P2&ygrj~{I#dp_gy{&Vn(;SY?^Vc#+6HN zZG$X2QVJ}fu|kr9*PP(ZxNP0S24{tVlXiwEMJ})>^$fMzpV0nKXM}OVip?vNy}k&5 zU7mw;U{?jg#`#6u$`|HL*J|Kz0!!zJy^}CgIAK6`-fPTtxRx|be}4(O+Z_q=+~JY7 z`YRX@ja3HRIGxe`W%Kf-U~}w6OOMJGWDz2!05W13S~3m48#HH5gV{|NGcr$gJlxrV%^qv`e98}haGc_aMk#;JGVzx3qIk^hnVwK*-G zxMuv9^+BsqbaqzeV`s;st=BEs`iCzt8Y96cUe&)=XkAapB?zxO&3L2C%NdBj)Q0_7 z5tI-UgR@OBzE(|5vd?j0A}0qQJ5Py?4O2ymsp)@NtMvzThNQVw0qk%;f>)`9Fmgez zk@80M;|*Rbp~_NW(c?|sxvR>cx|SB7pak=$@O+b3a|Amt=X+V+H#GAX;Q0|Ig{gqM zuX;o51k4hF1HaHsh<~#XV|w+2Tx0ZK#k^$sDd2J?W@kU)xL%a~Y$- z`({VQZ`dlN8NCbr>!n*2R}_h24r)jp7Pz?sRoOr-3)>r3L&)DQJwD;XH2xy6b9Vt04~Hj6>?{9lH6O^ z@ZIq|qxVe_6aF=!fQ)N}8a3+L>tEJ{-w}twpk^AW_g?;#_uoB-Zv7Jn*_D9z{0}m_ zag$!)4;Fi}pF`#({hsV9Sgccxk0Xr(8F+u@(HWnj%vCh_wt&Z;=mGK0 zZ$NyHpO19hZ!3rI^_|08R{p#$<-Q=R6r^n>9mgeZnc@;~A43hTFr{Wz zkvR@E#Swysc2c)JGmK>XMk13*sA!;CS%H_70 z9CEmu?x<28;8x;jSEK@lGKbX2Lxqk?e(8 zIAaDUj~!HSLFWtDntj{`*LL(ww-D*4>A2|j1H?Ig4$gQhAZnD?Y;;vMR|_zsxUFW- zkh?#a7V@P2-kOb1jH;LZYYuRdehL z-acy1Xb3+blro(eH4}YB+K#_>?7lzC^;Oa`Ze6S`S%Yh->DN?wyp&ZfjQ+A0QPXX3$4SqYec<6-vU(TuQh>6IcC zdY(N-MGWMY=YwrhN)|CAxYV9zsdg^#`VsCih;Cq`B2nf4VedVon%eq!UEIEcg`y(5 z5djq`B1J$zYCu$akt!k}(xgi#KoUEG^cF%1(tC%{2`W91NSCfa=ry5*w6ok>_kPbA z_l$c#-ZRF@mkdT=kjynR=lriZpXZ0>snZM0@*Ntr2CKP^g0^JBe2q6?yXCH3A(hlw zM0DpsswKtUYGv#}a=|6I+u=$8Pux$O0j+2_krq%2vG9{Q_z^T$F9F`dix+;oL3HpS zFEy;y9Y8pnCYCx0GK9fmv~>{cYPF8z(`N=PvDHYhnlgA#ZEQPD%5B;=#TZf{4zh1F zY2i_NEXL+esuzh8wSE9&nlK=FZnnB7xZmfAYJ7dKo%npGT+w*f0`Ax+nl8L~v}fv_ zu6X+qawfaLK!;iU{0`Zjoac{G4giA2+@$&*ac`ZX$t5A~HoZ?9i zw1qNg%-52K3?P;Hd)s|+1R(;g=xvWbpBDKE5z;CKjbr;MDJTq^0BhLS(F9uswVjRs_{T70bH%jQ>$nupHR)>E3z8m zh@X2wC@bOhgJqA(um)Rhpbf#ar7`&YC-04!-8;ak`IiTcrHVeqO0Eg?scGdl*gkY< zZ<6CGGBPn_WIKNK!~SIGz2O9)gQ13wbJImRo4{sDGvaKoSLoj8MKOIZVsaQx@W-B% zA^FKRWIjj*+(xy$Kqzcp??1}1UK()}RVCZ6nJlF1B5B({fo6u7%I3*IS?bga|_ zN1oNcZ93|L*_%nmleC%uSjb{Ja~_~oZ4t`#gs*y}sH>ACJlWY6wTn4C%(+W%xoG%# zX69_vud&zCcB@-3ostK5bM|WI`!}gd;GF_=?59=weH1leY;wrD2gO#}y~lO9T~Qnb z)s%tmvX6Prp}@^g=lrexV3#L|vUU@A+4;~@bVI*M3Y%p4zIG+8Kp)dVU9F_PM*FaH zO$VW<)-&*R*ctceaG9-T@h(Bw1a;eUqVIz*IpVVz6lfh<%UWF~@2LHgyNj!q;I&3; zcpCjf?Ua%L@P*e>}k?p8xx#Fk8{(B|^ z3gqF=?d2n!b=y^~*y%1(J$V!Jnocn1JOI&k{B8x|Hh@urBVl8lE)pfJvk$M_o>)mr zulvgulJ~`H4t>B!IzS&b5}s`~6J)>}yxpB4Fl+p$nOHZo()!M<**cJnz$$DaT`-IA zolU=R&d`2O{06v@HbaZ@qJM`2^g8|Z^-f`s`Y`EW7{HUp_5RX3EX|?mA!u=~2Jij} zv|gU_TnxZqHzG@V;KJ$jGW-vuKBkpU^{_3l5aZBy?_Iw8^=Ofw7`4i4tsz+Xuust5 zAOYv4UL!1Zh@Q#)T$?1d7f%8ColYY372OzC2^?pywX&;!U%OKUV06Gt>sHvKDWO#|+U@ ze|^uIUx&l|dco=#+Np@5FarWub#~GZF2Vtw zCXSaVv)t;5mM@LquCW(#1>_LvTca+5Ah&TC+F;aZf|y`O{u;W;<@d`B5i7#>pa*^C z2l?UZVg6f?RU>8XE@@kHo?K{8t!#pVNu0Z6=Np>oWvE&G`F|LZ2&l{b(@GI_wXUsG zH!7(iqiGqz1XC6wQ9nUFD4wzhD(^GS=mA<6w&Lyba{? zN_%mpRUGz%{L;#8uyeaPrvvluZZB@PA8vf9WK)AR#eG73C3`clXF2#Q&-%D!t=n9c zeyA1lsR}ZdWwTVHMwi~IycHv^@3)E&+$h{%-47t$5@t)6PB%g#X1zDqT6n3S!6TlU z39=rE|54g)gmJR%aw_yWDjudiSe%L6nk99mL8s9oFU8VpjH*4tgZtOtc=f>eX^Hrm zRQ1J6E@)vnGWP*KWP_-3Wus-ofMRLJBR|0Nu94gvG!gQATm4LdQtfpd_jR@_vkw;Y zZ6Nhd$b+#C*nYCB+2SLl?~K_Gq&I0va9BGB80Mp|)_T~W*?{%4H!@jN)<{r?gwE+H zwj#_W3Zot3&Sx@@b;dD>gVa#;k*mnEws9s4)V6vbP+$r!uR1LqWWixU-Ih@e8)VJT zcSZrI3VvptT=@148qLoG`g@_#zbpJJ(*_M($wb@>47d# z?p%HBk6Ok>qD9`u7=yhDcfmfbqNhxvzL3*K<-0%Rxr5`*e}==%*1@jIt)FhD-SQ%; zvtXX(&2FLyUt{)e-le%%PAI<#pV8x5ufGT0&;XYgSRdK7-@;8{l|bYJE)lfd)1s2m z2gJ@qU-EfLh{ZO{uAbRuiGbE0k%}v17QCeYM-P6mlFD`?5UiK`7wT331bc_H;gm;R7}YqV=)%-{cf2(?7k1y3RPRlNyDWqGgwQ)O&5kBBYb z#+cnDG^Kptlbd`AlgE<6xn@0&K{*=6MKV9KzwwK)f8?qHClf?U^)~1+Y$wBYAIF-3 zl1rPD-C0b+`Z}f@rdqH3$1=N*urRNAgIi?KO3?;W<#zaWB@&#jNFhK?cmS``jR*w} z86}$g4YrDewCAMFrJlV{SE1U#x~GvaX)J=jGIls@tgCZW7CIY@AmVh&X$tPH6CTDe z_cfBy7XT^NT+qK|gQ4e7(ie&+LFLx%4l=`G{7AdIO4c^_gmEWOY$(^cC?${^MlZ#_ zPno8dK%XyF*KLna*KLml{TLf6`-Z6!EU?`$w(>hj{dQ?n8)-P3! zIzjd?=;hT`^{rl_<3$|>j_<2~gH&#F?qf9n3y)wRa_$5+Og(RC!tpvp3nIC}_cRb*Svc;Vko$!ZXs) zX6E}~QVt_V%qgu<9JQ|YZD-04+RS+YMcoQ#F7#S+G+~m_tPg&KqT954wLAq#_kWrP z8&99ddJH_hPSp|K>*9Pe3|7Lw3d!M-&pMQD@uw>;2~l(Bl{#7BHL zsSpeR)<}R!anw;SLB4$AbE_?C!Ead>sdSy!QFvJf@Q0C^{1ia%%-B>o6UYu!N{{k> zcSC(#4QCOBSxLJ$U(vvGd}{0w;>1&`xqk=K)XQZb$s(_P1An0;J(vBE;btA z?ClDR5%tOCmNciY&2H(`)Z$Ia2}-)HL*lc*yh8bfs9$5KDARM3YbMrr64TRZU2-5G zl`cm1b^nrk88%Y4)YTq%5ht}N1I-K^7_p0IMbAsRQENY_FP z;ptXSNKvf|8>{Z&Gcm#7a@+nE^BL+iz`ER{sQNksHCdi0+xZNoC_`i%Fmulp6o~Kd z7K_(pc6dzZ!2^PqL>h;lv^sK;*mJ4jlyT}I(~{Yv8OuL?n5^@b8;5GtaY$qE!*1Mt(As!5Zxh&(hdV#J(LVrP zrK9mZ>c?MK&JhFp9AZh!H~HD?@9Rqg!D&^^Rm1BB2bqeKIW(I0elswG>36GH1H?Rg zn|nibk6FY?(ssZ#SFnC*%bOdZ z4Q5?=Yv;i!^6NN)%A!5rbQDc^kKBUFd;n&JgM^WrEpvXXMGfmSWs!?9@@ys>V%=Dm zT<_&pSN;-VdZb30Diq~!4an6vLg7r>KW0Py_Jj7T*%G&zh6;j0tYeQ1w*W8B;BhgqZqi$DelUl=I^ogad+>6jwP_g%5B!^*e@Z@0BdhqQ%@A5Y7NU$2){i># z_N}P=2e_mCsHAjM_}6-F@c815{=JN=h*fPa(%6~p#1O!r0Gu_TQW_2y$D7;$$@Wz9{q3!)g_>``7kx&o!Y;$E{5=wxRv;>2P@Sf`>#y8t!45xl{1hpKk)y@nH<7SGbCYueV&by zL0sH0l;eSM`p(wZ*sTa#;94~L3c?`ixfcN=3%UvkX{+RiwU!*8WxvXpSee)$CQ6YmUYGcz7?f0-ilNT zicNa?=d)*k88Pj8Ba=9K+$zL0eCMn#FYoD}D!vrjpS&`b9{on;GvgF&O6jmRsY;&* z>ihFnk^hH;eyje!)qptDQOg;o+vw)E#F`pUD4C58`kS78fqnXsS-zCa*t2FN0kP!i z!k@3Qd7}#~cOETwysY|rMajVMWR-|(V%{mRkUVG}Rbz&pNMZf`67cN9_rK!PYaq`d9ydVHzyeQ}_B02mJ#^t7j|S8Dm634c~&Cu+MaAQUdXU zpvgwfvW68FcgWcNOkh)d!9@z)T|clrl(T6XGJnG)LzyqVY$Ix;-2H<(CnA0{)sx?2 zk9HaHZ8>o3Z=8B^CGL5?*K>f^OnrXwy|mK(`y8WcgwG#(m4OK1SXo>_T+mWYiFNJ~ zhEIN=SgQCGd`y<7N4csXW=ej&J4lke*C{z74CD*`_4rtnYhFy3 zjhgGUceuCTv!|ex^#u2KK8dkWMd78np7EH{;{{LG>(K^d92tUC7aA=t#IXw^Nfdtn})SF=H&zf;J0R=`a5g9 zM3w37G^)C{Va)(8g3If=f3-{E!xdp!F-`XlZUG4j$uSki=gDX%bVJH(sMd8L0!WkQ z%C+4y02juc0rwX-S4w&tV-NHJ55J=qRypv~_;@lidh01z2FYjrpuU`dY;r{UKdNmv zDuv(JluXev`x8(f&|lQ)S;P$|drB1c*kbChZLVNWR`oEbz#6259I*}b4=8fM0!D6X zz{whR2WvI@+I|H8e5&>Lo~Fv_MYC@)7pzY^N+N_Xjo~_4)8@1`%Rh3K3qgG9#^WD= zXlW2QTk&Q_q!tK)7P^30o`HY3K=f+MByPPAMDLqx`p8FSPpaN<;iXKHWKTM(ysIR* zLpEj(J_XG(CvfTw&krWwC2Q2xI&l=jm7m@}O5gL-X3df^Uqg(A0k>d^}JDl)tM0gsp*pZz=YN2eGLhgz=q zw_>K)AVu>so_KYKCw&)2EyPBDhcDhdYL6r}ajrd3uvEQKpjjV4dvnQarj!|zF~)e* z&_{5WWS)RZ1rnn5djUH{r2hYom$vH$cA-2qpHRvjqNw=`MHj@GgcLbkJ6NEqz}J8rdI{@ymBloCu@@*gB zYL|!g zK9Gs@bNCb*qOhdSCjZ<)-~4}qn{_+o-#BpT$40(a=r^ExDX^6E-u7*?WO04S%90*8 z*MJ=ooyjS#$M4mZxhJe{0dTYElbt(DhsprdMPQ6hD;nCL6E6R;62r`qG4~fn>=c>u zrw{jjv%XT{ifvrLjZhYJXO&g14rar940$JOH<>9PKrfI|BaEcXDWt zV2%r(rkd3xEY4kD+il39+&yyzIp1c$6uT82%Zv(am zpLF_oy@xRnxvV5dL>B5jjI&~km4oht8#muW3;N$rj^Ne^6hO#dCwk-oe4Kg|nUR@= z_pH)S__!;}QUBS9VDR-tW|?G!oR~U0E@>o{|2}K8z)Yac!;xvS>L79a#MJqyV#sBi z5GnA{^hx``_!<*IDE+(8D2rU2!pcxP7wxF4X-G}^F2J6qd}5FS#`br@K^vFDADx*y zQ6PCcLmBB&$9FxMdE{YC?Va#fM8-0f^n+xNy;wenj_bp$&0`%mIh*IBd7kCZ`V%(4 zx=yv0$M6Myd>r&zwIP@CP#Xobpfxz3Ir={gJBTa9Ut4Spm;KyW;?2Rbj~!>;E7(Uc zr{A^)adj0-UCRxFEBC0pRVxFu@5Q}savHhNk&{ZcpL&wb4Rf=k&Io_TFkswp5c6du1lH6LUlXfwaP`-?Bfas z-6==zvTA~Rt--C<3R3L#Ri1LM8BMOBobj*cge(TS%1*K6wSmt6 z_(=g1_5li**XiROuoK)yZfD%iN&76$Fbih}yD83Wvq#Y~B+ z9P!W=nAYv~wb|^3T-`Tf6l*N#3YT*%u)l7rF{rR(!zHH;xxSnz+5Wx`vlj@EIlbsv zJple@rRd@XOx;aazX;A-gyMDxf1$m1@BfANUd?bo)m;HS&y&zt`^EOAwke3=-=#`O zZ5V?d;UwdA6cgQx8>Ifj{{�rK;M)dM|;j(mO`b6u2=>IBCsTcdz>s(jd^Hg_bJ;YEpSN^pP~ zz43|~IhCa^LOGr)Tr$ZWZ=Lk-xS9G=QWm;}9K4=BR4g=25D2-W=WS@kihp~d#PP`U zM2iL=O|RV7KEQ;KEM7AxE_hDz!hRtgg!WfS)@`PzdROxjWYQ6Xum?`3dv7 zH(P>hOOBY~n~FK-z2|x#cqZ4WRe5b^@(2OQR(mb>u@SIwbpL3|#q~xbe5aKH{?lW# z&tlYL4BwtMaX$UbNo`igqWxi#LOTD5x=Q4Z+V3zQ+YmnU3&AA|s^6d2pk?Zyq+WG2 zR$2M4EJPWzeEpMC?rNMf9ou#sNk}^ooCj1Tg#kW?!OCs1d~h$X$<=Hf=g_m<(?v<~Ld}V_L-bNYN!TEa#-`6qw7yl!bIgpy3 zmM}I(IYXHZAXVE%AQsgKVej0!B^-ykHMFSn&9gVJMKr$1Ced#m5@EII^w}4 z4VK$P#y=R_y{E@ubB9lhDCUvUa{CSnU+m)3Vc*xppPin|Nj2Yd3wSt!Z62%Oj0TQ+ z;yMu5+Hub?{4V=^v|Z1RXR9;1D9<2UH`A{Ur~dvYc{Jv$RF#f}jV zLu`q+fv+PQ3ubZ3?o7Q@Q(a?v0&%)!c#c8>JTH6Jx;Y>ycKyjE4WrelXLodYy%cXH@WHDUa5JY2+C#=3c^(R}>L?KWY+cABl-)AL}dcpQ3;t66=z_KFQY|dL|R&m!Pm& zv@!yggUv<@E4a(KM#;v#qrKPSR00+4p%>iuT?BbNWOB#-ata%Qv^lQVY~Q{!0ypS+ z&6?oHhyw#=?aDHybCI`?&QBhB=g@tHzh}+j0lyWr>80tOH3I8;Wh-b4yf=Az48l-jA^NN*G^AWCk(8W;n!;+QDbDnfV%C+$=x{7QN?l@imGoxi(|8c*y}eAd*P3K;as?5~Am{%g~7^2H3f76E)kPo>MASa?$|db{MvWx4CN-Z1b@B$7Ay7)WzvmNjJ9|-&_BJo}+Hd z3M(5j^+8k#IR#>!NXKeNw(RuZsdO7UpnWv?R(?@)=_`qP;UNoR=vj!zG zTWa70avWnG#d&)&e=N!Yl*Y8WE3ys6)@-U*0{pI#XXIi>wTBBpW)5dl3K*NOJ_6qc zgmKM|_Z*P86a2qyg}}vqpYE~D&oI?#sjMS@gQyc?I`UFfDQOu3rbDEJ?wEt&S6Hb< z5rmswcMUV)Uv2-9zA!LEp9A357bqZU_}%V|OQj@{rQ7o9Mmp^t_*b6+o*Fp+S8d>5 zVKD#x!q5Dce}DSli3r=AEd^!{9!@7cmc{)Sr#kgdkW0ph zccYfE_k0@;Lq7s`T<|n+u$SS=0urF-X)N{7JtMD$u(70@6Szs)y^7wJoi0W1Dahgp zFeKEAz1<;|Xe!S&*r3xwRoSzK=~yXAJwqLdNFrp84?NnCcFk*ehhRCmZqoFu3?(yh zc!&DCqW%bSUFNj++q8nmRqfk(Rd;s&k)gk=s}N9#S%XT)bGJsR*>N!#Vd|wjUl!4`J)->XCfva1vWi)cbK-3mSjk= zF~BC4S+7V`8BNz0IaUN*BeQ|dc@$q-=QxdYgJeK|x6xN#)N}FH7mwxUd~dmTotE-q zc=48HthOx0VQvK#aMsVoAkhk@Q(EMLSc3{N)xG=ucMJ>_KaXhZN9`-F;GL8c;eKpg z3Q|#+0nsWTnn2oqrU5ts$v%cA$BdQhmZGcFTF|$IYBcR<-i7l7Tn<4WX!vdJRl253 zHn*s?Km;x~S9-5sFMS`B3caHq} z(!7#fJxr}Mof@A?YFG1SK2#E>j&+&@%39v^?ez%0s^9&BGC!fPN^E;XR4fe}ZsWKP z{Xp5@q?0oVJK2FzVS`F&VGw+UI9=tjW{ZF<$4vgbw#}OIA-bd!eHi`CKVOrGgY3Y{ zIXBXO4X;1{LD*x8veQGuK&&rsHb4@Odmngt#URd}oI; zq?D&in{0xF69?lL%Vcj`r4XD_(jUwgoSC`IX}0|325mVuE0*XU&e8-$dlG03zB)$Ul02j6R@7+FF|%uE-BQm^}g$8Fo()PLuWrO->GaaG*_SjeDEc zWj(-YuA>Uo#dn2387y4`tJBKqH^G?-_xl`^)Q8jd&tUWeRstMBo(ch*$^wp%iX}p@ zB^&dJ)3l#Clhpx~*$p4R#6XMj>yFr^PSz7=?qnaj@EMY3=qfD-KWqsrt5Fz4j%vM^ zeWN@NM;0viKdS_km^ggUtdkh;&6<3iJ?eZ0M-$Hsz)d=I^1fcRZ)}+vgD(@qSst#f5)Ar2Z2#d%RXA^b6=eE=H*%n zx3MraR?}Kx?v+bQ@WXLaX^q81C=Rir7pLXegDor|d038JpMmCSE)MK+14$GePxB!w z*!n8R#wVOFzHY1c!URD#FgL4vaplDr{mQHG^zz>AK)C^XH#*arnik z7>zvda}QR^A~f(Y12HBM-dRV>=_cavO{Bx4PWBqe7UN0UtQ!04Lo@UgD5<<^1aAla zm>1Za%-L@&*9@%(Biwr$nzi-<9eJa2^4HbUYs`LNU_+}OhU*qX1VrkMnVBD~XM?*z zL9wzb25GgMwa#CDw}?t!ee+3+R~vT;8)DGU&Dw$fTa}FDcRq`61I@+NW;c_DqT(Xt zmnL&krvtJUlwK|~&5!H*FvVCow`$WX0yiO?T)rR$jjzn&R(U1{ow!}zd#l0yY2IcO z?%Sqr57p6}R!-_GRU_SevquJxYtor-YUdtf|GTKU9Hp42q=l+0Q zwk|`wqkHR{fY`DJj2XWbW9BD#G2QSH`&#c1*o(rQtF?(d{_v}wiCkZK3ieAe+Jc+i zTr9H%>vdtp0*OeY3up+)h|g*TA0V5z2ZDE>$j)t7Y*r_0ae;t7>35@M$+J$x+htIsve z1*&NACWBU8yS^G7&bSe5zU@B#0|3Ud34;Ja*$T0>-TI~V64fp<-#R6O2ra3KU~yFAAW#Kf6NgZtCbaNE z3b@<+qK%4{n@-u&bo#qEpe`*U4(iU_24T{Y1JA$d=`M0Ghh`EPuPDWew@lW>NpBO3 zT8wBfds}{hm^QZc^icKJ-n&6N&fpT#B#DvkG@ts2Ak8>8!Vk+`9FQ#1XybiXh;H)` zE=wnsgf7n01a@V!yVljtk$HN!JZ9o*XJ~ad7$5TYTk%7h0VEeS`;0faA_5c{aMrCX zS-SyxEa3+vi;ZieF)iTE@CDx09jl2YU@s{c-Oem)Ir1BTV)&|$Isghm3+;u{J!<-a z{MAW?YTuBK4eGxA-nunrVnIS7$l7Y$S_z|CPqf`XDkry^(=1peUV;g60Efyq?pn4B z>w;(Y=d%YxMZEUYItw=?TWTy@)+a-Y73_TSLPPP{oRi*=5Rn>*uh}slIe9YcYN`i& zC#$RyK-1e+pvJ0_#zHf8YUHN5?UriK+Sr(+q{@E7kT&f9({#UN|#d0 zIIbW}aGQ`7>F=Q4D)&6fA1&o#px{cBLIw!q_g?nub;vcotzAnIR`ZyIcWdrvFrD== z%p%PAfy>1_rgcuqLwptGJN)js@n-%BA+gJ{?S8L0nx>Tg^6ay+v*>B_QGbhSGf%B1 z=&|@8V{vR3HyTU+YP^Y^Lbs;iDGJ2D@v%!k*3UBNcZHzB8(Mpz;sz=A3v6t(-p4+# zNtu|`WHeaewZHMP$$RgF+}!3^W6hLikfpyN*rx%Z-;Ll!ESmUqCN+BGsdak$r#vLSY#_3o z??rYSeA!Hg&S7yvJK(`Z+kL(Q0B@=p0GQ@P%d$30921~2p+$AcW>EEaLV4fl%8(} zY*G6-AI-11k`)K|Y2V=qe*gWaKTm<2MTs@;&%npVfX?mF7zJEryV}8mrYf>y%aIo< zT#|boO2fJJQedZM=`FTj|Mdl}4bZ@=RF*0VNI5d$H>QqMsn$3CER>8K0v)>f8J^v* zadJ$5^8FWC!N8#Ol~g_rEVLS^q=Mg#%H=u`)P{R!-*Q+e(6%B8S(p7pQ5sNt?q-*O+&^P%nGjk?MzBsNgk|IZU~FT70Npy!G;J=PKCsT z>V5M-`USlr;ss-IqgYPfINRlFX3`*2$KYl!Noavz`@ez&zAmCD;gV`ZRW|noj+aB`T?v^+bS+I6^`=p_eNLN5U zbG7cKd1Ae5qUwJ&)<@Thmy&_GVa0aMoyJxt`I{jc1D(QYBtS`ozh&h92ZJskdBPW2 zMhdzSS z-B^coy-FYO9mNJ`Ts0O`mwLthOt`w50^}+$(Y73 zPu{(Imj>hOzT^XU0ubcSz@GXFAv0Cx=-2>LbByG5PU%7HIU;-Il69h!uSw#C#Z<_@ z*O1|~-S6b;#F*tfdIQ*Ej+!cyI!v`)$WHj5z+bJJjT+9Kl1%50HMG$ZaJDW^FiiJe zNy^5oHpgc_th=n;f~8g(Ye{C=dWo!C^WU9sP zi$jUkFSV?`!!!<3@#M-ExyvZb-fI$EV5E&gdFCY`npq5SQ#8tY{OZD)eq?z zd}~Xno{Y&XT+~E*{M)2d8yB3=+@E_a#!hNL`uD=__oH;|yK8BE-IP1_=7MHg9r}IR zVynE@vRLFWUA^KuA3XRuL!`c6&Qr)Qdd+=NOWPeu zyL!Pj9|dIjF2PK!>eb7IYo}OZxiQt&Sgxgp!#_W*4{+CetAmEvIm~ur{!^Zh#uHqT zjh*u1gG_yc7e`1=*&S0a)#cNaXZ~%IucakQ2KR9)l^>Z{gn%(3RFO;der}qcjnbFT z@pe__`w|K8Oe;#9oJ&D1`I-IgSDv)$8ZQ@TZ_e{8NL=y$fT?V167u$D zedSa)=wxI{6g?5KTnqd;jQg3|Fy4{4Va@50=XY=~c#Zwh5o4NQL-N zQ~2V1wSt%Mlbd&p@6|_7<|jM}>O-ehHVCm#Uq0t1wB9=zJ9relck!K7h0SPO1RBfX)`seyvT*67&YgK80xH=1a`V^biQniDWteQKo{@YsA^;M!lU<8XXmZ?aV_O+F<6pU={tXwR@^kPXnYJ+EBL)3$AtdHa-A- z2vx8R6RuiO6l;^=*V70*9fojfM3uB#qZgvWhiAeZG& zw=lFL8voV1--*f?MemfI=v=+DreibqNR8>y!AH|wBR21b59XtF}XQEEQ$s` z6pQKAX>Z?drjADkdOWJEKU6)wxBYE<54Py1lQt+_@vURVR1uH67`s!k-32!Es8O7I z*Eka&&g;!|ZwPC3E(q#(%CTI_YFJ)LqYYH0m-?>ndJ|tnB@U5b=cy;qqs>lY6u&p5 zyX!e9osX8mj|qVOolP!lT0rk3ZWkk_r=GS*GURPYJm9<(a}i_;II2O?CTU`utfj{@ z;bMrJRd#m6WME$fF|K+Xwf&8-N8g#YlP`DsS@Ol6BWyfjO^ERl#_EyqMGj~u&c5(4 zDbI*;Lj?)-0aAVaNbmQMJpS5v{#(Z=A>j4DeJAN2%Zw%t(dqg zinAg9vqS^VOSY*XPB-E24qVAA*m8I*Wc9Ub%-BCF2$NWyJzcBl|B3tkgD)~X7I^%N zfvnKpS2kQQh3(rIQBiSBD(A;r@k;#L5eepr{EjE%u5>R{uAsHH3&$^tJOk%^w0FJ8 za`(8UpL|csHh%-YXrumwFk3-?bxN7Pd&tFSXPy1vLNXP~ekooMW+5En^%Z!sb8LEz zAtKyPG7u~9@3jVU3fSVW`yZ~Dvs~P53zQ1U_O85{Rrnh*F}ce;;!fQcdl~f!~v6ClMxSKbTQ;)JZmgqgxF@ZcVu7o zu8F>ZL(VJwM}bDTX?2~AW-E4>yWG7*$QaC_I6fGKAM39FWQ@;i)*J5Zag|>NG&9Gl zbT=icXxVHXQr_BMh|+EZbfP`rPq0Xc{YM?;0kQ_Uwq?fr-+ZJtE$2#Lg$p4}epe@e^^M%S@uVFZG0*nT_dpL7*)HyxU4BKT`II)B`~MLiW_GcoqT%)x zBT}Yv4#xsz#GkfGFxX1qm9>06Hg;E-W>|5VRH!O>W|5v=1tKOH9GOHDh1 zBi&P#gVUEeOJ46j;H~u*SpeTaEi38K3inxZ(q4*SZOg!-ue`uAxZxPBitjhFOtkIq zb?MvsZda3)EpZddW&d8)a(iI2P3D8F#r^eSVHIA2Lx#=XugJNdzhoK4Y5$_jFcu-B z-E&U-rUoOGPfG06Ag5+duQz{AEy6w{%Gk< z`S-DN40y}`{>bqEEmO>=`5%d_x-@}liFas!q3fSI>4hmHp#P|{p>^bHX*vY!MCJ@i z-m;TP&@vRUiq2`bHJq zAHKbgc<6fW0WjLjXwCty4f_*G3IsCHd%Q|}f ztIybi*w$^l(>&Nxv@xFw18+UH97UzND@;Q%^sp;~k+{(sj-twdj3tV6ypqXiWOj#s z{cCrvV{(1}?0G=mzm&B6NU^PHS(OIyn@@G;@cwOM)uormBV_!i62oHCB}vP8Cr@^% zThhYr>Cb&1Pj-J;m^^ZfNF0ovOk#wnx1dYsbG{oVTmhUmW@kQ{_47P+(acqPY?+!r z#IhJ;#&1;Q#I>8{eA%#@gxuS)vc7?k6LP9jdf2MmtRBKB=ax3dazTG*PWQe7$$F~> zZnW}fU1m-#wxom|<6Rgr3dK4_7|gL7n$)%K z)$huLMZ^u8m|LaBg|=gTl3h6qJmnU8iPrIyyM2`s^7@McGBCLUL(P%E7RNg?{kW}A z?A8XezP5XY+G|r_z2^DZXObnyz8W85$TJH>^A7O%9mk`+Wipgpe*nr?X90Cgg&k|5 zmYGQMOEwGrT5bwA$@eW?f`?*PU=?i{QqtUX6m6xZ3FBPumD3IBS# z)sBA;*B-7Xt9II=Thx5+xVmTS+vnnkBxVdR0*W5-)WrHiq(Qpz{f^E^O?jnxno23) zZmNIYkV`b~*12#kkIYFJE9C8$8a(CP2=!NKa;6XF4bg5K(X4o`ns^=#@h=MObV14( zkn3fBR)=NEAB~eLlM$C)p{Dr0@jCg1Vg!e34vxu3KCi>xX4*&VmH}^3Pu{XoEOcCP zc;j(&r~qQNt?UnPkosbC7H!+LgBr)f_96=e=UZ7Wf;ltt1W4 zv>Yw`07{7MT}ry7O@;?v*DMH7)hl8v$qs+Sb9F)g>|*Wfrpl5(LboIRz1??J!it?w z^_rSx?R8fw3TdLuA`RB{?+HtZ*$M->DIRofXZr4DxxR;Y(;kWQ{CY+%=8yFZ-gTQl z`^NA0jVdpEh&AOj@fp|jZxrKg_bJV#y4K+OhFm(i_<7Lo3K!0kczqQ6#zi)zn>U*@ zu;j+op3A=)--lJI$F<=vyX>mo2cA9bv%J?ri<>@1LvcRtR_B{6?o(5--KlnvH0=0S zgGXL@pQmD`e4=$6#>dcTc5_~%pWj&4nQbN4r*+cHT!NUX+FUlzpNk?h_S!h!wbvQRYpHp%DI@k6R07~dqMte*YiHPVvbzJyIrt}y`D;{ z=Uo3H>hOCi6&E#}fP1zqib>&n3?2WdG~{FcQEA6#jueqeT(@C&G0T2v2+qu#_DVcw zWIx-J4E8*=f|^^erLa{7o1`LajV((XpNfbha?TA)YY#(8k{o{@`FKI=P(g^5HyNMa z-(OAby>Qa-;onu%#a(iC2R^#*rzRL^-VV?P6*ww#kbU&n5S+j!+I#s%l4Onh_HSSR{d_P z#nQvr_|JPW4;S-pr0HS++1hYLL+4X@XNu_s`_F4q3ijBVmbuZ1T%KJir8)6p@$#n~ z&aR`sH+ADAA#H?(;}+;USxVVT#i~I`r~SWH@mx!|XtMqwOQuRhRE`P}|6WnwdB(Js zyUb0|qCXk=rxy6p-iV3zFe+7My~NZGxGnhi13>a+zjB8dWkd+oK?Uh7)ra@MN#sNKFaa8ep8^{rJBJb_08 z(^c=3i%rORlwkUZ*WUX;(+_!ieRUV3bhP+uHqC%ZVxq0AT}9j$HgR=Ieq$e24F=b) zjalB_|1qJUSN<%0I5Uq$XK-yXP*n*cm;Ery>z6pfW{L631DM*PAoPhq(Fp~UR>NKl zpn$3;%D{2GGX14=hH=`Yt!S4uNL9Vi7nm$Ojv1pKbMBgP#s9@j?XI4IG<5)Tw_)Nv z=|&eXZq3d452?CH`P#qH{&@3L@s^mnUb+F7kMc`J1`4zlFfy$p@FO#A$4q-$c5@k3 zuje>@l2quKPcoYP`unwtL+pgksr3^n-rNdaBuX+2^Be9KE z8&6|XC^TM|G8O2+lq#Xm-|{t5gl(zTtHZD6_Ib!H4N^}83^rYJPDtU8?;{Ncn&4^N zZMY19D`QTU%NWl)_lBNNota33BD7rT#UH=f_>~7P&1>mKaq7 zE?V%)fVmzwZ_nbQz0Wmh=zm`aDy2X1T8+fxp(-jA>gGuHgin;1kDUMzqN0cBk?^9_ zvbmYyX)var-MDp;-(tTi)-@d5E~jP8L$2aZ;2 zZyk@rioMqBh=1RoF80Y-8>7dIiQ>)&CZo=tdTIGeae)u@VZS2e_aBj5ZQ7m(M!Q#f z)>n=Xrt7C6mvol$WeupfZY|g7Xy`+BLJrdBx0?QlpUmU}a8RTHh&8WEC{`IC9H;>} z!k<(V%wrc5kc0E)UHLg4dcDvb(5OqkmYTm&n=$;l4b;-~xP51X=_Eh@iE4YBgl(e8 z`D9>i?UA;|3~#H4a}U5}Q35}c4v|&Nze3Jhh?mx^)Qg&B>b&Ae1M=nNdgK;p<9M?{ z*p1*mN)aml;22-t8&f^}VSx`kj%4U*3jNBe4gqcfz_4J&;5cowgcc1Yhrp%|TUoxV z{n-4YL03id7Qk!3ebxoZFJcnBjK!__V!WWP9@6b!vJe zYxG@Gj-X!arml@~YaWSnWh0T@1TNHDRJ8xP@^6qwxDM#ki>OLOr$19GjvYxgcTi zNu@Gl_oT?rf3RqtzCV{jB9It7=hBht05Oxw zhj1adYOVe2Y593ATHONMmOn|^7f_*pI)6o8l(Gx)kxS}G-(O&1$RLZpHvMF;B|bu+ zPi3DH;_DGj`f~;=^aQgnSKP7&oMkZov;b$f6ZyRA1({^YQ_|o}$*+lV09wk`19_XHd)|EaiqbvOpDW{OF)c7P zlY>-u{DL!=tCW%Agm z*_1C$IOF4-@b$K-;Pm!CqbteCB>Ff*8MDM!{z)Nf%zMHUL-pchFZXJ8OUZpSA0t%p zdQ5HRdFd)lI^&8LcH3t>zx&7h;zVf(&25qaVK(Y^XW#oKLJBptnY>CA%6IV=uL4tGdy&z|-b+_(g5SYOT1byrjAeuGBBv{mK+;{m+Y?#2mpsvn z(Xhp>9Gn^u`1AQrj6l+w!F^R->CH&?qL;7@y?&u4nL}7QVz{5 zhD}-L_zJ(AvTiDu@ZpC;8jF08C4Y?q8MDmg_)mFCkUYpIBf2jV)YR5uIx|8me27dY zY^+zJ|Lxe#mhGVR`(^=HKW;YIPRa?uUL(UcThdlPGM*I=ke;lcjgD>AD0a@B8E|~5 zO$*caTZ{_!Z;>MQ)f^E#<@0?==Fk0a2My+TZOqf~lO!E1#6AqzaqyqO{a^^hK@glr z*yUN7I2XTwIVm7kw}%PGDr`PL#Q{!LGtP+g2sBQh=e2z6#LxUfTSESnJNywIW^x{0E8S6u1sC5iu+;*>+Y~Qr_GBsv=4`9&WKck`#Q_{zFold1*V%>;m`B>mfX z?Xw@>MpyhUJC!MDO~>W1&6%_LwQhAD|LIVE!3b1Rc$zzysyEgXR0;*zrkp1HR8ir1 z9(~{!PxYudW+$tsxVYBNa_do!Dnk<-B8b-tN}X+Yxss9e)D@)II}l z=gF5Xr?Wsp^THjT*cki#&<@FS!N+nO#B~_~NJmSYIwTv!iq5|T%7tq*!r0NFD{A_9 z3zqbRJqDa86&0wyfw#9e93&!gcdCf$Y2v`p{jt9D-}^sb8+2#0?@8FsD}$cJ$>mnt zH!h*_4X95)^8{D}H#&{D73U4!u5)2+e*IeY-PSYXbix{6%Rkjrz@CghQ z8se5h9*BP}Kyd^T6>il**$U6#5e;^jsB!#u$8zwZ?Qq+G@T*Z%ihD znwRTIaw?pNKVVI``$h{uy&~THtVa+sXRsky8QkNfRqwKVCA%AgnmEO%x>wDN_Sf47 z4OP~UUfjrSy_gQJ6H7q+(a;Lqea&+W2DF!>lP)Zj(EjG#!1&o!V*~Z5SJ|yy{Pgrs z82pNRt9(DvW`6hFcRpvU^!_)gMy&QuSl9#Um5ep1Z`wa`|U*mB0Io7D-_=*GYd;m@QADn#5WU_n;*e{L z*Z2Kz)=DZFFJkl_Pj0yMYy6SM+Hd=}bI$wFFH~%L`&bolvW=;!9k%!fM4vD}9Ng98 z0%lQ4Nk(=oY-q`HYwg}^NII{AA zVrEt&;}I3w0UiChuZ$9GQ}+d*;n6*W+FvQwhiSRdElZ{V60@Va6k5yYIyWMfRBmx< z=h2z;XE*kJv+FqaWAReot6a-txv!@8{$1ZU!fnpbNcxAPB#y~ppqS({V1(@Rp9l<% z2ITe=m*2HOe8NTw5S_h1bEy_-dZ@aq`T2ot7ad`9w9%xTLuh z@Z89Kh5+JMV;Yl5fd@1JM>2Bya_!mWc$qFgl{1nI3n>@%w3oV&LLSTc zc}3g&kzSaWc8RN=ydIx=B{|KBH?eIEj!aa;6sfG~B*?(#8KkDMyij(mH^HZ=J+u}e zq1f?#ylzO89V>!5Un#D7q&3R^DC)vaY26K_5sXHC{a|LvJhhL@yJa>VcyAs5!1yhs5^kNoR~@{vuGFnllrZBj^0<`XfCQ&n&e-U(c?Cm(R<3lFN$PAcDWVCoUpqxBZz>;>9K%Tg9rz5<QmujAvol+XBl(mp0|8QlaWc-R!L+++i zq5JPtYr*Zf3XCPNkZM#Ah+@vDck$Z3uQ)DIcO4Tnim(3O8%}3aE)Z=5v4r|if_ZJKs?XkqfzxE zCk_YVTYwKIjaF&oaj@5E-~iBSKVPjwqP+_r23Q(*>RA@Yzl^tTfjyqKo?&F4CmwFN zs!22+cJ?Hnylr4)n}xITxd+^O7#vtK=z4YKZ%v>l)j4DaR{vKd824uDT5{6k%$Tquix~7c`dw}&RcV)$N6EUPJa4aju$U`Qx%(fFw)l>_>j*hZ0m@ASL`k;CcqU( zjH~DM@H@Ee5aj`t;xO|00AHK#4t89x=Jz!+=T>f6;usU%MX8NjWu|}Pwd=^^812wDHY?e50dVPkbIU3;0w3lF5i1T&*ev zBvTcn$O=ft>1b5{$cf9C_;$(5fur@Sny*sr+w>$lubH}MrW<=eCMZ=0Qj*4jjTH*L zqx?GeO||dqXTOExE0i$ar2`6;S{z#Q*DZ1+Q4ytd5>@e0)8?-SnGn&d^+Nx8D<;5f z_X;k38V+k^(zosHdM%AGJXgX}8kvaq z&`Ss!m08hVExz_7=t9;)KG>0y4_)K8DO}E#ss&wKqI^}P%O|om;T&o+CB+&~daqJ- zVqY9ocT<@3#n4LU#ax!EV{3T=PLi)A!KA{ErFEi1tldL zIH?H}6?(7gK12OCIh-_h*Zw9-r9*&djM;{Z)`C+pQ*gWQ zRo1O?(A;Bk{23>}YWX%XcZlEn5)p52!OapNC%`1cQSF%Dh-rA5hy7B;di{ZufYv=D4e(;IluzN-AU2>^2W_MGi z&n&f~kk^?F`>b!7-Kv+mLo$C|*GtdG%S+EouegUuim2;eM6c`QtEZQ35!S_Pu2y9A z0|Q68274ekGHP6iRcYc-(}DYpe=DS4M4%qJ%r-bd|6A0m&j$i(asTtk;JNm1WBspp zUzAB_{9Eh)>y4(*MOWg{+uI^wh-LmAaW7?ch8cS(#S3rCV;S+`g{@qt!T*$@HtD!5 zb$;kwS=F;zt>mI*NKV+@T5_M_EnZIw%2!8jnPZD98yYyb&3&~7EZ|9g)g_hb?oJN6 z=WKebj(kse#&pX4HY+w63)!udjvfuu`+^YD|D*SwS1HIcC@)G{v#p73M0X;lUWME*Tq^p^qPoa zPkyG4giksqr{8t)d+>dsed5Ws)k*&RoclMQ@nO^Rb&PotF2O!V&JTh9I@SF1n$-D7 z_pkBd+HmdiB7LwR_x!gcV$fT)H}rb>yVtALagW?(kvi34m6xNhbTU1aO9lJLFnm~V zu@*R$@Jlm(KkSXnUV_+DBl_BmslE-(9v?$Qp?3B0rJV0iO~vhFX2Qd(uB{q-%6LJNKCrM^yq{BL{zlxbPm3Eby+UxpdIux$3v^yqAqpwPBrR@ z`+zv#=y8jwl+Ua&MKl-iHC{LImao_IRsBbcbJ*+MmO3F=Y}jP`c6{3G98WcBZp1h7 zyil!T`M@K0#q}F(n}8mOnRGA4-x2j3xDrwU_t~ttEMW~DpKoP;8)9A%j7SqJ6qy3=6-Ued_M`Ai(@jJJCr zHs^%C7q%A-FP514?zEV~%Uv_=X6WmQO7$ZC!7Bd8HeP{#8y@DT%~|Xs!Xy|wT?fA| zr6!3AD2a9cH6>&pUG;!NQC;?&54jj}5^yic;#Pn^+`O<~)ZJsp%^v!oc5a}H-`o$u2Tf=8XH8oSL-!g!He3B?;GHsur*YwBC73^AFPu< z*MsiqcFmG6EVOIvr|_h0&#U@ZR$4qXqKTs4tH!5hH7nGT`g+v@UJ82we)4@m<112Y zb3ulY4A^6-3X>uspS@zK=i2f<0Rk5ti!`pToopcd`{0Bp7m#)t(__fR9OVayt&;bT zPB>gh5k~VDDcuDs9Gf!r+P8{&&S$UtZdUBI5wg-Ok|%|vrylRast2Y~ z{Z;+4*V{>7HxQM*0sBb@wO;2g2mP2EdE6O3G3i?OcBgB^XemmcAr!C>yICA8s_jLS z`>m}oF!IJvu={8x2std)K!E02*eGZ>`yMP4((6`}_53!CMfR-LQRg)<;!6J***kwLC={amuZHN}Q&J#uimX%B6Zg9lW?d zn1c$P-rgufxld9KXh*k!#iKyb^?-QQ$(i(1Ga zS}=_znSLW|gz7wte0#xj;)vz1m6!H038IAV+7G=0S&lzJ!6G3K&u;B5JDJDKw98?KrBO8p3N zRt03W7=QPgUf1-CO~%rzBfU+M+l?tr^~EFwi{RKDK{+13g@0{UiDof}rVTMK$6cT) zl`Y#N|G9hK1dK0Ll}je-Nlz$El{u-k_9?$PNBZYy=9@WZTW>tFdJ{LrsZDYNX{+y6e+J2g9^Juvp~Utq2fFDL@|r z9y8b`MOjpw>31kWVy+CA5t(X;8TN?SsJ9sF`T8*>_55=`|07A(VouG@o`D!Ax>O=D z-FG+qV;Ry1WT_&Omvj5de94geVgUfwJp$2 zMB>CY(&2b`o2|zjYo6 zaOfC)B~$&Y@MNz{)x*($Fb!VvkDdbT(Q|D~Oy=Ul0LP{nrP}a7ELFp3$uZ|ANaYJ4{ z`#a(Dg{t1j`}JuTa)ys4%)CohPFK!J8ExaoU-o8Y@clThRzKu}@eRPN3Aox046EtR z*>WgQSPz@(x~dg#*d8mX*eq#m17}CCE52EMa7W6^+&R-=7xTe{WIdDW9gN*4IY@ev z5Y7*YUo$54nxVyPj-505onJQR+mF*!A@l!5K&&=NpOOQxM6`!mhvpfbK_?M--9^Pht5>h1+J_N8XZko_~<3E zG@}lSR}Ri^q4%r-9q-%4u2N2Q)ti`NX1Hq0`M-VgEp_la(I;6Q+rwPC#<~!_?z3Za z^~Sb==jj4tU0i1g9CPL;EjN!+XGs0q>Li~o!~;-%i^IAT)XUt;=j4-1ZEHE&t672e zst_7nOMo;}CK{*32E=^)J2i{!sW8%cw!v-cxJB%jdr27B+uhpZ&pR#XxzCX9rC@8W zeE0i#vO6m$Cao_gDQaYEu*s6I7mGwG))qMO)wigHzn`Np`qyV*t{YF#k{2?90Cl3Zj+lc7_7eX45XE9fk;l6&F@0NJ2;?gEy5t~?2Sp5`^znm{;W=GVg}p=NiM+#}ilU6i8UKk=aL-!J&<_35~AI_-aAM&O+nbZ7tL&IAAc ze-BDe|1ZOiF8-fiu=f9n3|^GQ4X*ZU7&?Ee|FxI@JU%b~gEZ*xYTp*;Jo34$Y=JPU zsbEg1RzU6brS-w&UhiJvx)P=vW&-i-P6`~3{IBfmJq@agV@gfz`Pg|ulM6X9Tz`7T zLuNcEGY>IUglQMN&--Ml)?{LHJN^58Bf^wHheyVE@v-3q5l_F82OYPPLk|+F_Br0D4fPuoYkTZuWR^1lxD32i8$q(G~XH>UjrO zJn9YOAvL?TuU(6CSI{cRs$6)YSs_+eJy-dOhT&T3zX1yO#Y6~uGZ^vJea z`Y2+S*b9!GK^%Q3A?>pn1&{K2I}x&u_f4XZu@4T#|Hv(Bhm* zoO{Jns6v8EYD}!EdQ{&-jkrT$GuH4Jw_2`{y|}gUo@2Rw;pf>NJH$~uK;;6CU0@9S#%mYyZ#jE+8a<~DXDTn#rCVH- zt_h_k{Z_t)VF~2Tng%A(7hIbSopo;VyvlZ!RX;344s;io48&JfbwBEL5Hh-%Q&Rny z^AeRD?L*X?aH@5^@pTH7#ScMlp-(*1!@k(0uV)^N%gRyD*M-$~%pPjuaHR@E`zdb) zzJ#_*zV$LOo-5IiXYVbNldzH3V;Tb&wEMBt?OM5ib~#^|W_Drf!F()yfmBy=#lMCt zRrN+A&0qs=6Xv_XYX-YWp`a^J)NK?}q+i;&Q|)kdG6Epsw%)SDf2XG}LHP5xvLhrO zup#uLG@v9kv{r8-BdIFz=PTUmH3tTEMVq;U3$cI9*^82I0MyYNa%)!b0e2e=y0T(z z<5^&pXSmMkAp~+zc&@m>Vp0-J#kE%|? z(C7VnR;v|{KrBgnPi6Fhj(Bm&VIHQ9V2fZiDSS4<3_o9wcd&_&6dQL5>Stt*vUXOWCdu)u#rUAP6GwKl6*Hs-m=mHU?YW zJCNKgK_|kP8s;AJk$H+H2JxMHyAfWmO`NwP^BwBy>Ea(4Y6oR#Qj1^EkP9p3Rkp#4 zFT{$2j}m#W6k67`$gIf@&G<-3_L#7L6km(d9j0%y`?`yDP;Ds;r3Xhheu-dnIikFbYW}%fe z`%#Ch)$4?Df%~0CK=){&EvXFQ5ZCs_N>1cxNUPnS`voLpz^O4<4l%?>`#h5Qks12Vx{`Put>jHJiB) z2osV&u4Nz2@Cb|7=#8z-nXp~nw-A#71uOim{U)>SyBI&U`|iSia+D~85-C|#BeFZ= z#4}gpoXVBc+Iw>2J+a&y)cU&kUq^q^)N^9je~yh-i|0Tx+f`GU<1~_DNCw1K?Czg} zxT_RUm(WtQBqM~=*sSwFbwt+MUpoYexSobEUKh9RPwNxcxyf+(0?2_GT(>a&YggPI zVqq%YE4O_DByH5g)1NBfP6_GhWmZRkXA4Z|}dTEN*tgtR3{+2K zJkkNI%O0l9tFD+^^}V+nqLH3+7V^ri>U80a%u(A`T%EAX1jXzrvgp@?-Lbb8j_8fB zWwjr-SIRM0%@9Whr&g`S6j!2__gDAg)quYt`rZE=7HR#7weX_MmjHIJ;NAV$&9Kg8 zj@#sIIQzT1Yy8BJ_|ou}OWJP8)K%S*V%U8603aXxym6!C=*0@Ia>g$M9w~}o#XPCr z&pB<~&UIYPIqa@IGkIo5YSja2@CyFZ;caWB8z8ef5LULQM#xrS%FwtcBrctRDNTvn z$+nrPeUN7gYyv!J`Y=XbK8p_?!9`Ti%u7baD9DAaT4T0@O(vixT_YYk0w*Q2C%`}f z{dqqN_adjkNj)R8`J*&7lVula??lumzSvYvh$6AECFE{LKKIkWB4&tQ)lViYi(}ivj#ruY+(LBd8a%ksVohH@~lRe}xoB70k_Q)ZfG;(#IUzTgVLvk6b|mG%ww|V+%e-vUmGacv z@%3gaose#N!7&j#$O;!)8g8Sb(;s2{NxoyT;xc4j&`t@ZhlUYfpI~R{V~^;?E9~(x z3zfuD!s}(?afIH()w$fvK`m}@BAiB#7VGFj2q}U2TAYjl(k{wesNG5&Q0vp(C@&=J zJ;F_)dW^Aaq5E^=`awk~uOR=;A9p1$xvGr)6y%8BV z{&|1Se?KLHmkByTDb|Qj+zUKO1DmMb&zan-7i`r|uE&%-Xw1rItc|iQYtWGrMF|y0 zJrr@ZHAPB8o!ZgWfUkM=sH^sSW6ZPn;*8$LK{rwxtvh!Zc6EPnO<7)fb?bfdyDqys zPtTJ{`xA})OMDkJdgk7ikV8|Y_X;ScLIpSr3Q#Bv_FAZ`-|f|HXJ%04mJu zx@d6;=YZ#((CedV4WmArzTDmuXB?YyVYIZuG_Q{2)vfP*|86ay*cvA@T<^g*@mfy?cQ|;Drq|9;Xe7{+RFS*9fqO#@6 zujn#>Na%n~a4HV%DO-**%c0a4YBug~EZBU`3oxEf&`;r^8k#tGC-!U z&#F}R-Xqwlb1EP+{=?G`fJVBALR@i&o(ucS>##IN?6^XbKoW~;y^s+17bA_q>jmqY&R21>qW<$v{h0b;IGi1j2LNZpAzk#7rHN%3X`-%Nx`r+dp`ti&0WaV8cNVKTUryRMxd-bCHq z(JPEgj^OPLZhDe9XGZWdl4?=dihQLS`T+1p#Ez52$wzjhb%;Sm*!oYS%#AX9ETjRf zv2Y%#IHebO<~p(ONxRJBAC|Fh zm|)OlJS7P=S)^ww`T+OIzlT*ZM>u#FGfcc zk4SybWBkI>+T~o)JK8)>rpVnNf1dbV@SJ6Q(=z-~!A8)d%`YzU$+;YDoJ?DUcpSEg z-ajQ*@W1X01(H)J`-=^CstlLfxD7SNi<&R$;1iia*xluDdp`y)A1F64qyr^gsP5hZ*u4W&Ct! ztIA#N+)dYWX%0uKDZCdiC1gbq%B~*tWbo(je=td+cL2k}zh6qIjtLP90X?pXYL_fN z(yK-NED&&%&2*ZsC}Z$!7anyggpTj>56n^n5qABu`q0uMrG>Q8oyPW{QOXBXdlXfB zXKd4ttnFnn3(8A@bfUO=R7`o3moc|<<$82+3wNXZ2>@gd&n&hj`j?6Pl$(zR{1W{} z2*dF+q#+8L0c9$=GVk|fLiN$9M*zjNUXZ&824(+Ts5(ai5>4OtgT9y@;tp`ohwr=1 z(Q#CVLEo?62ODW`SX3kh)M_(vH0xm=VZ`fJI#G6^1jBog;Ft!PH06k=vK&-NA|C~> zybfir{y-o_=BFI6Am3&OKKk<%ZX!kJ8r_?Os?YcCEgxU1U+^8fn1c2@5%&9@f?Iw5 zpRIyUZ*A4q}11C>>V{AN z)i2nKTYHZ@_m$R+=h)d<>+6b{(>9r)yB$AnR()8z{HWu?j&PLyQD#c#u#DP1^rB1e z^sc%7&uAB&xyZYyX*JjI`H3IXeSG-)Q?6#guJLMFbg~&jb~#>!R&ucBdy*EU%4L_7 zh1yitLdZfQTxI2eIf=jF6kpu|n~0mbKk@hqHXvH0WOCh}`Xi`xqO@1<80Fq%E{CHV zwRss6uoGcM<3q1%Mr3aV&_hvY@&Pv32wyxvgW6RlM*X%PX=ixUB`6}5(@C-Ytt2w( zf>#Lw7j-I8f87eQSmby0yruXQZs*#7+CwLyczUCQllC|bxJ#m1UKm&M-*DBW@dLdGyBaWe zjz=YvS425fo6Dv2>cA6&f2ku=6<*Z)2|r}AO1ar zz;Rqm{5#^kgc`sJ*Y5ezro!W|!=is=7QhOhyDsNyPUw&dnRq(7VGb8j^y{Jk0EN@| z@m~sOQf$}R$P6)?=x=*9vL;RL@UjGH3Zc1ZdJ)y=2<^t3_m|l$qM^e&L;rC}$;g;* z|AILgzFME4N5KM$xxTvcw%sRxT~^@d4F4-?DTurAXX(FRn*I;WN%sF28Q2vEO1VOf z-(S>*!xPc&pN`v_N;iYZ|J}~9-hYw>CCj;j-kST{x8vKSs_iUgxNW15XqYHWC>+o{ zZMvI0JI!yiQwUf21|OkE7cx{z#~Kl16yo$ds!cUn(OG0<(|Cv1D01vwUXwrqi*0thwD` zX6mS(9Cn*Awkcm>u_Qmit40YpU3)Jp1@C@&vUHGP;Th6pjkdM8H7IRRzH4Q4wCb8{ zQB`}K?%ESO&DOYBpKCkKd$}_3@CEOOz=>mtn*F2#XJ$1(d(h+Z2(C@0CK`$Z37D27`y&Cz{F{3&QQfZb7XQB5m^f zZ9g+5TJL_}T%Mq7B#wIRW@9CHBaLzf&YkXE(1t8DH<&{`g;%-$=3(qQ44lU?* z&WhQsCw&^pxw9FWy1F#%ETkRVgT6oR&&M4PsagiAuT*CyZTZhn88k^b4|)!@5tb3t zmgGHlLE>$=0_O#)`F9y5UCQ8k2 zJOZ)gb5M_IHi2#EPm$f2d2UcA8Dyd?FdNfvqs2n+Idz6f5uJM#;#&Rr@AfDC#!gk~ z|6r$t#~IMeS5+#jp3*QkEkX{zgylrCchU_yVA(slU)uu3zXW^naqK z!aY&x1vYYAl&M}tZ|4oDZW;N`a8Eq!*;sOpOHuePxAqHeq(YC>KY=^=?s&^^)Wozw zHC}$^r+v^n{NeuzG4*;_5SE5nd!Sm}J9`){UPif*k(9jBIank)!NDTids_O*c3Qgy zJYb6o-sRuug0kF*5o$xN-=DVv_f*N7^G_VN+QFNDu;5qbrz3=jyK-jxiO}@sT~BnK zQ~jmC$)S{rGmt(rqW<^KLj{L61XyYuY&bI#_N*|wufN-Fm@*U{U>%Ngy{Aw1RvTy$ zuuaKogt>`2@H1fIx|gsR9TDJ@IB&3&Js~!o{{hznX4_ZM%Xy~2sbp9!z{`7B zHE`3rP`}JHhG)g6hFbtHVb6Xo1}ml0ICg5v`r2HC>BthF5riy5avmMCBebtjbH(C3 zVgSuxwvVS+?(<^ct!5n@BcJ%YharXwpT$O}SW_rj!0Zr5!o`4#;X_8=yX1P_(*Ro8 zbx;3XXs zOiYc`eJ8)KaPLBQC3O=6m)29VxcRo6>4AQ+IGWaC>UnrQ(OY2o5@dX9ye;qv=B<17 zK-!>+Wl3Jk`j^plGRK;k5?lZ>F%|ROuq&IpxOeT$tUsI|%oP_GH{p?QFnntdwH>Ik zb-Za(^&kh@vmKoqzQ64Q_5Ns1%}V;^U3ckB#FGZBZg}@N_$f+unGk=j_FHI22dj>+ zzG#LbiW39L#joX!UsF|q-#FLaGz_X1*s zy;Bs{Qq8Ie3Q(|`pj6GXPCC~kYlWqo=43B!I&al3l)ARA!q`Nu}r?o!kb&&LX6U$2+MzFQ1`R*2%0V{0zNH7vVT~KL!qFW3-(jWG zIlr*ds9#v=k+S1)oyXLvJ6F!At*oeJ^Z^wr8n8G&pt|gq{A_O`zVc*3Y6z1&pi5Ue z&OzxaJv?jTI)ec@Lya%Og`pjNH{LcwvCaJAkFq9O7wSWb8oIKJEymR(HEN-dEY6O_ zSGwGnN+G)Ix+ee*7C^BS$(UKYikA z-tX2{lI%KmNMM@c@M*&)<6Fta)=(4{dN_QEQ3e{fLD(!P9^CzL=B1YrL1!ClV%0t< zd#NDYr{#7t@@3bHTeArv_P7~KR{AAJ(AG)@& z3`#k-v$WS22zAIl_bP>z*()(V4F7v5K9mmA%soCM{P=R@X7uaD3a7ayy*f&o4mca< zS{W7qNiQY;3zE9BEJjUfqBs;bFU4G<(!TQ$HE#9IPn`-)(KBYtkUT~s-2V_FB0O%g zTW3EB9W4>c=O_tJ%n}G&Z&FPvRb|!!Vfv=D_!9N2zWcJ34`#$mV9-Hh7ApMj7&xp5 z=rqcZ;r6S_OF1C#Yf+So6;(#OrZEQEl!;-iV0MnQ7e((KNKE|tCEgpCQbw#lZ5e!l zw{0_81%*~E9KDLh9pKAz=-r|eqCQ=ps?>}x{qNvp93z<+GyhZ4BzUCzd~HYE{SMN3 zu+j4qtFiDLSLAcFih()?%^WeIXFG~V+@kS`3qC^{4bbl>f^3>0}a5N{tu>F zYx6%b)zbi`%9yUkyh(@dE4byP%S+j-X?$f=IVj8h=oQr_2*8tKfT42&MmldUZzWNQ z6UKBbE?`8A00OKS<>AQ{4+o1owdGEerk4?MJHvN33}IOPhT;1w>PmX;#4NK}fvuf2 zotWw=xz}Ea$l?*9lS@adjUucEBHmA@c3Il3PE2r@dQR}P#ZCkD?VsVz8fg;e# z&%t`(&0I9A1*>czwZkd2^i`vOU z3B|S$uP%rNEq{md{z$rsNSEx8V1BWX(^U7-?k^vyg?*p-O^ktu^c6nBQiuRzIgrq(Y#2-GQg{})Yt>y~q zC!*1}yHq|BN~JUD04+zHe@R)n#$5d3#Db)_WC-o;pVvNGgFTd!a8H^S1k3IGMp+X1 zC#;LPgeeWXTTh}^@{49$DC6y~8Bz_qx2KbLrgZk0B%Q;#At@$SwUqUv%t)}g@yfbj3wD~s^aie>G%n0)pN7+rZ zy)1#X*73dY%S$GV@1MUP!`{gc%=&oFrwArRI}4t8^y1zX{pX5Wo}7;tx<3tlk6-== zY~}qIY-PLgkU?vIq+G)sKAlN#DO2XVs_veDX6o(-D*3*53_RK~hLf2(H`~E9fAQ2U z)8f}lY+F0;=u%0w=`TBQ$-E~{zie#`O3g;Av}u(S3wIK*_hvk;_`?k(|Xrv>M94yd3?s2#5FA zCDY?lWh}Mj`*te|^=#z5nCKG%ge^5^iD^tTu6eK`MK|AX!D(mva&P*axJ7lTqPnc7 zQ?FKGHr zU61PXe-=HIe2a5Q)iO>`|E6uz`%mxe}N~O`^+uv+Hc~yhu`KloSk!h;~#Fd!2GGV%f@bb&7wsaD-Jvs&|EU>?pq|Uv-vS1 zv2>44b}2FTXb`{B%O2fGyJ*-ebepinY~FP6u6Fw+tVlhw3KcA>GrjtsW|=r$mnJM(Q^-rn%O;n8Uizzf$)_bpKfv zHMeU}SXkLphOjN-4vp2NEp@M#9AE9M&`vE%NxPO>=0))>0!d|xv~;`g&v?qdKyGfE z$~|`mCUKw~haT^FgF}g!G|xAmZJ*i;kq~A}Ed>CFi--hdi^#v-L>c?0BmZ?X;^5j} zDJyT>kNY1x+OvvYPV#Cv4F>i+<#kRaC;29AC!92Y3J5DX=b{|iO#u3f^`Z`{=$CqZ zqeeq2I@VA)?5BASs-zhiXn;XV`Zqft)ZTd}nE#Pnsoe}RKQB_-qt;@__8ps1239w( z3>iQU#)HWL5To6K#7B3J&HLPSym`}iMqv6U4F3J@qs4nS0@ijpR77g$T;4er*V}h| z|7njuf1*fjVF8d7JP_`w-CmjQe>fSKV?SOa2TIyPS!+Qgx*4w5)+! zHMDEni@xPed;h<}s-HUk3#-NhbSYm^`kj$-OCN#Z&E&VIzBs^W8xP`>qx~ zN}BYV&E8o#{rg!mH%lYlNls1f z4+^3vZ{|)jx9S_n!%ogT-@p2DkI`W(_RjAMJLmT;e+>KT737xSqGZjY`YBi#T5ISs zX^=ZSj>s%aQQj`lm$#1~Z!mRJv~wpnl+Z||`PE#`Kyo#k_N_HGew4eh*K|3UTDQeQ zPxL@T^v;*PJ~}@o$2AZ&%}fXFr*Yj2^B>?{i}_CrRl{qO7uIQf_sN&ypXP zJMYO^-5yZ<`tE0BqOHO{Lqz_gpLiV#GnrmuE;$0ke~V5F>KRg4lYf6On&0B@uid`h zlW8{|XgNL;)8r@jlcjj(>-KIZImweJMQ5$|P;ucL^P;mFC9Zgop3cdOA1`N>MmbhH zLX&skkf=k?3Un-9u8J4!-pud3wxh6qp>#*1yhagP^&+nCg4cx?A5VC;e3rlaV&9Cv z@(}l;ZhYaR>?P5klNq>KeC3zJQ^n0n7rQ?VTtj4^T-wW64V_In;}V;5>-D3Oz1a!? z8=?OC*n|rORLl|zANuuttACGW+xZp~k|*NshQPPQ8?@Y(vktn<@vAr&Rgn|kV88nhf&GJ6(c_LKdu=WM&O z{7EOmeKPP=I6r6Y7cbAby8VrRd1co*5o2j7$`{FDy_o*fpWn86dns2$=N+$HaWTOJ z-`Fn0Bqku)`qCi3@Y*Sm-NGTe@JZst%~Z1gtpX1F{z1!EDfOqQ8y0Clw}N^;s4CfY zl$EOUNjM8nm2O8_?I_9k&?kC!=Fiy1?njT|%QMN+-5A*49a?zic+5du-NPCwmNoLH zhi~;h<(Qb1u04B{qDDh6-AQT{1Z&;?UU7wgH%-=5`tlQ(y(jaFH0lgY$*fs0lTY3u zulyogUPJML_INF&;mD_|!d=*ypRmEEP3q5ofU%ouJ*}RWl1V0XxdP-Ys?(bRnOpc zh0#pBa*-P%M5)# ztVj4>13w?SI>f5R1gDODpo}>>+F~wo5(mW79LZI?9OWA%Yfc>x<40n10$F4K+F>cI zvvJ=TVAH?1L~?1at^^zZzLmM~Yy!2{meRr-PnmNDT(JvO?Nc zk!@^v>v8LDBd>1)!Mg=+E;Ck;{nMacnK*=8$?_U+izB9nEk{13Zox6+I2)=4uORx3 z@IC=KtU+o09ON%o?VWyW)7F$PZhsHn<4d8}t8)+fV9cGcW@Rfd5Eq%r&-znuskOF#q^H_ z_10@smC{|O4G8`Hf;uKm5NInDNyjKZRc^Za9~R*zC9nP4y8q)Ujir%~K#S^X5?ag+ z@}{Wz;hCcGU&Yd7F1$Ie3G7p;^wpG?gIQkv&<_Jr@p6l3ft?7g>h{(6c? zSd}?yJ|N|-T^8jsbR(6dxDORRaRORpwQ@ai(k6WJk9(W1jC0qNqFKEnadTVe;x!{^N!7hvZGF9`MOHQ9}5NHFTJ5 zv2WEfF%J5B^VRHBHZ7+6er_ubf3Bz-tv~Cpqct|t35)fIpEqR9)8mA|Y&&3aC9h}> zMPH2JFZUdl%dEy!j1YX?f5%xOLY8~8cE8+yC{(nso9FD%1~x z9ji*LXJ##4rZaq&ZzaX0tgXkNv_>_OkAn5Vdi%mf%i`ZoP;Yz_v;zRs717LBf+_!o zdvATLWwyQCwnIe$Qwt-QJD}%N5@e_JAfFhLl|)Pcv0_$ojEXVSbBbJ- z`JOHpP2Ui4?NmnYZJBWAOsYl;-pQ<9cqABxG+wJL)>5~07ttCB87{DqXH-;edW@tj zIb0G)$=4+NJlgjzL|R|61zZ~4<2RvJX1O5i%p<*_U9>&Hd^#`nyE^f4(Z-UE80hvC zl^JCyGlsB}e1TFjP4!#c+aT5?(;*3t%X)9%@J}Zcju+DKss{Qw@p!cKh}Ge?`Jedj zu*%Nfle^Le%H_25;J#gv1MOLUUj*`!%Vty>fALd%9}KJ)xt-A>1bojl8d|URHNkbV zNja;F(LU8Vg7rbRBZIvyjBc=zQjclCsw-9wAAc64 zyO1VM3YVBkO)R#N=-4vNsyztCE(eD&s4j?QQ54yi=txv%IgVzF?wRI~wW&ZSUdeGg ztz0_)XZ^7+q*+}BugTdfpW{{*RxBLjrr+@u$2-p@r~Ecd+D|sJ78Y9_?5oy_peK zq!jsbRHt41Q+hks5Qeu}y9Mihe@Z87PUXQ`d7X)n5l)l36=40U-u%schX3QU{~7gQ zC9|E*+n$*|6kJuO>w z?@PVn+3)B5B1FQpvJm7j&I+Ibft3t~Dacn4YZwrAn%jt$6 z_E?yt1P}p zuz+1l{ecj1NP71Au=n`nR}#}bEre)Zumxgwz)l0?(QC#al^T9R>-RyK)j@MW#HIW8 z2Wa-vb*xF`H4gp-_|RU3I>_m|*ZNx~XJ!BJQ~hAV(A}x3VYR`PnLRD7I(jZ6ojqW` zl&{4fJAShJHZ^yhlug?cO>b?gKv0PJJNIyvCIk;Vo&+`&t^L5i=d4doWt24Vv5Jjy zU_H2s%r(n#**9xzDXl)mnWfnCRQt;AMuzy@ZYR0q%RzWJTB1dZ9O;g%GD_xblSd4+ z56|A;vE0;V+28QTox!U&o;+2N`s-=WW#?5~A{IYNstjJJUcAJ_xDlocC$P<^aM)r? zYw%-SNtm^5D`gd^D;GrLpdH*q&+!NxhE`!UCo&BT4sm}6dfO10ldfC3UmAPl3G9SS zQ=5b4IxUZNm^q6Vfs+|E(nixQJ0{AkCBaSL_T^y9;%)BJP=GtpLE7S-GA3RkMkp+P z!KqEOfVX&^f_RCMa>A&@*p-tOSpeu}MB}|Lg-;kkWuW8)xx~g<@>tYLGoL@=(lp|k z%WPcvnG4<5N?EpVPxxl`R8mQw86@ zW~w7Ebf9h{c>{7COX03^_Y@mOI>T|xbP_B>{zt9&g-^^dV>A;L)XItN+ReC~Q=M5Z z$r>KXNtK5CKp(&Q(U@IRSE~9={%8@!qbnOI!&Rw%q)dC^)1Zd&$a3bzPYd9xx7s^~ z*J3wi@c>v^KVcm%kBV zbHU8PSCWzezzkQv0dC42WXL-Q zhk9y8=DlOt42JElUwbq1Zb-p1_-AhV1@a7)qp@jI zNC4p|Fix|}J5pej+!r&x8J3;R06U*JuYFJsw|UqlgXv*-)!SfhEh6rx4A-W~oj-U) z+Vv_5^-~!(3(#*xnMhrf_wi1IMLX$C1W@A)l@{ZQXQh_+`Z0=3gW~V|b^{p$hszb-@q*;|YJf z<8b=_tEiAPtyc2=A334VP7O-HIN9$7U}<9iPT14^LHx_-Uupk}FV6q?n{Ikw%nea3 zYuIdj%3rE_A}2RHwG-IQ?_p~PXH+Q~&5jM1{@$D6#arp|cD9$a*@o77Kawj+=zJZf z9&B#(zTd-E+whRK>$K+PzNyW20C0p;gsdHp|6+=2l-QLPwDjqOLFl~OHKO25RH)Yr6}A0kEtNZ%U5R0Z?0UM}ygx2SZF;Nt&?4f%}O(YIo;a^E(ZdX!L> zpT}8d>P%6mJW8R4IBRJVYPbLgmTK(^pN$$upiOH>m)7m0SnT->_Ji%L6nF9&cLSwzQT)th}$$#4hH80S|JFlV$Frf zT|B4XXtGqabbViC+2>dnJ0e`<|1vot?Mv=zLt@{02_AK%T`y|xKa@GUzB~nA+4%|H z$Z<=%R0Ruf)tF~p$YxMgWEW8OU$^<6uP6u#Zs|~;bX6CM3Q*M31{r=bIAX6gVl&Yw zU@`3BRAFfC*StDKVBZRI*h!6xMlP`QWwBcmxJ*m{D6!7G0j}(^oN7}Af%&HV$S62^* zJ89Cw9GV*t2hq^Jo~E7H5i5L)>?lqu9PlyXi&Xiib?idY^;%NS3MUwo(171UA*(-? zPf|kwr|1}f)QIn&lcF70Ogk%NC4!xsq~b?!G_WI=m!!@At1l~yF`zk;Qh5;FE=`@| zR3m7}0kXY|mKLa!qR%_#X6IT2RLBAodtfWW7BwnqgXs(?oqFdtGHUdyqw(#QTp2a7rm{n$t=^k4@sOG^v5IGh=w9;Wf*IYyf=@v(DL0XFQu$3 zHFW*^Ek;PF-?d`PvaqiQvEXNi_pajMmJ=NcX0P1Ooy)>&7O3cb~LjZU9kZK8$FE6Ulsj%4tT zCJ--Gt}VeXX|>)#$iwG9A7ZgltmSPY=0MOG~RM2*&8>R2y^+vhUT#%$PX zLDOBsFIJ!KCbmyObAhjobd3nj@X4)Tq8@&USjLr{8l;_D<;ChDr`IOmobnI6zP`(a z)z&iKV~Ay&=`NP36&q&!$Pp`U{96QVe>eZaW$LXzPSO3wa*GAkl#j6Cr?SFF2`5#s zwz#lv2g2JOw)%ijlvK|JqT^GQFo(t;0+v6-k?SDzv0L(QF=dn=`c&EQhhcNB@!G_4 z6Samm7@_%eY%DhP>nC=oGosW#%a4SB*~7}M!h&L57#O~`ZR`N3&7ee%TZWi>bA2Qd zixU@>MvCaOFBKgKWA=lhua^y`OWluh&|8h!f9UV6BM9xG;7V>V&afP`IjmYg_hZZZ znv^g#LOOMJ$jC?WCxDg>D)DxNywq?c$ z+?bP#{b9rY29#ndb~mq__n7Ti#ZzK%+K~{##8V0f);SYvK_xx>6xj1O(R_&)soEj* zC`7zvkDb7|fm@J5Z5WEQF6zu!i;C-MO=#t~>x`S-tHfU z#aVe6Zz?rlfJhn3j{8`4imq6G*bd;p8KtBQGWddv?g}|!N<3Vyh|;bT*xg* zKE~NG+$y^@bP$UiB=08KJ1xW=X^&8}9`33Fu0!%Jp$EgDEo%7e zEL-j{h!aw{fAM11*ZT(kWI?zKHPujrcb*CVuMU`+Ab;WrW6J|wwFXobUMc|+6B>LI zal5ANrVZfTaES^uc98?94kURpkLD}^YnFh!&X@p0Ua(Jt%jlYspqIYQ;M}ybR}_p6 z6TBj2sur7&fo1(o%7vmS!Aojhy)lj`v~c&o4FEc?p&wu7gg8gvNyU@$d^3h6%oc>e z%eK%W!5OVVe0oV_u)31+Ju0|orO`}Ew;~A!3;!)ZRvxpRT23aA#08fU{oBs;^Uj&B zp?X?#VFQ+~7>D^Thnk_(8S!PFc@w`vZWvaJq*@C@yTvKyU@HmShINrMy3;Wg$+64Z zZNWlk=xg41C{|>MPCMO6x<0~UomNGGUql*vpX_1B$Mpl&*4I8)hE*E>NP10c87?a7hhn`F|wNf~^7t#woKad(B&U zg_lwk38yF2%uoqf!rpk2QY}h&y@-mJ50kl4jZ)}$r1=f;1OE(q(mP%;?taLQW&Jck zamTK2S3CH|#h`|5s_VIg6WoPD9T2I<-rq8F^N#D<9<@WmxXTo-yr^*%X_uT5#@BB+S-BA(=&cC&hdq4R7}S{XylN{U>$yxyPk?8$FxfA?a=2% z*|g+sN{!XVD&%qNCE)e&5gFFTL`zY{ER)37} zS+IDmRyY#T0RhVseOODC{{=0j@lgU#g$+;K637Q1JvQTecA9*OvQi2(|Y!bZ;}`40N5* zawOMl@oS6lHNkMSwoT{53ED8Bp<3K^DM*qeSQCKm)E^PAq?k!(Oxz}m_eKvK4BHqw zO-7woz<{h9^t+iBQ$(Zv)-iaC3i>epU;O zfvA`m=&IV5By`mvjJ$llc-PZJChvtF@ox`h^-`%6K2ln!?X{_+6(MMWe^SVs^%ef| zoP)m(&LNa@KfOzxYtZYK9$O2N9f`c*t#DT+reolL6RO^ zxob`IN&8S?jInIgu)F{B4dieKyQq_y>S>%n@|lO!dg%Ly-2XVf?_JZ* zt0kdpkmMouB;*vdT2H*T=Yil{#6}V>h#F}k_~o@Tsv`39G3;^HwA&3_#bB5)e@O0F zAM5$qFl5f3nQh0&fi9^JbwY%_kkVz0#>AlD7ku`WF!5@Lj!$%7=DsW*I;v9ky+DS# z?S7I#Z9h(wS9x+Xrv_|CJ({%!kjUwbWOgowYs~|XSMpyAhXoueHN9EkCU%)*t;!@> zOHcoQY{t3i4?0ELUItN9?$&>jbMHy*a`5uXlzNMm;lewgBE`YBoa^78;v8l4oO@iP zDg*&-?SnBgm#BQ*n9y#)-Y}n@i?@2kibBx<(D-VHdLfVZ(gg;75pC>!qJpOSk^OXf zI-4g?=JAw`ju9>MQcPs~x`(5O>Mig4P`sK@Muie$`dU=TFS>F|`~E=8PE14upDL-c zK${xQ!aI_`h>oYU)6GLDYB}QEDEM-nNWt-&zz5!T(9wt#Ona+6zvK{j3+YJl^_g_` ziX1bND`M2@zl0u7~<5{1_=D7U<3yt@4yT7N&5RPGU zm3kv4>48b869ReNXp7S@n>0pO$m5C*6QxZ6=SCO0&&jKW-^EzQx+(O5{Sb;~=YJ^V z^@cUrPcRZc@^2J@lHv?0@M~n|%O&%#wr;g{y(<*c62(578&7_vUqthIo^#jG>jfj= zvQj3C`}DrR2{6e@4iF{^?oxfI%<=}xME^8oL7Mb9_-klg3-tG#LKj|}K+`x4>qMAq z0`b0RFs{uhd|ddr>$iafaRk3H%(1X|i_rxeSJot>KOK41=cF0CUqhst(KIyJyc5+u zIW<*LQhn!C_yvstVOTBCl=3d$FR3@6$vWwJJ&R?g8Uwy|*VtRNf>u;TmyN})F7ny6 z>(w!+7#?|~ILGYbF=2y_5akwt^#!Z*Vi)(kX4@>~rmEUEN% zWkn-q{CuJIC3R7EQuAdd0vdFmu3fD4ZD}&7XE_Qep|AV zH{skyAmU~^B#kb{kXl8|0Xc?{`=PZz=|;x*g8{#5k%(l>gtJ4d<)Y;4L6owYC@Mtw z*+eY$*v&aF1js+#7m2~=#`^`WBbQuxn6S30iKEd=&!O%jtO+z;y@wa;uyy)Ya1cLO z_0;_IZ~dDNo)Ou6uQShUAr!nY0#!CLtIB$QJR0hF)RUcEUs~8N$3k{twY{{nmO!~k zZp9in*wFU$NSv-^v3fY<^VwVDUNs)(-bKui>S9?{C9B#PaJ+vw;71ky*IL0AugSD@ zvQcoZT{70oC?dz1pBqX?7^0BKjEb+)QCu}x{-;-NlW)5xl)zn%?sgeNpFkm=W)x;!`V}TVL<6Q z<_ut|bE1~~q$rD-4xur|!x!!oA~}ZtxKWla4PI>2ee?tDfZ6rhdY_o2s`#g|+V%+% zB@qrRR!BHy9pIQjOSdEe6mNt(_qRN_BdQH1-S35*^%2(_-n%A|-M<>GB7H<`Nez@E zi$d$)nTFK)G?IT_v29{z<>FAa_&~ZX+OjE0-YWH6#TEH;bkg$}e{!<++?f9p9aPQpWn`_TTrMJ!m)5k z-a^g(iO4UD_dW$3*oonXg&N;%i|ZjK`WTvAf79|Y`xym2wQ_hySeU6hHRwgkn`n_W zHJ8hu5yEV&6RAO|3sFV%6;xzH?y$^M)5SL6-S9&b%Tx5r2}&6cB71g~R5Xp+7=Rb< zivcS0>}N+SXkM44!NH|CUg2L8h_+i@ufPrd@MTs5DRXeflh=82lNN(bZd^7)?fhDY zDsk(K>&a?6T52L}VB{#J-T(a+mwi3r+4dnV-@JOlOeo25iy-VXoky<4tK-BWRddY1 zdq^^Y^;XHQ_N*#!;>7i;SD~6c=N8X^f}1^To~fWUDVmEykI9u1cdAcB&$@I6?%OLK zFC^p|Mfi1F;G*BdIDOd-_VPc(Dc3|edqK1?A~7q!ep$ea)=rwqemJBaV1PR7@S+H!Pewp(7q%12|rh zrXy6q7I%=A>A~Wi(nQ&$xRGUAVcJ2f+B*wpt*#uk$uKF$*e7lgjg<-LWwvT$s8(le z-a*yEb^t1cD|seV-!>95{e^vXglKLAPP~_(@=xMBP-mF`l!b{%AdT<`&4lnSp^Iin zEv$6pFMFmPbLfRK-A-6>3s}=ja&*rdq~6xHMb5vP^2qNP^;kQ>-W*&%U55wR2e8C}-}(4JO3r{VoXEnI z8qzk{kk6c;bp&Q{7M4d7JXB*p-F>n_;ldTyvG-lz;45FRx4cqL(#}IxuY%;n#g~ejEPrb?~oK-dbFsN&Y!Jbvea+_K;+F~I6(e$ki zrLxa#=XTzbMTs8uoIOyIuwMM`s)}b0qh8lrxMb{&+Eh(Z3sTVa?Kw8dF~Yce3tRW^mtj!C?sC&3ebXQ7 zwzE7ElEi-y2%i)Hb$%_q*Tz%l#-o`di8b?+qXCER8b@9-fTXk%h|ws!hrFy3^tu3| zFilm;Y6bXqB@pj?pqSbpe@nUoh8P#D0W8wZBgCJ-jQxHRLuj88zWLpFEs4WuVfA-c z&`D-fCp^t_1?!ZzkX#Yvrh-BO_^VLAPGUL*O;!~e@1;kTG12{DQ_MwAUD=t&{ zZzx|B_Wd%PsMaTVG_y?eW?CZX|9`9vK5%(+T8>EcH&g7|Gpx@{D)WkHe?modySGU~s6P(s3YWR`7DQtNz z-=uTHz<;XEp21PJRvW@(Q*#C|whovfqnmk4u89_N!Jy39#_5F5zh!BcUud%_j?YFe z9DF=JH!=E|>5QzIyLjy&Gxf2fMNSUXzR1%7wZ3!lcQc{Ym9@Ix9HvC^!Az zaT``-zNA-kiMS;}|pOEQ~ zgcm*@lGp5BNqGa78dty_NI(9p7Z`POI84q}YGEEPc*wdJo(9t{1-UHwAA+JG9K)x( zofB!qq1`E6*CyGi2Lz~L5xKw{ZK`bo89Yn1AEKr`Q8<722;H0u zt8q>`$bm$4Mx*s(&tP}aUB~~9$i)~M77$clEn+DL3lZ^evda`ht!+&DuEX9uwlbxt z;VlWchn{uQe$b%s?==asELhxaf#n$gb%S^pcJ7YUL-{#VFaq~)EAN+kc7X{E{;+U7 z_Ae_#1nTmB9Aa;9a2%VY=NBdQSXNvo>pvFV4t=-yTALfG-N$^MG_G^b9Ic;o+P1aq zeMh7+ZPxo7<4f6tnf%~(1=I`@^8Lja+VNi=?5e|g>By6FEoWM3nySwKR4ANvB5Mc= zDGxlh(crpTCPicL_AfW31|{`(Tqh~${?E7Pa+s6;R-Df^*>aW2Tf6@CcK^kH5B0cX z{UbBFR(wUqGi348mlH|`+BKJXO5sSxzb^OV!&0X2D0{&QB|vF|BWswULX!B= zBwruUJ<*ioE&pXq3m{t9+=TVTS0b3Zy0VHmdOdlPuc)*{6vnwJJ%P4OdcQQ09P<*& zb!v<-@4be)rz4R3C>csGU7Pl=e>Zv?MI%yNsR95;nU5e?(ax|QjfCrn@9VK!LV0~U ztM6nn*=Y8EU8N{+V=tWCUvp)hzaPJWVN?`jwy$me32q8rrO%7>7Ai)F4P_}fv$KX=^)$t;uD)F$4=4sY=6BMt(lW_R^_f8%Ziy3R@F%cZ% zD={k<^`tAhUNtD1;_W=auY_;3>oxgjd_#u2kBCHxo=k>CJ?xPmi3#1qCtH1M>v2@z zZFUcuzpr6(N8(o{9rpN@9`8kN2$H-mPjnS$R7SIJulo;Q^)U7oZJ@mJSm8M%iILpk z62=ngd(LeCXxj;Zi3{_QI%C7fiz9*Y#Rpepf!qDGIhVf|f*p~l0k>5{ zQBC6aI^;w0f&BNoYS-E~^S}F3`ck|Ny^NnvP#pv-l&bX&g*fqKt_e5hk*9+)h_t;> zJoVbOLqZ+|IIN0aM71e7_OjXBn;TMVcPjfeZSx0XaV7i>2wue(9*dN@togLQ#~ld) ztvHgL6J@G#QLwJx>RZ|J&Nh;{YSGo0rOS5;)2GFG4yeT~!!3@i$an#YB+G0b2KPJ7 zhK8Dt^xJPaKa&yJk6&9AY8{wbm34!U3+{*UvQAe(IBRm_G6y&M%+nQOCVu-+qdo4# z0CvX=)kjM1V%7&kgT~YWIdNxAHgl3q(a+aD8PIkVeAzg_>wTj3EkTzH$>dP6!dJ+< zk+6uf)0-=e|Gj^0A5Q;k^Ule?I?UM|7ohF5-@&Cmmh4^BfGQ!Q1)K7Yo|m9D)VY~+ z!H~ENW&C-55tZUC%4y-cQ_q+;jgopqa+sdlfK;RK$n-@Z(>&rX8<+WUi?y-bBbM-W z>PfNELeqwO6U*^Pk1R+N5Irg}g5D+6L7}$j@Zj>*+VdGmqDH{86;BRDqenq62mN6G zk|V1tyB9Ma(pssp1l;5j$7VLp$+&!k?EVyjg_c zl_KMHWtoX@=njRkJ2dM`vVxl<03*IG{ll-%OS%UK`{lmTOsT2iD3j@#r$bV)n{^S< ztIupBjsjQ`hn>DR?csx@UY}dD=QLqO@=b*a!9KtJ+JY*tmz|I7DW^_~KQ6BQ-UKZZ zNbrl2{8)b%oC)dK#_K%DNVVi+IW&t(g*%T(-iGKEUZvzki;N1vbyXncuS_bRcF{?M zU@incI>zRxDUbrKg<%5G;lGHEWc=B-g0~Y`t{V-24xBVS8W{XR5Im+~&i5hL(8mdW zq9%P{d5H16$Q*Ck&VypEiUvD3qSnglUF#5(M)=1SQyq^R?)XYOrd4cf^cCFePA1{( zp-6>zTW?Bbi@pDZ(cPB5=FBbKPe;f{8%>yccLWB+XawTx7fi?Hz45`)L^GW5_m0Ww ztu(W5q@AY|FqaskM@Fc{-HV`N;op8AC>O=PzZ3xUzZL+V{@|mo7{5u>HwKh)UpIU7 zMkmBneFg!}*R5#YLM52wtDo+dIf(Nev zpHS4mW4@w77G|CHNOOp$|DchcxDPbJDIuRg;KUOUVLL1Mc|zn-(aMhkzl)*n=5TF;+Gx1~=sb%6v$pBMa znI4N5*9U3RVgk(%i(d+n)Oz`kM?fXNyEwR)@ZGTj4y;UZo(Y0nFaZ^hRDmx!IC}HY zTUNiKryki=jL2=4scaJ9p@&>WVWn-L2vjxl`D@LJKp$7EKq|<9<{2u zqJHBe@TID*HUec$KX6gCNpz`COlGo6A*k~3Lh$2&*xx*MGe(mE(o=XiOdEQ4_w=+f)YoFF@jmf%B2h!=T0m1@r9bv0I_8p&b`hUK>>IW$>IRvS za5}Z(nVv!RT$MmPzCAAtBuS{ut1nT-Cd$LgK)cr--|6cQxd) zVn4Fcp|P7|RXfM@uZ^$-gVKta(2Q9g>Lg<>D4XD@iLyrA9Z;)Okfc{Otv@ur`xV^P ztS^b|3Y1RCM$FjO!iCM12qAju!xMI{u7nEcnt(n*mCdFLhdowTSQCKy(N+WZhG~@< zG8EV24^9bp)4;6@A{7FQ2B{nN44*J*JYUrm8#w0y^^8 z#OEV~e<$-~BvW#Nr>nuUs@(`4)&qYp{t=v!Wrph|Jjr>`23Q1`T`rcL^jcj9^gJcO z?Q!o>YW~#_Y=bG8#+}C3uC+|-jsFwb;oQqfVq08brWhlMY!ZFToLw#vM-C<0!-5=z zG?DrPkzxM7&RxNUZ&rkIQ0a7vvX7`aHzvMiQFj@%UW##tEAzoUdGY9!NePQ(V^AMZ z4l0CI0sGc2zva*+J`KywsAhZ`2wye3*MFlm9Vi3)i~trx;8PS@MOhMeBZ^Vzum=T* z$KuTvG5wB`KEi&!Q0|*pLPDhob3-i^%Gd?7;W&ilKW8`kDl5_%kjTl92chVaM zh^dVtq6y>(TJ|`>n)!<8F*QvPzVTz}&n`>8gS(Y07P^<~%{6KTkwWr%gM0BN)?eKy%74taM_x$hjS%6ikhc{vVehcq5M1GZW$Fh3ew< zB+iP0?IgSFZ#rB0{QAH^@)(KQ4{ouv<|vMeqk_yj=+X~TNQs)yq|sldQT`>VLs0w2 zA@mFOAqbgWyez3pxVqtP<~ZnkJ+Cgkp}Kp5KnN0zX}Fm9CB@BhLZZLG1eUqK+nhL7 ziE>zzc_cLPy&mRA8HKhKfxZML8Sw=zkHUqs;uU(Etb+kw32aP_nn?Pa*;F@W=}ukR zy-+$kvT=>MCexSK_M&ZPLB#kft5o)$pR?ugQ)jx?Hr#d8O_Hd4=$OL2x#zBne1U-*MqX8#C$mU`Ju5oRGX)p%PBgcZ3P4Pfi(M zaUw=17y{Z3&h8#MHVc&`PnHl3!`i6VLyF8GL(kt2J<(m;8iGE{1K zDn@NbFoee|?-J78mzwM8JYhomj<)j&E%79}azickka8BrFJD?n9R&t?~0! zzywOqa76a7Krqv|u~<^eMOfuH$q|` zrI0_v0Hm?9NJ>|k)pDPI8_*>C`LrYuCp>NZ?)@pKVA$g0mjP*_<$}7?6ZO|?5iUUq zaAX9Wf1rc?m#9fo^nlhQa7_wG1R9-tRS#+gkc&S=P{6NDU}9<0@sPT7{v_5pBO;*; zgeD7Uo!qEOnz3oT>vU}UE?rHjgB39fDjg|b;I9Vg6cageE$YTlVmd%WS-9e6MOqKv z5FMq%^TaFr!$q3$vV<3gyZqKxs@x|AP%%iSx`)$e5Wi!D(@~E?R4s*dT>k0oc~Kui zTCfJax-0}8?&3;$RWtb{C&T%ug*tXc)Wn(mOViMhI{JG21Za6hVFf<8r$Is#wxUIR z&sH&jV3N8fnPuL!;FnWtVijPE@j5{E3sdllHXd;i zmcej0<-2SBk)>Q~& z$LD@L60+7oW!IPqePm^!eqx`*1idTbW@f?edOgkH4>ftGg5aJ z<}IQ!_wm-@tNC2X8Wm^*R^HO>J&v|z1v_)|{=94gwsLQo@Oiu>g-`F3@1M(xTL9u& zEm0kYzEI?M-=?WKSx#Q1q2|>yhT-!fR3P+Memif;=oxNSEguKm{pRqa0PO=|YHx9U z{g)Z8;0CQ2*Ph&1+TzEm{;juA$0sy;&^VUo+vTmNvkq0tevm$(K3Uf3@7klp= z)zsRx{bE736%_?h0UHFVQlyiB(yM@g)Sz?_klqso8=xRvY7{BbdoPjRL3$_BJCTqO z($B)Z-TQgo=l#w(-x=RHW1Mlm^#_9zGP3Tq?t9I7&3Rux_%p?)aCJ~Pqw6|CZgU6F z`}^u3&jP9=_BfzYVJdR6)Eue+3D|5jvRCus0 zsa*hVt$jbe+|4_43VwLI$@8!)AWa8+zR~ED232P`Yvj$HORUDJe#lQVi(-cYwL4&D zWxZppSY4RP!R&toFeZWImvHlN6xb*b_HGJQ*Y0BK+MxZmvVH{oaVhuup;ki4AX8Rp$cQE3qrNjWKVdhwQTAw3HFXvCw(e{41pr6Usr??50_Fe`^;EO4>To+;RBUX?H*B|b4DwPqnJq;Gl(5u-E;hcRb zt-LRz!4-PTPToTkV4^~vrPvhX-T-^pJ@_ zAxgIxuR?iWO;HbRty%Hzbou$G$c5g8Lj+`VQ2wn_HYG9hJF82k!y!`odeay%*H48p zA)>Ew{q(fDo@BIT`#;)N#d$*I)&y@k;kVI8|1xDh^=bqwcEy?1CHl`%e}5Eub3e{vVOP%b$`hbIOgrUJdHL%U}>j>)~6)q7!GtBES-4W~` z7`+X;`5|DgN<`5L6b~JL!11=1TtDKuE}77)7<0CGQp7Fp;}#~eA&hf0W!P(KRpadF zn@arFsq>@pN;#;R0S=2^MMqLFk;O_9S});jn>>C^-B+5DiEHdP+LLDNj%v7O6?h3- z7fwCgTBmj2AqnzM7Vl!vuHHi3tqC0HS%*M(<4X_IH|*VO7F3@Wtj3y2>r{xVktA@w zKkW%W`}c^Wxf|{zN9w{1!NLVkb3hDJpeVMH^;fd$$+Z;eVrtwb>As9mTje+qzdQ|i zt3&%_=|I@2AAQ8m*XWa)4zZm87D7J!=6-0i2`zE2E`Juiacg$g4(Ba!VJ%xECs3a4 zr1aJ-Q}Rc4@B4V@Bdfy2P;e|pJ!i{I$h$XVG;xu+IyRzg_teo4Fa&irLqa`4IxuVASWv#%|74W<43UYO$?(dkvEdHZDwdwzTkuq z1q!5-q`22^DViVNKS#>lZJ&y1m4V<@u@u{ftnVT06X6G2cxHGHJQVo)P?H*PJDv%s zsQ4$y)jX29im?jr`aCpsB2z1;u=}#7TP}{=d>~7sI|j$hOTcs+50f|QLDSpr=5ZuP zUDsofEMslT-1ZBOQ8>Yc-s>Y{<@>Id*mSCZUZ%lJ&DC>VwK(718a`QLXa%fxgrOT* z1i8Rq(m5Q0xl~J}*LPu>)KYJo(-%q7y_1gZQqCyV6(3i{RS2`b z7Sv;KsOzZxIOGa8Luu(|WyFN7Jns0UxeGE+e@feNYRI?11X1HvPHJsu6l2}3n|uTx zlU-+=7z4?ZcvD2cE&1?Z%dU55+!m|1jQ01`guORMadZ1>m0+Q}4*hr6FO_5#X6Ng@ z+rowD(mphN-~X6BS7A*d6+&b1wPIHz5+|HfiqQ?o%asC3FGF=^*WAlJNYnT+`F{PW zpXIi_VsQC54*iAP%b>A3d-x6?k{B}WO>iiuH18RiL5rrc238`8fmbSJb2sqe^jcSj z3y>KL1Lw+-ifgn)9u|L@5Tft(<(-TE6ytZqsZGmG=a>p-)tCiYxen@k6#Vk=b*rcD zZMJNSOR}R^aBDvE;V$etME9s2nMFzYsmpbYLG3T%DgfiEcg`j7`Q-lbNzb}V*?aH^ zZi=$r()kpZTe7oXLXByU0NO{&m2gIW>`DhN0<=xA;$7MOvrg-++J*q?w0he#oovt6 zHChNtFm3P6C0qy!%#XEwQ@j0R7a$`i>IwPx&n6^`#G>ofL*tQ}tA z5Si_@6w#Xt2g2MZ^|f#AcEmIJVPl;5PYt|7uxSF@Zsl;@O~)*#*B46gwpxc7wO|^b zNu>J>rkgRuFAd0A%v5KB->pesN_(qE<=7PSmim0gW?hHl%sNuEmS+R$-R8_vs?beQ zi$z|deirugSL!_*tmLpo9GLgduSsBa~B z7VI!$TSL9?v5ndFLgAlg4Iws2a1GV%J#;$J-+GqgNvJMDVS?F%w*Fz~x)%i}gewV* zPkQoxp{AGCNUb;T*^yE8w>enDO!15RYM&}(Q{3N^XtM!2CP0*~l#N_F*|(FnCRm3i z&AMN2*Emg)ikyXisY8{|`8>py#Sz0cmsv?qUAyq6I5f~HHU=oY+l;582H3r(G;Q+- z3csMq9&s~)Wn%)%e@xkW2Fjvj=yS1TKXC)lPLMb-iv!Y?)gPwDJ!)Hk0BT&k zF(<2aQIzPr@5nbE7vOOdR#Hh?he%p2g0{5-Y7U4h@2s0bd`Nc+eks9{qECzX%kj{`rwk&hXlUf9!97KbXDv z`;+{7+5dh6pqJwRl|hti``gt$+^c(KcNvr?wrTSBwYrv?D3?%op{+|;Ht@dl^qq(` z``av-bwT}CC6r37n(Y*`6pZk)6C?V5W$&ZLF|mrEnBcD3h!P$hLBQb8O>T<8v8eFy zMsx~!+yjS8lFdtDnob=$MCmc$hsZLQS@%+B-|dp^kDF6IYM6LF{fQ5cG=u*!*absiQrX|uUz>kDs$8%cI=Qss#l7D?O;20Ttd?K&a)mP*I{t+ltbPUtX}HE;*Z62 zR+B@(CZDws!7c`PwFxY+ONQzeAMmrqwg<|Fh@3J=Fmv;dHd-uS3&WUf5KdEngM#cr zKo67{PHCvwTRngd73d;Q#c?fN8=G5St6;s|X?2;!>7+H_f7f9l2e_=y|8iO9{h4Nu zm>BJ>c^)Il3Na~$tzVJUVNv%Q^tiDDuYNowM|MQp#kv@k-=Lo(Im=*eCA?$-!AUuY49~e5Z_)32 z>cZ6>l(e+@-k#cqHckQi@H&{ZJ6w=EUjxXJyS01O+MVsF{hNfX#y7n|Tpk~KD%oQ> z34e1rf6wLgw$m}UV?dhU2VIyLS#rGtLN{cVi zJQ>GN6|rZ&BfkkDd@b`te%^H>8Bq^save|**^AWRqEoKUa67$ z=-iM{zB;Fij$ze1pVv1`B0g>1WrTiu`7fYLN8#PY6f+1f_6A88VWtIL+iqmxs_pUG8tYDi)m{-I! z$y1~*N=|jMUE)u;0?FNb);ssB{!8LRa6V-1Uxo52;u+|uvCrtOu<$nP#Q}P!s zv#=i?T2qn~ z9>-tgx*4mE%ZK2*F7oES!{WxuhJtSStNL@SdTq9Z<%o=4gqBxJr{+w|Iy@M(P{qBt zXS8`s#;Zu)59`6F^`Yk(U>(^4*t?2YU%q+P36_9s1ouxtenPTHE+i}VKq$b^)cc22KX+U z79Uw45=~HyeH%yL^O}!PKaol)r86wD$au$+=|zC0sTbHj<4t^wUYJbuCz&MU!yW~S zxCKVx#dE>4xLmNA^S({`q*nX@Z%Ny6O0sG~a&lSfhV1CjC5}zkQf$t10oo8KVBdr?8g(?fq`fyv-54)_IP#3{imX zw8)m13vK9-Fw(W2X$s|m96%nPXDj`?^?RD8W?Hb7n!yrQ6tbp`h&J?;vCE`}%y<>l zjTeiZKFqefJOYhrP;Tin5&`Y*9d^ulUlp&Ie~qr3(L>s6@gbW?{AQ2P_MTr)a)pTV z+w~(hIXP;@UYBzXR>M~JT6bR^XS>BzqHT$_^>?_y$-NL9jk_PKA||VKlJeNlh)ZAE zpi*uOUNT2ds#->we|SYtd4OC6+X`+!qbGeVNbY@F!|z>pTH?pbX3B)|Yo2CU%Icwq z!JIRQEd*eOt5Us52>hP3q{Id~U}L5c!wI#TTMP0tG{;OW_ZA=I_r{y@f$n}tH2wRR zxRAzt09y)>FQRhvR(A#q*E5fxnbU=JVxG9(c4e(4?J*qm+e1rcsB%fw)CtW-$?tq8uqP5V91&`S~W&Py6=@wek`BFL^f(R z2Y6^WRG6vzc4e@El20+6bX$roI>S^6Ny^q47IBDp-!`F`rasAWj6olr!rfKm{`4ZR z4N#<+d!kJemCE6;%_Zf9PiEck+mU_Rb~do*U=lczCamypvESuShdOlsKyD~+ykIfL z{HWmuXqPzv?NabdIA~_nYd#aqx|vi&Yq9F!*MPrF{9u_Yd#F`#XAis-)OoTjiqZbuV!& zo1f!l)p_vzI^CxuWaCk?E{$~CXwr)R*$lx_T1AOrA}{bC3rjnOjpeo$9@|_a@nst; za-L^-A^6I@$MK#!zW-vrUi*jRzXB~q1Jq*jL^_Ap#Ay}X(^aQ@kGv)xQ?wQS<-IQZ zc|g&`$!7S37!UIsYwq3GMyeihY$KL=Vq4LXMR|&@q|*7LrP%jZu}6(QZgSQjQk}1# zzhw{qdO@G#qiaNlF!E}<*Kjs=bJ>bkr(G*mNeW6N!H zT?nj2c*R@z@>t7J)@kVJsO$?vr=qg^r7UdE3h4!vr3T$(XN!3(d~@|%e8}G9>wM$|qv#7^Aei+Ba8oyo!5G;k!jrZo8^NIjeJ%h}P1de(P^3&=3;r9F{2J zwH`~i1NcIvEpR$~(j5hSO+LBTQxgL)kOr;|Y(K*PAvjwOWyjmT5%w12Q_=1j?or`A^37pIz$Kc5$6el?NJ+>pE9p=SdN(F_6KQPNV26@?KgR1D83fM zqIozrx#w)8FjG$_ua|L9&z|-2G~bExDU0<*ylST6%}Z|M+!!cilU& z-GAXCx5h&iG|D37eBh6Y18<3m)a#5lTd^PSIsXuE=ahKMJK&_mkLHr>7Gd#1rKYEg z?R!abB0saI~tRV@IGb)e#hZExBR#nL_Jqm*LF_kbB(P5My&&1EQSsep=) zr)j#42GRYGfs;CS4}g={3x5J9jShg5%yb7%=*vd(gA2b%lUld_CQagAupCOWlI~CRiHrEm7Mn{ZT6amGXp%jUOuDB0LAkF#+!)#e1( zUsuwOf%k#Za$9S-Ddz#DaM-YJ-kb|Kf&j;PJM>+_>H}2d``t zuSd#M+sZ1o7r8qSWNQB)p^ityh(|SBbmH|>=>a2>C-Jp(`JfSsEKV=ES{Cem`MmgU z*m`BdnV8ofEOLQczxi}jk;IT}IhQN^>xPT<5b83XL;S5g14wZj%w_jQ;7+>1-P*zM zV5*nX`-EBUpK}`8`jJn5pn}=?wW3Awq2nR2O_-xg)+}_yqG+`CRkS8I#eO{b7fjND zqa(K%u&v_9yni!d4lH`AQf9Cj-`EBNb+_@BiOe^AnpU3hect-t?0g*5lgp{ zTQ?mjupBZ_A8IUeX))w5YbfwJ;-0$Z%VlyVef4jGq^B-Gkj$Fg(ZW4BY>!oR^)}iX z=AZra?tmYeclb|!Wc=R0@gti6e&l?Zi2XJ0`}auuQ>8UfeUme!Bj-+7554x>`b5g_ zB&QMeiN(V2Gc7`=5VF&sL^EQ4@F1)xpLMf`v6b?%3hi?;kFYEE?k_vrdcp}ojmAHY-Ocz56qy7% z07c^NX1P~3b>^i((TqO z410LMhRnB>_{bXjc`i-kA(=&rg_%eA^C0lyRUE!Mxa83OsRuxg1+yJKTqJDUCQjQa*c{1*PoeB7QL~g3+d{;Gyt2 zg5(PY9kX@2fTtCJ6&qZuW!s*C z*@fLRa{ih!vqkb-Qn>?OUAxbvtRO4@T6nF)7a&5u`vm?Q^;wnI{d+5d!O6qc&nr6B zKj&o$>p=`c6RgwhUdyp^?*0hUiu~J)T+LPuWW_nLZ^6+Ia3P&mu#f*%e!g`rCHNhe z6Z0r)I$iDBY--Nbk?yhEpJsQw0Mwi1i7#$`Hd@O)Y4-@PV~#2EwB`#-Su)RB zPK?|Zv>b{So-cb7@0`SIyM*0;H+7&fy2L2xXT1o$P#Lb~ba(fakYp8H;h5tj>|uFe z{V@_x5qRrEI@q_omtm;|*6(g5&&*@Hv}%vLx}iF<2mN>{?(@#%-Mb`zkdFocm(bt2 zYKH^Z_T_NuYXYZ9H;NF)m))1Z3p4S{>}@*R{f4n)u*0_3rQ(2J;Jd@JTzTsBR-|+W z{|#|kDOSc^(apw{ceD7${v}h_6V}zA85&*zHvgBA4DEqgLY6KW+$PdXy~pXk>&K<; zrv+e#zhHJ&dV-)-rO|p}UPe=rZK)jT-88iu5;cV&{Vm#gJtd}*_e>(XqD1$9JMn+w zOnoQp^-bQsb&}dx zRzbjM)Nnhj&nkOaau5_4ye{edCMkNS^x{TVLnM>1!O(X?F_YKXp{AEtw>qw|dc=@F zZdT5#X%&*MB@C}wVqQ;I37L5(=U)|Z^Lrw1^^A66_88r_P+SWWX3E6}*iqIWs0OCC zadf&cf}b*r%Kz=;Z3R3qZpUbj#=Hu>0FZV^^OuEW_*%Py%)aLGcf71T07HgeY#I>1 z0SYj*wzV4Ox9xglWR+}Pp1^lPchu;5%&U;JQ%P>kt;caW4!B^>owmfP#_Kg&t&dB! zaNkJ|X*W)~n6yrEsRq_jobJWCBT79VIeGWG+bHRC3`4M;Z^fVR0(i7dmbk8o*bkZu z{A+{*nAs=gq>|2C2WViE{?rhB?_g~9vu6spw9sVEuMQ)Le@cmu4$;Xi|38-!`5rZ1 zz5iuCY0cJI%SP#qluc+xxUl%mFJ4FAU}Z~MiQ0{)4aYC5lFhA#B>zU$UWVUuT_ zUNXh@cI!!NJX3L1X{(o3N9XS7II?X;e*)j7I8CQoc7D{GGNUCY=rBE7Jbt+(nEk?t zGge!430vF6S<>n8vTkwLC_OMCQr)lUTJ9;jPbsOZu@N|AA(9J09_Y{iaVdXg5*|qs zq_hzU3;BJ+GI2 z#Av~|Bu8$jY`?HF%aNJbByzJ+(QoTm{zSOy~ufrb{q=bA8KW9I z8f?XuVbfY3gNR>h_Ls5NB%o!ux-(TiEM3Q41zLp>NRyxl ztV-)VPJz4 z^Lnw!B3-yChiwff<&IIRhgh5{dRN2D@eyvP0I;=KdL3NqAZ?~p*x|?Vy+0wQvr~1W z+KV)av9#%N{(eb|%I}Jpunm0|0LeV+-{UXhukigp*pND&|H_8E%aT%FXYQ@4G~5-~ ze*jYpd5$^c@j;_ACplHjUS%-ViL-Fs=eq`lsdmfMx7gC@^3lYe-Xyt%e*{8@$g21k zyOyN`DcP#?V<1;}k|e)Z5Zeet&qMv#CtIQhxhk|XM#M&r4h+|BZGb_XI)hbp zK_tLXLd}BIZ<@$h-_?ix%q{V7)qeJ@t3bBop>Dqr(s)UEuQLTLV*_(}amJ6(Xs90u zc4W5mqtt+<*Vjt}_T?LN&s0uRri6DBX0A43@Yt(hL%`JkSY+ zzlQUccBF-u%F8yya{g2a6N0V(h*TX}$;}u2Rm%RC&>P)|jKSK+jxMz9d_3+~ys?tQuYuu%W;%tc}77<69Q;WOUm`KO0;c_b> zGO!%5U(@l9q_5$K=A0*iRF#i89XOr#;SYK?A8tn+Q?_ak zav1x3ofW9wB_z1AvavF|Svz|H|25)s7- zeNftw0;veL^7zm@Irc}?U$y7wUPaE%8hLI8O13@vE7h#55J&Cz3%zG7ZXaMD{r**X zTz&el%A<#AQ-P=Le_|h(e@T!0X#B*Ng(cy3_Pg`NcB9=(JT7J{@UZzY=$HX54PFg?BP#yBxdr3i^KQh{I zLpnDXa}K=*3el#f8_7zr4dJ z@+v%D4>s4u`}vTN{4u;?Fi_MJ4Ypb@iH&aE)KwUw!L~?c{}>4l}=I@`ctF zYr;=t>%SGE>i09TlAXsEYcTzPs&WEE$-Im~+qSo;b-5hdgmP9JeiJH{i)`q=8h z^d&X%=a|sCD9lfKx^J?(M4+%p+fU3Jg21OgzL4#9hedaqh5jb`?$81j^*i1@jVHE?Mw0cpuP0eM@b0i z`RE5UzXbEnRJEa!_v>H;W;{DbiWS1*izdc;O1@rd+=kxFYx1Ig|3!602S5#lJHdg&e1W@1U28B8~Ra#e_=s+>v^I9 z-W8^iSK@s^l-=eq^z#q1`%lIF{Q@SYbs7cn3K6px-`$)(sw zI}M1aT1KOjWU)Zw?R7YufXZ2cloyyqYu6RPE&kR5I}$H)ivYFGjT>$TMmouB631?4 zO1fsD$9EoX<}r+VsEddQI|%%Nf1lhT@vZR}{4r4NANZrS&fBPZ5R_Zx^Cx#vE5WZ| zpq&V|`SRVWKEFnXj^mkhhqjQ-5<=ctuAA!HE&?jfeTdlBgi&sl?^w0RL)R3Sv`}Rm zLK`TXrc8h1*;s=_UZ(G(__tC|hL*59T@yM93A-E_ePycoe3^?KDh3?hIsE>qVK&VhkKQN_P4w zs^a{hy92pfXYhWF=78eQAETwL6U&$Ch!klCEBg+5e6!VZj9vINtk5DYk`W zso4xs0yFo%uU&fK_3e*;#UP*9Hsx5YZ9K*`tVOspR4J{+z8YF;_C?z4#bKOQwm!v% z&?r^gnROvaqo{M`Fo-_Pr>Q}1QhfdZ)oEDwLb4w#6RmpWwrQa7yb}xMyJux#TUU#Y zHr<;G>6>vPFGQt2Hk%m~bUvT&eXi$(P>oi6{8=|f2Zfshvkor}H@I%N-QNPap9}cx zCZ_v9`GL|~tNF1kSHK>As+-B@A$O8ko+Cp&yGU=`x*rsxK-~3%S~YU>_|%67lZE_ z{TqdyN5=9XZ!G^=G?K4y15byA`^~z^U*Do9bBP(cSL42lW5h-O;yLaMZVK={w7Qp% z6r*eE!anXIB=aaEy!j)zGHJ3+)BSU=8t&Q_d;&b=A zmptl2*Y9Tb9=?g6XRQ_37W-I3*7O62v4Do!rzV_X!YyJzh%g($11Z1VBU1Yvad3JY z1G+dZJo>oj(-d68SAa5^r(t@x-Re1!$lX_*G+3qU2*eBL+9`E1Y)CFq-ExjG6-g*B^Rq1JWUJiTVo`az>{h5S3&jp)4$-2 z^mKWAuS%x5Ppf5Am4hW1*;H&-23a$ht!_BTrKM_Fx+x4ZS#g&SiU2eu6GO#o_8}D>OEPA>FV5K1To-r#c#So~?e>!n^@eMdw5))C z(m&w8?i?^Dq~3NChqB=2AJlffzU%uy`u$OWzf1ksc1nu_YUDr1|J~NXZ!}~*?YQf$ zY9=I2yMLVBpV8k(Yr#O)tkh1s*ZoY;hPy6 zw==p_ZtG_cl3v|4(|x5IiP(>JQ?&XWOwvie2oe706zJ&2k8Au})95!Xi?mb!`U0=R zO#jwpTEz76-;H}xzA5}$Tk7UfE?`{$IV?bTbbcq@zkd7w-;cmuUpq)zWFn+{kI5*X z-;p2eO%m_brYjJhl=?kh9Sf7Qv$MRrrEYrLv!o32JjmM$j;izU;?joE90@a7i0+l; zj9TEy9V~rAdz+?`h)84+G;;N=xs3<=@}U)X-VPNLDpB2N_CTo+&D8=dbzknG7qia| zJbXB2;^}Oizc!2zhF|Q{*i7Qt+hvZ?cYujq*M>RgxKlI8%PhtCj)Y`cLOmI<-jQug z%G`mFR?U1@1~wxFh!lN`^`U4I=1YlamF{)$#}kG0Yk`}i)l<=s^^;Q*%9w5G2!YQ~ z)Dv-ZS3Q(^4|cwmoic?^bmJ5}M_=QMX7ZKUrMH_X9~GlGtdextUDzl_>~)lUU#}M{ zE*Q4&Ne!eE}4mlQ={WW6G z!&(etj)Shs_y~gS<#n{TWqkOMR>iMU`=RtA%mWm$saox(-$DTu1gR@dn4$@KL`Py|PZ ze1(8eCn|TM7DleW@K3Z}=_9;E{gf&UUREEri{A($f~BBO@UHbX)z- zD@&RzLGwvW+o@>=#%Qibid=q9UKzo)gG0H;xNSLR;=MI%ywUJuhIt?e#|FDa{RaEQn&y-BdCc z*}@m&Z(u^_H8TTk$npyTVf^DxPuM?}{iARKT{jeCv&-hIYzBo<1cy(WARvCWZhPDK4c>MA~`wLd_dPN0zmgo;Pd$c0ZL#22eg^^m@1V2!#0_6T!E)CC_+a4i74 zBrVsw!1!q%qP6oH!j!<2y$ouy%hDDDv{U}$Z$|vH7thk(RiA`>?Ag)MJ1S@ZGXGxhL{8Gxem?8pBRDn4iOWY>k7yZLS59Y8J z<>N+U^SvO*XJ!zQ*eE0hQij>APknh7!MWh>IlGG{l-YA#ZRI-sD)@6~sEH76H*>nK z(`h~I9AZni0%PnBOM@+SB1;lAZku`qN=NIE2h(l|+^xC+j)jjv&pgsUX4nZ>CVgRS zriJY-Gse5DU|(sGaydHQ^y91&To!HJnH+zw1olD5rdjfucF=K#>WbVdUv0leuQPLT zXVkqZ)xL%ql$P|nYv`HSSC&_JDD|@-Oy@1YK_gn*&u#3VgdPmCHbS`M$Y^hrXyxkWj0LT@PkGOr z+JHSUSS~JeU0c|wxYVybY(B3XUI86oEX|2%xI){($`XZWI&nKX2w!}v6sps9P1)y+ z`$_^Y#CKjKngf?=u4e02$wJadqV@vg8Csm>~5Y>>@F z!=Z>dzRFy?>SJG@;y{sL=ecsl;tuVBQ5_EOX`8Oi(sJg4m`a|xm+|q?Hm^>z`8D{* zkh>7-a?{q|6k40K9-t%gjvtq8X(bez?xzj*C8tgwM-^;#r!bu4N?(<)Y-fBpsrp1Q z1CD+nqs7)S&a<$y}U)CuCKZF_(Y|ltI=g3e8pHeK@YD1FF z_mvyn8CcwnOyJ%XKVM+9Tm7zyB`E*c(>Jf2XGw|=j!A`YOyqn6L#j8XvXIfwUa;vi z^cd)CZ}&S}mgnS5liufK@vLiuYXtFV_$}?!k~;1CQ{vK9E$bnqQHg-y6xDq`4MQ}gE}=Z=STuO<6ncx$pB!0u4kPv3-IifrcGlG;r;GHpG>@QC$w|1~M2kOY77=$?IYfJ?pn2h>37{ z!(`oXbtCqPun&=~)NDu`15b+oJ(MRn)xb#g5Mx{HywxU zHaX=cYEt$o*knD;N@OSp#6gflmVo}@ef?sycNz-mIGN{sa|plM-c)518sWy=wmh@% z`UU)i7EXR5d?QgD!n{-Ms|^zgp_SEnkefHtT_z&yfj!xJ=7~;$VVw`mqgCl|i&!^} z6!a~ksGPGen3E6z+iF#J#>igMC$p*dSENn9IzYEK;TK|BYah#l79y*`+SUaS_4O5) z8&~tg$Sq!5|KnCX&{BOZHNG_c7Y4K74Hp_0*>I$_T^+QS_lXY&invO+-FJDtd4+bP zF#y^5@`*)K0Iv&VpM$W!u(r?`T(}fx8ok0nm1CE^+bMue3?KmAt!g}cw|{qAuyj?V z;%{if1bel6zb>vIy1t+ozOe1)9#P0WAAsGk_tUJ;vY(B<^iX(hwaWjf#bh^-&ZlX$D1l& z-XtMHSlmS2lGM)pLC%}sL~?GBYwgl3sx%|q#7=bG04LmuDFxf+L*OLwlN+`Y$1M$t zw|KpQ!RAHTxh+xscSfSqj=x&3q10Yne>>}@qP?g~V4U|}X`HT55J#_vB-*D(FnZkY z4t2E?-)$h)!P_k7zfDXodD%z) zQ_?MSZZ31WeoW7|N&=iz!bi0~(riQ$_xD<|HbI>ijvWiyicdQ(Btm(%L+G5d@thR2 z60OBeVJ=o|Zj^4}gtm4er{gD8fR&`gN$^G9(iP3V(6MT}dvlvDe7tiG%8j?CGHd$o zj@&h_Pn<4U%_L)47#AA_4Uqm-Cn@9I=(_+cwmEv}a_#&a(&~0N2 z(#fDa$NYD*Db7!mG3vD?lvGlY^cJ6co?{zW@H+3jWcAmMfq7lyiS9JGNOD_Y!Jj8Q zs-Q=(W`kKz1f7P`9dN;>K}LSmVRx>esqlKLmEE4YAX8K zRdh&zm4TQKTAV)KHT3=H$~!O|t5tQk>y{TO=k|0Yp53(RmVoRdN_#uzN4@mWvV>!1 z1oz;D`C3`l;1j~P2<-5+CG(3#`{ z=Vw@6T55>pXvn73b1eu40WgL?5A~zGT?!@)n=EBEfgWTKzE_RUKEgXQ_pyOVa3i${@{^HKgO9aE!8)qd>;Ov;($lt8SsFLrfo?q%Zp z7}r6Y4Rq%^D8(5M%Nod?qk1;F-g$&3&w*-x0HvFz3K^5#rpY?;cF@*us3Gz$pU?Nb6d)B5os~BU^V*7#O?ny|Y0Y?wM zD6PWlh=5|*Jq7}|hdTW6@t8w$Jthp$gz~Iur(AvWZjcm6Xe8@zi9jrE{Lo;@HCzt{A zqj8};zvW2^A_KRTb(<+-yG!tkL`&u42i8~n-srZgKKvB4l;7{Zbj69&^PI?Wi;~H9 zz7ej6SJ9kYkhT4!*dxq|1vc1dC|AGb#er}6v-d2GD|lS+e$m8a0T)>dFyc%1miZld%SNjp&82p=GQSaGWlA>`A)?9_PLN3fS zh)tMtAudWpN5Cq(L|K=RRA~wG$;D3XB%7ob3dKUdlftvh+&`;AfV zB512Lu2>6E3Dz)}?H2niQJREbjD}EksQbz|Hi`RGRrm`hZKSG8;7w(Y04IaDgm5w` zDnZaCy9^KBkFJBOXT**zF@naLpugvF+zC(@H$n_`*JNTkk|6@e5#GD7W9j?M7E0yP zh|{ZTQuo7!O4UyKNY zt=7HyB7zwxTrhb*9Xop~B|+s2$X&yHqxaN&tKqkVww2=QT;gVDpgmVx*xpjOR#uly zR6HcdO*q)p7I@ejImc<62wmeKk@*T-(64qvHDzk1M1+#Vg2CYvHtHj}y{G%8##Vl6 z`No2E8oU=fql>2F!wmY51L=U@+Bk0;pG z^0=U9bUYsu2+Q65^gAIH<~n<*q){Cg?m5p*o?(QV~mK!MeqtuL@8K-7u^cFP}n>JO^kIKhQl#KVUDNP z;-~hotptNv*iQ&%r@gp6xNvbx9M=**wpfaS+1kANXhme;*~;hM0-ja&|B?=ODzFX& z_oC}~@escoA;pro#ZHR1^~jZa+&Txk&L0Z)I}fP>lbHxAY;do`FWK*e41!8`6}Evc z5WX*cYhyM|y(x#ebia-RQjp3vlGL|;qi_QdZk~JREKpM$Ak~qF{(u^P@WwiD)C6AV$i?z<*cX;Q zNO8nsgQVwziJxbYaOLgk;~bFeMATZy$=gnxp-a-;JXzrFr>uEhWiwW02Clw%<)K}( zTfi)912NMSQZ>v>^rSlklzojxE9=riE=&%}ubf_L@5(}+$3PVrv}W&y>ile2~FbTH&(45x}L^MkVp2;wEFV=uU?;q^|2%tgz#0-+k-1`)d}UCua%)X|eaR7nA>bFA zNvQbflY2UT*@Ix(PX;&I3QN}SI7`E9YHDhz^7`SbQ{Mg<<$B%bXj1&0$49E3X!!fN z(Ujvwj2Pmu>yCb{MIr<{HYP3GwGpcz4sd+Gy4aa;QOWh-S@6n^f7)zOfUcc$&6#YX zWvbabY`2rY-(NHLoBlZyonvy3w`Tznz#CV!)uKsBoanQfF;SwE?#$yU9T>^@Fw{F< z;YaTT-y^K7!-wx>V7%W4tmJR3bL zs7S?mM=a_o$*HdbLBwKlU8PGZPaa)ax4~D&KXt!ghfPgi1%toFsn zW0IiLkrX?*pPL$~^ z*Iy@IAo^cl@!!eKMFyyjdPXWA$26%;<{yopE~d4-zP{dYd4(UR77pGot)ncH?dt*v zHRQQURCE^9RHV>?lC}>8fQp*$*M}`eKLc9^IMh57=1k$NboG z=`H=<-4DBll=?io=*nshi|k!qyHbtc8FSkJZjY4+bMeAMVIPTU#6hFEyPKyyPm2h0 zV+}VH;hyCa6)cl~$5OR(YK7dp%SosX5+n5%{!B%Hz*(=D)R$NgkSamC2TYbZpS~Dd z8t8|AyTzKvSNED?WY*jRpv-jos zJADxjFRrop9o8N+ z0mMNFh0lu|&jW$`+-H~7%#^LdgM4YDyuQ>qXy+H&E{ppaqAU;P4a#t>*<)>368;m3 zx&ME#_nuKrWl{UMGmbD;Y^Vr=iWKPrg4B!!ks5l3fb=d3NKI5!q)YEa={?eW5a~5i zLJJV70RmzI0n&e$nQ>;;`|baJ`>*xhFRaCad+*t2-+lJ8pJ$)bz|SOOWLIn-yZc9; z^S@2*ON2%2cgHYM*^ep6fS5i<6Qod%RRNZ;EUQ?d zyqF%%L`(}}sT(8>t`P@@r#=O}`8-In&zNbr0b-bWC^55WG`PyL8l9^o#Cfx=7I|dN zaa)pMrVM3rlIFI0>5yOnqoE{nFB;T*jac=gex86d49@W{SKh0OSs>qxlQT}lvcbyN z%l&UxH^W3rTc{^D7RkVfVN5yS<8_o(xS#T zceA(qDXBl*z*?&ON4O+n>Dy|_lP_)kTRp9)=nSqPJ|4xdA@@e_wpd3*)V0R3@@UpI z{N56m1EW+mGk0Yx?s9CJ`)?;{Gto2S+HK!@sa4$ zx5NhuX)2jOMK*732bPq>C(PFDbNdF{|(y zR4T~7f@^a^y_|x;qzSP$T+p;5|7hRp-Nqq$(?OcMwzVYpI4HmP9%D$+Qcwd*B(c@a zFdVKp68>4_hZn&_B(i^R8Bd(wnC+pQVp!XU zxDsjYJUu8F!+3s?!AZ36MDIBCU0qAInWpY(nZ+wlPOl9D6r*C*$?%YrSP9&3T71@HZyt(czJnmFCyEP@wwvxTVxtaBBmpAPYw_E zT@T`+dEWU)DAzFn%lj;F(6&%#gAC+hhj9jt{0lymV=-_AE*0X(>0({UqHd?%2bk5wx_)gQj0bA+%KeZe4&+D+@zbuesh7wyHJyMzUM>fj zrfx^oetze9E*rm{qhda0CwgGI!5H1156`&$|FV`3`F`K-HLr1!s1V`}*>q)rwzSEU z9rlWe(}-s!A)tXllJgG1yjx)Yfa31MRoBys3sB5Y&D!Gjvk>;cmMTi`YzOgwft#%Z~u5|e`_$Url#}R+hF?5@o79NG8O!C~CXyfGFpJ_-ncKg} z4#rTbZUWA>@w&>VA_^x&K3Xg8PXRu1iOU^K^=O%^iw zg*Cndu~e3WSc1=KbEc3Veh>bU$i>H>(;OGFs=dkqp;4bN*(bm!uPl0A3F2d$8$aBO z=!G!?Q$h_}{>)Ao#zY1cW`>zvvk5{b)-l&g9t52-2%0-p`>>vWkv++{BBu3B$1^je ze3lKRlaO)3W!>;}{+nbwQy4LV|5~sxh|%2{Qu<~B-kV8;^oOu5*q_eYY6%qfIQp0q z<+Fra<9y1Lnx(G=G<_G;2O8uir)dI!)N*g*40EaY&bA91TW~#u5my6EOKQ1hdC?E)NjL(7+=)V zVNwu7>H6$SoB}~UebMTr06NOQF5g~uxF9dFGpwP6KF;i7x&B3=n)IS8y$%VaAIz8_ zH*PlWAU;p5dihfc4w~P#t3UH@!Y=qAQ(w}kYU#+3C0TX~4K7W>;`T!B+QlZrgT?)m z2^X=kSeED4@xO2=IXxy4k*PD=w5F$(UmnD(j2QX#EJno34)iHuZ!Db4+}dWtnyi(t zTM&%1KrqYBKdng-+e+?V;5g}`PP1S*I(!>1XOsd&;%l5NRW6Kn-U>EGo@08bbG*g{ z3Y*+^Vy`7wfT8)MU044no|z@)4=~6g`JFc7r(0J0U(X$tan_|n8k9{Y>YVb9WGbif zWOf@aZ}JfD>~}Ly+7=2R=yZpEDZ^jrhVv5&_+5_(hD+u#PsmN5{WdB@Tya;O+&gCV z*ln}~Nnk?_qQqNynb5l+^G3!`NzjWcm2%$SO{yaLZ2#CI52Ii17ApG@>2 zZvc5m&VJ9_MeNZL<%tl3GnwBR2N|>k>fFv}6Tc1XKLm)x+y3Rz^pf~V*RBIXpUQ(Q zSB5FmQ9NbJDO^WcFy2T52FL>Ag#dB~q&Uw|hUX6s`G+Onm8`G87D02r+sgOGhvOWD z7#b){LS{H!Pln{k-lqPm6zRIkJgGaO%`Nj(-+mo7HgRR%2{4xi)S!i4#_<_RLNBzT z6ulePh<@AV4LM55%~vO^G0YIJyJ)(r})oKKHCEb}rl(*4+5ziApfDI&0Oaa?@?07OB1i};la~j^rOe&TtZC*rA=IkNNc22(6)h}B`35cq zA2%kB)VeMJ8)g1=HH85ZuosV&=vgz&A3ne&r}^LYC#NYgfsC>Bwz=3ZpCyaI)E}&x zw4X5C`nUy9y*Tdq@}Fh$;7;WdM*rWXV<9J8M!xUmVaro-kFG1f%Q-kf^Y7~q(}NmG zpgnZ%sbLX1X6dU}uHG&2sO|-7!BfdgeLc+y28k}4kD}RZw_+Bx9Xhi1ybko!@edvP zG+5sAAeAqq#ccgWO=qBsJ0UHaVk2Tza*a@db!icwePUo@7S;CZsNAEAiETj^;;K`` z!%H@!JXY_G(l;@yY(M7%n8pg>3X)z8)Qu&edx2!O(CD%6RFjJ%2iUxptEuJS;-)SX zdB^Y7i#n4W=A<^>1*jFh3p|g}&NEB4mHM&)mlQnljkvtV)~+V5%X2AjOKozq;i(0J zrRdk4(4Pi6bf|WcJw;GI@o@Kvq20@pm%83Ms`7H2k{`J=P|B}g*8kD!^xdGf+wzjX z@0mzGl)OCD>Nt5Tq)S_W+QF^fcS-$6FBBX;Ws@g_c5!=gS?V3i_olhOWhW&*2V`L< zouGY(_n}ZC{TsGt`M(eS)F}Lk6i4@!8{WNYi{Lw%PWRzAkKP}_he_waXQNA%#Vq1$ zC(f&SMm)VfqHF{TU7US6)@+0hjAk=z6r48++A`2)X&*y#6$>%Dg-|wt2;rl*71tY_ zbe&;%xGU8p1HV$_3l)mu-N$R~VvpV038#+;I-c?|h zINcV@CBGgdu+}BH3ts*x|71b&*|GpQ$be=Tl;4@x_st;jZjfY%o@-!s#Et!dsz!O^ ziHs6;tZS+J*?a#=N|@QccG2;*iQsGG-jc%I$HL)`) zxQjRn_6g^E5&Zp)^p|Vf7n4GU+s_eRxodDq36=^-l?LkUza1l8g-WbqEVuso=; z)eo#iR7<5v9?nhUTTf>4y{V*zJHH*C6GDxS$-5g**A0%I7w^J7g_iTIhfy;21qP4B zAOyT`=o<#cG{i98C$`WmLunVqAeLm?>@%z zi0PG$FZT>w)z}o49pU-OT^%~~Il%HIyM2PoosZU%ONzUld+^?>g8QkarS~f=y=)!= z%^U0I@n&d7A`8$o*#N|U0e(_GP{1g@=??E*YLUH%?PCeSXV{27@B>u*i zp!~9{adjRRr{N}leW@ZVQDKApG+@~JS#{srIr*E)FO)8;b!td-1l_cV0U+sgX zq@!Re86gRMD{$@py7$eFA_t`0`;W~J!R@v98KW|Za{jvZR8gXIR&Vxp)vY5n$u&En zw`2!e|MkWvM=YYkB-6NWc-Aj8mRFl85fmY>ycVp5<4JBcIq2xe!nf~);^CT7-2JI! zuh{h{d|NAf9cTKaEw{t*gnB``pCW8Yw;tRdp1t|(OWEt}u!nT0`TBSc@nX+uxQ?L< zskX1ZlQ5NFPXMRW7C^!-_d+fGr^f4EYcAJVQ}d2s#Mt5RZ1)c>CVQa)=qw5P(izR# zVT`FyUh9V(;pYFu))%*%hZer3qPa>cn`^yo#+Ft*PzH!{+7VwYxKfAoI!%fhPC zO&9knzG(3*)1w_q<^B2HZ})_c@2RCWLT%#^f^>4kzIO=e^2y61>*N6LS05M5y*k{* zxb4KFt*#UbYfb3-ONbAKcNRFg#xO4vJ@gNh6*0r3Pp+#5@uis6XGHFYE!zPsc(!|G zi^vvR^6&hZ9RtsTr#S9GaQRI&*iZE!U|*T?r3a}+PRlGx9?DmcN>Vwagb6XDM2bzl zS?8oeY}+@RJF6C6%YNi(@qA0Ki{JW}r?gBF0cqHXPb-EQ!&f^5Ehr-@^o&`}dp|Lz-U*Y{|Na8g9`o3JERI-N=IMVDe&j&z zqy0A-%vtksH_rKX)@%BhVQSylf8xQ5DB)YW`QzU;nylhoZX@IF0xp7_x@_k_O>!a~ z^U9s_)DR`qh9tb4Pvbh0^Cf_&;+7X~qx$8D`k2==C)Z_e<>RblnfW_t9DgOGOwF>J zH!IUG;7Y+al|9NrzrpapqTGPB+NWyD;P1adqYC`FMT=dZKiQU~@t5kTMQQ4+w=8s#ufg`IY2CE(! zuG2m0h3=(Q2#8h%_b=_Q-}ye{1!NUp+U#1`2WAjbQr%&}-}P#0cMS6{*u!MR(E(V^ zzDLAh2$$+IBO7H1$E{tB3?{T6QL>|LJYZ?RdN2M`E=QoKbFP-(*X+fil$bn~pkkl3 z=d)vz7JkUoZ7EylpbOm~34KI1MP@HG9@S%_dZAC(YQ4&R9kV1QvYqDCU$i+msRMm# zZpLHeB+fBO3ca`!^0LCO?Ck?nU_1W*ICE$HS>e7D3ZiTN$jCwU>#>B%i--yJ)EhZb zc3Od4UdDS*<3@{(l5p{a}Qj)095SEjzGfa6+D6JZvV#MC{7WM0ciW zh+!1VUAMCzhrOH|{s(D?*>`)zb+AMckJUkrkx@xp-uKD9?#{s`A6x9+BS4xT=Rmtk zYn}YPle_=Pdvs)sAFFVbUNs{>o?5^1Z#+D7$oO-ji7pGr*!qa)MDAsCVtz41DzI9_ zvgK#)dho~Jj#;auTAnH6ihqm#2aW^3|9r}ij$l#{Owfv>g*`J^jKSPu#xWfc*J1RD zl^TKxZT=5#1}v1wlokK?qdauT!m#vz^g#SbBmcwR96Ygp|4IeqvG*0x4qh8931`#F5~xAHhGLrEZTG=!@)L3AHblU20?)^C=&<5^VbD zZDBk!^i`2n2WnWfNBZNN%DFwAQwbAK?ITe6MExa_G5o%qX@>B}~%MK`j+8djb2{s^v_n0B{hAZ>c~?X|Y~e<@w(ZSx2m*H&Zx6 zenX=E`X9jm)qW-$@pId)Ywz}3p4_CkSW+yFg(NQ^(5HiQe9Jb~QaTblTuA7&>KL}p zq6e6(k)3_;zr~ify&xFbJsEFgdpbiENqb#PLV8=WYawcWlG8GjLO1uW;`-&O`7N?W})}e;yeu7d#nj)5rO0pQm^qd@cf3+-J>CkY*^w zwo71bpnG|`5Y**%4OC0B+?xU)igO_-Fu7NN zN}6a?)B>Xs$qpjdA?6>glcc>3o&h(2A0MbAmWt?o;S=HIYd$(t{D2cS>C>! z>KyDnzgZ%tD#RRJ#Or^hqr=t_ddh7=pp|Ax7`)dC#H#|~DO=1*SfSbtQ9MVPtOU5b zuopGZtE? z*IL`f&sP}v@N!*EKlP$;QI@I)dGh1p;xjqW1khr!ESyzk+GhP_DdtDoP~e})UmD=* zU8L`@Lzv&VhImTMlJBcBzx6xM-Hm$9sW%a#-uC%EI6w&;1S|b=yJ|1Voy-!Fs^w+) zFenTrSrP7NOxaBHzt-!l&Dt*Xy`maZfr}PamGZ1ohZg|d6`?AmGFZxZ z{l-!e&|d@Le}ltTDzmy#LSJEw)6fFFrG-8FCPMb~)>-XM-eys!n6yEXtj<@GfpBe5 z$@=B##C(NkL5a-p9K55s9;huls0s(Pl%PrGK8&XdoA+Jo(4GbndvJaMcDLN%wMwnu z9%a5-M>MK#6M8e@kyytB3oM&>BusgnP@zNH>&+mw-oXCTk%6zG1A15S2Qi$ht$z5A zZ=R&*RUrL+9TfOU({{MLwCWJhW4yvW^FJ$Tq#Kv2iUz=DJ?&mstAX`Z!b5Vi@cs&*&h@2QStagnUxp-iH96v zR3$WZK6_%}H59lYb9sd0g#8_^LciM%-xX^`A&E%W@B2J~Bd)1C^7#~{syDi2{e&9~ zz=!r(L&2QihIg)v2xY407n)e^y{dG5bL864#v_}R9rF6K70RcjAYrM)_GFWBcr!t~ z;}OfrbDnkfj|&B!K0v&Qc|a0TWG0cOdgegqXOMcOo^hJdM9J`on7n&<)G>nYMAj5u z_s6bAU(A(syr8xp*n_erIfZ0$Yy- zGKtKibP@t@V%)63v`KRi;9ZM1z$Csl6@YP)gR$h%Gao2 zD*lEW@)jzcFy#E>B4z9@r>hW8{9=hM+d#$W_Y1X;iQ-W3%diL+wZmm@Q`9bO<6(OK>uMzO7O$QrSY@Av&cbXIj)y+O)|HaU8hiATN!QYBF zeB!P5#1~lY+%NARB`}??tITApT{P1rAAJyFc@}feM)D73sjoHRCURnr&5`kbLmpdC zUIMikPA56|z}T_87Lb@LrI{q>8f;>OedSwwN^U*Ioz|aUNW+z(YDphgck|>zN7W>u zTL+Z6N~Zv#63uk?jHx}=k|osHRV7ptU{0ZLkVa)dQ={UO(3@#5Bu72ZW0Bc3gI8tY zUu^dbOQ-b^gJ22Mn;;G(YRS-oUJ(Hndjo@4Bba%Vf88ZQSVGs~#)>ENDZT6E>MgYO ztF|eVu7JeU!_S@TF_dS4&QD;!-vSia*}yFB#(>gJXW`xFLNR)N9?jF}hH)SANfe*(wxMRlO^ zJC)WotT9AAJF_nra19sm1fAO2B&EK!XW=z_B&N$j;Q8b`BFu8@wjqDJqYy>CGd9HR zjc$^!{Uk|;xE0vDC(*+tQ>-Ny*YGt=dvx|_nzB31~;^*6%Zt{ksLliY*c z*e#IiYS_A#lMa-2!&Bv*gCzd-#Lw;lTN(UrS3m|=j2e9&JFRXEy%#t#J0#g6? zi+QV*<_3y`F|4u+HvHKzVYl+UA&dxomNIB$wRKb8ypFuXp{fOLjVWB;`B`pW`w}K* zjvMe$#x*m-7tm(UlY|UpMd=B*ZQFF^nWQeanks7ypS%91@ZC=5Nx72(9s~6WG`mm3 zyY)Ae$?V&~lAd&w%hMK(Fy=B8^i#Px&bRiLxFfKLxBwxr_f3jf45^sD{iDSn$1isR0=Bn{*5EcsT(+I)xWpJsgvAcgfeL3QJ z<(ji(I55`l7QT5Hgusw~foEO3IZ|`z);k~|$ulgC*}e>TUeqnj#y`tO@}Zt@l0@&* zE23a#SIWoNpK|3|7C3R%hu=}{efmzgnPD>S=z63MVbC63oqcz7(Z{_lp{%)-eTG}s zm?^q=ZpkrC8z(@E)aV0yT(p24H}Ynl){6XUKCOv)Y*MIlG)`v{~28U=2cRq`xl4FB!oJa z_~n<>@bo+2%(;!Z+j&ET`5TN`=V_<&9iYT&Ld(K=OZXy5)E{XpwOTD>dJwYo*Pqm_ z^^qGf;b+UkB_OB5cJ=(|6{V)<*Y&QCSICAbL zyXs7Sc<<7rrE{9vdD|^Jeh6m|_{$N}vvc$i!=T6NKmK$wkC1Eco<)F`bR<3J00mKNtCbxOZT=< zi(5l~zx7?Rj2;nC>JC2;6jfgH*Z#QKAB5BG*5;aBYK@iYFZi~rjC%#NjWOTzDxX_v z35&F^s@;nB$G5&A@0U!Tr~QM`thfH!vO=o-m6^F>Fzyr9JopdM7`26DWEnNp!!`ubwJPc5*{GwX7R&31PS~ssiEz#S; zD|&|{9;xn6EezTGQ;Ht+5C84Pa?ZXM{o{d|5~Ry^+d>~4gb*mxm}TBu{y6dY_^GA; z@#p7pe-H2H+aCOUVqfvU*(SxjoPQGvIY`>0Z!G*A2ynaD16=`P0A&o6o;Pj1GQ<7F zBzMh5E=3MVDWC%&jo)`nrYjex^0B@(wE|kIfC)!9(k{OADXi%nO?2H*E4lT4=Yqij z3){UFrg22G;$DzsCr`hZQuty(g{8Ws6}quHMo3b5t*17qMF>WRc*1^C`2g$Q#%^D% zxApuSY)REhJ}Jx33bJmveO5>Q{#w0QyxoQC%E_VlGkwKw6MJ0bDa5sm zKGGncv3_g}{I_NPD6(t`lLVkMvujHOu^e8)`VMWI*kq#`(zSF))YHw%azI82D8?=X z{AO)_{ww6?;-q6_{6vMrRxwG6Q5zFgtX|{ttfay{;85qyPfAafe*Gl6iun5phZMg3 zH5uEx#p8KcAN#4t1Y5-I=f-X|<4KG6RfUE*yKMl2ii>ih^m$AypMh`;!f?k>-|hA-}QD4p@) zUdNdXtn_1KS{`%2UCRE1pgY6ltitU~B(If~K=zQS(oyWk7{t)?yPbWP;mxFLcNlI~ zR$O5PY%;sK=t~_#=IGJgiaQg7`kOR^%QsoEn$Qhna1N2mN`~PN6MzSAK2diAg7bB!&MZq+2be#ND)OX^~5v4UD?$fRx~kH`T%V zE6hPo%|S)>3uqP0n&HEs>VeI3KaLwYlfHQ`yzcZ$AK9F+U-r53s_jjFeZ^h9GCi+c ze?=ILMw&k$)dM92NB<(#OX|dgK18koE9eTOTRCek!*lPE9_pK~f}-QLv?A2FIbnN{ zohKsN{ao~+Lt3ZR9^5scNB~EPaxdq>F2UDgcOD0c;cyf{{M~#5rPz6yIEr-7Dlc4J zbd@1gM8k@vJq>G$%B5#hDMfRpUf$5@H4;ciGjPTsr>$t8!isx>bT51==e@&*%qW*H zbQxxvbi-^(0_mTEO?OcCrJku&)WRZ%EH|V%GsTA{`%x2{TsoQTIZvpI1k0`8pWE8@ zC18P*@%}cU`KpU%0>{bR`NoBH)&6+v$p={Um6m+&-X@}+nQRjH(Pwe3q z1n>;As5rYva0O*sIha?{9ob;yVIO1>{8Gl2cPc{#m#&~}if>H}T^Jmjt4ofM(A=YGq5%9+xO>{KP=* z2-Fg=PV|b{3YeH{aQ`&SLnNzK1QDYm+`q@?>fc}YVILQbTAm0E!s0d;TVwzIa=#9F zwB+WM)k_|mxYwHSGDDNh(ek@-XyKi+k$gsM10(c_<%d6#cY_m+n1N#A3$O>+Lrl|J zP_63o9-|!@&MfZp@08NoPUF=1pMRr(&xKsQ*lWnsWm{oZCvZQ2x*xgZ?1%UM#IBc8 zD}JCZs7W{~@^+IvonKeR=G@Kmw(cIHQ=SNjCxyb5k)oJ=%mi2=o@P>mEgoBUH^hl< z>pu!?{uNL~0I@K(_tLiAui@hOl#tXKP(>8;As)={OMhmHHy>w68Q3YhFE=S)C=&r< zR>e*1<3&#%_*p;%g2jnqCRqOK19G|!ST}*E$hFiP$0wH^8zURJHBJ~&F9j|{3C~;Z z$Ah5e+TR36mOQ3(ypbA%&E0R^$w@WJBDc@Mft=36^6ZJXPQ_3IF0XXUHStNArICfI z56Tv1G}#Pz<+v<{@@b3(gFs3lH+}Nfijhxr$H(8p^Nj95<)ZjY&fN+1@$vW*@P1;f zZG?-E&l_UFdNv)hE%wIJPvvTEHBG4{H(l=pQ(5#g}8{e(Q_ z4r+-A7Wb?U@rwGNudjuqH|={X9;8{|c3ii*l$`1Yp6BXycNAfa#ztmQBz-;lV;efF zMM?~F!-!j&<1Zix=b%##--|8FMMyTH1db>>02i`8yzog0o4pWUJFA}St8nS;bYb?k zZ6HnMFjIHK*ghRzGlt^hC-+W5bJ&m2Mx##{w#1mLMgl|0?w4)-nBgby`7f+3R~=_` z$N<0`UXw5p$7ook7*%Iu7R~JHm6KGa*4xeco<|WN8bM+e%9_*?H!NdSkkwB}<|0 z99gYPz#J-O?N6&#&NVAZuk2fRwLjT~ZG@VIN5d=V56@lPNjf`PT`W45Lut-_g?gSy z6kP~7FY8PN^aKK~Tt-?&OlNwG5AAq4Xm{Cu3UqN6$sgW;#CgUX3C8vW23ObSVUtCs zl0E149U}{Tf{qQvl&QfIH>otCiKcu1&i=i>IS<6!1ExcE64zQ|Cr5E<@nj+E9|$xf z7%vXAgBjG2Myp*Vy6DshX1PF!1T}uORDnfLTiy8-+}2=TRMV-!AsT}NQDz!0J=oIf zkD*vX=pMsx;DjovoI>uDf}=Od6$rL`3#smD$ri()5xj}rpL$T}LT+;qB;&~qexPFL zz7SuaEF#bFE3Lb#gMgf`>#YbTgtUo}7OT#2oKSTgMBQ*m?N`d2C0yLOCgX2|Zp4x%CSx z1>4oaP5~YL*L7PZc%eafhLmxjwB%B^RrEg2@PXoOQd87es0baMHmzajOsg7aE!lUw z3NjLDMIlCz2|jzczn890gTB7_yx|Ym%CdOMt555KX;#AadJkrF{R6UaXCZ4*u_lZ~3lOKZr5RLijbqjZfBPN(ICWqtkmUtM_ONhTl z%G-MaeR35REcnhaNb^d%+=L+WIBRfLQ6(DZmQ^hKZYvM&IZQ|eK~s{uic-;DXDq=a zD#>yx+J8o(wcjR=XIrrkj7#-ze9T#;fE0s(^9`>>Qg66lxY*Gx&#-rC<#)d>EYBl4 zA@;EUd;qc}!SmKcjERagCS&&5=DcuY(U0a=<#7_9Gu&>=U`e^!x{PM{e&+D0YyM0TR(NU3d2GH^iE zl2KwntFT9>qT#CWA#2Wmp7HtmfkdS0tpK21ljKgvr3EYmDmj!rD25e;MNQD*tAV%pZqD)G6DR8{e&up=J`flMQEj4xXdgWdURc9ZBsp1@TKq+ z+^xy7j9KIB7qvNouKUBzhDy(M!#e)FcMtxUc0$pC5_`fXrJCk#y7n>&DKYelMzRsS zGQnOaC5=LqufLSyIE1*vdhrAcrmMOT3A5h(4oH1yVRrqzK(Gx z`)q(3N6d2RkA_=@MMlq^a~s`oO|FD)?Fg-3cY^7jK&_qvIV7ln{HI9vf>>W$SGU(k zYi{{dnAMwf!@0}K)x$8=rA)LtG}<6rHoAKOuADF=;r>=jYuZYmt5x88*dwY5os0Up zdePCaY%dRA9I|Fx;Fhd&047&JysnI#J*tv1eWnaz*8pGh#uc0$+=dg?OfZ%l2RH3sgz#K%ps+?z;j+INrUswJ@r9FQDx0iHE|#cSy;2MZK@EJK>6<=y)^TCXi1vJRkbE@~k*RjIiMI z={rDlb`jRa+!9Bn1!4C9$ZaQzX8+q|KE;=R50||7JL5Bg)%0-)2jxJ}e1IQw8?N~% z{eJd@{7&w~cE?`#dKo#{_tHsiK9j3owc{{kNpBK7 zssrb%Ks&K*fcP$ z^4VGbL&4%G91#-kt|)tk`44A%MZtgNCNTry&z);%j{#t!)vM{v54@+DQ#ubFZPCA; z$N2_KT^Ou{vWP6eL$!i?3_-_-TBEe8#Q`kYs}BpeON7Us;10;A-?qQ}J+0a>)yZ~i ze&HqK)%hu&m1F|-emQgb*Prg zf?ox%D_Q->_m9T+_7pAxgn6~`t=pJkTzIYKcu|qR5!^jz8 zzYjHWqV~02&vIP+a5g-?xscDm^L6~kuC_O)|8~X~G~cHxscYZv1^uC6C%w_rZe8s( zWgRPCP!i)Z$dUE>T#%4Q5Gls#GgM@*cSp*&zqRb~kZfzjc2&jxZ{2VY+fTKN6CSLG z%L5Ewb;z*geDu~H{N}d)$B4-gBtwgs6Ezmdl6jYsIl)-l*_Y3h+nrfgOd`y<(W~DR zf^zo7#jqNumz_8IN3Y)J^}ie$AjV@L@?egVdYf^i-x5@(#8jzrgY44H?-Ey3I~6j7 z_0QCCx+9h}5C#+dd@_1)sUX;=D5jlCESciDB35Qsv+F~~vh|N6-4^5mE^BAS;$;Vl zWdnXq6Uds7!Clch#vQ?9_508ATaUYAoz=+5_eWRWxVDOOi(HP$drvhPHF_s;n0vI1 zkZVD%GaMtY*vG=N7xl*oISTp00h|J0wzHkwgC8CayyMpbxWf})s)N_|FnS14Zixx= z{YfBMj~W9XH$^?4x`Nkb$hyY~FxR{nhGsr2I808U-n$12r@7N-c4-#}YlEl8;n}5e za2o+LzK5g2o!%n?d`Vv)u0MN=uW_dd1ly?7dEROuN`XAU@YPnRy(=VElfVT)*89bG z&JJW2ai^5EmG?(|yIzMFS%;yPkNn$nDLma9CP{VhXKd@y^Ma)taxikG8irW4z)PoG z_6tIKzttNJ(UQCK7Y+tS6TdAb_gLELwK<0-i@UI1(VYdErjfeYQ2Uk1TEa{trD1`5 z!L!YBXZV}5bYcB9|8<-mXVz-fE^!tZhx=WsnM=qmOHt5hV&(-cr&;-A19)=ya?vUU zGPG;Yq(3RraDq%N$jv&aqcV>gyW*s}f}HNkzyNO_3cCE4$GZH=t7-jHF3XGin=1M`-?rR3L1I!O;~nTxKa{xY-Vdve!p zv0;-`9FIhT6J}emBvei@bC%gU7miLhOq`Cyge;g1Zq{8`a)0w@WL##|YZXgbLyxF5L=Z9w<%B~OTy1O5y zb4Gxl(43JZLi?UbImj7)TcGio1y5^cq(+Tdd_WUO>XLc9Z>y+}&*#Z0^)l!s5B*Qq z@qQ~F5Py7o>Tl07Gcdws8B%9rn407?FG44fSg@9~?C|-n3%{$iG^@V%2dY$3CzfpU z2;=9V>1het7Rpy%Lcmq}eUa(?0H}t10Jf<$h62@j*>(ncOD$(KHni;qOK{5WD`?>d zzai6SFcWFaz5oU1ysk-q#sV~kKK4JJ3jNBvtjSWdbLHEx5&x`HLMJG(Z8z zHz)~u1NB@m)9OyWG-t9$-OcjUJFz-I8;7s0bJNHPmX8L|=J+=yXYGNOV)kicBt;_b z?pG%RO{%+dU3+NOodmEq4mkv?4Dv}8*H@5yPQkbRsBsIeM&LdRMzFAko_#wITYcm= zx5ZS^;Al#!{xy=#8;WVQCa$TChKpdT09BTaZT&J}Cs$O^n$MTFa+jqaTzNQ5D$kNX zlb_-@WcK_|19nq!c66R}^t_SY0^=@DRM5>CKBOyl#UhJfft(uP8hat)C8=yp$#aq4rA6c**22$h9 z$16&K#YYdY*^1)0H7~(GpXkAodv871O5?HyK88`6<##djQ@FA0u69Q2Blq;-Kk93f zHm|wV@muHinNhkwp;y3p6RY3=~2+UWoxL!9nAsHi$p+Z{>-h&Yu+>d&p-KXCnRGS57q0oAhZahB!h ztuPFSIv@@m9R^$Kg6oD>wjqB~eI0C7M(@C7qFG`C6#`{le>_}S9;_UM{Y4=!(b& za`7Ew%zt#UPbUGLY|h>L7If9U-W&nw4^Bux*aSKRG;p}&0N}kXnl^TwuC1qX7#q?q zj}p693_n~_u*B9Qt^22HA?4!#c^{S+Ed-OHOHCc>e@Sew+n2D7p)~aS@W)9%`?3b$0C;o6eHHa;6ohjw==0w4IpgR5U`|zTq>U>SE}^s`|9B?pEr? zF^QxKm=GJ9Gv8L;0srE-@2TIF6>z`G5@_&8=F#Z*ABzTzme3JX?>!&G&J_TbxhBYE zGJwWb0!?+yzdEL~!6VBB5aK@S^)B}lLYt}2LH=R%a+idg4wyhdwHk!}+1To|(v#XR z8>gfkn~j=?R$-I?-9VP{Ti&*aL~1$nVD4wFX;$mSU|o6|J@RT+aZmx;v~$Ox(>rBA zH5=N+hL~#|pDfum1nF&R7&4rbyo2@WUpLxB*Ujg4*(GrG^{qFNt;bTKrpbh`!e=_5 zZe^(&h6%}7zCk6>xN;m`Usx_yzNM>p(GsS~HL{+uOm;9F+D*(+wmXjk@8WZUp~Q~v z^?))93+!Xbakw)y;Ckr1|J!bCvM*L~PYKOd_Elio0$f%uF1%xGv{_P50S-(Hh;@R< z&|BSMakoIz*)kBy4-Q*t9!!utT_>Y1KGNME(WA7i1Z;ww~No&O~?U$Dcz#et!1;Rjo?RMONPu{|#sz7OdEAsy< z@4Vug%KmomjBOkh6b4YFIMO@Pq{jvbh)4}RBE3s*2~n{D(g{r=zVp+$DiDK&Y#6*J_>YSWE_4{IEs5l|uYyDi8!8do*_ie*( z8zwht)eSbMz%3?F7CLcH+Y;?BPi`YBb7pq;o&vGS

$yf|BZepNS=hOE|q_J&6%_%scYbl z#z7jx?S$8UH0gCqmrKGe>^m^uq4Za|YBATApVm&9i{-=&6%_>)XMIKUAfaz37pfZF zi~t4o&U^a&6kJux8MMcPgETYN7Xl==G0}J|gfJDTF1ROzPxy46H6fjNeST_o^-|r* z?!YlEofW)0-zY3so52^mOEw^!Fg7zNmqt>-d%uYNoHP={JUx(B?5 zof-V*qxr)jv6QUr)h-T}afv5R=GFeamKQRoo+5d_-^o(0l^X*L)>ElGyRPIgTKghb z`>96EUm1NDw{f$Mlf)$^AEKMF{v~ebe)@Lor-t0(aPL;uQRmZ}&3flJpGTDbDA=OD zLpF;dDSU}c=9!&R#0%uoXze;6_%OsxdgS+n-e{bd*_ntxc*u=#2et&)I5|&-Z7Hu(I^g=C@ z<#-1mCG>mVx2CE2waP|M!^8o#{*JWBjoRqsT&NS~yTgLRuR9s(yFX2ehq45!$N57Y z507g3;HardnbKrDyP~)j2$b(F`+H8k{S+I~@Agw{31d&tAyCL~Dn34PeIy+$!<0pyGA`FisYHNw8B6Pze?rn*< zB23iWhtcijZI4(H?1ZW>C85$PN6K*0F_cO;h#;IC`?&pgsDOe*jTED)2r~m~PLHt! z<19fUJw<%DV*5xtt0_32evW~}g-AtCC6J>X@)k>rkgXGn;8=O&kDq4m_@BRkj zMLAB73ZelM76xOFahPJ5V~U5W^h&x*0!&Fa7%ZjzkBEB%Q4j{HfvK2|2tPvXmpVsg zsSm3Ot)v)oslKDBw{x>gW#Y0Rp@GP4$)AtDOHh~+4$5+edBa(N;sTF8Ry?T0=_lOk zkV1G@uzL}G?<7K?KMj0=r%Xu3nEgifX+%BwCxVnVR|Hl)nhS>??-hf#6l@KA#HiCA zxq?|it;1-9@|mC&GlfwnYCCkQeZ2z356&fPfNmFjD2+SGV1(t5n0iVlf>AsAW;>V_ z0b?u+^?-X+h+WZMeL%0I)gz?J!L1~m3;z9vb@oj?o)E!|G@B{J9dkL9bQV6#X-Z`; zWgk8Sry^Jm#H&ZC)ItF2i7Kheuv&1bR&5cg6(q-CL~Ezz)x)(-Mx%$K@aIyzYtNij zkJt@xXBZs=58(Q%jVpv#VrLA$9=xJ=49(592kzV*B(O$eisgK1F{44%f zP6*5u4aD!D9s;_Mz?HotGeWwcMYwpNC|q3D50^Hq87>~&Ll}T}OMpBHg5(#~1(O(- zI=vW1D^?2%%1=@k3B=+N^BOJkzzA$F&MWXwBLDDuGXJnq5Bz~5{)sg2(puz)!3Rl$Ot-)l30laIHUL05AN6@CAE1agtEAFPS zE9xd%5wwv`Dme0Z2YA4nPIy3wIOgw1;VV3d--O5NN60T5 zt1xJ!V|mpK@nv;l1#%P%KkTqU1hiSf58EjaW=H62w{Z1Nmn))U)GMrGX+rh3h80Nx zjx~BTLUBg4<0or}t7aXH{)}sdA<`nM%l2vO!5-B>0h0&WQIb_HjLWv3CPCG&K-MdzjGR-@zLKso=yhpy_1O| zXXE2rE%8H}X+I1X4v5@y-L5Bm9=e>HnXJ-jK%T1j^5w%z8dRM%;`+I!m^=Q?fL}$O z>erz{7OH_NU}7dAKj-i^`sO&D(FXdo7*H!o58|QW%yPSe9ChGLn<)VMy<~^{5p$X< zX}s4v;b^XINkmqlI?DV=ZRS_rdY*boFuQluZJx})Qdig7+;1F9H>&O%2LbfSg{&paT`zcRxmuHI6&ytQW2yiYN<*ZVI^K+h_ z%R1xC!cABalk8RZbj0ENo{pxwqk41D8!K<8#eR~dPyH-WWIWu)z<{NNsshRe9gyeYEk=EVDikT$IVq-ER|YdWT-1& zT|6h22SC^ceX#ol9jH~Qb$unUMp_^+N+_zC%jb-RIE#?TMKd)D0mSi1ItizXtKlf^ zO@C9~O9$q9)G~oYNg|744%crvKjZ2uM5?7=kMNYzKIMoy=<;U)OPII4(F~;~vB-(M zc$k~5DifrUGEDwbqOARhqWJ5_GXplSvU1rDqac^g?4tPW*X^jnj9(u(QRU`JY4<*s z)Z^}zQae%&>S1d;|0yvS*g|uAR#7nhp{#o5TIU&d2|hZ;8&wX65~8&ME**Mo)m7d3 z-U;^Vof{Vb}?C(^V&L8op$()X8Q^Ol)a$g zry=J@r@ZP$k%1mR)l{V8Q! zAK01+Khq_zFLHfSraJh@gyFKHP{FaaY*+ob63@rX*qFm(?0NW7xIuR;;qRTy3ZX|g zTHIcr5ZKxgf)Y`*#u5C3SlA^wU&gZMh{|V3`-LO1s@~bLOdR;pO|dHzs^B#r&|lCf zsB@K<0C*}R^T<-yc6HgzO4#B7<>Z!$SPqqjI?q&E-+c~Ad4Vy{%{8(w)% zxdSr<*jKaE&p8lU^pVTPr~gXM*t(LvX8xp;l&C`jk_pm z4V#QaKxx;MNST;`eo7>oaXj;O7%O5T9Oq+Xm`K}8e6rnR97VU@#BeWt z*=#SI2Q;nn9LK$)SaIoN!0Crlx?kf@(H=e}n9XCQG^J_#$|-vxR^=hFS{cS%mm9*i zv3jw|RFG0$w3LU@q$&f|mchrl)WPvWA7-hk)Kth!UPAI`aq7ZnTeB7i_C>!)R2>hr=r?p}Ypvkcf$Pa(+ZW)Cij;>^PHyIw1p2Wt zr6RZN`V?Mf#B-dFXwC55Dqn^%*Y|0$rgZg^heCDwi`KY9Y6)mQR}G#ADV3W-iJnP4 z0aZUv2$^qA*fz>iBmVepGiH2Gdu((d5vPN|bfU|U$upZSyr7FkTP=jG>Uq}6QGCMOp zPhGLEj%~oz)vekqm|30`mXTX+(6GPSW@pfeeJ9Q)M$~+HOh+@hpFk216`>huV@`Y9aBVWu=K zIa=hxy<@>c7NDp6GASpsS$oC;gkDik2zGc=%9{Odo4@+-J1V595-qb5C1eu{G_Net z%pdAWtcQp>JFE(PAS-zmG!`#ITs>XV;Z=Ml1SFU(m}2d+PmD5pIUH`rWCVtED4M%g zF$`F=e`5KN6M}-MMaOd;;-$HI1LPI0PG$PhhHVH_r4s&Go8c)Y@5WFmqSF^=%?8U2q*~!V&cS`kb~Dc5q^Q9;(Th zMAQju0(JJ1$5fJz=ecI%d>nuCCO*#szT!?E{Zt}Ye&FqFu0xqfHM!hw^O(xOQ_gjw z;i;X7wbfGorc>FMZ&kccx2*zO_l;%?Sn+9!y*wErgov3ZP^^}+&ZnnR#tjNkEBH7t zS?dw&Xrda5NtO+Ixno66r?&1dcP~wAVpLjZ6wIq|a>SFLE7wHQ8Y#YWr){V*P*XYT zwZ~GY41VFe&C&iQeMr|*csfUvjf;(HZ?t31a`W(JCAe)OPUPfAyB_|f#bBI%?0eB+ zy;vvjeUHS{WeK7$51A-16F~_RJ|8s&xmt$?y6rUGIo!n%yWEQ3^ZmBXWt$Y*MEPG*CN02(RgaK%B#%n*@Oj+-HiPYye49=SF9RkK34@F;|L~hLzv2_ z__Ks7Cp{N~@APk-dR%HvMt?I3VsI`<7kK5uEKAj}?4B!0wsXGlz%TC`1yX`4)U4Pq z$>oCfaANdpzA?vi^r>ia?(T6V-`s7kExdl|v9mzJ^?g|XeCi|hkPUB#mH__?_(f3! zMWLy>H7*Vhej0_=<6W=Fr6|qZrn5)gLWDoFQ4~hmfSeo}Wvj2#A!Ui?@%cHgx+#z zx}tq`;6p8aR&BCj3$fZ9@5qTNI#XEL!6YkxOx+YvtWItGU1DN2@`q=yspV3<-R|&R zwyvL--Cl#0V10SjY9AY_$12Fp4w06!Q%Mx6Tdd_KbY@f-PAio=c|vOx|Ka5m_Ye8! zAOq?(Ft8urJy4(6>&7Z+PiBv5iTRmI`^r>{+tt&fwsI7%j?qj}yGFito zqvN&9XS?MsRuafbO6R-7l3Np36*JQqR(I@c8`|tqNO^Xv4KPF}f|U1-fs*!bnK6xo z=lwTVge#eMRNgS7|MF)f_ewlpXn;7U3MeAa`a~;mjmvcqMMMdQcvy$8xQ`3SQZF)v zdBSCi<0-7d$zi>8nStH!zNOtHVc{n*t1P`@gI5-j>a}JAT=S5xztdyLq}m zSH(4Bc$z2-zesilyr8l|V0rqiSy(ox>B2_}DPMaT1KC|>5-fVjDb6`+dTBnnM1*Ej zreQX!J>j`pr%bu1iff(tquWlq2*2sjfEe)zp&Xx zJ9&$xsXQ0)MA(I=?uohicZytT-yL_$6L&r~U%ue#NmI2h7i)@Yj`&JuQ(!ImL~fU7 zlFs$G5|@_M)RVdX^2Xo3EW%0)GbAjVs!?fBv$Ez2|6dy{^ZqTnCR`7<&US_o91$QP zWwmdgI+V){5={BB>gJzWW(fK#1)o}MAu5E2NSDNnJMoXX!Q!f{z(ssf8OxT-CDP}yLZKzh+un*u6+AE*NI=S2G)d6jMqvDh4c-%yjp=9 z`fzP4lV{Q6_A#0}9Y)MWnUk2*emW+}W13HiU*)pyKh6;P21u%8qQy1*ND|l?XuMEn zmJc364SXHQly6;#{Oe$imf#&^Vg}u>^YN z9ky53ua6g{t!S#(slCs)(#|ycn2^CtyWlrf(nRNJoT$gfz?l-6Uj$bwf1c4%Ca(;& zwbCK=pd+J@)=`ZKoQ}6E@>|2BV5j1G)FVT_;Ow@IYHVcgZ0CgCw9t^_+`dWC5UA(6 zlPJeh2=|T*E~mw{hYM$nH-zG5ezoaNpomJvJ#z^8rD0L3#PM~TPDMe2GqE>6Gy0bv z*5`M9=4~eaAF9PDy*-q&n4Jk5~JzppOpi&@nM|s{V)vBVF$W4ZA%#$+k(2ZCA z4m@g;L{@5y6m>x!`6>DqWDTQJf zi{koBAsbRE)9lsG^>5ZJMi~s${2+U$fpIRomlCX5`jpGUR-;w2w7ad>!uBKHyO*)! zMm7HEMNp*}rLuOFH&CU#v6oUClt~ZC>Qk-nWcQv2$rVo9JQ0e1(wpm1&&u@-%P$lI z_k|fQSt%Wz2QFo>noUltGE3YP)u;=sB}aB0RVoO!n|uCQ(rJ?NoHMUz*%nGES*k&C zPHGT)jdm)#H}j@FA3`z7I*7(jl&iuzYCWY9CSVX`A3;`bj4hYAWIh>k#+F{>pVe(1 zjmnhI+~e0AR4#bLGi+LuT0=UBR+Ny(Lpr)sJ60fLuv_?1k5^B;WxqRZjr&I6277IH za#jE*T7%B6a#i{0%qD`YR2<8cICtGF<~a57s7h3vdfOX{bu+C8baw@(WTneE)A^~) zcG^UiC^Zwri3x2_Xp*e6g60M8x|M#lH9U|zK6}2``ZEsz{OK?|s=<^d*GhK#<0Y5$ zb4Sn!qO`{E(_z4@ zX^Py>?9HPyQ|WwPI$Om`z1-y8<}zbR?QiCV*h_i}oK)vTey7LQgsNsOF3XX6_`bOD z4PIN81hV#Fn^3+dWKj>X!PrPxG)WH?=qm#b$&M1i_a1rHS6YhIU}gtA^*Yda-`6OM}$!^zpZ8 z3hO9c2q1s>&gcs%cWZ72A!srIvV#wmA=~kl>^m<~vw<;YCc83vp?*wS6a|8*2o*9L zSZ!B+n9bx!k*uEb=Qo+S6V%L!;!!o(y9^_tHj~n{Pj=WTiCxuGyHnrV$;`>zF{Fgr zoP}#g)lgisY7!L)DzHQ_((v@}bwc|)IpSkKo*e0<%6ZRnjZed=b(vh>N!&eTP?=bc z!0Y0P=i%+ z$JYuxQa|+e?WFe{$V&kmBurC*Qo(nXEGLwsQGlkXx}T;7LZfWr zI?$xa(QsRlEe5NY7g-r^zoOONXgnvetS~(DWYC+9zWFLUFl1U#EH9GnaOM;*(<)te zWwH#R$1a{+)uGLg@E^_ml4oB;sV7Tg2SR1|hCXX612{O46BiEec^3MU)rEVA?5xlz z8WUf3)+ku?-5zls$S0aX)L}LLBsa-%-;gWTlf6IVXbCTVB1Y50^vr96Ke9v@v_Hu2^sc{?4i-)qj!SgSD z)XNokA6g!_FI!jLQ2FvBx?af{8&TWMM;E%qru*V&65V8azPqJX4Sl<%b}&YcC30Jj z|LU^}`fV^~ecZ*1L~N`r>m${WHv9&4LP*C|{Pxr`Gb zE9&=I$pUHt!-5y+`1{ujj=XK3*;Y@Bwu3I?Dr3zKL3ks+8sCHV3wx`?fw4*=%Kp-G zIaumpHSymU+w;%W6LjeFLrdi4Y4X!bmWPD;-?yv;y2bh)z7_b<6706%fw^9P zidVcI>;JLyl4wR^a7hd5^WT)`es#dR%zlcOBNslw}v#we$Mn};{dzFM(9`lq6`4>No z1W=-rn5|9CT_~%UazU=FF%wGT3lj@k3&lSaf0*+hs=4bgf*h;ox2GFO9E&qNg1<6; zj(pVZ%3jV%liu$)>TrYvdZVpTdJmBlGKCSN7|1UE?(mpMR#%O z=o~Ut)_k35q=Vo|c{?1cm3PL|PmV2rpHE*QF?d%GYRBFTvMYXBOC~rhE+o4x5+@8~ zsnt%^*(DxVvQw4`1gy#wJAbO_W6P{cc@+Fnb+(8EdxuX9YG!%)s-2-p@FeEkke~*SjfB-(edkFCV9}WHE{*Hfh zQsDo(=j0!-*8T6lprL<^%KwCh{xkMJ*{=VWeCYo`Od`2Mpa2YsV)B8Jv@Md>6hz{f zNIX*j2<86^$_axap-Uvgi@ZMwIUx`rk~aOzr%31&c{>mindG}4yuZE^L~^44sK*C{ z{@LokfXRRNiR2akg?`dQk)Wv{6m(BTA{kT=9}tO*{=4?QCxU`tq@PGa69C~uqL;`X zVgHKbziCG#<@x7&{a=x7V16W0OOJ$bEleD+yfO_n29UHPLDL@BMJjEOeUIZKa%tzs zSEHC7w-2joUZK@8n_sWAmeaO}0u+E>A$f}=4qsE$6fw^_ZI|+bH8@RN#y)R8LRHg@ zZ+gJc;G5%hnnoejT)#{j?p7$Lx=Z=S|7Le>A0zzpD5@CYjU&2VF{^;zm*$>3+d=%{ zB5Lz|8;OIv~uYMnzR&cTgVnd`MpU zX+*%emL@J!D}-v+V)dJ@j`u4G2bqx+_@|KJliyK+n6JNrbF zq=X1>0BuVj2b3zg(J82W`i2ED)U(sMpt<@>P%a|}Yl|#cGBW$WlfYl@ruoOcIRBdj z<3D%9|0c%x$KLxp#`wo<{)OE3a<#C;LasOn7Vys(Js%7Pfzeyi|7pADz3*T2j{mfQ zp~wrZ{=)`DE>t9$`L}XV5D>}Q{@n%xL6Ow$-)+dcNKW_fHZXD_|7X8IekeZ@lD>ca z#TUN&sQzX{^1LwQPWqb-{6E>iU?hwDALWoNIC3BUy)Nv(V?$mhgrva#UJfG2cfW`K zRu@T!|5rI=Dn(M+e=mpR@A?1JXBZF){m*hRAfLd0#tQ?1{u+n7s|k|Rcf~@I@me-M v7WdQ3tLfx~R1@5zv&c022RDD8neUf^o4bjt`yUGdNtXi!u~=AS)#d&_t^)$( literal 0 HcmV?d00001 diff --git a/docs/storage.md b/docs/storage.md index 52581fb6..374235fb 100644 --- a/docs/storage.md +++ b/docs/storage.md @@ -1,3 +1,49 @@ +# Local Static Volume Provisioner +## Cluster Configurations +To use the local static volume provisioner, you need to set the `local_volume_provisioner_enabled ` to `true` in the group_vars/all.yml file. Then you can customerize your storage class name per your workload by setting `local_volume_provisioner_storage_class`, and if no change, the default value is local-static-storage. + +## Disk configuration +Then you need to configure the disk information for storage in the host_vars/\.yml based your systemd disk information. +``` +persistent_volumes: [] +#persistent_volumes: +# - name: "mnt-data-1" # PV identifier will be used for PVs names followed by node name(e.g., mnt-data-1-hostname) +# mountPath: /mnt/disks/disk0 # Mount path of a volume, for local provisioner, it musts match /mnt/disks/* pattern +# device: /dev/nvme0n1 # Target storage device name when creating a volume. Only set it when storage_deploy_test_mode is false +# fsType: ext4 # file system types, by default is ext4. + +``` +If you do not have real disks inserted into your machine, and you still want to try the k8s provisioner, you can just enable `storage_deployment_test_mode` in the group_vars/all.yml. RA will automitically create 6 10GiB fake loop devices for simulation and fill the persistent_volumes automatically inside. + +## Simple Verification +After the deployment successfully, you can claim a persistent-volume-claim for verification w/ below commd: +``` +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: user-data +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: local-static-storage +``` + +# Rook-Ceph Operator +## Operator Configuration +To use the rook-ceph operator as your k8s storage framework. You need to set the `rook_ceph.enabled` to `true` in the group_vars/all.yml file. Same as local volume static provisioner, you can customerize your `storage_class` name, otherwise, the default value is `rook-cephfs`. Also, you can choose the storage method for your workload/solution, by default the backend uses `cephfs` as default storage method. +``` +rook_ceph: + enabled: true + storage_class: "rook-cephfs" # Storage class name + storage_type: "cephfs" # Storage type for rook-ceph, supported values[cephfs, block, object] +``` +## Disk Configuration +Same as described in local volume static provisioner. + # MinIO Operator/Console/Tenant ## Cluster configuration options @@ -47,15 +93,14 @@ There's also a set of configuration options that are applied in per-node manner. First set of variables enable Persistent Volumes and also this info was used for sample tenant deployment. You would need your own tenant settings. ```yaml -minio_pv: - - name: "mnt-data-1" # PV name will be followed by kube_node name(e.g., mnt-data-1-hostname) - storageClassName: "local-storage" # default storage class name which PVC should match with - accessMode: "ReadWriteOnce" # ReadWriteOnce/ReadOnlyMany/ReadWriteMany/ReadWriteOncePod - persistentVolumeReclaimPolicy: "Retain" # Retain/Recycle/Delete - mountPath: /mnt/data1 # mount path - storage: file # file = file block device, nvme = nvme m.2 SSDs. - device: /dev/nvme0n1 # when storage=nvme, device will be used for block device name. when storage=file, loop devices will be populated with /root/diskimage* automatically. - capacity: 1GiB # size of the PV. support only GiB/TiB +persistent_volumes: + - name: "mnt-data-1" # PV identifier will be used for PVs names followed by node name(e.g., mnt-data-1-hostname) + storageClassName: "local-storage" # Storage class name to match with PVC + accessMode: "ReadWriteOnce" # Access mode when mounting a volume, e.g., ReadWriteOnce/ReadOnlyMany/ReadWriteMany/ReadWriteOncePod + persistentVolumeReclaimPolicy: "Retain" # Reclaim policy when a volume is released once it's bound, e.g., Retain/Recycle/Delete + mountPath: /mnt/disks/disk0 # Mount path of a volume, for local provisioner, it musts match /mnt/disks/* pattern + device: /dev/nvme0n1 # Target storage device name when creating a volume. Only set it when storage_deploy_test_mode is false + fsType: ext4 # file system types ``` ## Sample Tenants diff --git a/docs/vm_bm_mixed_config_guide.md b/docs/vm_bm_mixed_config_guide.md new file mode 100644 index 00000000..d12a8617 --- /dev/null +++ b/docs/vm_bm_mixed_config_guide.md @@ -0,0 +1,101 @@ +# What is VM + BM Mixed configuration + + +There is a requirement that a cluster is to be composited of VMs and Baremetal hosts. The role of VMs and BM hosts are flexible. + +In VMRA, we create VXLAN bridges on VM hosts. The one on first VM host is special -- it is a DHCP VXLAN bridge. Other VMs (even for those on other VM hosts) will get their VXLAN ip from this bridge through DHCP. + +In order to composite the mixed cluster, we reused the same VXLAN way on BM host. it is a simple VXLAN bridge interface.Instead of requesting DHCP from first VM host, we choose to assign fixed VXLAN IP for BM host. The default DHCP pool range is from 172.31.0.3 ~ 172.31.0.100 and thus leaves the IP above 172.31.0.100 to be assigned to BM hosts. + +## group_vars/all.yml specific options + +There is one special requirement for CNI selection: calico with VXLAN backend must be selected for the mixed cluster. Other CNI may work too, but not verified. + +``` +kube_network_plugin: calico +``` +``` +calico_network_backend: vxlan +``` +``` +calico_mtu: 1390 +``` + +## Mixed BM host specific options + +The Mixed BM host need define two new parameters in host_vars compared with normal BMRA host. +examples: + +``` +vxlan_gw_ip: "172.31.0.101/24" +``` +``` +vxlan_physical_network: "11.0.0.0/8" +``` +Note: please select the proper subnet of vxlan physical network according to your real situation. Subnet must be the same as it is used for vm_hosts. +Note: vxlan_gw_ip must belong to the same subnet as for vm_host. + + +The vxlan_gw_ip variable defines the VXLAN bridge IP on Mixed BM host. This is also the IP that the cluster will "see" the node. + +The vxlan_physical_network is used to auto select the physical NIC that will carry on the VXLAN transport. + +This two parameters are very similar with the ones on VM host host_vars unless only the first VM host requires vxlan_gw_ip. + +# Example inventory.ini for Mixed case +In this example, we have vm-ctrl-1 and vm-work-1 as they are from traditional VMRA configuration. The difference is we also defined a "bm-1" host which will be a kube_node in the coming cluster. +``` +[all] +host-for-vms-1 ansible_host=10.67.116.xxx ip=10.67.116.xxx ansible_user=root +localhost ansible_connection=local ansible_python_interpreter=/usr/bin/python3 +bm-1 ansible_host=10.67.117.xxx ip=10.67.117.xxx ansible_user=root + +[vm_host] +host-for-vms-1 + +[kube_control_plane] +#vm-ctrl-1 + +[etcd] +#vm-ctrl-1 + +[kube_node] +bm-1 + +[k8s_cluster:children] +kube_control_plane +kube_node + +[all:vars] +ansible_python_interpreter=/usr/bin/python3 +``` + +During the ansible execution, the new created inventory_vm.ini will look like this: +``` +[all] +host-for-vms-1 ansible_host=10.67.116.xxx ip=10.67.116.xxx ansible_user=root +localhost ansible_connection=local ansible_python_interpreter=/usr/bin/python3 +bm-1 ansible_host=172.31.0.101 ip=172.31.0.101 ansible_user=root +vm-ctrl-1 ansible_host=172.31.0.60 ip=172.31.0.60 ansible_user=root +vm-work-1 ansible_host=172.31.0.47 ip=172.31.0.47 ansible_user=root + +[vm_host] +host-for-vms-1 + +[kube_control_plane] +vm-ctrl-1 + +[etcd] +vm-ctrl-1 + +[kube_node] +bm-1 +vm-work-1 + +[k8s_cluster:children] +kube_control_plane +kube_node + +[all:vars] +ansible_python_interpreter=/usr/bin/python3 +``` diff --git a/docs/vm_cluster_expansion_guide.md b/docs/vm_cluster_expansion_guide.md index 19df33b4..34f18517 100644 --- a/docs/vm_cluster_expansion_guide.md +++ b/docs/vm_cluster_expansion_guide.md @@ -1,7 +1,7 @@ # VM cluster expansion guide -VM cluster expansion means that we can add additional vm-work node(s) to existing VM cluster. They can be added on any existing vm_host machines or new vm_host machine(s) can be added as well. By default VM cluster expanssion feature won't re-create existing VMs. They will remain untouched during VM creation phase, nevertheless standard ansible deployment tasks will run on them as well. Corresponding ansible playbooks should be idempotent to ensure that running tasks again won't corrupt target system. Group vars for VM case contain new variable `vm_recreate_existing` with value set to `false`. +VM cluster expansion means that we can add additional vm-work node(s) to existing VM cluster. They can be added on any existing vm_host machines or new vm_host machine(s) can be added as well. By default VM cluster expansion feature won't re-create existing VMs, which are in 'running' state. They will remain untouched during VM creation phase, nevertheless standard ansible deployment tasks will run on them as well. Corresponding ansible playbooks should be idempotent to ensure that running tasks again won't corrupt target system. Group vars for VM case contain new variable `vm_recreate_existing` with value set to `false`. In order to expand VM cluster, the original VM cluster have to be up and running. ``` @@ -11,7 +11,7 @@ vm_recreate_existing: false For VM cluster expansion deployment we should use the same ansible host, which was used for original VM cluster deployment. Deployment configuration can't be changed except adding definitions for new VMs. Configuration of existing VMs have to remain the same. -**_NOTE:_** If you want to add new vm-work node to existing vm_host then the vm_host needs to have enough free available resoures for it. +**_NOTE:_** If you want to add new vm-work node to existing vm_host then the vm_host needs to have enough free available resources for it. **_NOTE:_** VM cluster expansion deployment is not intended for any kind of configuration update on existing VMs. @@ -98,12 +98,49 @@ If you want to update VM cluster including current VMs then `vm_recreate_existin In that case all existing VMs will be destroyed and re-created again. **_NOTE:_** All data, configurations and logs stored on VMs will be lost. Re-created VMs will get new IPs. +**_NOTE:_** If `vm_recreate_existing` is set to `true` then following two lists `vm_recreate_listed_vms` and `vm_keep_listed_vms` are not evaluated. ``` vm_recreate_existing: true ``` -To run this option use following command: + +If you want to update just specific VM(s) then keep `vm_recreate_existing` parameter on `false` and use another configuration parameter +`vm_recreate_listed_vms` and insert VM names, which should be re-created, there. Default value for this paramater is empty list `[]` + +``` +vm_recreate_listed_vms: [] +``` + +VMs to be re-created should be provided as list items E.g.: + +``` +vm_recreate_listed_vms: + - vm-work-2 + - vm-1 +``` + + +If you want to keep existing VM, which is not in `running` state untouched from any reason then keep `vm_recreate_existing` parameter on `false` +and use another configuration parameter `vm_keep_listed_vms` and insert VM names, which should be kept, there. Default value for this paramater is empty list `[]` + +``` +vm_keep_listed_vms: [] +``` + +VMs to be kept should be provided as list items E.g.: + +``` +vm_keep_listed_vms: + - vm-work-3 + - vm-1 +``` + +**_NOTE:_** If the same VM name appears in both lists `vm_recreate_listed_vms` and `vm_keep_listed_vms` then `vm_keep_listed_vms` has precedence and VM will be kept. +**_NOTE:_** Deployment procedure tries to start not running VMs, which were kept. Nevertheless subsequent ansible plays can fail if VM start was not successful or if they are in some inconsistent state. + + +To run those options use following command: ``` ansible-playbook -i inventory.ini playbooks/vm.yml diff --git a/generate/playbook_templates/infra_playbook.j2 b/generate/playbook_templates/infra_playbook.j2 index 7bf770c2..3af8172c 100644 --- a/generate/playbook_templates/infra_playbook.j2 +++ b/generate/playbook_templates/infra_playbook.j2 @@ -1,17 +1,18 @@ --- # apply common cluster node configuration -- hosts: k8s_cluster,vm_host +- hosts: k8s_cluster,vms,vm_host handlers: - name: reboot server reboot: { reboot_timeout: 1200 } when: - inventory_hostname != "localhost" pre_tasks: - - name: End play for VM host + - name: End play for VM host and BM host meta: end_host when: - - "'vm_host' in group_names" + - "'vm_host' in group_names or 'bm_host' in group_names" - on_vms | default(false) | bool + tags: always roles: - role: cluster_defaults tags: always @@ -54,7 +55,7 @@ that: >- ( on_vms | default(false) and - ansible_play_hosts_all | difference(groups['vm_host']) | difference(ansible_play_hosts) | length() == 0 + ansible_play_hosts_all | difference(groups['vm_host']) | difference(groups['bm_host'] | default([])) | difference(ansible_play_hosts) | length() == 0 ) or ( not on_vms | default(false) and @@ -65,18 +66,19 @@ any_errors_fatal: true # apply worker node kernel configuration -- hosts: kube_node,vm_host +- hosts: kube_node,vms,vm_host handlers: - name: reboot server reboot: { reboot_timeout: 1200 } when: - inventory_hostname != "localhost" pre_tasks: - - name: End play for VM host + - name: End play for VM host and BM host meta: end_host when: - - "'vm_host' in group_names" + - "'vm_host' in group_names or 'bm_host' in group_names" - on_vms | default(false) | bool + tags: always roles: - role: cluster_defaults tags: defaults @@ -91,7 +93,6 @@ - kmra - istio-service-mesh - sst - - apply-intel-pstate when: - configure_sgx | default(false) | bool or kmra.oran.enabled | default(false) | bool or @@ -102,8 +103,7 @@ sst_bf_configuration_enabled | default(false) | bool or sst_cp_configuration_enabled | default(false) | bool or sst_tf_configuration_enabled | default(false) | bool or - sst_pp_configuration_enabled | default(false) | bool or - intel_pstate_enabled | default(true) | bool + sst_pp_configuration_enabled | default(false) | bool - role: bootstrap/install_realtime_kernel when: rt_kernel_enabled | default(false) | bool - role: bootstrap/configure_hugepages @@ -111,18 +111,20 @@ - hugepages - intel-platform-qat-setup when: hugepages_enabled | default(true) | bool + - role: bootstrap/configure_kpm_drivers + tags: + - power-manager + when: kubernetes_power_manager is defined and kubernetes_power_manager.enabled | default(false) | bool - role: bootstrap/configure_cpu_isolation when: isolcpus_enabled | default(false) | bool - role: bootstrap/configure_cpusets when: cpusets_enabled | default(false) | bool - - role: bootstrap/configure_intel_pstate - when: intel_pstate_enabled | default(true) | bool - - role: bootstrap/reset_qat_option - tags: - - reset_qat_option - - intel-platform-qat-setup + - role: bootstrap/configure_disks + tags: storage when: - - update_qat_drivers | default(false) | bool + - local_volume_provisioner_enabled | default(false) | bool or + minio_enabled | default(false) | bool or + rook_ceph.enabled | default(false) | bool - role: bootstrap/auto_detect_qat_devices tags: - auto-detect-qat-device @@ -170,7 +172,7 @@ when: - configure_fpga | default(false) | bool - not vm_enabled or on_vms | default(false) | bool - - role: bootstrap/install_gpu_driver + - role: install_gpu_driver when: - configure_gpu | default(false) | bool - not vm_enabled or on_vms | default(false) | bool @@ -191,7 +193,7 @@ that: >- ( on_vms | default(false) and - ansible_play_hosts_all | difference(groups['vm_host']) | difference(ansible_play_hosts) | length() == 0 + ansible_play_hosts_all | difference(groups['vm_host']) | difference(groups['bm_host'] | default([])) | difference(ansible_play_hosts) | length() == 0 ) or ( not on_vms | default(false) and @@ -203,31 +205,29 @@ {% if playbook_name in ['full_nfv', 'access', 'on_prem', 'on_prem_vss', 'remote_fp', 'build_your_own'] %} # install worker node qat software -- hosts: kube_node,vm_host +- hosts: kube_node,vms,vm_host handlers: - name: reboot server reboot: { reboot_timeout: 1200 } when: - inventory_hostname != "localhost" pre_tasks: - - name: End play for VM host + - name: End play for VM host and BM host meta: end_host when: - - "'vm_host' in group_names" + - "'vm_host' in group_names or 'bm_host' in group_names" - on_vms | default(false) | bool + tags: always roles: - role: cluster_defaults tags: defaults - - role: bootstrap/apply_intel_pstate - tags: apply-intel-pstate - when: intel_pstate_enabled | default(true) | bool - role: bootstrap/install_qat_drivers_services tags: - setup-qat - intel-platform-qat-setup when: + - configure_qat | default(false) | bool - update_qat_drivers | default(false) | bool - - qat_devices | default([]) | length > 0 - role: bootstrap/configure_dlb tags: dlb-dp when: @@ -246,7 +246,7 @@ that: >- ( on_vms | default(false) and - ansible_play_hosts_all | difference(groups['vm_host']) | difference(ansible_play_hosts) | length() == 0 + ansible_play_hosts_all | difference(groups['vm_host']) | difference(groups['bm_host'] | default([])) | difference(ansible_play_hosts) | length() == 0 ) or ( not on_vms | default(false) and @@ -258,13 +258,14 @@ {% endif %} # install worker node network software -- hosts: kube_node,vm_host +- hosts: kube_node,vms,vm_host pre_tasks: - - name: End play for VM host + - name: End play for VM host and BM host meta: end_host when: - - "'vm_host' in group_names" + - "'vm_host' in group_names or 'bm_host' in group_names" - on_vms | default(false) | bool + tags: always roles: - role: cluster_defaults tags: defaults @@ -276,24 +277,28 @@ - dataplane_interfaces | default([]) | length > 0 - role: bootstrap/update_nic_drivers tags: update-nic-drivers - when: update_nic_drivers | default(false) | bool + when: + - update_nic_drivers | default(false) | bool + - not configure_tdx | default(false) - role: bootstrap/update_nic_firmware tags: update-nic-firmware when: - dataplane_interfaces | default([]) | length > 0 - update_nic_firmware | default(false) | bool - update_nic_drivers | default(false) | bool + - not configure_tdx | default(false) environment: "{{ '{{' }} proxy_env | d({}) {{ '}}' }}" any_errors_fatal: true # install DPDK and apply SRIOV configuration -- hosts: kube_node,vm_host +- hosts: kube_node,vms,vm_host pre_tasks: - - name: End play for VM host + - name: End play for VM host and BM host meta: end_host when: - - "'vm_host' in group_names" + - "'vm_host' in group_names or 'bm_host' in group_names" - on_vms | default(false) | bool + tags: always roles: - role: cluster_defaults tags: defaults @@ -332,7 +337,7 @@ and container_runtime_only_deployment | default(false) | bool)) {% endif %} {% if playbook_name in ['full_nfv', 'access', 'on_prem', 'on_prem_vss', 'remote_fp', 'build_your_own'] %} - - role: bootstrap/install-qatlibs + - role: bootstrap/install_qatlibs tags: qatlibs when: - qat_devices | default([]) | length > 0 @@ -346,7 +351,6 @@ when: - qat_devices | default([]) | length > 0 - iommu_enabled | default(true) | bool - - not on_vms | default(false) | bool - configure_qat | default(false) | bool - role: bootstrap/configure_openssl tags: @@ -361,12 +365,24 @@ - sgx - intel-platform-sgx-setup when: - - configure_sgx | default(false) | bool + - configure_sgx | default(false) | bool or + configure_tdx | default(false) | bool {% endif %} + - role: imtl_install + tags: + - intel-media-transport-library + when: intel_media_transport_library_enabled | default(false) environment: "{{ '{{' }} proxy_env | d({}) {{ '}}' }}" any_errors_fatal: true - hosts: k8s_cluster + pre_tasks: + - name: End play for VM host and BM host + meta: end_host + when: + - "'vm_host' in group_names or 'bm_host' in group_names" + - on_vms | default(false) | bool + tags: always roles: - role: bootstrap/configure_adq tags: configure-adq diff --git a/generate/playbook_templates/intel_playbook.j2 b/generate/playbook_templates/intel_playbook.j2 index 756fd7dd..544897e6 100644 --- a/generate/playbook_templates/intel_playbook.j2 +++ b/generate/playbook_templates/intel_playbook.j2 @@ -34,7 +34,7 @@ - nfd - intel-platform-qat-setup - intel-platform-sgx-setup - when: nfd_enabled | default(true) | bool + when: nfd_enabled | default(false) | bool {% if playbook_name in ['full_nfv', 'remote_fp', 'build_your_own'] %} - role: intel_cpu_controlplane tags: cpu-ctlplane @@ -127,9 +127,9 @@ - tac.enabled | default(false) | bool {% endif %} {% if playbook_name in ['access', 'full_nfv', 'on_prem', 'on_prem_vss', 'remote_fp', 'regional_dc', 'build_your_own'] %} - - role: intel_power_manager + - role: kubernetes_power_manager tags: power-manager - when: intel_power_manager is defined and intel_power_manager.enabled | default(false) | bool + when: kubernetes_power_manager is defined and kubernetes_power_manager.enabled | default(false) | bool {% endif %} {% if playbook_name in ['access', 'full_nfv', 'on_prem', 'on_prem_vss', 'remote_fp', 'build_your_own'] %} - role: openssl_engine_install @@ -143,10 +143,10 @@ tags: platform-aware-scheduling when: tas_enabled | default(true) | bool or gas_enabled | default(true) | bool {% endif %} - - role: kube_prometheus - tags: kube-prometheus + - role: prometheus_install + tags: prometheus when: - - prometheus_operator | default(false) | bool + - prometheus_enabled | default(false) | bool - role: collectd_install tags: monitoring when: @@ -157,19 +157,6 @@ - telegraf_enabled | default(false) | bool - not (collectd_enabled | default(false) | bool) tags: monitoring -{% if playbook_name in ['access', 'full_nfv', 'build_your_own'] %} - - role: intel_sriov_fec_operator - tags: intel-sriov-fec-operator - when: - - intel_sriov_fec_operator_enabled | default(false) | bool - - not (intel_flexran_enabled | default(false) | bool and intel_flexran_type == "pod") -{% endif %} -{% if playbook_name in ['access', 'full_nfv', 'on_prem', 'on_prem_vss', 'regional_dc', 'remote_fp', 'build_your_own'] %} - - role: istio_service_mesh - tags: istio-service-mesh - when: - - istio_service_mesh.enabled | default(true) | bool -{% endif %} {% if playbook_name in ['full_nfv', 'on_prem', 'on_prem_vss', 'regional_dc', 'build_your_own'] %} - role: minio_install tags: minio @@ -209,6 +196,17 @@ tags: intel-eci when: - intel_eci_enabled | default(false) | bool + - role: intel_csl_excat + tags: intel_csl_excat + when: + - intel_csl_excat_enabled | default(false) | bool +{% endif %} +{% if playbook_name in ['on_prem_aibox'] %} + - role: intel_base_container + tags: + - base_container + when: + - intel_base_container_enabled | default(false) | bool {% endif %} environment: - "{{ '{{' }} proxy_env | d({}) {{ '}}' }}" @@ -246,6 +244,8 @@ tags: defaults - role: net_attach_defs_create tags: net-attach-defs + when: + - kubernetes | default(false) | bool - role: elasticsearch_install tags: elasticsearch when: @@ -267,6 +267,45 @@ tags: linkerd-service-mesh when: - linkerd_service_mesh.enabled | default(false) | bool +{% endif %} + - role: intent_driven_orchestration + tags: intent-driven-orchestration + when: + - ido.enabled | d(false) | bool + - registry_enable | d(false) | bool + - linkerd_service_mesh.enabled | d(false) | bool + - role: intel_xpumanager + tags: xpumanager + when: + - intel_xpumanager_enabled | default(false) | bool + - inventory_hostname == groups["kube_control_plane"][0] + - role: wait_for_kubernetes_ready + vars: + force_check: true + tags: k8s-ready-final + ignore_errors: yes + when: + - kubernetes | default(false) | bool + environment: "{{ '{{' }} proxy_env | d({}) {{ '}}' }}" + any_errors_fatal: true + +- hosts: k8s_cluster + tasks: [] + roles: + - role: cluster_defaults + tags: defaults +{% if playbook_name in ['access', 'full_nfv', 'build_your_own'] %} + - role: intel_sriov_fec_operator + tags: intel-sriov-fec-operator + when: + - intel_sriov_fec_operator_enabled | default(false) | bool + - not (intel_flexran_enabled | default(false) | bool and intel_flexran_type == "pod") +{% endif %} +{% if playbook_name in ['access', 'full_nfv', 'on_prem', 'on_prem_vss', 'regional_dc', 'remote_fp', 'build_your_own'] %} + - role: istio_service_mesh + tags: istio-service-mesh + when: + - istio_service_mesh.enabled | default(true) | bool {% endif %} - role: wait_for_kubernetes_ready vars: diff --git a/generate/playbook_templates/main_playbook.j2 b/generate/playbook_templates/main_playbook.j2 index 6cb74f01..24146ae8 100644 --- a/generate/playbook_templates/main_playbook.j2 +++ b/generate/playbook_templates/main_playbook.j2 @@ -21,12 +21,15 @@ - kubernetes | default(true) - not on_cloud | default(false) - kube_provisioner == "rke2" +{% if playbook_name == 'on_prem_sw_defined_factory' %} +- name: install Intel ECI for non-k8s vm nodes + import_playbook: intel/eci_basic.yml + when: on_vms | default(false) +{% endif %} - name: install Intel Container Experience Kit features import_playbook: intel/{{ playbook_name }}.yml +{% if playbook_name != 'on_prem_aibox' %} when: kubernetes | default(true) -{% if playbook_name == 'on_prem_aibox' %} -- name: install Intel Base Containers - import_playbook: intel/base_container.yml {% endif %} - name: run post deployment hooks import_playbook: k8s/post_deployment_hooks.yml diff --git a/generate/profiles_templates/cloud/profiles.yml b/generate/profiles_templates/cloud/profiles.yml index 11eef93d..1048f156 100644 --- a/generate/profiles_templates/cloud/profiles.yml +++ b/generate/profiles_templates/cloud/profiles.yml @@ -16,7 +16,6 @@ # - intel_cpu_controlplane # - native_cpu_manager # - bond_cni -# - topology_manager # - sriov_operator # - sriov_network_dp # - nic_drivers @@ -37,14 +36,14 @@ # - openssl # - tas # - gas -# - ddp +# - ddp_legacy # - network_userspace # - dpdk # - ovs_dpdk # - sst # - power: # manager -# pstate +# frequency_scaling # cstate # uncore_frequency # - telemetry: @@ -72,7 +71,7 @@ # - intel_ethernet_operator # enabled # flow_config -# ddp +# ddp_update # fw_update # - intel_sriov_fec_operator # - intel_oneapi @@ -81,6 +80,7 @@ # - intel_flexran # - tadk # - cadvisor +# - imtl --- access: name: access @@ -92,7 +92,6 @@ access: isolcpu: optional cpusets: optional native_cpu_manager: off - topology_manager: off sriov_operator: off sriov_network_dp: off nic_drivers: off @@ -118,14 +117,14 @@ access: tac: off tas: off gas: off - ddp: off + ddp_legacy: off network_userspace: off dpdk: on ovs_dpdk: off sst: off power: manager: off - pstate: off + frequency_scaling: off cstate: off uncore_frequency: off telemetry: @@ -154,7 +153,7 @@ access: intel_ethernet_operator: enabled: off flow_config: off - ddp: off + ddp_update: off fw_update: off intel_sriov_fec_operator: off intel_oneapi: @@ -163,6 +162,7 @@ access: intel_flexran: off adq_dp: off cadvisor: on + imtl: off basic: name: basic @@ -173,14 +173,13 @@ basic: kube_dashboard: off isolcpu: optional cpusets: optional - topology_manager: off sriov_operator: off sriov_network_dp: off nic_drivers: off dpdk: optional power: manager: off - pstate: optional + frequency_scaling: optional cstate: optional uncore_frequency: off telemetry: @@ -206,6 +205,7 @@ basic: base: optional ai: optional cadvisor: on + imtl: off full_nfv: name: full_nfv @@ -218,7 +218,6 @@ full_nfv: cpusets: optional intel_cpu_controlplane: optional native_cpu_manager: off - topology_manager: off sriov_operator: off sriov_network_dp: off nic_drivers: off @@ -244,14 +243,14 @@ full_nfv: tac: off tas: off gas: off - ddp: off + ddp_legacy: off network_userspace: on dpdk: on ovs_dpdk: on sst: off power: manager: off - pstate: optional + frequency_scaling: optional cstate: optional uncore_frequency: off telemetry: @@ -280,7 +279,7 @@ full_nfv: intel_ethernet_operator: enabled: off flow_config: off - ddp: off + ddp_update: off fw_update: off intel_sriov_fec_operator: off intel_oneapi: @@ -290,6 +289,7 @@ full_nfv: tadk: on adq_dp: off cadvisor: on + imtl: off on_prem: name: on_prem @@ -301,7 +301,6 @@ on_prem: isolcpu: optional cpusets: optional native_cpu_manager: off - topology_manager: off sriov_operator: off sriov_network_dp: off nic_drivers: off @@ -328,7 +327,7 @@ on_prem: sst: off power: manager: off - pstate: off + frequency_scaling: off cstate: optional uncore_frequency: off telemetry: @@ -362,6 +361,7 @@ on_prem: base: optional ai: optional cadvisor: on + imtl: off regional_dc: name: regional_dc @@ -372,7 +372,6 @@ regional_dc: kube_dashboard: off isolcpu: optional cpusets: optional - topology_manager: off sriov_operator: off sriov_network_dp: off nic_drivers: off @@ -394,7 +393,7 @@ regional_dc: dpdk: optional power: manager: off - pstate: optional + frequency_scaling: optional cstate: optional uncore_frequency: off telemetry: @@ -428,6 +427,7 @@ regional_dc: base: optional ai: optional cadvisor: on + imtl: off remote_fp: name: remote_fp @@ -440,7 +440,6 @@ remote_fp: cpusets: optional intel_cpu_controlplane: optional native_cpu_manager: off - topology_manager: off sriov_operator: off sriov_network_dp: off nic_drivers: off @@ -462,14 +461,14 @@ remote_fp: dlb_dp: off openssl: off tas: off - ddp: off + ddp_legacy: off bond_cni: off network_userspace: optional dpdk: on sst: off power: manager: off - pstate: off + frequency_scaling: off cstate: optional uncore_frequency: off telemetry: @@ -497,13 +496,14 @@ remote_fp: intel_ethernet_operator: enabled: off flow_config: off - ddp: off + ddp_update: off fw_update: off adq_dp: off intel_oneapi: base: optional ai: optional cadvisor: optional + imtl: off build_your_own: name: build_your_own @@ -516,7 +516,6 @@ build_your_own: cpusets: optional intel_cpu_controlplane: optional native_cpu_manager: off - topology_manager: off sriov_operator: off sriov_network_dp: off nic_drivers: off @@ -540,14 +539,14 @@ build_your_own: tac: off tas: off gas: off - ddp: off + ddp_legacy: off network_userspace: optional dpdk: optional ovs_dpdk: optional sst: off power: manager: off - pstate: off + frequency_scaling: off cstate: optional uncore_frequency: off telemetry: @@ -576,7 +575,7 @@ build_your_own: intel_ethernet_operator: enabled: off flow_config: off - ddp: off + ddp_update: off fw_update: off intel_sriov_fec_operator: off intel_oneapi: @@ -586,3 +585,4 @@ build_your_own: tadk: optional adq_dp: off cadvisor: optional + imtl: off diff --git a/generate/profiles_templates/common/group_vars.j2 b/generate/profiles_templates/common/group_vars.j2 index 7f3ee48b..caec1b60 100644 --- a/generate/profiles_templates/common/group_vars.j2 +++ b/generate/profiles_templates/common/group_vars.j2 @@ -38,6 +38,17 @@ project_root_dir: /opt/cek/ # When vm_recreate_existing is true, existing VMs are destroyed and created again during cluster update/scaling vm_recreate_existing: false +# When vm_recreate_listed_vms is not empty, listed VMs are destroyed and created again during cluster update/scaling +# even if vm_recreate_existing is false and VMs are in running state +vm_recreate_listed_vms: [] +#Example how to use it. +#vm_recreate_listed_vms: +# - vm-work-1 + +# When vm_keep_listed_vms is not empty, listed VMs are kept during cluster update/scaling even if they are not running +# It has precedens over vm_recreate_listed_vms. Nevertheless it is overwritten by vm_recreate_existing: true +vm_keep_listed_vms: [] + {% endif %} # Improve deployment stability of Kubespray by increasing wait between retries of failed ops like pushing/downloading retry_stagger: 20 @@ -109,10 +120,10 @@ kubernetes: true # Kubernetes provisioner, Support: rke2(work with os ubuntu22.04 and containerd as container_runtime only), kubespray(default option) kube_provisioner: kubespray -kube_version: v1.26.3 # test placeholder: n version -#kube_version: v1.25.8 # test placeholder: n-1 version -#kube_version: v1.24.12 # test placeholder: n-2 version -rke2_version: v1.26.2+rke2r1 # test placeholder: n version +kube_version: v1.27.1 # test placeholder: n version +#kube_version: v1.26.4 # test placeholder: n-1 version +#kube_version: v1.25.9 # test placeholder: n-2 version +rke2_version: v1.26.9+rke2r1 # test placeholder: n version {% if kube_dashboard in ['on', 'optional'] %} # Kubernetes Dashboard @@ -168,8 +179,15 @@ kube_pods_subnet: 10.244.0.0/16 {% endif %} kube_service_addresses: 10.233.0.0/{{ mask }} -# Supported plugins: calico, flannel, cilium for kubespray; canal, calico, cilium for rke2 +# Supported plugins: calico, flannel, cilium, cni for kubespray; canal, calico, cilium for rke2 +{% if calico_vpp in ['on'] %} +# Currently, calico vpp dataplane is not supported by kubespray, +# so need to install a very basic setup without any actual mesh capable CNI in kubespray. +# Later, install calico and calico vpp dataplane with operator based installations. +kube_network_plugin: cni +{% else %} kube_network_plugin: calico +{% endif %} # Calico settings {% if vm_mode in ['on'] %} @@ -179,7 +197,7 @@ calico_network_backend: vxlan calico_network_backend: vxlan # Supported backends: [vxlan, bird(kubespray only)] {% endif %} -kube_network_plugin_multus: {% if multus == 'on' %}true{% else %}false{% endif %} +kube_network_plugin_multus: {% if multus == 'on' and calico_vpp != 'on' %}true{% else %}false{% endif %} # Set on true if you want to enable the eBPF dataplane support @@ -255,29 +273,43 @@ native_cpu_manager_enabled: {% if native_cpu_manager == 'on' %}true{% else %}fal {% endif %} -{% if topology_manager in ['on', 'optional'] %} -# Enable Kubernetes built-in Topology Manager -topology_manager_enabled: {% if topology_manager == 'on' %}true{% else %}false{% endif %} - +# Kubernetes built-in Topology Manager settings # There are four supported policies: none, best-effort, restricted, single-numa-node. topology_manager_policy: "best-effort" -{% endif %} +# There are two supported scopes: container, pod +topology_manager_scope: "container" ###################### ## Storage Features ## ###################### +{% if lpvsp in ['on', 'optional'] or rook_ceph in ['on', 'optional'] or minio in ['on', 'optional'] %} +## Storage simulation configurations ## +# Enable this option, RA uses a file as loop device for storage deployment +# RA create 6 10G loop disk on each node for storage simulation by default. +storage_deploy_test_mode: {% if lpvsp == 'on' or rook_ceph == 'on' or minio == 'on' %}true{% else %}false{% endif %} + +storage_nodes: [] #if no setting, all kubenode will be used as storage node. +# storage_nodes: +# - node0 +# - node1 +{% endif %} + {% if lpvsp in ['on', 'optional'] %} # The local persistence volume static provisioner # https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner local_volume_provisioner_enabled: {% if lpvsp == 'on' %}true{% else %}false{% endif %} +local_volume_provisioner_storage_class: "local-static-storage" {% endif %} {% if rook_ceph in ['on', 'optional'] %} +## rook_ceph configuration ## rook_ceph: enabled: {% if rook_ceph == 'on' %}true{% else %}false{% endif %} + storage_class: "rook-cephfs" # Storage class name + storage_type: "cephfs" # Storage type for rook-ceph, supported values[cephfs, block, object]. log_level: "DEBUG" # The logging level for the operator: ["ERROR", "WARNING", "INFO", "DEBUG"] allow_loop_devices: true # Allow using loop devices for osds in test clusters enable_nfs: true # Enable the CSI NFS drivers @@ -302,7 +334,7 @@ minio_enabled: {% if minio == 'on' %}true{% else %}false{% endif %} minio_tenant_enabled: true # Specifies whether to install MinIO Sample Tenant minio_tenant_servers: 4 # The number of MinIO Tenant nodes -minio_tenant_volumes_per_server: 4 # The number of volumes per servers +minio_tenant_volumes_per_server: 2 # The number of volumes per servers minio_tenant_volume_size: 5 # The size of each volume (unit: GiB) minio_deploy_test_mode: true # When true, use a file as loop device when creating storage @@ -448,7 +480,7 @@ gpu_dp_prefered_allocation: 'none' # Available policies are: ['balanced', 'pack {% else %} gpu_dp_enabled: false {% endif %} -{% if sgx_dp in ['on', 'optional'] and arch in ['icx', 'spr'] %} +{% if sgx_dp in ['on', 'optional'] and arch in ['icx', 'spr', 'emr'] %} # Intel SGX Device Plugin for Kubernetes sgx_dp_enabled: {% if sgx_dp == 'on' %}true{% else %}false{% endif %} @@ -571,7 +603,7 @@ linkerd_service_mesh: # jaeger, opentelemetry, kibana. Collectd has to be disabled in that case. # If Collectd is enabled then all Telegraf stack components need to be disabled. {% if telemetry.prometheus in ['on', 'optional'] %} -prometheus_operator: {% if telemetry.prometheus == 'on'%}true{% else %}false{% endif %} +prometheus_enabled: {% if telemetry.prometheus == 'on'%}true{% else %}false{% endif %} {% endif %} {% if telemetry.collectd in ['on', 'optional'] %} @@ -610,30 +642,39 @@ cadvisor_pik_perf_events_enabled: false {% endif %} +{% if telemetry.intel_xpumanager in ['on', 'optional'] %} +# intel_xpumanager plugin collects information about Intel data center GPUs. +intel_xpumanager_enabled: {% if telemetry.intel_xpumanager == 'on'%}true{% else %}false{% endif %} + +{% endif %} + ###################### ## Power Management ## ###################### {% if power.manager in ['on', 'optional'] and arch in ['icx', 'clx', 'spr', 'emr'] %} -# Intel Kubernetes Power Manager -intel_power_manager: +# Kubernetes Power Manager +kubernetes_power_manager: enabled: {% if power.manager == 'on' %}true{% else %}false{% endif %} # Enable/Disable power manager power_nodes: [] # List of power_nodes that should be considered during Operator work and profiles deployment # - node1 # - node2 - build_image_locally: false # Build Intel Power Manager image locally + build_image_locally: false # Build Power Manager image locally deploy_example_pods: true # Deploy example Pods that will utilize special resources global_shared_profile_enabled: true # Deploy custom Power Profile with user defined frequencies that can be applied to all power nodes # to make use of Shared Profile fill Shared Workload settings in host vars global_max_frequency: 1500 # Max frequency that will be applied for cores by Shared Workload global_min_frequency: 1000 # Min frequency that will be applied for cores by Shared Workload -{% if power.pstate in ['on', 'optional'] and arch in ['icx', 'clx', 'spr', 'emr'] %} - # P-State governor decides what frequency within the CPUfreq policy should be used +{% if power.frequency_scaling in ['on', 'optional'] and arch in ['icx', 'clx', 'spr', 'emr'] %} + # !Please set up scaling driver in host_vars.yml file! + # available governors: # "powersave" - Lowest frequency within the borders of min_frequency and max_frequency. # "performance" - Highest frequency within the borders of min_frequency and max_frequency. - global_pstate_governor: "powersave" + # "userspace" - !ACPI ONLY! - Allow user space to set CPU frequency in scaling_setspeed attribute + # "schedutil" - !ACPI ONLY! - Uses data from CPU scheduler to set up frequency + global_governor: "powersave" {% endif %} {% endif %} @@ -733,6 +774,22 @@ intel_oneapi: {% endif %} +{% endif %} + +{% if imtl in ['on', 'optional'] %} +###################### +## System Libraries ## +###################### + +# IMTL: DPDK-based solution designed for high-throughput, low-latency transmission and reception of media data +# Supported only for Intel® E810 Series Network Cards! +# Source: https://github.com/OpenVisualCloud/Media-Transport-Library +intel_media_transport_library_enabled: {% if imtl == "on" %}true{% else %}false{% endif %} # enables IMTL deployment +intel_media_transport_library: + # Patch of ICE driver is needed, set to false if ICE driver is already patched + # update_nic_drivers option in host_vars must be set to true in order to patch, build and load ICE driver + patch_nic_driver: true + {% endif %} ########################## @@ -812,12 +869,41 @@ adq_dp: interface_name: "ens107" {% endif %} +{% if calico_vpp in ['on', 'optional'] %} +# Calico VPP dataplane is in beta and should not be used in production clusters. +# It requires the used interface must be the name of a Linux interface, up and configured with an address. +# The address configured on this interface(e.g. CVL interface) must be the node address in Kubernetes. +# In inventory.ini set "ip=" to IP address of CVL interface. +# Additional requirements and details can be found in docs/calico_vpp.md +calico_vpp: + enabled: {% if calico_vpp == 'on' %}true{% else %}false{% endif %} +{% endif %} + {% if intel_eci and (intel_eci.values() | reject('eq', 'off')) | list | length() > 0 %} # Please contact eci-support@intel.com on how to access this repo. # Also refer to ESH (https://www.intel.com/content/www/us/en/edge-computing/edge-software-hub.html) intel_eci_repo: {% endif %} +{% if intel_csl_excat in ['on', 'optional'] %} +# Please contact eci-support@intel.com on how to get the csl_excat release +# Also refer to ESH (https://www.intel.com/content/www/us/en/edge-computing/edge-software-hub.html) +intel_csl_excat_enabled: {% if intel_csl_excat == 'on' %}true{% else %}false{% endif %} + + +{% endif %} +{% if ido in ['on', 'optional'] %} +# Intent Driven Orchestration (IDO) +# Installs IDO and the linkerd-viz extension, with optional support for deploying a demo workload. +# Follows the installation steps in https://github.com/intel/intent-driven-orchestration +# Requirements: LinkerD and local Docker registry must be enabled +ido: + enabled: {% if ido == 'on' %}true{% else %}false{% endif %} # Installs IDO and linkerd-viz extension + demo_workload: false # Installs 'ido-example-deployment' and IDO intent linked to workload. + # IDO can be tested by changing the number of replicas to 5+ + # After a few minutes the number of replicas should decrease towards 1. + +{% endif %} {% if mirrors == 'true' %} ######################## ## Repository Mirrors ## @@ -868,11 +954,15 @@ mirror_urls: ffmpeg_install_enabled: {% if intel_ffmpeg == 'on' %}true{% else %}false{% endif %} ffmpeg_patches: - - url: "https://github.com/intel/cartwheel-ffmpeg/archive/refs/tags/2023q1.tar.gz" + - url: "https://github.com/intel/cartwheel-ffmpeg/archive/refs/tags/2023q2.tar.gz" type: "tar.gz" - sha256: "5f4c34edbc32298cd3f8fb8ebf789a2d2dbceb5864ec20e5cd501b3cba130871" + sha256: "8c9a1b33bf1e034cd5ec0a9cf208cdb6e0846dae9d621040b83b1e5e31e59799" subdirectory: "patches/" patchset_enabled: true apply_all_patches: true {% endif %} + +{% if base_container in ['on', 'optional'] %} +intel_base_container_enabled: {% if base_container == 'on' %}true{% else %}false{% endif %} +{% endif %} diff --git a/generate/profiles_templates/common/host_vars.j2 b/generate/profiles_templates/common/host_vars.j2 index 7501f98c..0fee2afa 100644 --- a/generate/profiles_templates/common/host_vars.j2 +++ b/generate/profiles_templates/common/host_vars.j2 @@ -46,7 +46,7 @@ isolcpus: "4-15" {% if vm_mode == 'on' %} # isolcpus variable can't be enabled in case of VMRA deployment. # Its content is generated automatically. -# isolcpus: "" +# isolcpus: "" {% else %} isolcpus: "4-11" {% endif %} @@ -69,7 +69,7 @@ cpusets: "4-11" install_dpdk: {% if dpdk == 'on' %}true{% else %}false{% endif %} # DPDK version (will be in action if install_dpdk: true) -dpdk_version: {% if intel_flexran == 'on' %}"22.11.1"{% elif arch == "emr" %}"22.11.1"{% else %}"23.03"{% endif %} # Note: dpdk_version is also dependent on ovs_dpdk when enabled (see preflight) +dpdk_version: {% if (intel_flexran == 'on' or ovs_dpdk == 'on') %}"22.11.1"{% elif arch == "emr" %}"22.11.1"{% elif imtl == 'on'%}"23.03"{% else %}"23.07"{% endif %} # Note: dpdk_version is also dependent on ovs_dpdk when enabled (see preflight) # Custom DPDK patches local path {% if intel_flexran == 'on' %}dpdk_local_patches_dir: "/tmp/flexran"{% else %}#dpdk_local_patches_dir: "/tmp/patches/dpdk"{% endif %} @@ -117,10 +117,10 @@ dataplane_interfaces: [] {% else %} # - bus_info: "18:00.0" # PCI bus info # pf_driver: {% if nic == 'cvl' %}ice{% else %}i40e{% endif %} # PF driver, "i40e", "ice" -{% if ddp in ['on', 'optional'] %} -# ddp_profile: {% if nic == 'cvl' %}"ice_comms-1.3.40.0.pkg"{% else %}gtp.pkgo{% endif %} # DDP package name to be loaded into the NIC +{% if ddp_legacy in ['on', 'optional'] or intel_ethernet_operator.ddp_update in ['on', 'optional'] %} +# ddp_profile: {% if nic == 'cvl' %}"ice_comms-1.3.45.0.pkg"{% else %}gtp.pkgo{% endif %} # DDP package name to be loaded into the NIC # For i40e(XV710-*) allowable ddp values are: "ecpri.pkg", "esp-ah.pkg", "ppp-oe-ol2tpv2.pkgo", "mplsogreudp.pkg" and "gtp.pkgo", replace as required - # For ice(E810-*) allowable ddp values are: ice_comms-1.3.[17,20,22,24,28,30,31,35,37,40].0.pkg such as "ice_comms-1.3.40.0.pkg", replace as required + # For ice(E810-*) allowable ddp values are: ice_comms-1.3.[17,20,22,24,28,30,31,35,37,40,45].0.pkg such as "ice_comms-1.3.45.0.pkg", replace as required # ddp_profile must be defined for first port of each network device. bifurcated cards will appear as unique devices. {% endif %} {% if intel_ethernet_operator.enabled in ['on', 'optional'] %} @@ -149,8 +149,8 @@ dataplane_interfaces: [] # - bus_info: "18:00.1" # pf_driver: {% if nic == 'cvl' %}ice{% else %}i40e{% endif %} -{% if ddp in ['on', 'optional'] %} -# ddp_profile: {% if nic == 'cvl' %}"ice_comms-1.3.40.0.pkg"{% else %}gtp.pkgo{% endif %} +{% if ddp_legacy in ['on', 'optional'] or intel_ethernet_operator.ddp_update in ['on', 'optional'] %} +# ddp_profile: {% if nic == 'cvl' %}"ice_comms-1.3.45.0.pkg"{% else %}gtp.pkgo{% endif %} {% endif %} {% if intel_ethernet_operator.enabled in ['on', 'optional'] %} @@ -172,10 +172,10 @@ dataplane_interfaces: [] update_nic_drivers: {% if nic_drivers == 'on' %}true{% else %}false{% endif %} # The below options can be used to downgrade drivers. This is not recommended and users should proceed at their own risk. -#i40e_driver_version: "2.22.18" -#i40e_driver_checksum: "sha1:0c94bd91014a0d81bd6b99fb41d0e4f1c12b09ff" -#ice_driver_version: "1.11.14" -#ice_driver_checksum: "sha1:730cd04fcfd0ba1b33ba21aaf671d0e1654c999a" +#i40e_driver_version: "2.23.17" +#i40e_driver_checksum: "sha1:57904a541212174c20bacfbf109401c19c59c25c" +#ice_driver_version: "1.12.7" +#ice_driver_checksum: "sha1:b286f3bdf48c2a355f6c1b0ed2a3d82bce21c6df" #iavf_driver_version: "4.8.2" #iavf_driver_checksum: "sha1:fcc997aebeee3744e621e0fd3290205bd18f6a45" @@ -186,9 +186,9 @@ update_nic_firmware: false {% if nic == 'fvl' %} #nvmupdate: # i40e: -# nvmupdate_pkg_url: "https://downloadmirror.intel.com/769287/700Series_NVMUpdatePackage_v9_20_Linux.tar.gz" -# nvmupdate_pkg_checksum: "sha1:87F0BDA58BAAEE0ADF1FADBBCC485AF0A2F0777F" -# required_fw_version: "9.20" +# nvmupdate_pkg_url: "https://downloadmirror.intel.com/786060/700Series_NVMUpdatePackage_v9_30_Linux.tar.gz" +# nvmupdate_pkg_checksum: "sha1:929987FAB30394C86AA1BBADDAA62BB7D4E8CC0E" +# required_fw_version: "9.30" # # min fw version for ddp was taken from: # # https://www.intel.com/content/www/us/en/developer/articles/technical/dynamic-device-personalization-for-intel-ethernet-700-series.html # min_ddp_loadable_fw_version: "6.01" @@ -216,19 +216,20 @@ update_nic_firmware: false {% endif %} {% if cloud_mode == 'on' %} -# install Intel x700 & x800 series NICs DDP packages +# install Intel x700 & x800 series NICs DDP packages (Legacy) # For Cloud RA, the install_ddp_packages option must be false install_ddp_packages: false -{% endif %} -{% if ddp in ['on', 'optional'] %} -# install Intel x700 & x800 series NICs DDP packages -install_ddp_packages: {% if ddp == 'on' and nic == 'fvl'%}true{% else %}false{% endif %} +{% elif ddp_legacy in ['on', 'optional'] %} +# install Intel x700 & x800 series NICs DDP packages (Legacy) +install_ddp_packages: {% if ddp_legacy == 'on' and nic == 'fvl'%}true{% else %}false{% endif %} # If following error appears: "Flashing failed: Operation not permitted" # run deployment with update_nic_firmware: true # or Disable DDP installation via install_ddp_packages: false -enable_ice_systemd_service: {% if ddp == "on" %}true{% else %}false{% endif %} # Enable custom ddp package to be loaded after reboot +{% endif %} +{% if ddp_legacy in ['on', 'optional'] or intel_ethernet_operator.ddp_update | default("") in ['on', 'optional'] %} +enable_ice_systemd_service: {% if ddp_legacy == "on" or intel_ethernet_operator.ddp_update | default("") == "on" %}true{% else %}false{% endif %} # Enable custom ddp package to be loaded after reboot {% endif %} ####################### @@ -252,7 +253,7 @@ userspace_cni_enabled: {% if network_userspace == 'on' %}true{% else %}false{% e ovs_dpdk_enabled: {% if ovs_dpdk == 'on' %}true{% else %}false{% endif %} # Should be enabled with Userspace CNI, when VPP is set to "false"; 1G hugepages required -ovs_version: "v3.1.1" # OVS version has to be compatible/functional with the DPDK version set by 'dpdk_version' +ovs_version: "v3.2.0" # OVS version has to be compatible/functional with the DPDK version set by 'dpdk_version' # CPU mask for OVS-DPDK PMD threads ovs_dpdk_lcore_mask: 0x1 # Hugepages allocated by OVS-DPDK per NUMA node in megabytes @@ -269,8 +270,8 @@ vpp_enabled: {% if vpp == 'on'%}true{% else %}false{% endif %} # Should be enab {% if native_cpu_manager in ["on", "optional"] %} # Native CPU Manager (Kubernetes built-in) # These settings are relevant only if in group_vars native_cpu_manager_enabled: true -native_cpu_manager_system_reserved_cpus: 2000m # Amount of CPU cores reserved for the housekeeping (2000m = 2000 millicores = 2 cores) -native_cpu_manager_kube_reserved_cpus: 1000m # Amount of CPU cores reserved for Kubelet +native_cpu_manager_system_reserved_cpus: 2000m # Amount of CPU cores reserved for the housekeeping (2000m = 2000 millicores = 2 cores) +#native_cpu_manager_kube_reserved_cpus: 1000m # BUG NPF-8495 - uncommenting causes deployment to fail; Amount of CPU cores reserved for Kubelet #native_cpu_manager_reserved_cpus: "0,1,2" # Explicit list of the CPUs reserved for the host level system threads and Kubernetes related threads # Note: All remaining unreserved CPU cores will be consumed by the workloads. @@ -279,53 +280,52 @@ native_cpu_manager_kube_reserved_cpus: 1000m # Amount of CPU cores reserved ## Storage Features ## ###################### -{% if minio in ['on', 'optional'] %} -# MinIO storage configuration -minio_pv: [] -#minio_pv: +# When group_vars: storage_deploy_test_mode == true, RA will create loop device for storage, no need to configure the list. +# otherwise, an actual NVME or SSD device for storage on the device name. + +{% if minio in ['on', 'optional'] or lpvsp in ['on', 'optional'] or rook_ceph in ['on', 'optional'] %} +# node storage configuration +persistent_volumes: [] +#persistent_volumes: # - name: "mnt-data-1" # PV identifier will be used for PVs names followed by node name(e.g., mnt-data-1-hostname) +{% if minio in ['on', 'optional'] %} # storageClassName: "local-storage" # Storage class name to match with PVC # accessMode: "ReadWriteOnce" # Access mode when mounting a volume, e.g., ReadWriteOnce/ReadOnlyMany/ReadWriteMany/ReadWriteOncePod # persistentVolumeReclaimPolicy: "Retain" # Reclaim policy when a volume is released once it's bound, e.g., Retain/Recycle/Delete -# mountPath: /mnt/data0 # Mount path of a volume -# device: /dev/nvme0n1 # Target storage device name when creating a volume. - # When group_vars: minio_deploy_test_mode == true, use a file as a loop device for storage - # otherwise, an actual NVME or SSD device for storage on the device name. +{% endif %} +# mountPath: /mnt/disks/disk1 # Mount path of a volume, for local provisioner, it musts match /mnt/disks/* pattern +# device: /dev/nvme1n1 # Target storage device name when creating a volume. Only set it when storage_deploy_test_mode is false +# fsType: ext4 # file system types, [ext4, xfs] # - name: "mnt-data-2" +{% if minio in ['on', 'optional'] %} # storageClassName: "local-storage" # accessMode: "ReadWriteOnce" # persistentVolumeReclaimPolicy: "Retain" -# mountPath: /mnt/data1 -# device: /dev/nvme1n1 - -# - name: "mnt-data-3" -# storageClassName: "local-storage" -# accessMode: "ReadWriteOnce" -# persistentVolumeReclaimPolicy: "Retain" -# mountPath: /mnt/data2 +{% endif %} +# mountPath: /mnt/disks/disk2 # device: /dev/nvme2n1 +# fsType: ext4 -# - name: "mnt-data-4" +# - name: "mnt-data-3" +{% if minio in ['on', 'optional'] %} # storageClassName: "local-storage" # accessMode: "ReadWriteOnce" # persistentVolumeReclaimPolicy: "Retain" -# mountPath: /mnt/data3 +{% endif %} +# mountPath: /mnt/disks/disk3 # device: /dev/nvme3n1 +# fsType: ext4 -# - name: "mnt-data-5" +# - name: "mnt-data-4" +{% if minio in ['on', 'optional'] %} # storageClassName: "local-storage" # accessMode: "ReadWriteOnce" # persistentVolumeReclaimPolicy: "Retain" -# mountPath: /mnt/data4 +{% endif %} +# mountPath: /mnt/disks/disk4 # device: /dev/nvme4n1 - -# - name: "mnt-data-6" -# storageClassName: "local-storage" -# accessMode: "ReadWriteOnce" -# persistentVolumeReclaimPolicy: "Retain" -# mountPath: /mnt/data5 -# device: /dev/nvme5n1 +# fsType: ext4 {% endif %} ########################## @@ -343,7 +343,7 @@ configure_fpga: {% if fpga == 'on' %}true{% else %}false{% endif %} {% endif %} -{% if sgx in ['on', 'optional'] and arch in ['icx', 'spr'] %} +{% if sgx in ['on', 'optional'] and arch in ['icx', 'spr', 'emr'] %} # Intel Software Guard Extensions (SGX) configure_sgx: {% if sgx == 'on' %}true{% else %}false{% endif %} @@ -449,6 +449,14 @@ qat_devices: [] {% endif %} {% endif %} + +{% if arch in ['spr', 'emr'] and tdx in ['on', 'optional']%} +# EMR TDX configuration +configure_tdx: {% if tdx == 'on' %}true{% else %}false{% endif %} + +tdx_version: "1.5" # only ["1.0", "1.5"] supported +{% endif %} + ############### ## Operators ## ############### @@ -456,8 +464,8 @@ qat_devices: [] {% if intel_ethernet_operator.enabled in ['on', 'optional'] %} # Intel Ethernet Operator for Intel E810 series ethernet network adapters intel_ethernet_operator: -{% if intel_ethernet_operator.ddp in ['on', 'optional'] %} - ddp_update: {% if intel_ethernet_operator.ddp == 'on' and nic == 'cvl' %}true{% else %}false{% endif %} # Perform DDP update on PFs listed in dataplane_interfaces using selected DDP profile +{% if intel_ethernet_operator.ddp_update in ['on', 'optional'] %} + ddp_update: {% if intel_ethernet_operator.ddp_update == 'on' and nic == 'cvl' %}true{% else %}false{% endif %} # Perform DDP update on PFs listed in dataplane_interfaces using selected DDP profile {% endif %} fw_update: {% if intel_ethernet_operator.fw_update == 'on' and nic == 'cvl' %}true{% else %}false{% endif %} # Perform firmware update on PFs listed in dataplane_interfaces # ClusterFlowConfig does not require additional configuration and can be used in conjunction with NodeFlowConfig @@ -504,8 +512,12 @@ exclude_collectd_plugins: [] ###################### {% if power.manager in ['on', 'optional'] and arch in ['icx', 'clx', 'spr', 'emr'] %} # The performance profile is available for nodes that has CPU max MHz > 3500.0000 - use 'lscpu' command to see your node details -# To use PowerProfiles in this list as sample pods on this node, please set 'deploy_example_pods' to true in group_vars -power_profiles: [balance-performance] # Possible PowerProfiles are: [performance, balance-performance, balance-power] +# To use PowerProfiles in this list as sample pods on this node, please set 'deploy_example_pods' to true in group_vars +power_profiles: [balance-performance] # Possible PowerProfiles are: [performance, balance-performance, balance-power] + +{% if power.frequency_scaling in ['on', 'optional'] and arch in ['icx', 'clx', 'spr', 'emr'] %} +frequency_scaling_driver: intel_pstate # Possible values: [intel_pstate, acpi_cpufreq] +{% endif %} # Power Manager Shared Profile/Workload settings. # It is possible to create node-specific Power Profile @@ -514,22 +526,24 @@ local_shared_profile: local_max_frequency: 2000 local_min_frequency: 1500 -{% if power.pstate in ['on', 'optional'] and arch in ['icx', 'clx', 'spr', 'emr'] %} - # P-State governor decides what frequency within the CPUfreq policy should be used - # "powersave" - Lowest frequency within the borders of min_freq and max_freq. - # "performance" - Highest frequency within the borders of min_freq and max_freq. - local_pstate_governor: "powersave" +{% if power.frequency_scaling in ['on', 'optional'] and arch in ['icx', 'clx', 'spr', 'emr'] %} + # available governors: + # "powersave" - Lowest frequency within the borders of min_frequency and max_frequency. + # "performance" - Highest frequency within the borders of min_frequency and max_frequency. + # "userspace" - !ACPI ONLY! - Allow user space to set CPU frequency in scaling_setspeed attribute + # "schedutil" - !ACPI ONLY! - Uses data from CPU scheduler to set up frequency + local_governor: "powersave" {% endif %} # Shared Workload is required to make use of Shared Power Profile shared_workload: - enabled: true # Enable/Disable shared workload + enabled: {% if name == "on_prem_sw_defined_factory" %}false{% else %}true{% endif %} # Enable/Disable shared workload reserved_cpus: [] # The CPUs in reserved_cpus should match the value of the reserved system CPUs in your Kubelet config file, if none please # set a dummy core here - the last one to avoid AppQos bug. shared_workload_type: "global" # Set to node name to make use of node-specific Power Profile, 'global' means use cluster-specific custom Power Profile -# EMR uncore_frequency has not supported in the kernel driver yet. {% if power.uncore_frequency in ['on', 'optional'] and arch in ['icx', 'clx', 'spr'] %} +# EMR uncore_frequency has not supported in the kernel driver yet. uncore_frequency: enabled: {% if power.uncore_frequency == "on" %}true{% else %}false{% endif %} # Enable/Disable uncore frequency @@ -556,7 +570,7 @@ cstates: profile_exclusive: balance-performance: C1: false - + # If needed, you can choose specific C-State for each core: core: {} # core: @@ -565,24 +579,13 @@ cstates: # C6: false {% endif %} -{% endif %} -{% if (power.pstate in ['on', 'optional'] or sst in ['on', 'optional']) and arch in ['icx', 'clx', 'spr', 'emr'] %} -# Enable/Disable Intel PState scaling driver -intel_pstate_enabled: {% if power.pstate == "on" or sst == "on" %}true{% else %}false{% endif %} - -intel_pstate: {% if power.pstate == "on" or sst == "on" %}hwp_only{% else %}disable{% endif %} # Supported values: [disable, passive, force, no_hwp, hwp_only, support_acpi_ppc, per_cpu_perf_limits] - -# Enable/Disable Intel Turbo Boost PState attribute -turbo_boost_enabled: {% if on_vms != 'on' %}true{% else %}false{% endif %} - - {% endif %} {% if sst in ['on', 'optional'] %} {% if arch in ['icx', 'spr', 'emr'] %} # Intel(R) SST-PP (perf-profile) configuration sst_pp_configuration_enabled: {% if sst == "on" %}true{% else %}false{% endif %} -sst_pp_config_list: +sst_pp_config_list: - sst_bf: "enable" # enable/disable Intel(R) SST-BF (base-freq) configured through SST-PP. - sst_cp: "enable" # enable/disable Intel(R) SST-CP (core-power) configured through SST-PP. - sst_tf: "enable" # enable/disable Intel(R) SST-TF (turbo-freq) configured through SST-PP. @@ -655,10 +658,16 @@ adq_dp: enabled: false # IP address of CVL interface located on the worker node interface_address: "192.168.0.11" +{% endif %} + +{% if intel_csl_excat in ['on', 'optional'] and vm_mode != "on" %} +# please mark the node to support excat dp by change the value to true +excat_dp_enabled: false +{% endif %} -{% if intel_eci and (intel_eci.values() | reject('eq', 'off')) | list | length() > 0 %} +{% if intel_eci %} # Intel ECI (Edge Controls for Industrial) -intel_eci_enabled: {% if (intel_eci.values() | select('eq', 'on')) | list | length() > 0 %}true{% else %}false{% endif %} # if true, deploy Intel ECI +intel_eci_enabled: {% if intel_eci.enable == "on" %}true{% else %}false{% endif %} # if true, deploy Intel ECI intel_eci: eci-process-automation: {% if intel_eci.process_automation == 'on' %}true{% else %}false{% endif %} @@ -666,17 +675,6 @@ intel_eci: eci-discrete-manufacturing: {% if intel_eci.discrete_manufacturing == 'on' %}true{% else %}false{% endif %} - eci-realtime: {% if intel_eci.realtime == 'on' %}true{% else %}false{% endif %} - - eci-connectivity: {% if intel_eci.connectivity == 'on' %}true{% else %}false{% endif %} - - eci-infra-clients: {% if intel_eci.infra_clients == 'on' %}true{% else %}false{% endif %} - - eci-inference: {% if intel_eci.inference == 'on' %}true{% else %}false{% endif %} - - eci-softplc: {% if intel_eci.softplc == 'on' %}true{% else %}false{% endif %} - - eci-acrn: {% if intel_eci.acrn == 'on' %}true{% else %}false{% endif %} # The following ECI meta-packages aren't included becuase they are NOT suported yet on Ubuntu (as of 05/2023): # eci-robotics-control @@ -690,9 +688,18 @@ opcua_framework: standalone_opcua_server: {% if opcua_framework.standalone_opcua_server == 'on' %}true{% else %}false{% endif %} -{% endif %} +{% if intel_eci.enable == 'on' %} + +cat_enable: true +# Change this CAT configuration according to your CPU architecture +cat_define: "llc:0=0x0f;llc:1=0xf0" +cat_affinity: "llc:0=0;llc:1=1,3" +ethercat_mac: "" + +{% endif %} {% endif %} + {% if vm_mode in ['on'] and on_vms != 'on' %} ######################## ## VMRA Configuration ## @@ -729,7 +736,11 @@ vm_hashed_passwd: 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' vxlan_physical_network: "11.0.0.0/8" #cpu_host_os will change number of CPUs reserved for host OS. Default value is 16 +{% if name == "on_prem_sw_defined_factory" %} +cpu_host_os: 2 +{% else %} #cpu_host_os: 8 +{% endif %} # VM cluster name is used to group all VMs from single deployment together #vm_cluster_name: "cluster1.local" @@ -792,7 +803,7 @@ vms: - "18:02.4" - "18:02.5" {% if qat == "on" %} - - "3d:01.1" # 3x:xx.x are example VFs for QAT + - "3d:01.1" # 3x:xx.x are example VFs for QAT - "3f:01.1" {% endif %} {% else %} @@ -820,5 +831,28 @@ vms: {% else %} # pci: [] {% endif %} +# - type: "vm" +{% if secondary_host == 'true' %} +# name: "vm-2" +{% else %} +# name: "vm-1" +{% endif %} +# cpu_total: 4 +# memory: 61440 +# vxlan: 120 +{% if name not in ['build_your_own'] %} +# pci: +# - "18:02.0" # 18:xx.x are example VFs for networking +# - "18:02.1" +# - "18:02.6" +# - "18:02.7" +{% if qat == "on" %} +# - "3d:01.2" # 3x:xx.x are example VFs for QAT +# - "3f:01.2" +{% endif %} +{% else %} +# pci: [] +{% endif %} + {% endif %} diff --git a/generate/profiles_templates/k8s/profiles.yml b/generate/profiles_templates/k8s/profiles.yml index 81503547..74b7a1e6 100644 --- a/generate/profiles_templates/k8s/profiles.yml +++ b/generate/profiles_templates/k8s/profiles.yml @@ -16,7 +16,6 @@ # - intel_cpu_controlplane # - native_cpu_manager # - bond_cni -# - topology_manager # - sriov_operator # - sriov_network_dp # - nic_drivers @@ -40,14 +39,14 @@ # - openssl # - tas # - gas -# - ddp +# - ddp_legacy # - network_userspace # - dpdk # - ovs_dpdk # - sst # - power: # manager -# pstate +# frequency_scaling # cstate # uncore_frequency # - telemetry: @@ -58,6 +57,7 @@ # opentelemetry # elasticsearch # kibana +# intel_xpumanager # - wireguard # - multus # - minio @@ -79,7 +79,7 @@ # - intel_ethernet_operator # enabled # flow_config -# ddp +# ddp_update # fw_update # - intel_sriov_fec_operator # - intel_oneapi @@ -100,10 +100,16 @@ # - opcua_framework # codesys_opcua_client # standalone_opcua_server +# - intel_csl_excat # - tadk # - sigstore_policy_controller # - cadvisor # - fpga +# - tdx +# - calico_vpp +# - ido +# - imtl + access: name: access vm_mode: optional @@ -114,7 +120,6 @@ access: isolcpu: optional cpusets: optional native_cpu_manager: on - topology_manager: on sriov_operator: off sriov_network_dp: on nic_drivers: on @@ -122,8 +127,8 @@ access: qat: optional qat_dp: optional openssl: on - dsa: on - dsa_dp: on + dsa: optional + dsa_dp: optional dlb: optional dlb_dp: optional gpu: off @@ -140,14 +145,14 @@ access: tac: off tas: off gas: off - ddp: off + ddp_legacy: off network_userspace: off dpdk: on ovs_dpdk: off sst: off power: manager: off - pstate: off + frequency_scaling: off cstate: off uncore_frequency: off telemetry: @@ -158,6 +163,7 @@ access: opentelemetry: on elasticsearch: on kibana: on + intel_xpumanager: off istio_service_mesh: enabled: off tcpip_bypass_ebpf: off @@ -180,7 +186,7 @@ access: intel_ethernet_operator: enabled: optional flow_config: optional - ddp: optional + ddp_update: optional fw_update: optional intel_sriov_fec_operator: on intel_oneapi: @@ -191,6 +197,10 @@ access: adq_dp: optional sigstore_policy_controller: on cadvisor: on + tdx: off + calico_vpp: optional + imtl: optional + base_container: off basic: name: basic @@ -201,14 +211,13 @@ basic: rancher_manager: optional isolcpu: optional cpusets: optional - topology_manager: on sriov_operator: optional sriov_network_dp: optional nic_drivers: on dpdk: optional power: manager: optional - pstate: optional + frequency_scaling: optional cstate: optional uncore_frequency: optional telemetry: @@ -219,6 +228,7 @@ basic: opentelemetry: on elasticsearch: on kibana: on + intel_xpumanager: off wireguard: optional multus: on firewall: optional @@ -240,6 +250,10 @@ basic: base: optional ai: optional cadvisor: on + tdx: off + calico_vpp: optional + imtl: optional + base_container: off full_nfv: name: full_nfv @@ -252,7 +266,6 @@ full_nfv: cpusets: optional intel_cpu_controlplane: optional native_cpu_manager: on - topology_manager: on sriov_operator: on sriov_network_dp: optional nic_drivers: on @@ -278,14 +291,14 @@ full_nfv: tac: on tas: on gas: optional - ddp: on + ddp_legacy: on network_userspace: on dpdk: on ovs_dpdk: on sst: optional power: manager: on - pstate: on + frequency_scaling: on cstate: on uncore_frequency: on telemetry: @@ -296,6 +309,7 @@ full_nfv: opentelemetry: on elasticsearch: on kibana: on + intel_xpumanager: optional istio_service_mesh: enabled: on tcpip_bypass_ebpf: on @@ -318,7 +332,7 @@ full_nfv: intel_ethernet_operator: enabled: on flow_config: optional - ddp: on + ddp_update: on fw_update: optional intel_sriov_fec_operator: optional tadk: on @@ -329,6 +343,11 @@ full_nfv: ai: optional cadvisor: on fpga: optional + tdx: off + calico_vpp: optional + ido: optional + imtl: optional + base_container: off on_prem: name: on_prem @@ -340,7 +359,6 @@ on_prem: isolcpu: optional cpusets: optional native_cpu_manager: on - topology_manager: on sriov_operator: on sriov_network_dp: optional nic_drivers: on @@ -370,7 +388,7 @@ on_prem: sst: optional power: manager: optional - pstate: optional + frequency_scaling: optional cstate: optional uncore_frequency: optional telemetry: @@ -381,6 +399,7 @@ on_prem: opentelemetry: on elasticsearch: on kibana: on + intel_xpumanager: optional istio_service_mesh: enabled: on tcpip_bypass_ebpf: on @@ -411,6 +430,11 @@ on_prem: ai: optional cadvisor: on fpga: optional + tdx: optional + calico_vpp: optional + ido: optional + imtl: optional + base_container: off on_prem_vss: name: on_prem_vss @@ -421,7 +445,6 @@ on_prem_vss: isolcpu: on cpusets: optional native_cpu_manager: optional - topology_manager: optional sriov_operator: on sriov_network_dp: optional nic_drivers: on @@ -447,14 +470,14 @@ on_prem_vss: tac: optional tas: on gas: on - ddp: optional + ddp_legacy: optional network_userspace: optional dpdk: on ovs_dpdk: optional sst: optional power: manager: optional - pstate: optional + frequency_scaling: optional cstate: optional uncore_frequency: optional telemetry: @@ -465,6 +488,7 @@ on_prem_vss: opentelemetry: optional elasticsearch: optional kibana: optional + intel_xpumanager: optional istio_service_mesh: enabled: on tcpip_bypass_ebpf: optional @@ -486,13 +510,17 @@ on_prem_vss: intel_ethernet_operator: enabled: optional flow_config: optional - ddp: optional + ddp_update: optional fw_update: optional adq_dp: optional sigstore_policy_controller: optional intel_oneapi: base: optional ai: optional + tdx: off + calico_vpp: optional + imtl: optional + base_container: off on_prem_sw_defined_factory: name: on_prem_sw_defined_factory @@ -505,7 +533,6 @@ on_prem_sw_defined_factory: cpusets: optional native_cpu_manager: optional intel_cpu_controlplane: optional - topology_manager: optional sriov_operator: optional sriov_network_dp: optional nic_drivers: optional @@ -531,14 +558,14 @@ on_prem_sw_defined_factory: tac: optional tas: optional gas: optional - ddp: optional + ddp_legacy: optional network_userspace: optional dpdk: optional ovs_dpdk: optional sst: optional power: manager: optional - pstate: optional + frequency_scaling: optional cstate: optional uncore_frequency: optional telemetry: @@ -549,6 +576,7 @@ on_prem_sw_defined_factory: opentelemetry: optional elasticsearch: optional kibana: optional + intel_xpumanager: optional istio_service_mesh: enabled: optional tcpip_bypass_ebpf: optional @@ -570,7 +598,7 @@ on_prem_sw_defined_factory: intel_ethernet_operator: enabled: optional flow_config: optional - ddp: optional + ddp_update: optional fw_update: optional intel_sriov_fec_operator: optional rt_kernel: optional @@ -588,10 +616,15 @@ on_prem_sw_defined_factory: opcua_framework: codesys_opcua_client: on standalone_opcua_server: optional + intel_csl_excat: optional tadk: optional adq_dp: optional sigstore_policy_controller: optional cadvisor: optional + tdx: off + calico_vpp: optional + imtl: optional + base_container: off on_prem_aibox: name: on_prem_aibox @@ -604,7 +637,6 @@ on_prem_aibox: cpusets: off native_cpu_manager: off intel_cpu_controlplane: off - topology_manager: off sriov_operator: off sriov_network_dp: off nic_drivers: off @@ -630,7 +662,7 @@ on_prem_aibox: tac: off tas: off gas: off - ddp: off + ddp_legacy: off network_userspace: off dpdk: off ovs_dpdk: off @@ -641,13 +673,14 @@ on_prem_aibox: cstate: off uncore_frequency: off telemetry: - prometheus: off + prometheus: on collectd: off telegraf: off jaeger: off opentelemetry: off elasticsearch: off kibana: off + intel_xpumanager: on istio_service_mesh: enabled: off tcpip_bypass_ebpf: off @@ -669,28 +702,19 @@ on_prem_aibox: intel_ethernet_operator: enabled: off flow_config: off - ddp: off + ddp_update: off fw_update: off intel_sriov_fec_operator: off rt_kernel: off intel_flexran: off - intel_eci: - process_automation: off - manufacturing_equipment: off - discrete_manufacturing: off - realtime: off - connectivity: off - softplc: off - infra_clients: off - inference: off - acrn: off - opcua_framework: - codesys_opcua_client: off - standalone_opcua_server: off tadk: off adq_dp: off sigstore_policy_controller: off cadvisor: off + tdx: off + calico_vpp: off + imtl: optional + base_container: on regional_dc: name: regional_dc @@ -701,7 +725,6 @@ regional_dc: rancher_manager: optional isolcpu: optional cpusets: optional - topology_manager: on sriov_operator: optional sriov_network_dp: optional nic_drivers: on @@ -723,7 +746,7 @@ regional_dc: dpdk: optional power: manager: optional - pstate: optional + frequency_scaling: optional cstate: optional uncore_frequency: optional telemetry: @@ -734,6 +757,7 @@ regional_dc: opentelemetry: on elasticsearch: on kibana: on + intel_xpumanager: optional istio_service_mesh: enabled: on tcpip_bypass_ebpf: on @@ -764,6 +788,11 @@ regional_dc: ai: optional cadvisor: on fpga: optional + tdx: off + calico_vpp: optional + ido: optional + imtl: optional + base_container: off remote_fp: name: remote_fp @@ -776,7 +805,6 @@ remote_fp: cpusets: optional intel_cpu_controlplane: optional native_cpu_manager: on - topology_manager: on sriov_operator: on sriov_network_dp: optional nic_drivers: on @@ -796,14 +824,14 @@ remote_fp: dlb_dp: optional openssl: on tas: on - ddp: on + ddp_legacy: on bond_cni: optional network_userspace: optional dpdk: on sst: optional power: manager: optional - pstate: on + frequency_scaling: on cstate: optional uncore_frequency: optional telemetry: @@ -814,6 +842,7 @@ remote_fp: opentelemetry: optional elasticsearch: optional kibana: optional + intel_xpumanager: off istio_service_mesh: enabled: optional tcpip_bypass_ebpf: optional @@ -836,7 +865,7 @@ remote_fp: intel_ethernet_operator: enabled: on flow_config: optional - ddp: on + ddp_update: on fw_update: optional adq_dp: optional sigstore_policy_controller: optional @@ -844,6 +873,11 @@ remote_fp: base: optional ai: optional cadvisor: optional + tdx: off + calico_vpp: optional + ido: optional + imtl: optional + base_container: off build_your_own: name: build_your_own @@ -856,7 +890,6 @@ build_your_own: cpusets: optional native_cpu_manager: optional intel_cpu_controlplane: optional - topology_manager: optional sriov_operator: optional sriov_network_dp: optional nic_drivers: optional @@ -882,14 +915,14 @@ build_your_own: tac: optional tas: optional gas: optional - ddp: optional + ddp_legacy: optional network_userspace: optional dpdk: optional ovs_dpdk: optional sst: optional power: manager: optional - pstate: optional + frequency_scaling: optional cstate: optional uncore_frequency: optional telemetry: @@ -900,6 +933,7 @@ build_your_own: opentelemetry: optional elasticsearch: optional kibana: optional + intel_xpumanager: optional istio_service_mesh: enabled: optional tcpip_bypass_ebpf: optional @@ -922,7 +956,7 @@ build_your_own: intel_ethernet_operator: enabled: optional flow_config: optional - ddp: optional + ddp_update: optional fw_update: optional intel_sriov_fec_operator: optional intel_oneapi: @@ -943,8 +977,14 @@ build_your_own: opcua_framework: codesys_opcua_client: optional standalone_opcua_server: optional + intel_csl_excat: optional tadk: optional adq_dp: optional sigstore_policy_controller: optional cadvisor: optional fpga: optional + tdx: off + calico_vpp: optional + ido: optional + imtl: optional + base_container: off diff --git a/generate/profiles_templates/vm/inventory.j2 b/generate/profiles_templates/vm/inventory.j2 index 84ed2e88..28bc4153 100644 --- a/generate/profiles_templates/vm/inventory.j2 +++ b/generate/profiles_templates/vm/inventory.j2 @@ -5,6 +5,10 @@ localhost ansible_connection=local ansible_python_interpreter=/usr/bin/python3 [vm_host] host-for-vms-1 +# for generic VMs that are not part of k8s cluster +[vms] +#vm-1 + [kube_control_plane] #vm-ctrl-1 diff --git a/generate/profiles_templates/vm/vm_host_profiles.yml b/generate/profiles_templates/vm/vm_host_profiles.yml index 4844c2b7..a99d95d3 100644 --- a/generate/profiles_templates/vm/vm_host_profiles.yml +++ b/generate/profiles_templates/vm/vm_host_profiles.yml @@ -14,7 +14,6 @@ # - cpusets # - native_cpu_manager # - bond_cni -# - topology_manager # - sriov_operator # - sriov_network_dp # - nic_drivers @@ -34,13 +33,13 @@ # - openssl # - tas # - gas -# - ddp +# - ddp_legacy # - network_userspace # - dpdk # - ovs_dpdk # - power: # manager -# pstate +# frequency_scaling # cstate # uncore_frequency # - sst @@ -52,6 +51,7 @@ # opentelemetry # elasticsearch # kibana +# intel_xpumanager # - wireguard # - multus # - cert_manager @@ -68,13 +68,15 @@ # - intel_ethernet_operator # enabled # flow_config -# ddp +# ddp_update # fw_update # - sigstore_policy_controller # - intel_oneapi # base # ai # - cadvisor +# - tdx +# - imtl # sriov_operator is permanently disabled in VM mode # sriov_network_dp and dpdk are enabled for all VM mode profiles except build_your_own @@ -92,7 +94,6 @@ access: isolcpu: optional cpusets: optional native_cpu_manager: on - topology_manager: on sriov_operator: optional sriov_network_dp: on nic_drivers: on @@ -102,7 +103,7 @@ access: dpdk: on power: manager: off - pstate: off + frequency_scaling: off cstate: off uncore_frequency: off telemetry: @@ -113,6 +114,7 @@ access: opentelemetry: on elasticsearch: on kibana: on + intel_xpumanager: off istio_service_mesh: enabled: off tcpip_bypass_ebpf: off @@ -136,6 +138,8 @@ access: base: optional ai: optional cadvisor: on + tdx: off + imtl: optional basic: name: basic @@ -145,14 +149,13 @@ basic: kube_dashboard: on isolcpu: optional cpusets: optional - topology_manager: on sriov_operator: optional sriov_network_dp: on nic_drivers: on dpdk: on power: manager: optional - pstate: optional + frequency_scaling: optional cstate: optional uncore_frequency: optional telemetry: @@ -163,6 +166,7 @@ basic: opentelemetry: on elasticsearch: on kibana: on + intel_xpumanager: off wireguard: on multus: on firewall: optional @@ -178,6 +182,8 @@ basic: base: optional ai: optional cadvisor: on + tdx: optional + imtl: optional full_nfv: name: full_nfv @@ -188,7 +194,6 @@ full_nfv: isolcpu: optional cpusets: optional native_cpu_manager: on - topology_manager: on sriov_operator: optional sriov_network_dp: on nic_drivers: on @@ -210,13 +215,13 @@ full_nfv: tac: on tas: on gas: optional - ddp: on + ddp_legacy: on network_userspace: on dpdk: on ovs_dpdk: on power: manager: optional - pstate: optional + frequency_scaling: optional cstate: optional uncore_frequency: optional sst: optional @@ -228,6 +233,7 @@ full_nfv: opentelemetry: on elasticsearch: on kibana: on + intel_xpumanager: optional istio_service_mesh: enabled: on tcpip_bypass_ebpf: on @@ -245,13 +251,15 @@ full_nfv: intel_ethernet_operator: enabled: optional flow_config: optional - ddp: optional + ddp_update: optional fw_update: optional sigstore_policy_controller: optional intel_oneapi: base: optional ai: optional cadvisor: on + tdx: off + imtl: optional on_prem: name: on_prem @@ -262,7 +270,6 @@ on_prem: isolcpu: optional cpusets: optional native_cpu_manager: on - topology_manager: on sriov_operator: optional sriov_network_dp: on nic_drivers: on @@ -284,7 +291,7 @@ on_prem: bond_cni: optional power: manager: optional - pstate: optional + frequency_scaling: optional cstate: optional uncore_frequency: optional sst: optional @@ -296,6 +303,7 @@ on_prem: opentelemetry: on elasticsearch: on kibana: on + intel_xpumanager: off istio_service_mesh: enabled: on tcpip_bypass_ebpf: on @@ -319,6 +327,85 @@ on_prem: base: optional ai: optional cadvisor: on + tdx: off + imtl: optional + +on_prem_sw_defined_factory: + name: on_prem_sw_defined_factory + vm_mode: on + on_vms: optional + nfd: optional + kube_dashboard: optional + isolcpu: optional + cpusets: optional + native_cpu_manager: optional + sriov_operator: optional + sriov_network_dp: optional + nic_drivers: optional + sgx: optional + sgx_dp: optional + kmra: + sbx: optional + oran: optional + pccs: optional + apphsm: optional + ctk_demo: optional + tcs: optional + tac: optional + qat: optional + qat_dp: optional + openssl: optional + tas: optional + dpdk: optional + bond_cni: optional + power: + manager: optional + frequency_scaling: optional + cstate: optional + uncore_frequency: optional + sst: optional + telemetry: + prometheus: optional + collectd: optional + telegraf: optional + jaeger: optional + opentelemetry: optional + elasticsearch: optional + kibana: optional + intel_xpumanager: off + istio_service_mesh: + enabled: optional + tcpip_bypass_ebpf: optional + tls_splicing: optional + sgx_signer: optional + intel_preview: optional + linkerd_service_mesh: + enabled: optional + wireguard: optional + multus: optional + firewall: optional + cert_manager: optional + registry: optional + hugepages: optional + intel_eci: + enable: on + process_automation: optional + manufacturing_equipment: optional + discrete_manufacturing: optional + opcua_framework: + codesys_opcua_client: optional + standalone_opcua_server: optional + intel_ethernet_operator: + enabled: optional + flow_config: optional + fw_update: optional + sigstore_policy_controller: optional + intel_oneapi: + base: optional + ai: optional + cadvisor: optional + tdx: off + imtl: optional regional_dc: name: regional_dc @@ -328,7 +415,6 @@ regional_dc: kube_dashboard: on isolcpu: optional cpusets: optional - topology_manager: on sriov_operator: optional sriov_network_dp: on nic_drivers: on @@ -350,7 +436,7 @@ regional_dc: dpdk: on power: manager: optional - pstate: optional + frequency_scaling: optional cstate: optional uncore_frequency: optional telemetry: @@ -361,6 +447,7 @@ regional_dc: opentelemetry: on elasticsearch: on kibana: on + intel_xpumanager: optional istio_service_mesh: enabled: on tcpip_bypass_ebpf: on @@ -384,6 +471,8 @@ regional_dc: base: optional ai: optional cadvisor: on + tdx: off + imtl: optional remote_fp: name: remote_fp @@ -394,7 +483,6 @@ remote_fp: isolcpu: optional cpusets: optional native_cpu_manager: on - topology_manager: on sriov_operator: optional sriov_network_dp: on nic_drivers: on @@ -412,13 +500,13 @@ remote_fp: qat_dp: on openssl: on tas: on - ddp: on + ddp_legacy: on bond_cni: optional network_userspace: optional dpdk: on power: manager: optional - pstate: optional + frequency_scaling: optional cstate: optional uncore_frequency: optional sst: optional @@ -430,6 +518,7 @@ remote_fp: opentelemetry: optional elasticsearch: optional kibana: optional + intel_xpumanager: off istio_service_mesh: enabled: optional tcpip_bypass_ebpf: optional @@ -447,13 +536,15 @@ remote_fp: intel_ethernet_operator: enabled: optional flow_config: optional - ddp: optional + ddp_update: optional fw_update: optional sigstore_policy_controller: optional intel_oneapi: base: optional ai: optional cadvisor: optional + tdx: off + imtl: optional build_your_own: name: build_your_own @@ -464,7 +555,6 @@ build_your_own: isolcpu: optional cpusets: optional native_cpu_manager: optional - topology_manager: optional sriov_operator: optional sriov_network_dp: optional nic_drivers: optional @@ -486,13 +576,13 @@ build_your_own: tac: optional tas: optional gas: optional - ddp: optional + ddp_legacy: optional network_userspace: optional dpdk: optional ovs_dpdk: optional power: manager: optional - pstate: optional + frequency_scaling: optional cstate: optional uncore_frequency: optional sst: optional @@ -504,6 +594,7 @@ build_your_own: opentelemetry: optional elasticsearch: optional kibana: optional + intel_xpumanager: optional istio_service_mesh: enabled: optional tcpip_bypass_ebpf: optional @@ -521,10 +612,12 @@ build_your_own: intel_ethernet_operator: enabled: optional flow_config: optional - ddp: optional + ddp_update: optional fw_update: optional sigstore_policy_controller: optional intel_oneapi: base: optional ai: optional cadvisor: optional + tdx: off + imtl: optional diff --git a/generate/profiles_templates/vm/vms_profiles.yml b/generate/profiles_templates/vm/vms_profiles.yml index 8c2739d51..a1873181 100644 --- a/generate/profiles_templates/vm/vms_profiles.yml +++ b/generate/profiles_templates/vm/vms_profiles.yml @@ -14,7 +14,6 @@ # - cpusets # - native_cpu_manager # - bond_cni -# - topology_manager # - sriov_operator # - sriov_network_dp # - nic_drivers @@ -34,14 +33,14 @@ # - openssl # - tas # - gas -# - ddp +# - ddp_legacy # - network_userspace # - dpdk # - ovs_dpdk # - sst # - power: # manager -# pstate +# frequency_scaling # cstate # uncore_frequency # - telemetry: @@ -52,6 +51,7 @@ # opentelemetry # elasticsearch # kibana +# intel_xpumanager # - wireguard # - multus # - cert_manager @@ -68,13 +68,15 @@ # - intel_ethernet_operator # enabled # flow_config -# ddp +# ddp_update # fw_update # - sigstore_policy_controller # - intel_oneapi # base # ai # - cadvisor +# - tdx +# - imtl # sriov_operator is permanently disabled in VM mode # sriov_network_dp and dpdk are enabled for all VM mode profiles except build_your_own @@ -98,7 +100,6 @@ access: isolcpu: optional cpusets: optional native_cpu_manager: on - topology_manager: on sriov_operator: optional sriov_network_dp: on nic_drivers: on @@ -108,7 +109,7 @@ access: dpdk: on power: manager: off - pstate: off + frequency_scaling: off cstate: off uncore_frequency: off telemetry: @@ -119,6 +120,7 @@ access: opentelemetry: on elasticsearch: on kibana: on + intel_xpumanager: off istio_service_mesh: enabled: off tcpip_bypass_ebpf: off @@ -142,6 +144,8 @@ access: base: optional ai: optional cadvisor: on + tdx: off + imtl: optional basic: name: basic @@ -151,14 +155,13 @@ basic: kube_dashboard: on isolcpu: optional cpusets: optional - topology_manager: on sriov_operator: optional sriov_network_dp: on nic_drivers: on dpdk: on power: manager: optional - pstate: optional + frequency_scaling: optional cstate: optional uncore_frequency: optional telemetry: @@ -169,6 +172,7 @@ basic: opentelemetry: on elasticsearch: on kibana: on + intel_xpumanager: off wireguard: on multus: on firewall: optional @@ -184,6 +188,8 @@ basic: base: optional ai: optional cadvisor: on + tdx: optional + imtl: optional full_nfv: name: full_nfv @@ -194,7 +200,6 @@ full_nfv: isolcpu: optional cpusets: optional native_cpu_manager: on - topology_manager: on sriov_operator: optional sriov_network_dp: on nic_drivers: on @@ -216,14 +221,14 @@ full_nfv: tac: on tas: on gas: optional - ddp: optional + ddp_legacy: optional network_userspace: on dpdk: on ovs_dpdk: on sst: optional power: manager: optional - pstate: optional + frequency_scaling: optional cstate: optional uncore_frequency: optional telemetry: @@ -234,6 +239,7 @@ full_nfv: opentelemetry: on elasticsearch: on kibana: on + intel_xpumanager: optional istio_service_mesh: enabled: on tcpip_bypass_ebpf: on @@ -251,13 +257,15 @@ full_nfv: intel_ethernet_operator: enabled: optional flow_config: optional - ddp: optional + ddp_update: optional fw_update: optional sigstore_policy_controller: optional intel_oneapi: base: optional ai: optional cadvisor: on + tdx: off + imtl: optional on_prem: name: on_prem @@ -268,7 +276,6 @@ on_prem: isolcpu: optional cpusets: optional native_cpu_manager: on - topology_manager: on sriov_operator: optional sriov_network_dp: on nic_drivers: on @@ -291,7 +298,7 @@ on_prem: sst: optional power: manager: optional - pstate: optional + frequency_scaling: optional cstate: optional uncore_frequency: optional telemetry: @@ -302,6 +309,7 @@ on_prem: opentelemetry: on elasticsearch: on kibana: on + intel_xpumanager: off istio_service_mesh: enabled: on tcpip_bypass_ebpf: on @@ -325,6 +333,85 @@ on_prem: base: optional ai: optional cadvisor: on + tdx: off + imtl: optional + +on_prem_sw_defined_factory: + name: on_prem_sw_defined_factory + vm_mode: on + on_vms: on + nfd: optional + kube_dashboard: optional + isolcpu: optional + cpusets: optional + native_cpu_manager: optional + sriov_operator: optional + sriov_network_dp: optional + nic_drivers: optional + sgx: optional + sgx_dp: optional + kmra: + sbx: optional + oran: optional + pccs: optional + apphsm: optional + ctk_demo: optional + tcs: optional + tac: optional + qat: optional + qat_dp: optional + openssl: optional + tas: optional + dpdk: optional + bond_cni: optional + sst: optional + power: + manager: optional + pstate: optional + cstate: optional + uncore_frequency: optional + telemetry: + prometheus: optional + collectd: optional + telegraf: optional + jaeger: optional + opentelemetry: optional + elasticsearch: optional + kibana: optional + intel_xpumanager: off + istio_service_mesh: + enabled: optional + tcpip_bypass_ebpf: optional + tls_splicing: optional + sgx_signer: optional + intel_preview: optional + linkerd_service_mesh: + enabled: optional + wireguard: optional + multus: optional + firewall: optional + cert_manager: optional + registry: optional + hugepages: optional + intel_eci: + enable: on + process_automation: optional + manufacturing_equipment: on + discrete_manufacturing: optional + opcua_framework: + codesys_opcua_client: optional + standalone_opcua_server: optional + intel_ethernet_operator: + enabled: optional + flow_config: optional + fw_update: optional + sigstore_policy_controller: optional + intel_oneapi: + base: optional + ai: optional + cadvisor: optional + tdx: off + imtl: optional regional_dc: name: regional_dc @@ -334,7 +421,6 @@ regional_dc: kube_dashboard: on isolcpu: optional cpusets: optional - topology_manager: on sriov_operator: optional sriov_network_dp: on nic_drivers: on @@ -356,7 +442,7 @@ regional_dc: dpdk: on power: manager: optional - pstate: optional + frequency_scaling: optional cstate: optional uncore_frequency: optional telemetry: @@ -367,6 +453,7 @@ regional_dc: opentelemetry: on elasticsearch: on kibana: on + intel_xpumanager: optional istio_service_mesh: enabled: on tcpip_bypass_ebpf: on @@ -390,6 +477,8 @@ regional_dc: base: optional ai: optional cadvisor: on + tdx: off + imtl: optional remote_fp: name: remote_fp @@ -400,7 +489,6 @@ remote_fp: isolcpu: optional cpusets: optional native_cpu_manager: on - topology_manager: on sriov_operator: optional sriov_network_dp: on nic_drivers: on @@ -418,14 +506,14 @@ remote_fp: qat_dp: on openssl: on tas: on - ddp: optional + ddp_legacy: optional bond_cni: optional network_userspace: optional dpdk: on sst: optional power: manager: optional - pstate: optional + frequency_scaling: optional cstate: optional uncore_frequency: optional telemetry: @@ -436,6 +524,7 @@ remote_fp: opentelemetry: optional elasticsearch: optional kibana: optional + intel_xpumanager: off istio_service_mesh: enabled: optional tcpip_bypass_ebpf: optional @@ -453,13 +542,15 @@ remote_fp: intel_ethernet_operator: enabled: optional flow_config: optional - ddp: optional + ddp_update: optional fw_update: optional sigstore_policy_controller: optional intel_oneapi: base: optional ai: optional cadvisor: optional + tdx: off + imtl: optional build_your_own: name: build_your_own @@ -470,7 +561,6 @@ build_your_own: isolcpu: optional cpusets: optional native_cpu_manager: optional - topology_manager: optional sriov_operator: optional sriov_network_dp: optional nic_drivers: optional @@ -492,14 +582,14 @@ build_your_own: tac: optional tas: optional gas: optional - ddp: optional + ddp_legacy: optional network_userspace: optional dpdk: optional ovs_dpdk: optional sst: optional power: manager: optional - pstate: optional + frequency_scaling: optional cstate: optional uncore_frequency: optional telemetry: @@ -510,6 +600,7 @@ build_your_own: opentelemetry: optional elasticsearch: optional kibana: optional + intel_xpumanager: optional istio_service_mesh: enabled: optional tcpip_bypass_ebpf: optional @@ -527,10 +618,12 @@ build_your_own: intel_ethernet_operator: enabled: optional flow_config: optional - ddp: optional + ddp_update: optional fw_update: optional sigstore_policy_controller: optional intel_oneapi: base: optional ai: optional cadvisor: optional + tdx: off + imtl: optional diff --git a/library/check_nic_firmware.py b/library/check_nic_firmware.py index 6243b850..bbe34db3 100644 --- a/library/check_nic_firmware.py +++ b/library/check_nic_firmware.py @@ -48,6 +48,12 @@ required: false default: false type: bool + fail_msg: + description: + - custom fail message when fw version is not sufficient + required: false + default: "" + type: string author: - Jiri Prokes (jirix.prokes@intel.com) @@ -64,6 +70,12 @@ check_nic_firmware: pci_id: "18:00.1" min_fw_version: "5.02" + +- name: check nic firmware version with custom fail message + check_nic_firmware: + pci_id: "18:00.1" + min_fw_version: "5.02" + fail_msg: "FW version not sufficient for feature X, please update FW version to at least 5.02" ''' RETURN = r''' @@ -128,7 +140,8 @@ def run_module(): module_args = dict( pci_id=dict(type='str', required=True), min_fw_version=dict(type='str', required=True), - ddp=dict(type='bool', required=False, default=False) + ddp=dict(type='bool', required=False, default=False), + fail_msg=dict(type='str', required=False) ) encoding = 'utf-8' @@ -165,6 +178,20 @@ def run_module(): nic_name = str(nic_name_result.stdout.rstrip(), encoding) result['interface_name'] = nic_name + fail_msg = module.params['fail_msg'] + if fail_msg is None and module.params['ddp']: + fail_msg = ( + "Current nic firmware version doesn't allow loading of " + "DDP profile. Set 'update_nic_firmware' " + "to 'true' and run deployment again." + ) + elif fail_msg is None: + fail_msg = ( + "Current nic firmware version is lower than minimum " + "version needed for automatic firmware update. " + "Update nic firmware manually and run deployment again." + ) + # Bandit: # Issue: [B607:start_process_with_partial_path] Starting a process with a partial executable path. # pylint: disable=line-too-long # Severity: Low Confidence: High @@ -186,15 +213,7 @@ def run_module(): if b'firmware-version' in line: result['current_firmware_version'] = str(line.rstrip().split()[1], encoding) if float(result['current_firmware_version']) < float(module.params['min_fw_version']): - if module.params['ddp']: - module.fail_json(msg=("Current nic firmware version doesn't allow loading of " - "DDP profile. Set 'update_nic_firmware' " - "to 'true' and run deployment again."), **result) - else: - module.fail_json(msg=("Current nic firmware version is lower than minimum " - "version needed for automatic firmware update. " - "Update nic firmware manually and run deployment again."), - **result) + module.fail_json(msg=(fail_msg), **result) else: result['msg'] = "nic firmware version is sufficient to proceed" module.exit_json(**result) diff --git a/playbooks/dockerfiles.yml b/playbooks/dockerfiles.yml index f08fe510..e3141cd4 100644 --- a/playbooks/dockerfiles.yml +++ b/playbooks/dockerfiles.yml @@ -36,6 +36,7 @@ base_container_sudo: false profile_name: "on_prem_aibox" prc_network: false + gpu_type: "Arc" ansible.builtin.include_role: name: intel_base_container tasks_from: main diff --git a/playbooks/dyna_config.yml b/playbooks/dyna_config.yml index 803f929d..bfa32bb3 100644 --- a/playbooks/dyna_config.yml +++ b/playbooks/dyna_config.yml @@ -17,8 +17,21 @@ # This playbook contains those tasks can be executed separatedly after # RA installation, to dynamically adjust some system configurations. -# Preflight check for config dpdk link operaton +# Execute config dpdk role bind/unbind tasks on worker nodes - hosts: "{{node | default('kube_node')}}" + roles: + - role: configure_dpdk + dpdk_link_node1: "{{ groups.kube_node[0] }}" + dpdk_link_node2: "{{ groups.kube_node[1] }}" + tags: + - dyna_config_dpdk + when: + - dyna_config_dpdk_bind | default(false) | bool or + dyna_config_dpdk_unbind | default(false) | bool + + +# Pre-flight check for link operation +- hosts: localhost tasks: - name: check config dpdk link nodes count vars: @@ -31,20 +44,64 @@ when: - dyna_config_dpdk_link | default(false) | bool -# Execute config dpdk role tasks according to tag and conditions +# Execute config dpdk role pre-link tasks on worker nodes - hosts: "{{node | default('kube_node')}}" roles: - role: configure_dpdk dpdk_link_node1: "{{ groups.kube_node[0] }}" dpdk_link_node2: "{{ groups.kube_node[1] }}" + dpdk_link_pre: true tags: - dyna_config_dpdk when: - - dyna_config_dpdk_bind | default(false) | bool or - dyna_config_dpdk_link | default(false) | bool or - dyna_config_dpdk_unbind | default(false) | bool + - dyna_config_dpdk_link | default(false) | bool + +# Execute config dpdk role link tasks on local host +- hosts: localhost + roles: + - role: configure_dpdk + dpdk_link_node1: "{{ groups.kube_node[0] }}" + dpdk_link_node2: "{{ groups.kube_node[1] }}" + tags: + - dyna_config_dpdk + when: + - dyna_config_dpdk_link | default(false) | bool +# Execute config dpdk role post-link tasks on worker nodes +- hosts: "{{node | default('kube_node')}}" + roles: + - role: configure_dpdk + dpdk_link_node1: "{{ groups.kube_node[0] }}" + dpdk_link_node2: "{{ groups.kube_node[1] }}" + dpdk_link_post: true + tags: + - dyna_config_dpdk + when: + - dyna_config_dpdk_link | default(false) | bool + +# Execute gpu driver role on worker nodes +- hosts: "{{node | default('kube_node')}}" + roles: + - role: install_gpu_driver + tags: + - install_gpu_driver + become: true + +# Execute base container role on worker nodes +- hosts: "{{node | default('kube_node')}}" + roles: - role: intel_base_container profile_name: "{{ container_set | default('on_prem_aibox') }}" + gpu_type: "Arc" tags: - base_container + +- hosts: "{{node | default('kube_node')}}" + roles: + - role: prometheus_install + tags: + - prometheus + + - role: intel_xpumanager + tags: + - xpumanager diff --git a/playbooks/infra/prepare_ipu.yml b/playbooks/infra/prepare_ipu.yml index afb5b071..308e7ab2 100644 --- a/playbooks/infra/prepare_ipu.yml +++ b/playbooks/infra/prepare_ipu.yml @@ -40,8 +40,10 @@ - hosts: ipu_imc roles: + - role: ipu/common - role: ipu/imc - hosts: ipu_acc roles: + - role: ipu/common - role: ipu/acc diff --git a/playbooks/infra/prepare_vms.yml b/playbooks/infra/prepare_vms.yml index 9bf4d790..c479b6aa 100644 --- a/playbooks/infra/prepare_vms.yml +++ b/playbooks/infra/prepare_vms.yml @@ -14,6 +14,15 @@ ## limitations under the License. ## --- +# add those bm host in mixed cluster to a group +- hosts: k8s_cluster + tasks: + - name: Add hosts to inventory - bm_host + ansible.builtin.add_host: + hostname: "{{ inventory_hostname }}" + groups: "bm_host" + inventory_dir: '{{ inventory_dir }}' + - hosts: vm_host roles: - role: vm/compile_libvirt @@ -24,6 +33,20 @@ environment: "{{ proxy_env | d({}) }}" any_errors_fatal: true +# prepare vxlan unicast peers +- hosts: vm_host,k8s_cluster + gather_facts: false + tasks: + - include_role: + name: vm/manage_bridges # noqa role-name[path] + tasks_from: vxlan_interface_name + - include_role: + name: vm/manage_bridges # noqa role-name[path] + tasks_from: vxlan_unicast + when: inventory_hostname == groups['vm_host'][0] + environment: "{{ proxy_env | d({}) }}" + any_errors_fatal: true + - hosts: vm_host gather_facts: false roles: @@ -50,6 +73,22 @@ environment: "{{ proxy_env | d({}) }}" any_errors_fatal: true +# need handle those baremetal hosts need connect to VMs +- hosts: k8s_cluster + vars: + - vms: + - name: dummy + vxlan: "{{ hostvars[groups['vm_host'][0]]['dhcp'][0] }}" + - dhcp: [] + gather_facts: false + serial: 1 + roles: + - role: vm/install_bm_libvirt + - role: vm/manage_bridges + - role: vm/prepare_bm_host_config_vxlan + environment: "{{ proxy_env | d({}) }}" + any_errors_fatal: true + - hosts: vm_host gather_facts: false serial: 1 @@ -66,6 +105,13 @@ tasks: - name: Test updated inventory ping: + - name: Check kernel commandline + command: cat /proc/cmdline + changed_when: false + register: cmdline_info + - name: Print kernel commandline + debug: + msg: "{{ cmdline_info.stdout }}" - name: Check if reboot is needed command: needs-restarting -r register: needs_reboot @@ -73,6 +119,6 @@ notify: - reboot VMs changed_when: needs_reboot.rc == 1 - when: vm_image_distribution == "rocky" + when: vm_image_distribution | default("") == "rocky" environment: "{{ proxy_env | d({}) }}" any_errors_fatal: true diff --git a/playbooks/intel/eci_basic.yml b/playbooks/intel/eci_basic.yml new file mode 100644 index 00000000..c1edecfa --- /dev/null +++ b/playbooks/intel/eci_basic.yml @@ -0,0 +1,48 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +# intel_eci may install intel RT kernel and reboot, so vm_host needs go first +- hosts: vm_host + tasks: [] + roles: + - role: intel_eci + tags: intel-eci + when: + - intel_eci_enabled | default(false) | bool + environment: + - "{{ proxy_env | d({}) }}" + any_errors_fatal: true + +# This tasks is used to configure host and extract some information used for vm +- hosts: vm_host + tasks: + - include_role: + name: intel_eci + tasks_from: eci_host + when: + - intel_eci_enabled | default(false) | bool + - vms | length != 0 + +- hosts: vms + tasks: [] + roles: + - role: intel_eci + tags: intel-eci + when: + - intel_eci_enabled | default(false) | bool + environment: + - "{{ proxy_env | d({}) }}" + any_errors_fatal: true diff --git a/playbooks/intel/tdx.yml b/playbooks/intel/tdx.yml new file mode 100644 index 00000000..05e416e1 --- /dev/null +++ b/playbooks/intel/tdx.yml @@ -0,0 +1,44 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +- hosts: kube_node, vm_host + handlers: + - name: reboot server + ansible.builtin.debug: + msg: "TDX kernel successfully installed, you can reboot and change your bios to enable TDX now!!!" + when: + - inventory_hostname != "localhost" + tasks: [] + roles: + - role: cluster_defaults + tags: always + - role: bootstrap/install_tdx_drivers + tags: tdx + when: + - configure_tdx | default(false) | bool + - not on_vms | default(false) | bool + - role: bootstrap/set_tdx_kernel_flags + tags: tdx + when: + - configure_tdx | default(false) | bool + - not on_vms | default(false) | bool + - role: bootstrap/update_grub + tags: + - tdx + - grub-update + - intel-platform-qat-setup + environment: + - "{{ proxy_env | d({}) }}" diff --git a/playbooks/k8s/k8s.yml b/playbooks/k8s/k8s.yml index b1d48a73..1cb822e4 100644 --- a/playbooks/k8s/k8s.yml +++ b/playbooks/k8s/k8s.yml @@ -29,13 +29,37 @@ - hosts: k8s_cluster gather_facts: false tasks: - - name: prepare additional kubespray facts - set_fact: + - name: Prepare native CPU manager settings + when: native_cpu_manager_enabled | default(false) + ansible.builtin.set_fact: + # Config Variant 1 - set CPU time reservation for system/kube daemons + system_reserved: >- + {% if not native_cpu_manager_reserved_cpus | default(false) -%} + true{% else %}false{% endif %} + system_master_cpu_reserved: "{{ native_cpu_manager_system_reserved_cpus | default('1000m') }}" + system_cpu_reserved: "{{ native_cpu_manager_system_reserved_cpus | default('1000m') }}" + kube_reserved: >- + {% if not native_cpu_manager_reserved_cpus | default(false) and native_cpu_manager_kube_reserved_cpus | default(false) -%} + true{% else %}false{% endif %} + kube_master_cpu_reserved: "{{ native_cpu_manager_kube_reserved_cpus | default('1000m') }}" + kube_cpu_reserved: "{{ native_cpu_manager_kube_reserved_cpus | default('1000m') }}" + # Config Variant 2 - set explicit list of CPUs to run system/kube daemons exclusive kubelet_node_custom_flags_prepare: >- - {%- if native_cpu_manager_enabled | default(true) and native_cpu_manager_reserved_cpus is defined -%} + {%- if native_cpu_manager_reserved_cpus | default(false) -%} reservedSystemCPUs: "{{ native_cpu_manager_reserved_cpus }}" {%- endif -%} + - name: set kube_config dir + set_fact: kube_config_dir: /etc/kubernetes + - name: Set default kube feature gates + set_fact: + kube_default_feature_gates: + - RotateKubeletServerCertificate=true + - name: Set kube CPU manager feature gate for kubernetes older than v1.26 + set_fact: + kube_cpu_manager_feature_gate: + - CPUManager=true + when: kube_version | regex_replace('^v(.*)', '\\1') is version('1.26', '<') - name: set kube_cert_dir set_fact: kube_cert_dir: "{{ kube_config_dir }}/ssl" @@ -47,6 +71,12 @@ {% if ansible_distribution_version >= '21.04' -%}host-gw{%- else -%}vxlan{% endif %}{% endif %} when: kube_network_plugin == "flannel" + - name: Disable DNS stub listener when needed + set_fact: + systemd_resolved_disable_stub_listener: true + when: + - dns_disable_stub_listener | default(true) | bool + - name: print flannel_backend_type debug: msg: "flannel_backend_type: {{ flannel_backend_type }}" @@ -120,7 +150,7 @@ krew_enabled: true multus_conf_file: /host/etc/cni/net.d/templates/00-multus.conf multus_image_tag: "v3.9.3-amd64" - nginx_image_tag: 1.23.3-alpine + nginx_image_tag: 1.24.0-alpine calico_node_livenessprobe_timeout: 15 calico_node_readinessprobe_timeout: 15 kube_proxy_mode: iptables @@ -128,17 +158,11 @@ override_system_hostname: false cilium_ipam_mode: kubernetes enable_nodelocaldns: false - system_reserved: true dashboard_enabled: "{{ kube_dashboard_enabled | default(true) }}" - system_cpu_reserved: "{{ native_cpu_manager_system_reserved_cpus | default('1000m') }}" - kube_cpu_reserved: "{{ native_cpu_manager_kube_reserved_cpus | default('1000m') }}" kubelet_custom_flags: "--cpu-cfs-quota=false" - kubelet_node_config_extra_args: "{{ kubelet_node_custom_flags_prepare | from_yaml }}" + kubelet_node_config_extra_args: "{{ kubelet_node_custom_flags_prepare | default('') | from_yaml }}" kube_api_anonymous_auth: true - kube_feature_gates: - - CPUManager=true # feature gate can be enabled by default, default policy is none in Kubernetes - - TopologyManager={{ topology_manager_enabled | default(true) }} - - RotateKubeletServerCertificate=true + kube_feature_gates: "{{ kube_default_feature_gates + kube_cpu_manager_feature_gate | default([]) }}" # Kubernetes cluster hardening kubernetes_audit: true audit_log_maxbackups: 10 @@ -151,10 +175,11 @@ service-account-key-file: "{{ kube_cert_dir }}/sa.key" kube_kubeadm_controller_extra_args: service-account-private-key-file: "{{ kube_cert_dir }}/sa.key" + kubelet_cpu_manager_policy: "{% if native_cpu_manager_enabled | default(false) %}static{% else %}none{% endif %}" + kubelet_topoloy_manager_policy: "{{ topology_manager_policy | default('none') }}" + kubelet_topology_manager_scope: "{{ topology_manager_scope | default('container') }}" kubelet_config_extra_args: protectKernelDefaults: true - cpuManagerPolicy: "{% if native_cpu_manager_enabled | default(true) %}static{% else %}none{% endif %}" - topologyManagerPolicy: "{% if topology_manager_enabled | default(true) %}{{ topology_manager_policy | default('none') }}{% else %}none{% endif %}" eventRecordQPS: 0 kube_apiserver_request_timeout: 60s kube_apiserver_enable_admission_plugins: ["EventRateLimit", "DefaultStorageClass", "NodeRestriction", "{% if always_pull_enabled %}AlwaysPullImages,{% endif %}PodSecurity"] # noqa yaml[line-length] @@ -336,6 +361,9 @@ when: - cert_manager_enabled | default(false) or registry_enable | default(false) + - role: calico_vpp_install + tags: calico-vpp + when: calico_vpp.enabled | default(false) | bool - role: container_registry tags: registry when: diff --git a/playbooks/k8s/patch_kubespray.yml b/playbooks/k8s/patch_kubespray.yml index c4897326..49f4a496 100644 --- a/playbooks/k8s/patch_kubespray.yml +++ b/playbooks/k8s/patch_kubespray.yml @@ -14,16 +14,6 @@ ## limitations under the License. ## --- -- hosts: k8s_cluster,vm_host - tasks: - - name: set target vars - set_fact: - target_distribution_version: "{{ ansible_distribution_version }}" - target_distribution: "{{ ansible_distribution }}" - target_container_runtime: "{{ container_runtime }}" - delegate_to: localhost - delegate_facts: True - - hosts: 127.0.0.1 connection: local tasks: [] diff --git a/playbooks/k8s/templates/rke2_config.yaml.j2 b/playbooks/k8s/templates/rke2_config.yaml.j2 index bf02f67e..da2dc3e6 100644 --- a/playbooks/k8s/templates/rke2_config.yaml.j2 +++ b/playbooks/k8s/templates/rke2_config.yaml.j2 @@ -39,14 +39,19 @@ cni: {{ kube_network_plugin }} {% endif %} {% endif %} -{% if native_cpu_manager_enabled == true or topology_manager_enabled == true %} kubelet-arg: -{% if native_cpu_manager_enabled == true %} +{% if native_cpu_manager_enabled | default(false) %} - "cpu-manager-policy=static" +{% if native_cpu_manager_reserved_cpus | default(false) %} +- "reserved-cpus={{ native_cpu_manager_reserved_cpus }}" +{% else %} +{% if native_cpu_manager_kube_reserved_cpus | default(false) %} - "kube-reserved=cpu={{ native_cpu_manager_kube_reserved_cpus | default('1000m') }}" +{% endif %} +{% if not native_cpu_manager_reserved_cpus | default(false) %} - "system-reserved=cpu={{ native_cpu_manager_system_reserved_cpus | default('1000m') }}" {% endif %} -{% if topology_manager_enabled == true %} -- "topology-manager-policy={{ topology_manager_policy }}" {% endif %} {% endif %} +- "topology-manager-scope={{ topology_manager_scope | default("container") }}" +- "topology-manager-policy={{ topology_manager_policy | default("none") }}" diff --git a/playbooks/preflight.yml b/playbooks/preflight.yml index 986142c8..44b4ef90 100644 --- a/playbooks/preflight.yml +++ b/playbooks/preflight.yml @@ -56,7 +56,7 @@ # On Role specific Only (k8s_cluster) # - Check Observability compontents # - Check MinIO requirements - +# - Check Intent Driven Orchestration (IDO) dependencies # additional vars required: # cek_supported_ansible_base: # must be version @@ -78,15 +78,15 @@ tasks: - - debug: msg="Ansible version is {{ ansible_version.string }}" + - ansible.builtin.debug: msg="Ansible version is {{ ansible_version.string }}" - name: Check Ansible Version - assert: + ansible.builtin.assert: that: (ansible_version.full is version_compare(cek_supported_ansible_base, '>=')) msg: "Ansible version must be {{ cek_supported_ansible_base }}. Please update" - - debug: msg="Python version is {{ ansible_python_version }}" + - ansible.builtin.debug: msg="Python version is {{ ansible_python_version }}" - name: Check Python Version - assert: + ansible.builtin.assert: that: (ansible_python_version is version_compare(cek_supported_python, '>=')) msg: "Python version must be at least {{ cek_supported_python }}. Please update" @@ -115,49 +115,49 @@ - kubernetes - name: read Group Vars - stat: + ansible.builtin.stat: path: "{{ inventory_dir }}/group_vars/" register: group_vars_details - name: Check Group Vars - assert: + ansible.builtin.assert: that: "group_vars_details.stat.exists and group_vars_details.stat.isdir" msg: "File group_vars/all.yml does NOT exist. Must be created per Guide" - name: read Host Vars - stat: + ansible.builtin.stat: path: "{{ inventory_dir }}/host_vars/{{ item }}.yml" register: host_vars_details with_items: "{{ groups['kube_node'] }}" - name: check Host Vars - assert: + ansible.builtin.assert: that: "item.stat.exists and item.stat.isreg" msg: "File host_vars/{{ item.item }}.yml does NOT exist. Must be created per Guide" with_items: "{{ host_vars_details.results }}" - name: read VM Host Vars - stat: + ansible.builtin.stat: path: "{{ inventory_dir }}/host_vars/{{ item }}.yml" register: vm_host_vars_details with_items: "{{ groups['vm_host'] }}" - name: check VM Host Vars - assert: + ansible.builtin.assert: that: "item.stat.exists and item.stat.isreg" msg: "File host_vars/{{ item.item }}.yml does NOT exist. Must be created per Guide" with_items: "{{ vm_host_vars_details.results }}" - name: set group_vars_profile variable - set_fact: + ansible.builtin.set_fact: group_vars_profile: "{{ profile_name | default('not generated') }}" - name: show profile_name from group_vars - debug: + ansible.builtin.debug: msg: "group_vars profile_name is: '{{ group_vars_profile }}'" - name: check ip is defined in inventory.ini - shell: "set -o pipefail && cat {{ inventory_file }} | grep -v \"^#\"" + ansible.builtin.shell: "set -o pipefail && cat {{ inventory_file }} | grep -v \"^#\"" args: executable: /bin/bash changed_when: false @@ -168,14 +168,15 @@ - name: handle the error for check passwordless access to VM hosts block: - name: check passwordless access to VM hosts - command: >- - ssh -o PasswordAuthentication=no {{ hostvars[item]['ansible_user'] | default(ansible_env.USER, True) }}@{{ hostvars[item]['ip'] }} + ansible.builtin.command: >- + ssh -o PasswordAuthentication=no -o StrictHostKeyChecking=no + {{ hostvars[item]['ansible_user'] | default(ansible_env.USER, True) }}@{{ hostvars[item]['ip'] }} /bin/true with_items: "{{ groups['vm_host'] }}" changed_when: false rescue: - name: Print when passwordless authentication failed - debug: + ansible.builtin.debug: msg: > "Current user: '{{ ansible_env.USER }}' does not have password less access to '{{ (ansible_failed_result.results[0].stderr).split(':')[0] }}'. Password less authentication have to be configured for all VM hosts. Use: 'ssh-copy-id @' @@ -189,13 +190,13 @@ - name: handle the error for checking hostname and passwordless access to VM hosts via hostname block: - name: set hostname configuration - set_fact: + ansible.builtin.set_fact: hostname_check: "{{ ansible_hostname }}" hostname_fqdn_check: "{{ ansible_fqdn }}" with_items: "{{ groups['vm_host'] }}" - name: verify user defined hostname in inventory with in system hostname configuration - assert: + ansible.builtin.assert: that: - (ansible_host == hostname_fqdn_check) or (ansible_host == hostname_check) @@ -206,14 +207,15 @@ with_items: "{{ groups['vm_host'] }}" - name: check passwordless access to VM hosts via FQDN - command: >- - ssh -o PasswordAuthentication=no {{ hostvars[item]['ansible_user'] | default(ansible_env.USER, True) }}@{{ hostvars[item]['ansible_host'] }} + ansible.builtin.command: >- + ssh -o PasswordAuthentication=no -o StrictHostKeyChecking=no + {{ hostvars[item]['ansible_user'] | default(ansible_env.USER, True) }}@{{ hostvars[item]['ansible_host'] }} /bin/true changed_when: false with_items: "{{ groups['vm_host'] }}" rescue: - name: Print when passwordless authentication failed - debug: + ansible.builtin.debug: msg: > "Current user: '{{ ansible_env.USER }}' does not have password less access to '{{ (ansible_failed_result.results[0].stderr).split(':')[0] }}'. Password less authentication have to be configured for all VM hosts. Use: 'ssh-copy-id @' @@ -225,14 +227,14 @@ - "'ip=' not in check_ip_status.stdout | regex_findall('^' + item + '.*$', multiline=True, ignorecase=True) | join(',')" - name: check scale variable value - assert: + ansible.builtin.assert: that: "{{ scale | bool }}" fail_msg: "scale variable must be set to one of the following values { yes, on, 1, true }, case insensitive" success_msg: "scale variable is set to {{ scale }} \ncluster scaling is enabled" when: scale is defined - name: check vm_recreate_existing variable value - assert: + ansible.builtin.assert: that: "not {{ vm_recreate_existing | bool }}" fail_msg: "vm_recreate_existing has to be false for cluster scaling case" success_msg: "vm_recreate_existing variable is set to {{ vm_recreate_existing }} for cluster scaling" @@ -272,23 +274,24 @@ tasks: - name: end play for VM host - meta: end_host + ansible.builtin.meta: end_host when: - "'vm_host' in group_names" - on_vms is defined and on_vms - name: fail if deployment is VMRA and isolcpus is enabled - assert: + ansible.builtin.assert: that: - isolcpus is not defined fail_msg: - "isolcpus variable can't be used on VMRA deployment" when: - vm_enabled and (not on_vms | default(false)) + - inventory_hostname in groups['vm_host'] - isolcpus_enabled - name: read Host Vars for VMs - stat: + ansible.builtin.stat: path: "{{ inventory_dir }}/host_vars/{{ item.name }}.yml" register: vm_host_vars_details with_items: "{{ vms }}" @@ -296,22 +299,24 @@ become: false when: - vm_enabled and (not on_vms | default(false)) + - inventory_hostname in groups['vm_host'] - item.type == 'work' - name: check Host Vars for VMs - assert: + ansible.builtin.assert: that: "item.stat.exists and item.stat.isreg" msg: "File host_vars/{{ item.item.name }}.yml does NOT exist. Must be created per Guide" with_items: "{{ vm_host_vars_details.results }}" when: - vm_enabled and (not on_vms | default(false)) - vm_host_vars_details + - inventory_hostname in groups['vm_host'] - not item.skipped | default(false) - name: Check VM root password block: - name: check vm_hashed_passwd presence - assert: + ansible.builtin.assert: that: - vm_hashed_passwd is defined fail_msg: @@ -319,7 +324,7 @@ success_msg: "vm_hashed_passwd is present" - name: check vm_hashed_passwd is not a placeholder - assert: + ansible.builtin.assert: that: - vm_hashed_passwd is defined - vm_hashed_passwd != "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" @@ -328,15 +333,71 @@ success_msg: "vm_hashed_passwd verified" when: - vm_enabled and (not on_vms | default(false)) + - inventory_hostname in groups['vm_host'] -# VMs multinode setup for VM Hosts +# Host mixed with VM hosts setup, must have VXLAN physical interface to connect to VMs + - name: Check requirements to enable mixed deployment + when: + - vm_enabled and (not on_vms | default(false)) + - inventory_hostname in groups['k8s_cluster'] + block: + - name: check if mandatory parameter 'vxlan_physical_network' is present for the baremetal host + assert: + that: + - vxlan_physical_network is defined + msg: | + "vxlan_physical_network parameter is not defined for host '{{ inventory_hostname }}'" + "Please correct the configuration" + + - name: check if mandatory parameter 'vxlan_physical_network' for the baremetal host has valid IP subnet + assert: + that: + - "vxlan_physical_network | ansible.utils.ipaddr('net')" + msg: | + "vxlan_physical_network parameter for the host '{{ inventory_hostname }}' does not have valid IP subnet" + "Current value is '{{ vxlan_physical_network }}'" + "Please correct the configuration" + + - name: check if VXLAN physical network is available + assert: + that: "hostvars[inventory_hostname]['ansible_all_ipv4_addresses'] | ansible.utils.ipaddr(vxlan_physical_network) | length > 0 " + msg: | + "vxlan_physical_network '{{ vxlan_physical_network }}' is not available on host '{{ inventory_hostname }}'" + "Please correct the configuration or update network setup to ensure proper connectivity with vm_hosts" + + - name: check if mandatory parameter 'vxlan_gw_ip' is present for the baremetal host + assert: + that: + - vxlan_gw_ip is defined + msg: | + "vxlan_gw_ip parameter is not defined for host '{{ inventory_hostname }}'" + "Please correct the configuration" + + - name: check if mandatory parameter 'vxlan_gw_ip' for the baremetal host has valid IP + assert: + that: + - "vxlan_gw_ip | ansible.utils.ipaddr('address')" + msg: | + "vxlan_gw_ip parameter for the host '{{ inventory_hostname }}' does not have valid IP" + "Current value is '{{ vxlan_gw_ip }}'" + "Please correct the configuration" + +# VMs multinode setup for VM Hosts (including VM + BM mixed case) - name: Check requirements to enable VMs multinode setup when: - vm_enabled and (not on_vms | default(false)) - - "groups['vm_host'] | length > 1" + - "(groups['vm_host'] | length > 1) or (groups['k8s_cluster'] | length > 0)" block: - - name: check if mandatory parameter 'vxlan_physical_network' is present for the first vm_host + - name: check if mandatory parameter 'dhcp' is present on vm_host[0] assert: + that: + - "hostvars[groups['vm_host'][0]]['dhcp'] | default([]) | length" + msg: | + "vxlan dhcp must be defined on {{ groups['vm_host'][0] }} for multiple hosts deployment" + run_once: true + + - name: check if mandatory parameter 'vxlan_physical_network' is present for the first vm_host + ansible.builtin.assert: that: - "hostvars[groups['vm_host'][0]]['vxlan_physical_network'] is defined" msg: | @@ -345,7 +406,7 @@ run_once: true - name: check if mandatory parameter 'vxlan_physical_network' for the first vm_host has valid IP subnet - assert: + ansible.builtin.assert: that: - "hostvars[groups['vm_host'][0]]['vxlan_physical_network'] | ansible.utils.ipaddr('net')" msg: | @@ -355,7 +416,7 @@ run_once: true - name: check if VXLAN physical network is available - assert: + ansible.builtin.assert: that: "hostvars[inventory_hostname]['ansible_all_ipv4_addresses'] | ansible.utils.ipaddr(hostvars[groups['vm_host'][0]]['vxlan_physical_network']) | length > 0 " msg: | @@ -363,7 +424,7 @@ "Please correct the configuration or update network setup to ensure proper connectivity between vm_hosts" - name: check for ip and ansible_host mismatch - assert: + ansible.builtin.assert: that: "ansible_host == ip" msg: "Configuration mismatch detected between ansible_host={{ ansible_host }} and ip={{ ip }} on target '{{ inventory_hostname }}'" when: @@ -371,10 +432,12 @@ - ip is defined - ansible_host | ansible.utils.ipaddr('bool') - not adq_dp['enabled'] | default(false) + - not calico_vpp['enabled'] | default(false) - - debug: msg="Linux distribution on target is {{ ansible_distribution }} {{ ansible_distribution_version }} {{ ansible_distribution_release }}" + - ansible.builtin.debug: + msg: "Linux distribution on target is {{ ansible_distribution }} {{ ansible_distribution_version }} {{ ansible_distribution_release }}" - name: Check Linux Distro and Version - assert: + ansible.builtin.assert: that: "ansible_distribution in cek_supported_distros and ansible_distribution_version in cek_supported_distros_versions" msg: - Linux distribution {{ ansible_distribution }} {{ ansible_distribution_version }} on target '{{ inventory_hostname }}' is NOT supported @@ -385,11 +448,11 @@ # TODO ?? Linux OS must be the same on all targets (no mix-n-match) - name: regather network facts in case hostname recently changed - setup: + ansible.builtin.setup: gather_subset: network - - debug: msg="Inventory target '{{ inventory_hostname }}' has the actual system hostname '{{ ansible_hostname }}'" + - ansible.builtin.debug: msg="Inventory target '{{ inventory_hostname }}' has the actual system hostname '{{ ansible_hostname }}'" - name: Check Inventory Hostnames - debug: + ansible.builtin.debug: msg: - "Target '{{ inventory_hostname }}' in inventory does NOT match the actual system hostname '{{ ansible_hostname }}'." - "If it's done intentionally, please ignore this message." @@ -399,8 +462,8 @@ # Early check if SELinux is configured properly - block: - name: "Collect packages facts" - package_facts: - - debug: + ansible.builtin.package_facts: + - ansible.builtin.debug: msg: - "Current SELinux status:" - "status: {{ ansible_selinux.status | default('') }}" @@ -410,7 +473,7 @@ - "config_mode: {{ ansible_selinux.config_mode | default('') }}" - name: check SELinux condition possibly causing system boot failure - debug: + ansible.builtin.debug: msg: - "Current SELinux setup might cause the system possibly will not boot up on next reboot." - "Please, check SELinux settings and set it up according to the documentation." @@ -419,7 +482,7 @@ - "'selinux-policy-targeted' not in ansible_facts.packages" - name: check SELinux configuration for CRI-O runtime - assert: + ansible.builtin.assert: that: - (ansible_selinux.status == "enabled" and selinux_state is defined and selinux_state == "disabled") or (ansible_selinux.status != "enabled" and selinux_state is defined and selinux_state != "enabled") @@ -431,7 +494,7 @@ - ansible_os_family == "RedHat" - name: check kubernetes and container runtime variables - assert: + ansible.builtin.assert: that: - (not kubernetes and container_runtime_only_deployment) or (not kubernetes and not container_runtime_only_deployment) @@ -442,7 +505,7 @@ - not kubernetes or container_runtime_only_deployment - name: check k8s version - assert: + ansible.builtin.assert: that: "{{ kube_version is version('v1.22', '>=') }}" msg: "Minimum supported k8s version is 1.22, please update kube_version variable with correct version" when: kubernetes and not container_runtime_only_deployment @@ -454,7 +517,7 @@ when: kube_provisioner == "rke2" - name: assert that Multus is enabled in the config - assert: + ansible.builtin.assert: that: - "kube_network_plugin_multus" fail_msg: "SRIOV and the Userspace CNI plugin require Multus for a fully functional cluster deployment" @@ -463,7 +526,7 @@ userspace_cni_enabled is defined and userspace_cni_enabled - name: assert that SRIOV Network Operator/SRIOV Network DP are mutually exclusive - assert: + ansible.builtin.assert: that: - "not sriov_net_dp_enabled" fail_msg: @@ -472,7 +535,7 @@ # check cpusets CPU core/threads shielding configuration - block: - - debug: + - ansible.builtin.debug: msg: - cpusets_enabled = {{ cpusets_enabled }} (host_vars) - cpusets = {{ cpusets }} (host_vars) @@ -484,7 +547,7 @@ # - CPUs Reserved for OS = {{ lookup('sequence','0-{{ ansible_processor_count - 1 }}').split(',') }} # [E207] Nested jinja pattern - name: check OS for cpusets based on cgroups v1 only - assert: + ansible.builtin.assert: that: - (ansible_distribution == 'Ubuntu' and ansible_distribution_version == '20.04') msg: @@ -492,50 +555,50 @@ - CPUs isolation ('cpusets') can be only enabled on Ubuntu 20.04.x. - name: check CPUs isolation - assert: + ansible.builtin.assert: that: ( "{{ cpusets }}" | length > 0 ) msg: - Incorrect configuration. Conflicting or improper values detected - CPUs isolation ('cpusets') must be set according to the example file for host_vars. Please correct the configuration - name: split cpusets Groups - set_fact: + ansible.builtin.set_fact: cpusets_groups: "{{ cpusets.split(',') }}" - - debug: msg="cpusets_groups = {{ cpusets_groups }}" + - ansible.builtin.debug: msg="cpusets_groups = {{ cpusets_groups }}" - name: filter cpusets Ranges - set_fact: + ansible.builtin.set_fact: cpusets_ranges: "{{ cpusets_ranges + [item] }}" with_items: "{{ cpusets_groups }}" when: ("-" in item) - - debug: msg="cpusets_ranges = {{ cpusets_ranges }}" + - ansible.builtin.debug: msg="cpusets_ranges = {{ cpusets_ranges }}" - name: filter cpusets Discretes - set_fact: + ansible.builtin.set_fact: cpusets_discretes: "{{ cpusets_discretes + [item] }}" with_items: "{{ cpusets_groups }}" when: ("-" not in item) - - debug: msg="cpusets_discretes = {{ cpusets_discretes }}" + - ansible.builtin.debug: msg="cpusets_discretes = {{ cpusets_discretes }}" - name: build cpusets List - set_fact: + ansible.builtin.set_fact: cpusets_list: "{{ cpusets_list | default([]) | union(cpusets_discretes) | union([item]) }}" with_sequence: "{{ cpusets_ranges }}" - - debug: msg="cpusets_list = {{ cpusets_list }}" + - ansible.builtin.debug: msg="cpusets_list = {{ cpusets_list }}" - name: check cpusets Total - assert: + ansible.builtin.assert: that: "{{ cpusets_list | length }} <= ansible_processor_vcpus" msg: - Incorrect configuration pertaining cpusets. Conflicting or improper values detected - The number of cpusets {{ cpusets_list | length }}, exceeds total CPUs on target {{ ansible_processor_vcpus }}. Please correct the configuration - name: check cpusets IDs - assert: + ansible.builtin.assert: that: "item | int <= ansible_processor_vcpus" msg: - Incorrect configuration pertaining cpusets. Conflicting or improper values detected @@ -543,7 +606,7 @@ with_items: "{{ cpusets_list }}" - name: check cpusets OS Reserved - assert: + ansible.builtin.assert: that: "item not in cpusets_list" msg: - Incorrect configuration pertaining cpusets. Conflicting or improper values detected @@ -555,7 +618,7 @@ # check isolcpus CPU core/threads configuration - block: - - debug: + - ansible.builtin.debug: msg: - isolcpus_enabled = {{ isolcpus_enabled }} (host_vars) - isolcpus = {{ isolcpus }} (host_vars) @@ -567,50 +630,50 @@ # - CPUs Reserved for OS = {{ lookup('sequence','0-{{ ansible_processor_count - 1 }}').split(',') }} # [E207] Nested jinja pattern - name: check CPUs isolation - assert: + ansible.builtin.assert: that: ( "{{ isolcpus }}" | length > 0 ) msg: - Incorrect configuration. Conflicting or improper values detected - CPUs isolation ('isolcpus') must be set according to the example file for host_vars. Please correct the configuration - name: split isolcpus Groups - set_fact: + ansible.builtin.set_fact: isolcpus_groups: "{{ isolcpus.split(',') }}" - - debug: msg="isolcpus_groups = {{ isolcpus_groups }}" + - ansible.builtin.debug: msg="isolcpus_groups = {{ isolcpus_groups }}" - name: filter isolcpus Ranges - set_fact: + ansible.builtin.set_fact: isolcpus_ranges: "{{ isolcpus_ranges + [item] }}" with_items: "{{ isolcpus_groups }}" when: ("-" in item) - - debug: msg="isolcpus_ranges = {{ isolcpus_ranges }}" + - ansible.builtin.debug: msg="isolcpus_ranges = {{ isolcpus_ranges }}" - name: filter isolcpus Discretes - set_fact: + ansible.builtin.set_fact: isolcpus_discretes: "{{ isolcpus_discretes + [item] }}" with_items: "{{ isolcpus_groups }}" when: ("-" not in item) - - debug: msg="isolcpus_discretes = {{ isolcpus_discretes }}" + - ansible.builtin.debug: msg="isolcpus_discretes = {{ isolcpus_discretes }}" - name: build isolcpus List - set_fact: + ansible.builtin.set_fact: isolcpus_list: "{{ isolcpus_list | default([]) | union(isolcpus_discretes) | union([item]) }}" with_sequence: "{{ isolcpus_ranges }}" - - debug: msg="isolcpus_list = {{ isolcpus_list }}" + - ansible.builtin.debug: msg="isolcpus_list = {{ isolcpus_list }}" - name: check isolcpus Total - assert: + ansible.builtin.assert: that: "{{ isolcpus_list | length }} <= ansible_processor_vcpus" msg: | Incorrect configuration pertaining isolcpus. Conflicting or improper values detected The number of isolcpus {{ isolcpus_list | length }}, exceeds total CPUs on target {{ ansible_processor_vcpus }}. Please correct the configuration - name: check isolcpus IDs - assert: + ansible.builtin.assert: that: "item | int <= ansible_processor_vcpus" msg: - Incorrect configuration pertaining isolcpus. Conflicting or improper values detected @@ -618,7 +681,7 @@ with_items: "{{ isolcpus_list }}" - name: check isolcpus OS Reserved - assert: + ansible.builtin.assert: that: "item not in isolcpus_list" msg: - Incorrect configuration pertaining isolcpus. Conflicting or improper values detected @@ -630,7 +693,7 @@ - isolcpus is defined and isolcpus - name: check Intel CPU Control Plane configuration - include_role: + ansible.builtin.include_role: name: intel_cpu_controlplane tasks_from: preflight_cpu_controlplane.yml when: @@ -640,7 +703,7 @@ - cpu_ctlplane - name: check Intel Media Analytics configuration - import_role: + ansible.builtin.import_role: name: intel_media_analytics tasks_from: preflight_intel_media_analytics.yml when: @@ -661,18 +724,18 @@ tasks: - name: end play for VM host - meta: end_host + ansible.builtin.meta: end_host when: - "'vm_host' in group_names" - on_vms is defined and on_vms # STORY: "nic bus info specified is present on system" - - debug: + - ansible.builtin.debug: msg: "Dataplane (DP) interface(s) defined in host_vars = {{ dataplane_interfaces }}" when: dataplane_interfaces is defined - name: assert that (SRIOV Network DP/SRIOV CNI) and (SRIOV Network Operator) are mutually exclusive - assert: + ansible.builtin.assert: that: "not (sriov_net_dp_enabled and (sriov_network_operator_enabled | default(false)))" fail_msg: - "SRIOV Network DP/SRIOV CNI and SRIOV Network Operator are mutually exclusive. One must be disabled" @@ -681,7 +744,7 @@ (sriov_net_dp_enabled is defined and sriov_net_dp_enabled) - name: check DP Interfaces - assert: + ansible.builtin.assert: that: "dataplane_interfaces != []" msg: "Dataplane (DP) interface(s) on target '{{ ansible_hostname }}' must be set in host_vars. Please correct the configuration" when: @@ -694,7 +757,7 @@ # FlexRAN needs 2 NIC PFs - name: check DP Interfaces PF numbers for FlexRAN - assert: + ansible.builtin.assert: that: - dataplane_interfaces[0].bus_info is defined - dataplane_interfaces[1].bus_info is defined @@ -702,19 +765,19 @@ when: - intel_flexran_enabled | default(false) - - debug: + - ansible.builtin.debug: msg: "Network interfaces present on target '{{ ansible_hostname }}' = {{ ansible_interfaces }}" - name: read Physical NICs PCIIDs - set_fact: + ansible.builtin.set_fact: phy_nics_pciids: "{{ phy_nics_pciids + [ ansible_facts[item]['pciid'] ] }}" with_items: "{{ ansible_interfaces }}" when: ansible_facts[item]['pciid'] is defined and ansible_facts[item]['type'] == "ether" - - debug: msg="PCI Slots for the NICs on target '{{ ansible_hostname }}' = {{ phy_nics_pciids }}" + - ansible.builtin.debug: msg="PCI Slots for the NICs on target '{{ ansible_hostname }}' = {{ phy_nics_pciids }}" - name: check DP Interfaces Bus Info - assert: + ansible.builtin.assert: that: ("{{ item.bus_info }}" in "{{ phy_nics_pciids }}") msg: "Dataplane interface defined with PCI ID '{{ item.bus_info }}' does NOT exist on target. Please correct the configuration" with_items: "{{ dataplane_interfaces }}" @@ -747,7 +810,7 @@ - name: Check first DP interface driver and DDP package block: - name: Check first DP interface driver - assert: + ansible.builtin.assert: that: "dataplane_interfaces[0].pf_driver == ansible_facts[item]['module']" msg: >- "Dataplane interface '{{ dataplane_interfaces[0].bus_info }}' 'pf_driver' is set to '{{ dataplane_interfaces[0].pf_driver }}'. @@ -758,7 +821,7 @@ - dataplane_interfaces[0].bus_info in ansible_facts[item]['pciid'] - name: check if selected DDP package corresponds PF driver ice - assert: + ansible.builtin.assert: that: "{{ dataplane_interfaces[0].ddp_profile is regex('^ice_comms*') }}" msg: "ddp_profile '{{ dataplane_interfaces[0].ddp_profile }}' doesn't correspond pf_driver '{{ dataplane_interfaces[0].pf_driver }}'" when: @@ -768,7 +831,7 @@ - install_ddp_packages is defined and install_ddp_packages - name: check if selected DDP package corresponds PF driver i40e - assert: + ansible.builtin.assert: that: "{{ dataplane_interfaces[0].ddp_profile in ddp_profiles_allowed }}" msg: "ddp_profile '{{ dataplane_interfaces[0].ddp_profile }}' doesn't correspond pf_driver '{{ dataplane_interfaces[0].pf_driver }}'" when: @@ -787,22 +850,22 @@ - dataplane_interfaces is defined and dataplane_interfaces | length > 0 - name: Print processor info - debug: + ansible.builtin.debug: msg: "ansible_processor model: {{ ansible_processor[2] }}" when: (not vm_enabled) or (vm_enabled and (not on_vms | default(false))) - name: set CPU ID in preflight - set_fact: + ansible.builtin.set_fact: cpu_id: "{{ ansible_processor[2] | regex_search('\\$?\\d\\d\\d\\d\\%?\\@?\\w?|\\d\\d/\\d\\w') }}" when: (not vm_enabled) or (vm_enabled and (not on_vms | default(false))) - name: print CPU ID - debug: + ansible.builtin.debug: msg: "CPU ID: {{ cpu_id }}" when: (not vm_enabled) or (vm_enabled and (not on_vms | default(false))) - name: check if CPU has confirmed support (preflight) - assert: + ansible.builtin.assert: that: "cpu_id in {{ lookup('ansible.builtin.vars', 'confirmed_' + configured_arch + '_cpus') }} \ {% if configured_arch == 'clx' %} or cpu_id in {{ confirmed_clx_ncpus }} {% endif %} \ or cpu_id in {{ unconfirmed_cpu_models }}" @@ -815,7 +878,7 @@ - configured_arch not in ['atom', 'core'] - name: check ubuntu pro token is not a placeholder - assert: + ansible.builtin.assert: that: - ubuntu_pro_token is defined - ubuntu_pro_token != "ffffffffffffffffffffffffffffff" @@ -854,7 +917,7 @@ - File {{ (emr_qat_driver_staging_folder, emr_qat_driver_package) | path_join }} on localhost is NOT the expected one. - Please provide the correct file. See docs/emr.md when: - - update_qat_drivers is defined and update_qat_drivers + - update_qat_drivers | default(false) - configured_arch == "emr" - name: check QAT SVM precheck @@ -867,78 +930,78 @@ - name: check QAT Devices list is configured properly block: - - debug: + - ansible.builtin.debug: msg: "QAT device(s) defined in host_vars = {{ qat_devices }}" - name: read QAT PCIIDs - shell: "set -o pipefail && lshw -businfo -numeric | egrep -i '{{ qat_supported_pf_dev_ids | join('|') }}'" + ansible.builtin.shell: "set -o pipefail && lspci -d 8086: -nn -D | egrep -i '{{ qat_supported_pf_dev_ids | join('|') }}'" args: executable: /bin/bash - register: lshw_qat_host + register: lspci_qat_host changed_when: false failed_when: false when: - on_vms is not defined or not on_vms - name: assert QAT PCIIDs - assert: - that: "lshw_qat_host.rc == 0" + ansible.builtin.assert: + that: "lspci_qat_host.rc == 0" fail_msg: "No QAT devices were found in system. Please configure properly the QAT PCIIDs in group_vars or disable this feature" success_msg: "QAT PCIIDs verification completed" when: - on_vms is not defined or not on_vms - name: read QAT PCIIDs on VMs - shell: "set -o pipefail && lshw -businfo -numeric | egrep -i '{{ (qat_supported_vf_dev_ids + qat_supported_pf_dev_ids) | join('|') }}'" + ansible.builtin.shell: "set -o pipefail && lspci -d 8086: -nn -D | egrep -i '{{ (qat_supported_vf_dev_ids + qat_supported_pf_dev_ids) | join('|') }}'" args: executable: /bin/bash - register: lshw_qat_vms + register: lspci_qat_vms changed_when: false when: - on_vms is defined and on_vms - - name: store lshw_qat - set_fact: - lshw_qat: "{% if (on_vms is defined and on_vms) %}{{ lshw_qat_vms }}{% else %}{{ lshw_qat_host }}{% endif %}" + - name: store lspci_qat + ansible.builtin.set_fact: + lspci_qat: "{% if (on_vms is defined and on_vms) %}{{ lspci_qat_vms }}{% else %}{{ lspci_qat_host }}{% endif %}" - - debug: - msg: "QAT devices found on target = {{ lshw_qat.stdout }}" + - ansible.builtin.debug: + msg: "QAT devices found on target = {{ lspci_qat.stdout }}" - name: check QAT Devices' Bus Info - assert: - that: ("{{ item.qat_id }}" in """{{ lshw_qat.stdout }}""") + ansible.builtin.assert: + that: ("{{ item.qat_id }}" in """{{ lspci_qat.stdout }}""") msg: "QAT device defined with PCI ID '{{ item.qat_id }}' does NOT exist on target. Please correct the configuration" with_items: "{{ qat_devices }}" # STORY: "qat_sriov_numvfs should not exceed max supported (16) per each dev_ID" - - debug: + - ansible.builtin.debug: msg: - qat_sriov_numvfs for {{ item.qat_id }} = {{ item.qat_sriov_numvfs }} (host_vars) - update_qat_drivers = {{ update_qat_drivers }} (host_vars) with_items: "{{ qat_devices }}" - name: check QAT SRIOV VFs - assert: + ansible.builtin.assert: that: ({{ item.qat_sriov_numvfs }} <= 16) msg: - Incorrect configuration pertaining QAT SRIOV. Conflicting or improper values detected - When SRIOV VFs are set for QAT, max value is 16 for each ID (max 48 total per card). Please correct the configuration with_items: "{{ qat_devices }}" when: - - update_qat_drivers is defined and update_qat_drivers + - update_qat_drivers | default(false) - qat_devices is defined and qat_devices != [] - name: check QAT SRIOV VFs requirement for qat device auto detection - assert: + ansible.builtin.assert: that: ({{ qat_sriov_numvfs_required }} <= 16) msg: - Incorrect configuration in qat_sriov_numvfs_required for requested number of QAT VTs - When SRIOV VFs are set for QAT, max value is 16 for each ID (max 48 total per card). Please correct the configuration when: - - update_qat_drivers is defined and update_qat_drivers + - update_qat_drivers | default(false) - qat_devices is defined and qat_devices == [] - name: Check GPU hardware requirements - include_role: + ansible.builtin.include_role: name: gpu_dp_install tasks_from: preflight_gpu_dp when: @@ -946,7 +1009,7 @@ - profile_name != "on_prem_aibox" - name: Check GPU device plugin state - assert: + ansible.builtin.assert: that: - gpu_dp_enabled fail_msg: "GPU device plugin is disabled. Please set gpu_dp_enabled to true in group_vars." @@ -988,7 +1051,7 @@ - configure_fpga is defined and configure_fpga - name: check OpenSSL and OpenSSL*Engine requirements when OOT or Intree QAT setup configured - assert: + ansible.builtin.assert: that: - openssl_install fail_msg: "OpenSSL & OpenSSL*Engine must be configured if configure_qat is set to 'true'" @@ -997,7 +1060,7 @@ - configure_qat | default(false) | bool - name: check gas(gpu aware scheduling) configuration - assert: + ansible.builtin.assert: that: - gpu_dp_enabled fail_msg: "gas installation requires gpu_dp_enabled set to true" @@ -1006,7 +1069,7 @@ - gas_enabled | default(false) - name: check KMRA sgx_dp requirements - assert: + ansible.builtin.assert: that: - sgx_dp_enabled fail_msg: "KMRA installation requires sgx_dp_enabled set to 'true'" @@ -1017,14 +1080,14 @@ kmra.apphsm.enabled | default(false) - name: check KMRA sbx requirements - include_role: + ansible.builtin.include_role: name: kmra_install tasks_from: kmra_sbx_preflight when: - kmra.sbx | default(false) - name: make sure netopeer2 server/client is off when oran disabled - assert: + ansible.builtin.assert: that: - not (kmra.oran_netopeer2_server.enabled | default(false)) - not (kmra.oran_netopeer2_client.enabled | default(false)) @@ -1033,14 +1096,14 @@ - not kmra.oran.enabled | default(false) - name: check KMRA oran requirements - include_role: + ansible.builtin.include_role: name: kmra_install tasks_from: kmra_oran_preflight when: - kmra.oran.enabled | default(false) - name: check Intel SGX DP configuration - assert: + ansible.builtin.assert: that: - configure_sgx is defined - configure_sgx @@ -1050,7 +1113,7 @@ - sgx_dp_enabled is defined and sgx_dp_enabled - name: check if Cert Manager is required - assert: + ansible.builtin.assert: that: - cert_manager_enabled | default(false) fail_msg: @@ -1067,13 +1130,14 @@ qat_dp_enabled | default(false) or jaeger_operator | default(false) or telegraf_enabled | default(false) or - prometheus_operator | default(false) or + prometheus_enabled | default(false) or opentelemetry_enabled | default(false) or elasticsearch_enabled | default(false) or linkerd_service_mesh.enabled | default(false) + - kubernetes | default(false) | bool - name: check NFD configuration - assert: + ansible.builtin.assert: that: - nfd_enabled fail_msg: "Deployment of Intel Device Plugins requires nfd_enabled set to 'true' in group_vars" @@ -1084,7 +1148,7 @@ (gpu_dp_enabled | default(false)) - name: check kmra.pccs.api_key presence - assert: + ansible.builtin.assert: that: - kmra.pccs.api_key is defined fail_msg: @@ -1094,7 +1158,7 @@ - kmra.pccs.enabled | default(false) - name: check PCCS API key length - assert: + ansible.builtin.assert: that: - kmra.pccs.api_key | length == 32 fail_msg: "PCCS API Key should be 32 bytes long" @@ -1103,7 +1167,7 @@ - kmra.pccs.enabled | default(false) - name: check PCCS API key is not a placeholder - assert: + ansible.builtin.assert: that: - kmra.pccs.api_key is defined - kmra.pccs.api_key != "ffffffffffffffffffffffffffffffff" @@ -1116,7 +1180,7 @@ - kmra.pccs.enabled | default(false) - name: aesmd-demo shouldn't be enabled if tcs is enabled - assert: + ansible.builtin.assert: that: not sgx_aesmd_demo_enable | default(false) fail_msg: "Please, set 'sgx_aesmd_demo_enable' to 'false' because it conflicts with on-host aesmd daemon required by TCS." when: @@ -1126,25 +1190,25 @@ - name: validate hugepage settings block: - name: validate default hugepage size setting - assert: + ansible.builtin.assert: that: default_hugepage_size == "2M" or default_hugepage_size == "1G" fail_msg: "Valid sizes for hugepage are: [2M, 1G], given: {{ default_hugepage_size }}." - name: validate that at least one 1G hugepage is requested - assert: + ansible.builtin.assert: that: number_of_hugepages_1G > 0 fail_msg: "number_of_hugepages_1G should not be equal to 0. Please update host vars settings." when: default_hugepage_size == "1G" - name: validate that at least one 2M hugepage is requested - assert: + ansible.builtin.assert: that: number_of_hugepages_2M > 0 fail_msg: "number_of_hugepages_2M should not be equal to 0. Please update host vars settings." when: default_hugepage_size == "2M" when: hugepages_enabled | default(false) | bool # STORY: "vpp/ovsdpdk require hugepage enabled and configured" - - debug: + - ansible.builtin.debug: msg: - vpp_enabled = {{ vpp_enabled }} (host_vars) - ansible_distribution = {{ ansible_distribution }} (basic facts) @@ -1167,19 +1231,19 @@ # W/A Disabled until userspace CNI compilation is fixed - name: check OS for VPP compilation - fail: + ansible.builtin.fail: msg: "VPP is temporarily not supported." when: vpp_enabled is defined and vpp_enabled - name: check OS for VPP compilation - assert: + ansible.builtin.assert: that: - (ansible_distribution == 'Ubuntu' and ansible_distribution_version >= '22.04') msg: "Unsupported configuration. VPP can be only enabled on Ubuntu >= 22.04" when: vpp_enabled is defined and vpp_enabled - name: check OVS DPDK Dependencies - assert: + ansible.builtin.assert: that: >- ({{ ovs_dpdk_enabled }} and not {{ vpp_enabled }} and {{ hugepages_enabled }} and "{{ default_hugepage_size }}" == "1G" and {{ number_of_hugepages_1G }} >= 0) @@ -1192,7 +1256,7 @@ when: ovs_dpdk_enabled is defined and ovs_dpdk_enabled - name: check VPP Dependencies - assert: + ansible.builtin.assert: that: >- ({{ vpp_enabled }} and not {{ ovs_dpdk_enabled }} and {{ hugepages_enabled }} and "{{ default_hugepage_size }}" == "2M" and {{ number_of_hugepages_2M }} >= 0) @@ -1207,7 +1271,7 @@ # STORY: "cnis require net-attach-defs to be enabled" - name: check CNI Config - assert: + ansible.builtin.assert: that: >- ({{ userspace_cni_enabled }} and {{ ovs_dpdk_enabled }} and {{ example_net_attach_defs['userspace_ovs_dpdk'] }} and not {{ vpp_enabled }} and not {{ example_net_attach_defs['userspace_vpp'] }} and {{ hugepages_enabled }} and @@ -1228,13 +1292,13 @@ # STORY: "If SST enabled, confirm minimum kernel or kernel_update specified" - name: check platform before SST-PP verification - command: "cat /sys/devices/cpu/caps/pmu_name" + ansible.builtin.command: "cat /sys/devices/cpu/caps/pmu_name" when: sst_pp_configuration_enabled is defined and sst_pp_configuration_enabled changed_when: true register: verify_platform_for_sst_pp - name: check Intel(R) SST-PP (feature perf-profile) requirements - assert: + ansible.builtin.assert: that: - (ansible_distribution == "Ubuntu" and ansible_distribution_version >= "20.04") or (ansible_os_family == "RedHat" and ansible_distribution_version >= "8.3") @@ -1250,7 +1314,7 @@ - "'skylake' not in verify_platform_for_sst_pp.stdout" - name: Intel(R) SST-PP (feature perf-profile) not available - fail: + ansible.builtin.fail: msg: - "SST-PP is not supported on {{ verify_platform_for_sst_pp.stdout }} platform" - "Make sure sst_pp_configuration_enabled is set to false in host vars" @@ -1262,17 +1326,17 @@ - name: check Intel VT-d on BMs block: - name: Check Intel VT-d - shell: "set -o pipefail && dmesg | grep DMAR | grep remapping" + ansible.builtin.shell: "set -o pipefail && dmesg | grep DMAR | grep remapping" args: executable: /bin/bash register: dmesg_dmar_remap changed_when: false failed_when: false - - debug: msg="dmesg >> {{ dmesg_dmar_remap.stdout }}" + - ansible.builtin.debug: msg="dmesg >> {{ dmesg_dmar_remap.stdout }}" - name: Check Intel VT-d via iommu_groups - shell: "set -o pipefail && compgen -G \"/sys/kernel/iommu_groups/*/devices/*\" > /dev/null" + ansible.builtin.shell: "set -o pipefail && compgen -G \"/sys/kernel/iommu_groups/*/devices/*\" > /dev/null" args: executable: /bin/bash register: iommu_groups_out @@ -1282,7 +1346,7 @@ - dmesg_dmar_remap.stdout|length == 0 - name: Check intel_iommu on kernel commandline - shell: "set -o pipefail && cat /proc/cmdline" + ansible.builtin.shell: "set -o pipefail && cat /proc/cmdline" args: executable: /bin/bash register: iommu_on_cmdline @@ -1291,14 +1355,15 @@ when: - dmesg_dmar_remap.stdout|length == 0 - - debug: msg="Result of 'Check Intel VT-d via iommu_groups' is not valid because of missing 'intel_iommu' parameter on kernel commandline" + - ansible.builtin.debug: + msg: "Result of 'Check Intel VT-d via iommu_groups' is not valid because of missing 'intel_iommu' parameter on kernel commandline" when: - dmesg_dmar_remap.stdout|length == 0 - iommu_groups_out.rc != 0 - "'intel_iommu' not in iommu_on_cmdline.stdout" - name: warn about Intel VT-d - fail: + ansible.builtin.fail: msg: "Warning: Intel VT-d appears DISABLED on target. Please check BIOS under 'Advanced > Integrated IO Configuration' and Enable if necessary" when: - dmesg_dmar_remap.stdout|length == 0 @@ -1306,24 +1371,25 @@ when: - on_vms is not defined or not on_vms - on_cloud is not defined or not on_cloud + - profile_name != "on_prem_aibox" # STORY: Intel Virtualization Technology should be enabled in BIOS - name: check Intel Virtualization Technology on BMs block: - name: Check Intel Virtualization Technology - shell: "set -o pipefail && lscpu | grep 'Virtualization:'" + ansible.builtin.shell: "set -o pipefail && lscpu | grep 'Virtualization:'" args: executable: /bin/bash register: virtualization_tech changed_when: false failed_when: false - - debug: msg="{{ virtualization_tech.stdout }}" + - ansible.builtin.debug: msg="{{ virtualization_tech.stdout }}" when: - virtualization_tech|length > 0 - name: warn about Intel Virtualization Technology - fail: + ansible.builtin.fail: msg: | "Warning: Intel Virtualization Technology is DISABLED on target." "Please check BIOS under 'Advanced > Processor Configuration' and Enable if necessary" @@ -1332,22 +1398,24 @@ when: - vm_enabled | default(false) - on_cloud is not defined or not on_cloud + - not configure_tdx | default(false) # STORY: CPU Hyper-Threading should be enabled in BIOS - - debug: msg="ansible_processor_threads_per_core={{ ansible_processor_threads_per_core }}" - - debug: msg="CPU={{ ansible_processor[2] }} cores={{ ansible_processor_cores }} count={{ ansible_processor_count }} nproc={{ ansible_processor_nproc }} tpc={{ ansible_processor_threads_per_core }} vcpus={{ ansible_processor_vcpus }}" # noqa yaml[line-length] + - ansible.builtin.debug: msg="ansible_processor_threads_per_core={{ ansible_processor_threads_per_core }}" + - ansible.builtin.debug: msg="CPU={{ ansible_processor[2] }} cores={{ ansible_processor_cores }} count={{ ansible_processor_count }} nproc={{ ansible_processor_nproc }} tpc={{ ansible_processor_threads_per_core }} vcpus={{ ansible_processor_vcpus }}" # noqa yaml[line-length] - name: warn about Hyper-Threading - fail: + ansible.builtin.fail: msg: "Warning: Intel Hyper-Threading Tech is DISABLED on target. Please check BIOS under 'Advanced > Processor Configuration' and Enable if necessary" when: - ansible_processor_threads_per_core != 2 - on_cloud is not defined or not on_cloud - configured_arch not in ['atom', 'core'] + - profile_name not in ['on_prem_sw_defined_factory'] # STORY: "collectd and telegraf are mutually exclusive" - name: fail if collectd and telegraf are both enabled - assert: + ansible.builtin.assert: that: >- (({{ collectd_enabled | bool }}) and (not ({{ telegraf_enabled | bool }}))) or (not {{ collectd_enabled | bool }} and {{ telegraf_enabled | bool }}) @@ -1359,7 +1427,7 @@ # STORY: "supported k8s versions require istio in >= 1.10" - name: fail if istio version is not compatible with current k8s version - assert: + ansible.builtin.assert: that: - "{{ istio_service_mesh.version is version('1.10', '>=') }}" msg: | @@ -1373,7 +1441,7 @@ # STORY: "TCS depends on sgx dp and cert manager" - name: check if sgx dp and cert manager are enabled when TCS enabled - assert: + ansible.builtin.assert: that: - "{{ sgx_dp_enabled | default(false) }}" - "{{ cert_manager_enabled | default(false) }}" @@ -1384,7 +1452,7 @@ # STORY: "TAC depends on TCS" - name: check if TCS is enabled when TAC enabled - assert: + ansible.builtin.assert: that: - "{{ tcs.enabled | default(false) }}" - "{{ kmra.apphsm.enabled | default(false) }}" @@ -1396,7 +1464,7 @@ # STORY: "istio_service_mesh.sgx_signer' option is available only for icx platforms" - name: particular service mesh options are available only for specific platforms - assert: + ansible.builtin.assert: that: - "{{ not istio_service_mesh.sgx_signer.enabled | default(false) }}" msg: "'istio_service_mesh.sgx_signer' option is not available for the configured platform architecture." @@ -1406,7 +1474,7 @@ # STORY: "istio_service_mesh.sgx_signer' option must be true when profile is ca_custom" - name: particular service mesh options must be set together - assert: + ansible.builtin.assert: that: - "{{ istio_service_mesh.sgx_signer.enabled | default(false) }}" msg: "'istio_service_mesh.sgx_signer' must be enabled for custom-ca profile." @@ -1418,7 +1486,7 @@ # STORY: TCS is available only for icx platforms" - name: TCS is available only for specific platforms - assert: + ansible.builtin.assert: that: - "{{ not tcs.enabled | default(false) }}" msg: "TCS is not available for the configured platform architecture." @@ -1426,7 +1494,7 @@ - configured_arch not in ['icx'] - name: Make sure istio and linkerd are not enabled at the same time - assert: + ansible.builtin.assert: that: - "{{ not linkerd_service_mesh.enabled | default (false) }}" fail_msg: "You should not have enabled Istio and LinkerD service mesh on at the same time. @@ -1436,7 +1504,7 @@ # STORY: TAC is available only for icx platforms" - name: TAC is available only for specific platforms - assert: + ansible.builtin.assert: that: - "{{ not tac.enabled | default(false) }}" msg: "TAC is not available for the configured platform architecture." @@ -1445,7 +1513,7 @@ # STORY: "istio_service_mesh.sgx_signer' option depends on KMRA AppHSM, KMRA PCCS, TCS, TAC. - name: check if KMRA Apps, TCS and TAC are enabled when service mesh sgx_signer option is enabled - assert: + ansible.builtin.assert: that: - "{{ kmra.apphsm.enabled | default(false) }}" - "{{ kmra.pccs.enabled | default(false) }}" @@ -1457,7 +1525,7 @@ - configured_arch in ['icx'] # STORY: TEMPORARY: "ovs dpdk version requirements" - - debug: + - ansible.builtin.debug: msg: - install_dpdk = {{ install_dpdk }} (host_vars) - dpdk_version = {{ dpdk_version }} (host_vars) @@ -1469,10 +1537,13 @@ - ovs_version is defined # host_vars - ovs_dpdk_enabled is defined and ovs_dpdk_enabled # host_vars + # Refer https://docs.openvswitch.org/en/latest/faq/releases/ to get OVS DPDK compatibility - name: check OVS DPDK compatibility - assert: + ansible.builtin.assert: that: - ovs_version == 'v3.1.1' and (dpdk_version == '23.03' or dpdk_version == '22.11.1') + ovs_version == 'v3.2.0' and dpdk_version == '22.11.1' + or ovs_version == 'v3.1.1' and dpdk_version == '22.11.1' + or ovs_version == 'v3.0.1' and dpdk_version == '21.11.2' or (ovs_version >= 'v2.17.0' and ovs_version <= 'v3.0.3') and (dpdk_version >= '21.11' and dpdk_version <= '22.07') or (ovs_version < 'v2.16.2' and ovs_version >= 'v2.16.0') and dpdk_version == '21.08' or ovs_version == 'v2.15.0' and dpdk_version == '20.11' @@ -1489,73 +1560,20 @@ - ovs_version is defined # host_vars - ovs_dpdk_enabled is defined and ovs_dpdk_enabled # host_vars - - name: check settings for Intel Power Manager - assert: - that: - - intel_power_manager.power_nodes | length > 0 - fail_msg: "Intel Power Manager is enabled, but Power Nodes are not specified in group vars." - when: intel_power_manager is defined and intel_power_manager.enabled - - - name: check if power_nodes are available in inventory - assert: - that: - - item in groups['kube_node'] - fail_msg: "Intel Power Manager power_nodes have to be present in inventory. '{{ item }}' is not there: {{ groups['kube_node'] }}" - loop: "{{ intel_power_manager.power_nodes }}" - when: intel_power_manager is defined and intel_power_manager.enabled - - # kubelet cpuManagerPolicy should be 'static' for IPM 2.2.0 and higher - - name: check if native_cpu_manager is enabled - assert: - that: - - native_cpu_manager_enabled - fail_msg: "Please set 'native_cpu_manager_enabled' to 'true' in group vars" - when: intel_power_manager is defined and intel_power_manager.enabled - - # kubelet reservedSystemCPUs should be set to desired value - - name: check if reserved cpus are defined in host vars - assert: - that: - - (native_cpu_manager_system_reserved_cpus is defined - and native_cpu_manager_kube_reserved_cpus is defined - and native_cpu_manager_reserved_cpus is not defined) or - (native_cpu_manager_system_reserved_cpus is not defined - and native_cpu_manager_kube_reserved_cpus is not defined - and native_cpu_manager_reserved_cpus is defined) - - fail_msg: "Reserved cpus are not defined. - Please configure ('native_cpu_manager_system_reserved_cpus' and 'native_cpu_manager_kube_reserved_cpus') - or 'native_cpu_manager_reserved_cpus' in host vars." - when: intel_power_manager is defined and intel_power_manager.enabled - - - name: check if Intel Power Manager is enabled, the ISST features should be disabled - assert: - that: - - not (sst_bf_configuration_enabled is defined and sst_bf_configuration_enabled or - sst_cp_configuration_enabled is defined and sst_cp_configuration_enabled or - sst_tf_configuration_enabled is defined and sst_tf_configuration_enabled or - sst_pp_configuration_enabled is defined and sst_pp_configuration_enabled) - fail_msg: - - "Currently Intel Power Manager and Intel SST features are mutually exclusive." - - "Please disable ISST (SST-BF, SST-CP, SST-TF and SST-PP) in host vars." - when: intel_power_manager is defined and intel_power_manager.enabled - - - name: check if Intel Power Manager is build locally on containerd/cri-o runtime - assert: - that: intel_power_manager.build_image_locally - fail_msg: - - "Currently Intel Power Manager must be build locally on containerd and cri-o runtime" - - "Please set build_image_locally as true in Intel Power Manager settings in group_vars" - when: intel_power_manager is defined and intel_power_manager.enabled and container_runtime in ["crio", "containerd"] + - name: check Kubernetes Power Manager configuration + ansible.builtin.include_role: + name: kubernetes_power_manager + tasks_from: kpm_preflight + when: kubernetes_power_manager is defined and kubernetes_power_manager.enabled - name: check Intel Ethernet Operator configuration - include_role: + ansible.builtin.include_role: name: intel_ethernet_operator tasks_from: preflight_ethernet_operator when: intel_ethernet_operator_enabled is defined and intel_ethernet_operator_enabled - name: make sure isolcpus and cpusets are not enabled simultaneously - assert: + ansible.builtin.assert: that: - (not isolcpus_enabled and cpusets_enabled) or (isolcpus_enabled and not cpusets_enabled) @@ -1567,7 +1585,7 @@ (cpusets_enabled is defined and cpusets_enabled) - name: check Intel SRIOV-FEC Operator requirements - include_role: + ansible.builtin.include_role: name: intel_sriov_fec_operator tasks_from: preflight_sriov_fec_operator when: @@ -1575,19 +1593,25 @@ - intel_flexran_type != "pod" - name: check Intel FlexRAN requirements - include_role: + ansible.builtin.include_role: name: intel_flexran tasks_from: flexran_preflight when: intel_flexran_enabled | default(false) | bool - name: check Intel ECI requirements - include_role: + ansible.builtin.include_role: name: intel_eci tasks_from: eci_preflight when: intel_eci_enabled | default(false) | bool + - name: check Intel CSL EXCAT requirements + include_role: + name: intel_csl_excat + tasks_from: preflight + when: intel_csl_excat_enabled | default(false) | bool + - name: check OS when DLB or DSA is enabled - assert: + ansible.builtin.assert: that: - (ansible_distribution == "Ubuntu" and ansible_distribution_version == '20.04' and (update_kernel or ansible_kernel[0:4] is version('5.14', '>='))) or @@ -1606,14 +1630,14 @@ - name: Check requirements to enable Intel SGX on VMs block: - name: Intel SGX - check if ansible_host distro is Ubuntu 22.04 - assert: + ansible.builtin.assert: that: - ansible_distribution == "Ubuntu" - ansible_distribution_version == "22.04" msg: "Deploying SGX on VMRA is supported only on Ubuntu 22.04 VM host. Please change the o/s for VM host" - name: Check if configured SGX memory is not bigger than total memory - assert: + ansible.builtin.assert: that: - item.memory > sgx_memory_size msg: | @@ -1627,6 +1651,42 @@ - sgx_dp_enabled | default(false) - inventory_hostname in groups['vm_host'] +# Storage preflight check + - name: check storage persistent volumes + ansible.builtin.assert: + that: "persistent_volumes != []" + msg: "Persistent volumes on target '{{ ansible_hostname }}' must be set in host_vars. Please correct the configuration" + when: + - local_volume_provisioner_enabled | default(false) or + rook_ceph.enabled | default(false) or + minio_enabled | default(false) + - not storage_deploy_test_mode | default(false) + - storage_nodes | default([]) | length == 0 or + inventory_hostname in storage_nodes | default([]) + +# TDX preflight check + - name: check tdx requirement + include_role: + name: bootstrap/install_tdx_drivers # noqa role-name[path] - role in bootstrap + tasks_from: tdx_preflight.yml + when: + - configure_tdx | default(false) + - not on_vms | default(false) + + - name: check Intel ECI VM Operating System + assert: + that: + - vm_image_distribution | default("ubuntu") == "ubuntu" + - vm_image_version_ubuntu | default("22.04") == "22.04" + when: intel_eci_enabled | default(false) | bool + +# STORY: Check requirements for Intel Media Transport Library + - name: check Intel Media Transport Library requirements + ansible.builtin.include_role: + name: imtl_install + tasks_from: preflight.yml + when: intel_media_transport_library_enabled | default(false) + #################################### # Prerequisites for FlexRAN BM oRU # #################################### @@ -1634,7 +1694,7 @@ any_errors_fatal: true tasks: - name: check Intel FlexRAN requirements - include_role: + ansible.builtin.include_role: name: intel_flexran tasks_from: flexran_preflight when: @@ -1649,13 +1709,13 @@ any_errors_fatal: true tasks: - name: Include Sigstore policy controller checks - import_role: + ansible.builtin.import_role: name: sigstore_policy_controller tasks_from: preflight.yml when: sigstore_policy_controller_install | default(false) | bool - name: Include oneAPI kits checks - import_role: + ansible.builtin.import_role: name: intel_oneapi_install tasks_from: preflight.yml tags: intel-oneapi @@ -1663,7 +1723,7 @@ # STORY: "Observability: assert that all required compontents are enabled" - name: assert that all observability/monitoring variables are disabled - assert: + ansible.builtin.assert: that: - not elasticsearch_enabled | default(false) - not jaeger_operator | default(false) @@ -1676,7 +1736,7 @@ - not collectd_enabled | default(false) - name: assert that all observability/monitoring variables are enabled for telegraf - assert: + ansible.builtin.assert: that: - telegraf_enabled | default(false) - elasticsearch_enabled | default(false) @@ -1689,7 +1749,7 @@ - telegraf_enabled | default(false) - name: assert that all observability/monitoring variables are disabled for collectd - assert: + ansible.builtin.assert: that: - not telegraf_enabled | default(false) - not elasticsearch_enabled | default(false) @@ -1719,15 +1779,27 @@ tasks_from: preflight when: cadvisor_enabled | default(false) + - name: Check XPUManager configuration + ansible.builtin.include_role: + name: intel_xpumanager + tasks_from: xpumanager_preflight + when: intel_xpumanager_enabled | default(false) + - name: check ADQ configuration - include_role: + ansible.builtin.include_role: name: adq_dp_install tasks_from: preflight_adq when: adq_dp.enabled | default(false) + - name: check Calico VPP configuration + ansible.builtin.include_role: + name: calico_vpp_install + tasks_from: calico_vpp_preflight + when: calico_vpp.enabled | default(false) + # STORY: "MinIO requires number of nodes should be more than the minimum number of nodes defined in group_vars/all/minio_tenant_servers" - name: display MinIO requirement with multus-service - fail: + ansible.builtin.fail: msg: | MinIO deployment for k8s service on additional interfaces with multiple interface/Multus CNI(multus-service) requires: - group_vars/kube_network_plugin: flannel @@ -1746,7 +1818,7 @@ # When MinIO enabled, container runtime should be set to crio - name: Check requirements for MinIO - assert: + ansible.builtin.assert: that: - container_runtime == "crio" - (kube_network_plugin == "flannel") or @@ -1784,7 +1856,7 @@ - minio_enabled is defined and minio_enabled - name: check MinIO configuration - import_role: + ansible.builtin.import_role: name: minio_install tasks_from: preflight_minio_main tags: minio @@ -1792,7 +1864,18 @@ - kubernetes - minio_enabled is defined and minio_enabled - - meta: end_play +# STORY: Check Intent Driven Orchestration (IDO) dependencies + - name: assert that all IDO dependencies are enabled + assert: + that: + - registry_enable | default(false) + - linkerd_service_mesh.enabled | d(false) + fail_msg: | + Local Docker registry (registry_enable) and LinkerD (linkerd_service_mesh.enable) must be enabled when using IDO. + when: + - ido.enabled | default(false) + + - ansible.builtin.meta: end_play # - name: Print all variables/facts known for a host # ansible.builtin.debug: # var: hostvars[inventory_hostname] diff --git a/playbooks/redeploy_cleanup.yml b/playbooks/redeploy_cleanup.yml index 1b2414a5..8321873b 100644 --- a/playbooks/redeploy_cleanup.yml +++ b/playbooks/redeploy_cleanup.yml @@ -16,4 +16,3 @@ --- - name: prepare for re-deployment import_playbook: infra/redeploy_cleanup.yml - when: kubernetes | default(true) diff --git a/playbooks/versions.yml b/playbooks/versions.yml index 90cb182a..0f6d95e1 100644 --- a/playbooks/versions.yml +++ b/playbooks/versions.yml @@ -56,20 +56,24 @@ } - { 'description' : 'Prometheus', 'shortname' : 'prometheus_stack_version', - 'var_file_path' : 'roles/kube_prometheus/defaults/main.yml' + 'var_file_path' : 'roles/prometheus_install/defaults/main.yml' } - { 'description' : 'Kube State Metrics', 'shortname' : 'kube_state_metrics_version', - 'var_file_path' : 'roles/kube_prometheus/defaults/main.yml' + 'var_file_path' : 'roles/prometheus_install/kube_prometheus/defaults/main.yml' } - { 'description' : 'Grafana', 'shortname' : 'grafana_version', - 'var_file_path' : 'roles/kube_prometheus/defaults/main.yml' + 'var_file_path' : 'roles/prometheus_install/defaults/main.yml' } - { 'description' : 'CollectD', 'shortname' : "image_collectd\\'\\]\\[\\'digest", 'var_file_path' : 'roles/collectd_install/defaults/main.yml' } + - { 'description' : 'Intel XPUManager', + 'shortname' : 'xpumanager_version', + 'var_file_path' : 'roles/intel_xpumanager/defaults/main.yml' + } - { 'description' : 'Docker', 'shortname' : 'docker_version', 'var_file_path' : 'roles/container_engine/docker/defaults/main.yml' @@ -93,15 +97,15 @@ 'var_file_path' : 'roles/rke2_kubernetes_apps/rancher/defaults/main.yml' } - { 'description' : 'k8s node-exporter', - 'var_file_path' : 'roles/kube_prometheus/defaults/main.yml', + 'var_file_path' : 'roles/prometheus_install/defaults/main.yml', 'shortname' : 'node_exporter_version' } - { 'description' : 'k8s prometheus-operator', - 'var_file_path' : 'roles/kube_prometheus/defaults/main.yml', + 'var_file_path' : 'roles/prometheus_install/kube_prometheus/defaults/main.yml', 'shortname' : "prometheus_operator_version" } - { 'description' : 'k8s prometheus-adapter', - 'var_file_path' : 'roles/kube_prometheus/files/kube-prometheus-stack/prometheusAdapter-clusterRole.yaml', + 'var_file_path' : 'roles/prometheus_install/kube_prometheus/files/kube-prometheus-stack/prometheusAdapter-clusterRole.yaml', 'shortname' : "metadata\\'\\]\\[\\'labels\\'\\]\\[\\'app.kubernetes.io/version" } - { 'description' : 'k8s kube-rbac-proxy', @@ -124,6 +128,10 @@ 'shortname' : 'calico_version', 'var_file_path' : 'collections/ansible_collections/kubernetes_sigs/kubespray/roles/download/defaults/main.yml' } + - { 'description' : 'calico vpp dataplane', + 'shortname' : 'k8s_calico_vpp_version', + 'var_file_path' : 'roles/calico_vpp_install/defaults/main.yml' + } - { 'description' : 'flannel', 'shortname' : 'flannel_version', 'var_file_path' : 'collections/ansible_collections/kubernetes_sigs/kubespray/roles/download/defaults/main.yml' @@ -220,10 +228,6 @@ 'var_file_path' : 'roles/intel_ethernet_operator/defaults/main.yml', 'shortname' : 'intel_ethernet_operator_git_ref' } - - { 'description' : 'Intel® Ethernet Operator image', - 'var_file_path' : 'roles/intel_ethernet_operator/defaults/main.yml', - 'shortname' : 'intel_ethernet_operator_img_ver' - } - { 'description' : 'Intel® Ethernet UFT', 'var_file_path' : 'roles/intel_ethernet_operator/defaults/main.yml', 'shortname' : 'uft_git_ref' @@ -504,6 +508,18 @@ 'var_file_path' : 'roles/jaeger_install/defaults/main.yml', 'shortname' : "jaeger_version" } + - { 'description' : 'Intel csl-excat', + 'var_file_path' : 'roles/intel_csl_excat/defaults/main.yml', + 'shortname' : 'csl_excat_version' + } + - { 'description' : 'Intent Driven Orchestration (IDO)', + 'var_file_path' : 'roles/intent_driven_orchestration/defaults/main.yml', + 'shortname' : "ido_git_version" + } + - { 'description' : 'Intel Media Transport Library', + 'var_file_path' : 'roles/imtl_install/defaults/main.yml', + 'shortname' : 'imtl_version' + } - name: Remove old version parsing results file: path: "{{ item }}" diff --git a/requirements.txt b/requirements.txt index b0ce6b1a..61bf442d 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,11 +1,11 @@ -ansible==5.7.1 -ansible-core==2.12.5 -cryptography==41.0.2 +ansible==7.7.0 +ansible-core==2.14.9 +cryptography==41.0.4 jinja2==3.1.2 -netaddr==0.7.19 -pbr==5.4.4 -jmespath==0.9.5 -ruamel.yaml==0.17.21 -ruamel.yaml.clib==0.2.6 -MarkupSafe==2.1.1 +netaddr==0.8.0 +pbr==5.11.1 +jmespath==1.0.1 +ruamel.yaml==0.17.32 +ruamel.yaml.clib==0.2.7 +MarkupSafe==2.1.3 ipaddr diff --git a/roles/bond_cni_install/defaults/main.yml b/roles/bond_cni_install/defaults/main.yml index 789c7067..eda20b84 100644 --- a/roles/bond_cni_install/defaults/main.yml +++ b/roles/bond_cni_install/defaults/main.yml @@ -16,4 +16,4 @@ --- bond_cni_git_url: "https://github.com/intel/bond-cni.git" bond_cni_dir: "{{ (project_root_dir, 'bond-cni') | path_join }}" -bond_cni_version: "98008131ca7c6bfb3cdf0021846b58891375686f" +bond_cni_version: "1578bc19a1cb8ea137abab77f682fc3d5ebfdc54" diff --git a/roles/bootstrap/auto_detect_qat_devices/tasks/main.yml b/roles/bootstrap/auto_detect_qat_devices/tasks/main.yml index 46d126bd..6c1989e1 100644 --- a/roles/bootstrap/auto_detect_qat_devices/tasks/main.yml +++ b/roles/bootstrap/auto_detect_qat_devices/tasks/main.yml @@ -61,9 +61,19 @@ - name: Create new_qat_devices list set_fact: - new_qat_devices: "{{ new_qat_devices | default([]) + [{ 'qat_id' : '0000:' + item, 'qat_sriov_numvfs' : qat_sriov_numvfs_required, 'qat_default_vf_driver' : qat_vf_driver_required }] }}" # noqa yaml[line-length] + new_qat_devices: + "{{ new_qat_devices | default([]) + [{ 'qat_id' : '0000:' + item, 'qat_sriov_numvfs' : qat_sriov_numvfs_required, + 'qat_default_vf_driver' : qat_vf_driver_required }] }}" with_items: "{{ qat_id_values }}" changed_when: true + when: not on_vms | default(false) + +- name: Create new_qat_devices list on VMs + set_fact: + new_qat_devices: "{{ new_qat_devices | default([]) + [{ 'qat_id' : '0000:' + item, 'qat_sriov_numvfs' : qat_sriov_numvfs_required }] }}" + with_items: "{{ qat_id_values }}" + changed_when: true + when: on_vms | default(false) - name: replace original qat_devices set_fact: diff --git a/roles/bootstrap/configure_intel_pstate/tasks/main.yml b/roles/bootstrap/configure_disks/defaults/main.yml similarity index 82% rename from roles/bootstrap/configure_intel_pstate/tasks/main.yml rename to roles/bootstrap/configure_disks/defaults/main.yml index 34bb9470..ced24738 100644 --- a/roles/bootstrap/configure_intel_pstate/tasks/main.yml +++ b/roles/bootstrap/configure_disks/defaults/main.yml @@ -14,7 +14,9 @@ ## limitations under the License. ## --- -- name: setup intel_pstate driver - include_tasks: setup_intel_pstate.yml - when: - - intel_pstate_enabled | default(false) | bool +disk_mount_path: /mnt/disks/ra_disk + +simulated_disk_num: 6 + +# Unit GiB +size_per_fake_volume: 10 diff --git a/roles/bootstrap/configure_disks/files/ra_loopdevices.service b/roles/bootstrap/configure_disks/files/ra_loopdevices.service new file mode 100644 index 00000000..8f36b70c --- /dev/null +++ b/roles/bootstrap/configure_disks/files/ra_loopdevices.service @@ -0,0 +1,18 @@ +[Unit] +Description=RA Setup loop devices +DefaultDependencies=no +Conflicts=umount.target +Before=local-fs.target +After=systemd-udevd.service home.mount +Requires=systemd-udevd.service + +[Service] +Type=oneshot +ExecStart=/sbin/loopdevice_bind +ExecStop=/sbin/loopdevice_bind -d +TimeoutSec=60 +RemainAfterExit=yes + +[Install] +WantedBy=local-fs.target +Also=systemd-udevd.service \ No newline at end of file diff --git a/roles/bootstrap/configure_disks/tasks/configure_loopdevices.yml b/roles/bootstrap/configure_disks/tasks/configure_loopdevices.yml new file mode 100644 index 00000000..ed063d9b --- /dev/null +++ b/roles/bootstrap/configure_disks/tasks/configure_loopdevices.yml @@ -0,0 +1,109 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +- name: get ra_loopdevices service status + ansible.builtin.systemd: + name: ra_loopdevices + register: service_stsatus + +- name: stop the ra_loopdevices service + ansible.builtin.service: + name: "ra_loopdevices" + state: stopped + enabled: false + when: + - service_stsatus.status.ActiveState == 'active' + +- name: initialize the fake loop device persistent volumes list + ansible.builtin.set_fact: + fake_pvs: [] + +- name: set iteration number + ansible.builtin.set_fact: + num: 1 + req_num: "{{ simulated_disk_num }}" + +- name: add directory for disk image + ansible.builtin.file: + path: /opt/cek/disks/tmp/ + state: directory + mode: '0755' + +- name: create file block device for loop devices + ansible.builtin.command: >- + dd + if=/dev/zero of=/opt/cek/disks/tmp/diskimage{{ item }} + bs=1M + count="{{ size_per_fake_volume |int * 1024 }}" + changed_when: true + with_sequence: start=1 end={{ req_num }} stride=1 + +- name: format created loop device to ext4 file format + community.general.filesystem: + fstype: ext4 + dev: /opt/cek/disks/tmp/diskimage{{ item }} + force: yes + state: present + with_sequence: start=1 end={{ req_num }} stride=1 + +- name: create mounted folder + ansible.builtin.file: + path: "{{ disk_mount_path }}{{ item }}" + state: directory + mode: '0755' + with_sequence: start=1 end={{ req_num }} stride=1 + +- name: populate loopdevice_bind script + ansible.builtin.template: + src: loopdevice_bind.j2 + dest: /sbin/loopdevice_bind + mode: 0755 + +- name: copy the service file to storage node + ansible.builtin.copy: + src: ra_loopdevices.service + dest: /lib/systemd/system/ra_loopdevices.service + mode: 0755 + become: yes + +- name: ensure that systemd service is enabled and restarted + ansible.builtin.systemd: + name: ra_loopdevices.service + state: restarted + enabled: yes + daemon_reload: yes + become: yes + +- name: list loop devices + ansible.builtin.shell: >- + set -o pipefail && losetup -l |grep diskimage |awk -F " " '{ print $6,$1}' |sort |awk -F " " '{ print $2}' + args: + executable: /bin/bash + register: loopdevice_output + changed_when: true + +- name: generate new persistent volumes + include_tasks: setup_fake_pvs.yml + with_indexed_items: + - "{{ loopdevice_output.stdout_lines }}" + +- name: show loop device information + ansible.builtin.debug: + msg="{{ fake_pvs }}" + +- name: set persistent volumes for storage component deployment + ansible.builtin.set_fact: + persistent_volumes: "{{ fake_pvs }}" diff --git a/roles/bootstrap/configure_disks/tasks/main.yml b/roles/bootstrap/configure_disks/tasks/main.yml new file mode 100644 index 00000000..29eca98c --- /dev/null +++ b/roles/bootstrap/configure_disks/tasks/main.yml @@ -0,0 +1,56 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +- name: set all work node as storage node by default + delegate_to: localhost + run_once: true + ansible.builtin.set_fact: + storage_nodes: "{{ groups['kube_node'] }}" + when: + - local_volume_provisioner_enabled | default(false) or + rook_ceph.enabled | default(false) or + minio_enabled | default(false) + - storage_nodes | length() == 0 + +- name: block to configure loop device + block: + - name: set fact to configure loop devices + ansible.builtin.set_fact: + use_loopdevice: true + + - name: generate loop device persistent volumes for storage deployment + include_tasks: configure_loopdevices.yml + when: + - storage_deploy_test_mode | default(false) | bool + - hostvars[inventory_hostname]['persistent_volumes'] | default([]) | length == 0 + - inventory_hostname in storage_nodes | default([]) + +- name: block to configure real disks + block: + - name: set iteration number + ansible.builtin.set_fact: + num: 1 + req_num: "{{ hostvars[inventory_hostname]['persistent_volumes'] | length }}" + + - name: mount storage disks + include_tasks: mount_disks.yml + loop: "{{ range(num, req_num|int + 1) | list }}" + loop_control: + extended: true + when: + - not use_loopdevice | default(false) + - hostvars[inventory_hostname]['persistent_volumes'] | default([]) | length > 0 + - inventory_hostname in storage_nodes | default([]) diff --git a/roles/bootstrap/configure_disks/tasks/mount_disks.yml b/roles/bootstrap/configure_disks/tasks/mount_disks.yml new file mode 100644 index 00000000..2d569a4f --- /dev/null +++ b/roles/bootstrap/configure_disks/tasks/mount_disks.yml @@ -0,0 +1,44 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +- name: wipe all existing filesystem on the device + ansible.builtin.command: "wipefs -a {{ hostvars[inventory_hostname]['persistent_volumes'][ansible_loop.index0].device }}" + changed_when: false + failed_when: false + +- name: format disks with filesystem + block: + - name: format the storage disks + community.general.filesystem: + fstype: "{{ hostvars[inventory_hostname]['persistent_volumes'][ansible_loop.index0].fsType }}" + dev: "{{ hostvars[inventory_hostname]['persistent_volumes'][ansible_loop.index0].device }}" + force: yes + state: present + + - name: create mount point for the file block devices + ansible.builtin.file: + path: "{{ hostvars[inventory_hostname]['persistent_volumes'][ansible_loop.index0].mountPath }}" + state: directory + mode: '0755' + - name: mount the parition + ansible.posix.mount: + fstype: "{{ hostvars[inventory_hostname]['persistent_volumes'][ansible_loop.index0].fsType }}" + src: "{{ hostvars[inventory_hostname]['persistent_volumes'][ansible_loop.index0].device }}" + path: "{{ hostvars[inventory_hostname]['persistent_volumes'][ansible_loop.index0].mountPath }}" + state: mounted + when: + - minio_enabled | default(false) | bool or + local_volume_provisioner_enabled | default(false) | bool diff --git a/roles/bootstrap/configure_disks/tasks/setup_fake_pvs.yml b/roles/bootstrap/configure_disks/tasks/setup_fake_pvs.yml new file mode 100644 index 00000000..b86238c8 --- /dev/null +++ b/roles/bootstrap/configure_disks/tasks/setup_fake_pvs.yml @@ -0,0 +1,30 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +- name: generate new persistent volumes + ansible.builtin.set_fact: + new_pv: + - name: "mnt-data-{{ item.0 }}" + accessMode: "ReadWriteOnce" + storageClassName: "local-storage" + persistentVolumeReclaimPolicy: "Delete" + mountPath: "{{ disk_mount_path }}{{ item.0 | int + 1 }}" + device: "{{ item.1 }}" + fsType: "ext4" + +- name: add new persistent volumes to list + ansible.builtin.set_fact: + fake_pvs: "{{ fake_pvs | default([]) + new_pv }}" diff --git a/roles/bootstrap/configure_disks/templates/loopdevice_bind.j2 b/roles/bootstrap/configure_disks/templates/loopdevice_bind.j2 new file mode 100644 index 00000000..c3b8dcd8 --- /dev/null +++ b/roles/bootstrap/configure_disks/templates/loopdevice_bind.j2 @@ -0,0 +1,24 @@ +#! /bin/sh + +max_devices={{ simulated_disk_num }} + +bind_loopdevices() { + for i in `seq 1 $max_devices` + do + mount -t ext4 -o loop /opt/cek/disks/tmp/diskimage$i /mnt/disks/ra_disk$i + echo "mounted /opt/cek/disks/tmp/diskimage$i to /mnt/disks/ra_disk$i" + done +} + +detach_loopdevices() { + umount /mnt/disks/ra_disk? +} + +option=${1:-""} + +if [ "$option" = "-d" ]; +then + detach_loopdevices +else + bind_loopdevices +fi diff --git a/roles/bootstrap/configure_dlb/defaults/main.yml b/roles/bootstrap/configure_dlb/defaults/main.yml index 013e3ccb..ac9212c2 100644 --- a/roles/bootstrap/configure_dlb/defaults/main.yml +++ b/roles/bootstrap/configure_dlb/defaults/main.yml @@ -14,6 +14,6 @@ ## limitations under the License. ## --- -intel_dlb_driver_ver: "dlb_linux_src_release_8.4.0" -intel_dlb_driver_url: "https://downloadmirror.intel.com/779942/{{ intel_dlb_driver_ver }}.txz" -intel_dlb_driver_checksum: "sha1:AED28FE711213913D2B34160A7047199BEA46D97" +intel_dlb_driver_ver: "dlb_linux_src_release_8.5.1" +intel_dlb_driver_url: "https://downloadmirror.intel.com/787629/{{ intel_dlb_driver_ver }}.txz" +intel_dlb_driver_checksum: "sha1:53BA0D21CDB5EBAAAB247FC9C73FAED7F0899B95" diff --git a/roles/bootstrap/configure_dlb/tasks/main.yml b/roles/bootstrap/configure_dlb/tasks/main.yml index 76b58de3..10a7e465 100644 --- a/roles/bootstrap/configure_dlb/tasks/main.yml +++ b/roles/bootstrap/configure_dlb/tasks/main.yml @@ -63,14 +63,16 @@ executable: /bin/bash when: ansible_os_family == "RedHat" -# Build workaround on Rocky 9.1 +# Build workaround on Rocky 9.1/9.2 # Driver's Makefile condition on disabling SIOV on RHEL does not work on Rocky. As a workaround disabling SIOV manually. - name: "Disable SIOV on Rocky 9.1 (workaround)" - replace: + ansible.builtin.replace: dest: "{{ project_root_dir }}/dlb/driver/dlb2/Makefile" regexp: '^\s*ccflags-y \+= -DCONFIG_INTEL_DLB2_SIOV' replace: '#ccflags-y += -DCONFIG_INTEL_DLB2_SIOV' - when: ansible_distribution == 'Rocky' and ansible_distribution_version == '9.1' + when: + - ansible_distribution == 'Rocky' + - ansible_distribution_version == '9.1' or ansible_distribution_version == '9.2' - name: build Intel DLB driver make: diff --git a/roles/bootstrap/configure_kpm_drivers/tasks/configure_drivers.yml b/roles/bootstrap/configure_kpm_drivers/tasks/configure_drivers.yml new file mode 100644 index 00000000..73b10f0e --- /dev/null +++ b/roles/bootstrap/configure_kpm_drivers/tasks/configure_drivers.yml @@ -0,0 +1,59 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +- name: Uncore Frequency driver + when: hostvars[inventory_hostname]['uncore_frequency']['enabled'] | default(false) | bool + block: + - name: enable Uncore Frequency driver + community.general.modprobe: + name: intel_uncore_frequency + state: present + persistent: present + +- name: acpi_cpufreq scaling driver + when: + - hostvars[inventory_hostname]['frequency_scaling_driver'] == "acpi_cpufreq" + - inventory_hostname in kubernetes_power_manager.power_nodes + block: + - name: set kernel flags to disable intel_pstate scaling driver + ansible.builtin.set_fact: + intel_pstate_cmdline: 'GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} intel_pstate=disable intel_pstate=no_hwp"' + + - name: disable intel_pstate in /etc/default/grub + ansible.builtin.lineinfile: + dest: /etc/default/grub + line: '{{ intel_pstate_cmdline }}' + state: present + mode: '0664' + + - name: run update-grub command on Ubuntu + ansible.builtin.command: update-grub + changed_when: false + when: ansible_distribution == 'Ubuntu' + + - name: update grub on RHEL + ansible.builtin.command: grub2-mkconfig -o /boot/grub2/grub.cfg + changed_when: false + when: ansible_distribution == 'Rocky' or + ansible_distribution == 'RedHat' + +- name: reboot server + debug: + msg: Rebooting server + notify: + - reboot server + when: + - hostvars[inventory_hostname]['frequency_scaling_driver'] == "acpi_cpufreq" or + hostvars[inventory_hostname]['uncore_frequency']['enabled'] | default(false) | bool diff --git a/roles/bootstrap/configure_kpm_drivers/tasks/main.yml b/roles/bootstrap/configure_kpm_drivers/tasks/main.yml new file mode 100644 index 00000000..f436dc6f --- /dev/null +++ b/roles/bootstrap/configure_kpm_drivers/tasks/main.yml @@ -0,0 +1,17 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +- name: manage power manager drivers + ansible.builtin.include_tasks: configure_drivers.yml diff --git a/roles/bootstrap/configure_openssl/defaults/main.yml b/roles/bootstrap/configure_openssl/defaults/main.yml index 28fe7f07..83441d99 100644 --- a/roles/bootstrap/configure_openssl/defaults/main.yml +++ b/roles/bootstrap/configure_openssl/defaults/main.yml @@ -15,6 +15,6 @@ ## --- openssl_url: "https://github.com/openssl/openssl.git" -openssl_version: "openssl-3.1.1" +openssl_version: "openssl-3.1.2" openssl_dir: "{{ (project_root_dir, 'openssl') | path_join }}" openssl_pkg_subdir: "{{ openssl_dir }}/{{ openssl_version }}" diff --git a/roles/bootstrap/configure_openssl/tasks/main.yml b/roles/bootstrap/configure_openssl/tasks/main.yml index bb0c6cc6..a97d0e1f 100644 --- a/roles/bootstrap/configure_openssl/tasks/main.yml +++ b/roles/bootstrap/configure_openssl/tasks/main.yml @@ -52,7 +52,10 @@ msg: "Failed to start {{ enabled_qat_service }} service on system. Please check if QAT configuration in host vars is correct." when: - "'up' not in service_check.stdout" - when: update_qat_drivers | default(false) + when: + - update_qat_drivers | default(false) + - qat_devices | default([]) | length > 0 + - not qat_oot_driver_build_failed | default(false) # OpenSSL build - name: create directory {{ openssl_dir }} for all OpenSSL dependencies diff --git a/roles/bootstrap/configure_qat/tasks/check_qat_status.yml b/roles/bootstrap/configure_qat/tasks/check_qat_status.yml index 76354b92..8da0b175 100644 --- a/roles/bootstrap/configure_qat/tasks/check_qat_status.yml +++ b/roles/bootstrap/configure_qat/tasks/check_qat_status.yml @@ -14,42 +14,53 @@ ## limitations under the License. ## --- -- name: confirm QAT module is loaded - ansible.builtin.shell: "set -o pipefail && lsmod | grep qat" - args: - executable: /bin/bash - register: qat_confirm_mod - changed_when: false - ignore_errors: true +- name: skip QAT module check for InTree driver + when: + - update_qat_drivers | default(false) + block: + - name: confirm QAT module is loaded + ansible.builtin.shell: "set -o pipefail && lsmod | grep qat" + args: + executable: /bin/bash + register: qat_confirm_mod + changed_when: false + ignore_errors: true -- name: QAT kernel module not found - ansible.builtin.fail: - msg: "No QAT module found. Please set update_qat_drivers to true in host vars to resolve the issue." - when: '"intel_qat" not in qat_confirm_mod.stdout' + - name: QAT kernel module not found + ansible.builtin.fail: + msg: "No QAT module found. Please set update_qat_drivers to true in host vars to resolve the issue." + when: '"intel_qat" not in qat_confirm_mod.stdout' -- name: make sure {{ enabled_qat_service }} service is started and enabled - ansible.builtin.service: - name: "{{ enabled_qat_service }}" - state: started - enabled: true +- name: skip {{ enabled_qat_service }} service check on VMs + when: + - not on_vms | default(false) + block: + - name: make sure {{ enabled_qat_service }} service is started and enabled + ansible.builtin.service: + name: "{{ enabled_qat_service }}" + state: started + enabled: true -- name: disable the multi-user.target in qat.service to avoid order cycle - ansible.builtin.lineinfile: - path: /lib/systemd/system/qat.service - regexp: "^After=multi-user.target" - line: "#After=multi-user.target" - mode: 0644 - become: true + - name: disable the multi-user.target in {{ enabled_qat_service }} service to avoid order cycle + ansible.builtin.lineinfile: + path: "/lib/systemd/system/{{ enabled_qat_service }}.service" + regexp: "^After=multi-user.target" + line: "#After=multi-user.target" + mode: 0644 + become: true -- name: restart the qat.service - ansible.builtin.systemd: - name: qat.service - state: restarted - enabled: true + - name: restart the {{ enabled_qat_service }} service + ansible.builtin.systemd: + name: "{{ enabled_qat_service }}" + state: restarted + daemon_reload: true + enabled: true -# ansible_facts.services is not supported currently on Ubuntu 20.04, once sorted will remove and use ansible service module - name: block to check {{ enabled_qat_service }} service + when: + - update_qat_drivers | default(false) block: + # ansible_facts.services is not supported currently on Ubuntu 20.04, once sorted will remove and use ansible service module - name: check status of {{ enabled_qat_service }} service ansible.builtin.shell: "set -o pipefail && service {{ enabled_qat_service }} status | grep qat_dev" args: @@ -62,5 +73,3 @@ ansible.builtin.fail: msg: "Failed to start {{ enabled_qat_service }} service on system. Please set update_qat_drivers to true in host vars to resolve the issue." when: "'up' not in qat_status_check.stdout" - when: - - update_qat_drivers diff --git a/roles/bootstrap/configure_qat/templates/cek_sriov_qat_init.service.j2 b/roles/bootstrap/configure_qat/templates/cek_sriov_qat_init.service.j2 index 815c6852..99d9b0ad 100644 --- a/roles/bootstrap/configure_qat/templates/cek_sriov_qat_init.service.j2 +++ b/roles/bootstrap/configure_qat/templates/cek_sriov_qat_init.service.j2 @@ -1,9 +1,11 @@ [Unit] Description=Intel Container Experience Kits SR-IOV configuration for QAT devices AssertPathExists=/usr/local/bin/cek_sriov_qat_init +{% if not qat_oot_driver_build_failed | default(false) %} After={{ enabled_qat_service }}.service Requires={{ enabled_qat_service }}.service PartOf={{ enabled_qat_service }}.service +{% endif %} [Service] Environment=QAT_SRIOV_NUMVFS_MAPPINGS={{ sriov_config_path }}/cek_sriov_qat_numvfs diff --git a/roles/bootstrap/configure_sgx/defaults/main.yml b/roles/bootstrap/configure_sgx/defaults/main.yml index 358f4db2..065be77a 100644 --- a/roles/bootstrap/configure_sgx/defaults/main.yml +++ b/roles/bootstrap/configure_sgx/defaults/main.yml @@ -15,9 +15,9 @@ ## --- # Intel SGX SDK for Ubuntu -sgx_sdk_version_ubuntu: "sgx_linux_x64_sdk_2.19.100.3.bin" -sgx_sdk_url_ubuntu: "https://download.01.org/intel-sgx/sgx-dcap/1.16/linux/distro/ubuntu22.04-server/{{ sgx_sdk_version_ubuntu }}" -sgx_sdk_checksum_ubuntu: "sha256:B99B66A2E7D3842D106CF37747A124C53A9B49B07649E1EE26C0DA2BEB5AB3CE" +sgx_sdk_version_ubuntu: "sgx_linux_x64_sdk_2.21.100.1.bin" +sgx_sdk_url_ubuntu: "https://download.01.org/intel-sgx/sgx-dcap/1.18/linux/distro/ubuntu22.04-server/{{ sgx_sdk_version_ubuntu }}" +sgx_sdk_checksum_ubuntu: "sha256:53e75ad08baad4f74c9f78e33bff30d3d3518160bf8729a4b213e8514c0fd0ec" # Intel SGX-SGX Key configuration for Ubuntu >= 18.04.4 sgx_apt_source_list: "intel-sgx" @@ -25,20 +25,20 @@ sgx_apt_repo_url: "https://download.01.org/intel-sgx/sgx_repo/ubuntu" sgx_apt_repo_key: "{{ sgx_apt_repo_url }}/intel-sgx-deb.key" # Intel SGX SDK for RHEL -sgx_sdk_version_rhel: "sgx_linux_x64_sdk_2.19.100.3.bin" -sgx_sdk_url_rhel: "https://download.01.org/intel-sgx/sgx-dcap/1.16/linux/distro/rhel8.6-server/{{ sgx_sdk_version_rhel }}" -sgx_sdk_checksum_rhel: "sha256:E293D0179F81264AAD81E5A9864065B117C99A6EAD2388BC2A807093CB7E837A" +sgx_sdk_version_rhel: "sgx_linux_x64_sdk_2.21.100.1.bin" +sgx_sdk_url_rhel: "https://download.01.org/intel-sgx/sgx-dcap/1.18/linux/distro/rhel8.6-server/{{ sgx_sdk_version_rhel }}" +sgx_sdk_checksum_rhel: "sha256:2ae85c535118a5ff5c2b39cf5652eabae460b8e8d8094340919737eb31bc4021" # Intel SGX RPM local repository for RHEL sgx_rpm_local_repo_version_rhel: "sgx_rpm_local_repo.tgz" -sgx_rpm_local_repo_url_rhel: "https://download.01.org/intel-sgx/sgx-dcap/1.16/linux/distro/rhel8.6-server/{{ sgx_rpm_local_repo_version_rhel }}" -sgx_rpm_local_repo_checksum_rhel: "sha256:CFC42AC150E167510190F6D56677B17619869E1B528A03DB713C3D41C85E3EA5" +sgx_rpm_local_repo_url_rhel: "https://download.01.org/intel-sgx/sgx-dcap/1.18/linux/distro/rhel8.6-server/{{ sgx_rpm_local_repo_version_rhel }}" +sgx_rpm_local_repo_checksum_rhel: "sha256:57f24d1f25d1ae100a4fcbfbdb7d49ef0a744f2edb69e9bb6fcf6e5667160444" sgx_config_dir: "{{ project_root_dir }}" sgx_rpm_directory: "{{ (project_root_dir, 'sgx_rpm_local_repo') | path_join }}" -sgx_pkg_version: "2.19.100.3" -sgx_pkg_dcap_version: "1.16.100.2" +sgx_pkg_version: "2.21.100.1" +sgx_pkg_dcap_version: "1.18.100.1" protobuf_version: protobuf-3.5.0-13.el8.x86_64.rpm protobuf_repository: https://dl.rockylinux.org/vault/rocky/8.6/AppStream/x86_64/os/Packages/p diff --git a/roles/bootstrap/configure_sgx/tasks/ubuntu.yml b/roles/bootstrap/configure_sgx/tasks/ubuntu.yml index 39f0bffa..aec20d8c 100644 --- a/roles/bootstrap/configure_sgx/tasks/ubuntu.yml +++ b/roles/bootstrap/configure_sgx/tasks/ubuntu.yml @@ -62,6 +62,14 @@ - libsgx-ra-uefi={{ sgx_pkg_dcap_version }}-{{ sgx_release_name }}1 - sgx-ra-service={{ sgx_pkg_dcap_version }}-{{ sgx_release_name }}1 +- name: install tdx software + ansible.builtin.apt: + name: + - tdx-qgs={{ sgx_pkg_dcap_version }}-{{ sgx_release_name }}1 + - sgx-pck-id-retrieval-tool={{ sgx_pkg_dcap_version }}-{{ sgx_release_name }}1 + when: + - configure_tdx | default(false) + - name: start aesmd service, if not started service: state: started diff --git a/roles/bootstrap/golang_install/defaults/main.yml b/roles/bootstrap/golang_install/defaults/main.yml index 2ffa5dad..2d27d17b 100644 --- a/roles/bootstrap/golang_install/defaults/main.yml +++ b/roles/bootstrap/golang_install/defaults/main.yml @@ -14,8 +14,8 @@ ## limitations under the License. ## --- -golang_version: "1.20.4" -golang_download_checksum: "sha256:698ef3243972a51ddb4028e4a1ac63dc6d60821bf18e59a807e051fee0a385bd" +golang_version: "1.21.1" +golang_download_checksum: "sha256:b3075ae1ce5dab85f89bc7905d1632de23ca196bd8336afd93fa97434cfa55ae" golang_download_url: >- {{ 'https://mirrors.aliyun.com/golang/go' + golang_version + '.linux-amd64.tar.gz' diff --git a/roles/bootstrap/install_gpu_driver/defaults/main.yml b/roles/bootstrap/install_gpu_driver/defaults/main.yml deleted file mode 100644 index ed258e66..00000000 --- a/roles/bootstrap/install_gpu_driver/defaults/main.yml +++ /dev/null @@ -1,130 +0,0 @@ -## -## Copyright (c) 2020-2023 Intel Corporation. -## -## Licensed under the Apache License, Version 2.0 (the "License"); -## you may not use this file except in compliance with the License. -## You may obtain a copy of the License at -## -## http://www.apache.org/licenses/LICENSE-2.0 -## -## Unless required by applicable law or agreed to in writing, software -## distributed under the License is distributed on an "AS IS" BASIS, -## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -## See the License for the specific language governing permissions and -## limitations under the License. -## ---- -gpu_repo_key_url: https://repositories.intel.com/graphics/intel-graphics.key -gpu_key_text_path: /tmp/intel-graphic-key.txt -gpu_usr_key_path: /usr/share/keyrings/intel-graphics.gpg - - -# repo for different OS and different GPU type -gpu_repo_ubuntu_url: https://repositories.intel.com/graphics/ubuntu -gpu_repo_list_path: /etc/apt/sources.list.d/intel.gpu.list - -gpu_repo_spec_u2204_flex: "jammy flex" -gpu_repo_spec_u2204_arc: "jammy arc" - -kernel_dkms_packages: - - gawk - - dkms - - libc6-dev - -# intel dgpu release 20230714 for Ubuntu 22.04 -gpu_kmd_packages_u2204_20230714: - - {pkg: intel-platform-vsec-dkms, ver: 2023.20.0-21} - - {pkg: intel-platform-cse-dkms, ver: 2023.11.1-36} - - {pkg: intel-i915-dkms, ver: 1.23.5.19.230406.21.5.17.0.1034+i38-1} - - {pkg: intel-fw-gpu, ver: 2023.25.6-231~22.04} - -gpu_umd_rt_packages_u2204_20230714: - - {pkg: intel-opencl-icd, ver: 23.17.26241.33-647~22.04} - - {pkg: intel-level-zero-gpu, ver: 1.3.26241.33-647~22.04} - - {pkg: level-zero, ver: 1.11.0-647~22.04} - - {pkg: intel-media-va-driver-non-free, ver: 23.2.1-647~22.04 } - - {pkg: libmfx1, ver: 23.2.1-647~22.04} - - {pkg: libmfxgen1, ver: 23.2.1-647~22.04} - - {pkg: libvpl2, ver: 2023.2.1.0-647~22.04} - - {pkg: libegl-mesa0, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: libegl1-mesa, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: libegl1-mesa-dev, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: libgbm1, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: libgl1-mesa-dev, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: libgl1-mesa-dri, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: libglapi-mesa, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: libgles2-mesa-dev, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: libglx-mesa0, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: libigdgmm12, ver: 22.3.5-647~22.04} - - {pkg: libxatracker2, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: mesa-va-drivers, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: mesa-vdpau-drivers, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: mesa-vulkan-drivers, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: va-driver-all, ver: 2.18.0.2-61~u22.04} - -gpu_dev_packages_u2204_20230714: - - {pkg: libigc1, ver: 1.0.13822.8-647~22.04} - - {pkg: libigc-dev, ver: 1.0.13822.8-647~22.04} - - {pkg: intel-igc-cm, ver: 1.0.176+i600~22.04} - - {pkg: libigdfcl1, ver: 1.0.13822.8-647~22.04} - - {pkg: libigdfcl-dev, ver: 1.0.13822.8-647~22.04} - - {pkg: libigfxcmrt7, ver: 23.2.1-647~22.04} - - {pkg: libigfxcmrt-dev, ver: 23.2.1-647~22.04} - - {pkg: level-zero-dev, ver: 1.11.0-647~22.04} - -gpu_tool_packages_u2204_20230714: - - {pkg: xpu-smi, ver: 1.2.13-24~22.04} - - -# intel dgpu release 20230526 for Ubuntu 22.04 -gpu_kmd_packages_u2204_20230526: - - {pkg: intel-platform-vsec-dkms, ver: 2023.20.0-3} - - {pkg: intel-platform-cse-dkms, ver: 2023.11.1-36} - - {pkg: intel-i915-dkms, ver: 1.23.4.15.230307.15.5.17.0.1030+i28-1} - - {pkg: intel-fw-gpu, ver: 2023.12.2+207} - -gpu_umd_rt_packages_u2204_20230526: - - {pkg: intel-opencl-icd, ver: 23.13.26032.26-627~22.04} - - {pkg: intel-level-zero-gpu, ver: 1.3.26032.26-627~22.04} - - {pkg: level-zero, ver: 1.9.9-625~22.04} - - {pkg: intel-media-va-driver-non-free, ver: 23.1.6-622~22.04} - - {pkg: libmfx1, ver: 23.1.6-622~22.04} - - {pkg: libmfxgen1, ver: 23.1.5-622~22.04} - - {pkg: libvpl2, ver: 2023.1.3.0-622~22.04} - - {pkg: libegl-mesa0, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: libegl1-mesa, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: libegl1-mesa-dev, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: libgbm1, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: libgl1-mesa-dev, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: libgl1-mesa-dri, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: libglapi-mesa, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: libgles2-mesa-dev, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: libglx-mesa0, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: libigdgmm12, ver: 22.3.5-622~22.04} - - {pkg: libxatracker2, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: mesa-va-drivers, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: mesa-vdpau-drivers, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: mesa-vulkan-drivers, ver: 23.2.0.20230414.1+2061~u22.04} - - {pkg: va-driver-all, ver: 2.18.0.2-60~u22.04} - -gpu_dev_packages_u2204_20230526: - - {pkg: libigc1, ver: 1.0.13700.13-627~22.04} - - {pkg: libigc-dev, ver: 1.0.13700.13-627~22.04} - - {pkg: intel-igc-cm, ver: 1.0.176+i600~22.04} - - {pkg: libigdfcl1, ver: 1.0.13700.13-627~22.04} - - {pkg: libigdfcl-dev, ver: 1.0.13700.13-627~22.04} - - {pkg: libigfxcmrt7, ver: 23.1.6-622~22.04} - - {pkg: libigfxcmrt-dev, ver: 23.1.6-622~22.04} - - {pkg: level-zero-dev, ver: 1.9.9-625~22.04} - -gpu_tool_packages_u2204_20230526: - - {pkg: xpu-smi, ver: 1.2.3-13~u22.04} - -# intel dgpu release independent test packages -gpu_test_packages: - - hwinfo - - vainfo - - clinfo - - mesa-utils - - vulkan-tools - - intel-gpu-tools diff --git a/roles/bootstrap/install_gpu_driver/tasks/main.yml b/roles/bootstrap/install_gpu_driver/tasks/main.yml deleted file mode 100644 index 8d88a649..00000000 --- a/roles/bootstrap/install_gpu_driver/tasks/main.yml +++ /dev/null @@ -1,69 +0,0 @@ -## -## Copyright (c) 2020-2023 Intel Corporation. -## -## Licensed under the Apache License, Version 2.0 (the "License"); -## you may not use this file except in compliance with the License. -## You may obtain a copy of the License at -## -## http://www.apache.org/licenses/LICENSE-2.0 -## -## Unless required by applicable law or agreed to in writing, software -## distributed under the License is distributed on an "AS IS" BASIS, -## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -## See the License for the specific language governing permissions and -## limitations under the License. -## ---- -- name: Install gpu type detection script to /usr/local/bin - copy: - src: "{{ item }}" - dest: "/usr/local/bin/" - mode: 0700 - owner: root - group: root - force: true - with_items: - - 'cek_detect_gpu_type.py' - become: true - -- name: Detect gpu type - command: "python /usr/local/bin/cek_detect_gpu_type.py" - register: gpu_type_result - changed_when: false - -- name: Output gpu type detection result - debug: - msg: "{{ gpu_type_result.stdout_lines }}" - -- name: Set repo for Flex GPU installation on Ubuntu 22.04 - set_fact: - gpu_repo_spec: "{{ gpu_repo_spec_u2204_flex }}" - when: - - (ansible_distribution == "Ubuntu" and ansible_distribution_version == "22.04") - - gpu_type_result.stdout_lines[0] == "Flex" - -- name: Set repo for Arc GPU installation on Ubuntu 22.04 - set_fact: - gpu_repo_spec: "{{ gpu_repo_spec_u2204_arc }}" - when: - - (ansible_distribution == "Ubuntu" and ansible_distribution_version == "22.04") - - gpu_type_result.stdout_lines[0] == "Arc" or gpu_type_result.stdout_lines[0] == "iGPU" - -- name: Set gpu package version to specific release on Ubuntu 22.04 - set_fact: - gpu_kmd_packages: "{{ gpu_kmd_packages_u2204_20230714 }}" - gpu_umd_rt_packages: "{{ gpu_umd_rt_packages_u2204_20230714 }}" - gpu_dev_packages: "{{ gpu_dev_packages_u2204_20230714 }}" - gpu_tool_packages: "{{ gpu_tool_packages_u2204_20230714 }}" - when: - - (ansible_distribution == "Ubuntu" and ansible_distribution_version == "22.04") - -- name: Install GPU drivers on Ubuntu - include_tasks: debian.yml - when: - - (ansible_distribution == 'Ubuntu') - -- name: Install GPU drivers on RHEL 8.x - include_tasks: rhel.yml - when: - - (ansible_os_family == "RedHat" and ansible_distribution_version >= "8.6") diff --git a/roles/bootstrap/install_packages/tasks/debian.yml b/roles/bootstrap/install_packages/tasks/debian.yml index aa36ea0e..07957d04 100644 --- a/roles/bootstrap/install_packages/tasks/debian.yml +++ b/roles/bootstrap/install_packages/tasks/debian.yml @@ -27,6 +27,14 @@ - {type: 'https', value: "{{ https_proxy | default('') }}"} when: http_proxy is defined or https_proxy is defined +- name: fix broken packages for tdvm + ansible.builtin.apt: + state: fixed + when: + - configure_tdx | default(false) + - on_vms | default(false) + - ansible_distribution == "Ubuntu" and ansible_distribution_version == '22.04' + - name: disable automatic package updates apt: name: unattended-upgrades @@ -158,6 +166,7 @@ name: udev when: - ansible_os_family == "Debian" + - not configure_tdx | default(false) - name: perform dist-upgrade on Debian OS family apt: @@ -187,6 +196,14 @@ - jq when: ansible_distribution == "Ubuntu" +- name: install pytdxmeasure for tdvm + ansible.builtin.pip: + name: pytdxmeasure + when: + - configure_tdx | default(false) + - on_vms | default(false) + - ansible_distribution == "Ubuntu" and ansible_distribution_version == '22.04' + # hirsute (21.04) package for (image & headers) is 20.04. (Note: ansible_distribution_version will not be returned as the correct version) # Depending on the needs, we can split tasks for future Ubuntu releases if necessary. # Ref: https://launchpad.net/ubuntu/hirsute/+package/linux-image-generic-hwe-20.04 diff --git a/roles/bootstrap/install_qat_drivers_services/defaults/main.yml b/roles/bootstrap/install_qat_drivers_services/defaults/main.yml index d74c2f3f..6f79aff6 100644 --- a/roles/bootstrap/install_qat_drivers_services/defaults/main.yml +++ b/roles/bootstrap/install_qat_drivers_services/defaults/main.yml @@ -15,14 +15,14 @@ ## --- # QAT 1.x (for QAT add-on cards) -qat_drivers_version: 'QAT.L.4.22.0-00001' -qat_drivers_download_url: 'https://downloadmirror.intel.com/780675/{{ qat_drivers_version }}.tar.gz' -qat_drivers_pkg_checksum: 'sha1:36EF82C00802F6A6D1494BC31B5C1264E2DFDFC4' +qat_drivers_version: 'QAT.L.4.23.0-00001' +qat_drivers_download_url: 'https://downloadmirror.intel.com/788561/{{ qat_drivers_version }}.tar.gz' +qat_drivers_pkg_checksum: 'sha1:07830339CD5F1903C727323768CEF4B01CCA10DD' # QAT 2.x (for QAT embedded into SPR) -qat_spr_drivers_version: 'QAT20.L.1.0.40-00004' -qat_spr_drivers_download_url: 'https://downloadmirror.intel.com/781387/{{ qat_spr_drivers_version }}.tar.gz' -qat_spr_drivers_pkg_checksum: 'sha1:27B687C0D72D83BF5A8DE10A7FEF1FF3D57828F2' +qat_spr_drivers_version: 'QAT20.L.1.0.50-00003' +qat_spr_drivers_download_url: 'https://downloadmirror.intel.com/783270/{{ qat_spr_drivers_version }}.tar.gz' +qat_spr_drivers_pkg_checksum: 'sha1:1B84397E1ADB191A150F274F961895615BD85BAB' # If updating mentioned below folder location kindly update similar in roles/redeploy_cleanup/defaults/main.yml qat_drivers_dir: "{{ (project_root_dir, 'qat_drivers') | path_join }}" diff --git a/roles/bootstrap/install_qat_drivers_services/tasks/main.yml b/roles/bootstrap/install_qat_drivers_services/tasks/main.yml index 9f13a7f3..231f71fd 100644 --- a/roles/bootstrap/install_qat_drivers_services/tasks/main.yml +++ b/roles/bootstrap/install_qat_drivers_services/tasks/main.yml @@ -14,183 +14,27 @@ ## limitations under the License. ## --- -- name: install dependencies for QAT - include_role: - name: install_dependencies - -- name: install libudev-dev package on Ubuntu - apt: - name: libudev-dev - when: ansible_distribution == "Ubuntu" - -- name: create directory {{ qat_drivers_dir }} for all QAT dependencies - file: - path: "{{ qat_drivers_dir }}" - state: directory - mode: "u=rwx,g=rx,o=rx" - -- name: block for QAT 1.x - block: - - name: download QAT drivers package {{ qat_drivers_version }} - get_url: - url: "{{ qat_drivers_download_url }}" - checksum: "{{ qat_drivers_pkg_checksum }}" - dest: "{{ qat_drivers_dir }}" - mode: 0755 - register: qat_driver_sw - until: qat_driver_sw is not failed - retries: 5 - - - name: unarchive QAT drivers package - unarchive: - src: "{{ qat_drivers_dir }}/{{ qat_drivers_version }}.tar.gz" - dest: "{{ qat_drivers_dir }}" - remote_src: yes - mode: 0755 +- name: make qat driver on VMs consistent with VM host on kube_control_plane + ansible.builtin.set_fact: + qat_oot_driver_build_failed: true + update_qat_drivers: false + with_items: "{{ groups['kube_control_plane'] }}" + delegate_to: "{{ item }}" + delegate_facts: true + run_once: true when: - - configured_arch not in ["spr", "emr"] - -- name: block for QAT 2.x - block: - - name: download QAT drivers package {{ qat_spr_drivers_version }} - get_url: - url: "{{ qat_spr_drivers_download_url }}" - checksum: "{{ qat_spr_drivers_pkg_checksum }}" - dest: "{{ qat_drivers_dir }}" - mode: 0755 - register: qat_driver_sw - until: qat_driver_sw is not failed - retries: 5 + - on_vms | default(false) + - hostvars[hostvars[inventory_hostname]['vm_host']]['qat_oot_driver_build_failed'] | default(false) - - name: unarchive QAT drivers package - unarchive: - src: "{{ qat_drivers_dir }}/{{ qat_spr_drivers_version }}.tar.gz" - dest: "{{ qat_drivers_dir }}" - remote_src: yes - mode: 0755 +- name: make qat driver on VMs consistent with VM host + ansible.builtin.set_fact: + qat_oot_driver_build_failed: true + update_qat_drivers: false when: - - configured_arch in ["spr"] - -# Due to EMR is not lauched yet, EMR QAT driver temporally copy from ansible host -# When external driver offically support the EMR platform, converge w/ upper task -- name: block for EMR QAT driver package - block: - - name: copy EMR QAT driver package - ansible.builtin.copy: - src: "{{ (emr_qat_driver_staging_folder, emr_qat_driver_package) | path_join }}" - dest: "{{ (qat_drivers_dir, emr_qat_driver_package) | path_join }}" - mode: 0644 - - name: unarchive EMR QAT driver package - ansible.builtin.unarchive: - src: "{{ (qat_drivers_dir, emr_qat_driver_package) | path_join }}" - dest: "{{ qat_drivers_dir }}" - remote_src: yes - mode: 0755 - when: - - configured_arch == "emr" - -- name: check all packages are present for QAT drivers installation - command: ./configure - args: - chdir: "{{ qat_drivers_dir }}" - register: qat_requirements - ignore_errors: true - changed_when: true - -- name: playbook terminated packages for QAT drivers installation are missing - fail: - msg: - - "Missing requirements for QAT drivers (i.e. kernel sources)" - - "If failure persists, consider setting update_kernel: true in group_vars" - when: "'error' in qat_requirements.stderr" - -- name: block for QAT 1.x drivers and samples compilation - block: - - name: make install QAT drivers - command: "make -j install" - args: - chdir: "{{ qat_drivers_dir }}" - become: yes - changed_when: true - notify: - - reboot server - - - name: make performance sample application after QAT drivers - make: - chdir: "{{ qat_drivers_dir }}" - target: samples-install - become: yes - when: - - configured_arch not in ["spr", "emr"] - -# Reboot with driver ver: QAT20.L.0.8.0-00071 causing issues, there is no need to reboot. -- name: block for QAT 2.x drivers and samples compilation - block: - - name: make install QAT drivers - command: "make -j install" - args: - chdir: "{{ qat_drivers_dir }}" - become: yes - changed_when: true - - - name: make performance sample application after QAT drivers - make: - chdir: "{{ qat_drivers_dir }}" - target: samples-install - become: yes - when: - - configured_arch in ["spr", "emr"] - -- name: confirm QAT module installed - shell: "set -o pipefail && lsmod | grep qat" - args: - executable: /bin/bash - register: qat_confirm - failed_when: '"intel_qat" not in qat_confirm.stdout' - changed_when: false - -- name: enable SRIOV QAT devices on VMs - lineinfile: - path: "/etc/default/qat" - line: "SRIOV_ENABLE=1" - regexp: "^#SRIOV_ENABLE=1" - create: yes - owner: root - group: root - mode: '0644' - when: - - on_vms is defined and on_vms - -- name: make sure {{ disabled_qat_service }} service is stopped and disabled - service: - state: stopped - name: "{{ disabled_qat_service }}" - enabled: no - -- name: make sure {{ enabled_qat_service }} service is started and enabled - service: - state: started - name: "{{ enabled_qat_service }}" - enabled: yes - -- name: configuration for QAT Shared Virtual Memory (SVM) - block: - - name: set QAT SVM is enabled - set_fact: - svm_value: 1 + - on_vms | default(false) + - hostvars[hostvars[inventory_hostname]['vm_host']]['qat_oot_driver_build_failed'] | default(false) - - name: enable address translation services for QAT Shared Virtual Memory (SVM) - replace: - path: "{{ item }}" - regexp: '(^SVMEnabled\s)(.*)$' - replace: 'SVMEnabled = {{ svm_value }}' - mode: 0600 - with_items: - - "{{ qat_drivers_dir }}/quickassist/utilities/adf_ctl/conf_files/4xxxvf_dev0.conf.vm" - - "{{ qat_drivers_dir }}/quickassist/utilities/adf_ctl/conf_files/4xxxvf_dev0.conf.sym.vm" - - "{{ qat_drivers_dir }}/quickassist/utilities/adf_ctl/conf_files/4xxxvf_dev0.conf.dc.vm" - - "{{ qat_drivers_dir }}/quickassist/utilities/adf_ctl/conf_files/4xxxvf_dev0.conf.asym.vm" - - "{{ qat_drivers_dir }}/quickassist/utilities/adf_ctl/conf_files/4xxxvf_dev0.conf.dc.sym.vm" +- name: install QAT OOT driver + ansible.builtin.include_tasks: qat_oot_driver_install.yml when: - - configured_arch in ["spr", "emr"] - - enable_qat_svm | default(false) + - qat_devices | default([]) | length > 0 diff --git a/roles/bootstrap/install_qat_drivers_services/tasks/qat_oot_driver_install.yml b/roles/bootstrap/install_qat_drivers_services/tasks/qat_oot_driver_install.yml new file mode 100644 index 00000000..4c13284a --- /dev/null +++ b/roles/bootstrap/install_qat_drivers_services/tasks/qat_oot_driver_install.yml @@ -0,0 +1,229 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +- name: install dependencies for QAT + include_role: + name: install_dependencies + +- name: install libudev-dev package on Ubuntu + apt: + name: libudev-dev + when: ansible_distribution == "Ubuntu" + +- name: create directory {{ qat_drivers_dir }} for all QAT dependencies + file: + path: "{{ qat_drivers_dir }}" + state: directory + mode: "u=rwx,g=rx,o=rx" + +- name: block for QAT 1.x + block: + - name: download QAT drivers package {{ qat_drivers_version }} + get_url: + url: "{{ qat_drivers_download_url }}" + checksum: "{{ qat_drivers_pkg_checksum }}" + dest: "{{ qat_drivers_dir }}" + mode: 0755 + register: qat_driver_sw + until: qat_driver_sw is not failed + retries: 5 + + - name: unarchive QAT drivers package + unarchive: + src: "{{ qat_drivers_dir }}/{{ qat_drivers_version }}.tar.gz" + dest: "{{ qat_drivers_dir }}" + remote_src: yes + mode: 0755 + when: + - configured_arch not in ["spr", "emr"] + +- name: block for QAT 2.x + block: + - name: download QAT drivers package {{ qat_spr_drivers_version }} + get_url: + url: "{{ qat_spr_drivers_download_url }}" + checksum: "{{ qat_spr_drivers_pkg_checksum }}" + dest: "{{ qat_drivers_dir }}" + mode: 0755 + register: qat_driver_sw + until: qat_driver_sw is not failed + retries: 5 + + - name: unarchive QAT drivers package + unarchive: + src: "{{ qat_drivers_dir }}/{{ qat_spr_drivers_version }}.tar.gz" + dest: "{{ qat_drivers_dir }}" + remote_src: yes + mode: 0755 + when: + - configured_arch in ["spr"] + +# Due to EMR is not lauched yet, EMR QAT driver temporally copy from ansible host +# When external driver offically support the EMR platform, converge w/ upper task +- name: block for EMR QAT driver package + block: + - name: copy EMR QAT driver package + ansible.builtin.copy: + src: "{{ (emr_qat_driver_staging_folder, emr_qat_driver_package) | path_join }}" + dest: "{{ (qat_drivers_dir, emr_qat_driver_package) | path_join }}" + mode: 0644 + - name: unarchive EMR QAT driver package + ansible.builtin.unarchive: + src: "{{ (qat_drivers_dir, emr_qat_driver_package) | path_join }}" + dest: "{{ qat_drivers_dir }}" + remote_src: yes + mode: 0755 + when: + - configured_arch == "emr" + +- name: check all packages are present for QAT drivers installation + command: ./configure + args: + chdir: "{{ qat_drivers_dir }}" + register: qat_requirements + ignore_errors: true + changed_when: true + +- name: playbook terminated packages for QAT drivers installation are missing + fail: + msg: + - "Missing requirements for QAT drivers (i.e. kernel sources)" + - "If failure persists, consider setting update_kernel: true in group_vars" + when: "'error' in qat_requirements.stderr" + +- name: workaround for 1.x QAT 4.23.0-00001 compilation error in VMRA + block: + - name: temporary workaround for VMRA qat compilation + ansible.builtin.lineinfile: + path: "{{ qat_drivers_dir }}/quickassist/qat/compat/qat_compat.h" + insertbefore: '^(#if \S+KERNEL_VERSION\S+6, 3, 0\S+(.*)$)' + line: "/***" + + - name: temporary workaround for VMRA qat compilation + ansible.builtin.lineinfile: + path: "{{ qat_drivers_dir }}/quickassist/qat/compat/qat_compat.h" + insertbefore: "#ifdef QAT_DBG" + line: "***/" + when: + - on_vms | default(false) + - configured_arch not in ["spr", "emr"] + +- name: block for QAT 1.x drivers and samples compilation + block: + - name: make install QAT drivers + command: "make -j install" + args: + chdir: "{{ qat_drivers_dir }}" + become: yes + changed_when: true + notify: + - reboot server + + - name: make performance sample application after QAT drivers + make: + chdir: "{{ qat_drivers_dir }}" + target: samples-install + become: yes + when: + - configured_arch not in ["spr", "emr"] + +- name: set QAT OOT 2.x driver build status + set_fact: + qat_oot_driver_build_failed: false + when: not qat_oot_driver_build_failed | default(false) + +# Reboot with driver ver: QAT20.L.0.8.0-00071 causing issues, there is no need to reboot. +- name: block for QAT 2.x drivers and samples compilation + block: + - name: make install QAT drivers + command: "make -j install" + args: + chdir: "{{ qat_drivers_dir }}" + become: yes + changed_when: true + + - name: make performance sample application after QAT drivers + make: + chdir: "{{ qat_drivers_dir }}" + target: samples-install + become: yes + rescue: + - name: QAT 2.x OOT driver build error + debug: + msg: "QAT 2.x OOT driver build or installation failed. Rolling back to use inbox driver - functionality might be limited" + + - name: set QAT OOT 2.x driver build status + set_fact: + qat_oot_driver_build_failed: true + update_qat_drivers: false + when: + - configured_arch in ["spr", "emr"] + - not qat_oot_driver_build_failed | default(false) + +- name: confirm QAT module installed + shell: "set -o pipefail && lsmod | grep qat" + args: + executable: /bin/bash + register: qat_confirm + failed_when: '"intel_qat" not in qat_confirm.stdout' + changed_when: false + +- name: enable SRIOV QAT devices on VMs + lineinfile: + path: "/etc/default/qat" + line: "SRIOV_ENABLE=1" + regexp: "^#SRIOV_ENABLE=1" + create: yes + owner: root + group: root + mode: '0644' + when: + - on_vms is defined and on_vms + +- name: make sure {{ disabled_qat_service }} service is stopped and disabled + service: + state: stopped + name: "{{ disabled_qat_service }}" + enabled: no + +- name: make sure {{ enabled_qat_service }} service is started and enabled + service: + state: started + name: "{{ enabled_qat_service }}" + enabled: yes + when: not qat_oot_driver_build_failed | default(false) + +- name: configuration for QAT Shared Virtual Memory (SVM) + block: + - name: set QAT SVM is enabled + set_fact: + svm_value: 1 + + - name: enable address translation services for QAT Shared Virtual Memory (SVM) + replace: + path: "{{ item }}" + regexp: '(^SVMEnabled\s)(.*)$' + replace: 'SVMEnabled = {{ svm_value }}' + mode: 0600 + with_items: + - "{{ qat_drivers_dir }}/quickassist/utilities/adf_ctl/conf_files/4xxxvf_dev0.conf.vm" + - "{{ qat_drivers_dir }}/quickassist/utilities/adf_ctl/conf_files/4xxxvf_dev0.conf.sym.vm" + - "{{ qat_drivers_dir }}/quickassist/utilities/adf_ctl/conf_files/4xxxvf_dev0.conf.dc.vm" + - "{{ qat_drivers_dir }}/quickassist/utilities/adf_ctl/conf_files/4xxxvf_dev0.conf.asym.vm" + - "{{ qat_drivers_dir }}/quickassist/utilities/adf_ctl/conf_files/4xxxvf_dev0.conf.dc.sym.vm" + when: + - configured_arch in ["spr", "emr"] + - enable_qat_svm | default(false) diff --git a/roles/bootstrap/install-qatlibs/defaults/main.yml b/roles/bootstrap/install_qatlibs/defaults/main.yml similarity index 95% rename from roles/bootstrap/install-qatlibs/defaults/main.yml rename to roles/bootstrap/install_qatlibs/defaults/main.yml index f6cf3ed9..ab3ed57e 100644 --- a/roles/bootstrap/install-qatlibs/defaults/main.yml +++ b/roles/bootstrap/install_qatlibs/defaults/main.yml @@ -16,7 +16,7 @@ --- # QATLibs intel_qatlib_download_url: "https://github.com/intel/qatlib.git" -intel_qatlib_download_url_version: "23.02.0" +intel_qatlib_download_url_version: "23.08.0" intel_qatlib_download_url_dir: "{{ (project_root_dir, 'intel_qatlibs') | path_join }}" intel_qat_4xxx_firmware_download_url: https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/qat_4xxx.bin diff --git a/roles/bootstrap/install-qatlibs/tasks/main.yml b/roles/bootstrap/install_qatlibs/tasks/main.yml similarity index 85% rename from roles/bootstrap/install-qatlibs/tasks/main.yml rename to roles/bootstrap/install_qatlibs/tasks/main.yml index 224c6a8e..bdcb7e40 100644 --- a/roles/bootstrap/install-qatlibs/tasks/main.yml +++ b/roles/bootstrap/install_qatlibs/tasks/main.yml @@ -14,12 +14,18 @@ ## limitations under the License. ## --- -- name: install dependencies for qatlibs +- name: install dependencies for Intel QATLibs include_role: name: install_dependencies - name: download qat_4xxx firmware if not exists block: + - name: ensure that directory /lib/firmware exists + file: + path: /lib/firmware + state: directory + mode: 0755 + - name: check qat_4xxx.bin firmware existence ansible.builtin.stat: path: /lib/firmware/qat_4xxx.bin @@ -66,20 +72,16 @@ - not qat_4xxx_mmp_exist.stat.exists - not qat_4xxx_mmp_xz_exist.stat.exists - - name: uninstall the qat_4xxx driver - community.general.modprobe: - name: qat_4xxx - state: absent - when: - - (not qat_4xxx_mmp_exist.stat.exists and - not qat_4xxx_mmp_xz_exist.stat.exists) or - (not qat_4xxx_exist.stat.exists and - not qat_4xxx_xz_exist.stat.exists) - - - name: reinstall the qat_4xxx driver - community.general.modprobe: - name: qat_4xxx - state: present + - name: reload the qat driver to install the firmware + block: + - name: uninstall the qat_4xxx driver + community.general.modprobe: + name: qat_4xxx + state: absent + - name: reinstall the qat_4xxx driver + community.general.modprobe: + name: qat_4xxx + state: present when: - (not qat_4xxx_mmp_exist.stat.exists and not qat_4xxx_mmp_xz_exist.stat.exists) or @@ -95,7 +97,7 @@ file: path: "{{ intel_qatlib_download_url_dir }}" state: directory - mode: '0700' + mode: 0700 - name: download Intel QATLib git: @@ -106,7 +108,7 @@ # using shell module instead of comand as it was giving aclocal: warning: causing playbook failure - name: run autogen before configure QATLibs - shell: './autogen.sh' + shell: './autogen.sh' # noqa 305 # command-instead-of-shell args: chdir: "{{ intel_qatlib_download_url_dir }}" executable: /bin/bash @@ -118,7 +120,7 @@ chdir: "{{ intel_qatlib_download_url_dir }}" changed_when: true - - name: make install QAT drivers + - name: make install QATLibs make: chdir: "{{ intel_qatlib_download_url_dir }}" target: install diff --git a/roles/bootstrap/install-qatlibs/vars/main.yml b/roles/bootstrap/install_qatlibs/vars/main.yml similarity index 100% rename from roles/bootstrap/install-qatlibs/vars/main.yml rename to roles/bootstrap/install_qatlibs/vars/main.yml diff --git a/roles/install_ddp_pkgs/tasks/install_pkgs.yml b/roles/bootstrap/install_tdx_drivers/defaults/main.yml similarity index 64% rename from roles/install_ddp_pkgs/tasks/install_pkgs.yml rename to roles/bootstrap/install_tdx_drivers/defaults/main.yml index 9033be83..8d95de22 100644 --- a/roles/install_ddp_pkgs/tasks/install_pkgs.yml +++ b/roles/bootstrap/install_tdx_drivers/defaults/main.yml @@ -14,17 +14,12 @@ ## limitations under the License. ## --- -- name: create DDP package installation directory - become: yes - file: - path: "{{ install_dir }}" - state: directory - mode: 0700 - owner: root - group: root +intel_tdx_repo_url: "https://github.com/intel/tdx-tools.git" +intel_tdx_download_dir: "{{ (project_root_dir, 'tdx-tools') | path_join }}" +intel_tdx_packages_dir: "{{ (project_root_dir, 'intel-tdx') | path_join }}" -- name: install ddp package from a URL - include_tasks: install_a_pkg.yml - loop: "{{ pkgurls }}" - loop_control: - loop_var: pkgurl +tdx_1_0_kernel_version: 5.19.17-mvp29v4+4 +tdx_1_0_tag: "2023ww33" + +tdx_1_5_kernel_version: 6.2.16-mvp30v3+7 +tdx_1_5_tag: "2023ww27" diff --git a/roles/bootstrap/install_tdx_drivers/tasks/main.yml b/roles/bootstrap/install_tdx_drivers/tasks/main.yml new file mode 100644 index 00000000..821e552f --- /dev/null +++ b/roles/bootstrap/install_tdx_drivers/tasks/main.yml @@ -0,0 +1,21 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +- name: Install tdx drivers on Ubuntu + include_tasks: ubuntu.yml + when: + - ansible_distribution == "Ubuntu" + - ansible_distribution_version == "22.04" diff --git a/roles/minio_install/tasks/mount_loopdevices.yml b/roles/bootstrap/install_tdx_drivers/tasks/tdx_preflight.yml similarity index 54% rename from roles/minio_install/tasks/mount_loopdevices.yml rename to roles/bootstrap/install_tdx_drivers/tasks/tdx_preflight.yml index b4f2142d..cda92a7b 100644 --- a/roles/minio_install/tasks/mount_loopdevices.yml +++ b/roles/bootstrap/install_tdx_drivers/tasks/tdx_preflight.yml @@ -14,18 +14,22 @@ ## limitations under the License. ## --- -- name: list loop devices - shell: >- - set -o pipefail && losetup -l |grep diskimage |awk -F " " '{ print $6,$1}' |sort |awk -F " " '{ print $2}' +- name: get the tdx dmesg log + ansible.builtin.shell: "set -o pipefail && dmesg | grep -i tdx" args: executable: /bin/bash - register: loopdevice_output + register: kernel_message changed_when: true -- name: mount file block devices -# noqa command-instead-of-module - Ansible mount module doesn't support 'fstype:loop' - command: >- - mount -o loop {{ loopdevice_output.stdout_lines[ansible_loop.index0] }} {{ hostvars[inventory_hostname]['minio_pv'][ansible_loop.index0].mountPath }} - loop_control: - extended: yes - changed_when: true +- name: check whether tdx is configured correctly + ansible.builtin.assert: + that: + - kernel_message.rc == 0 + msg: "Please refer the doc/emr.md to run the tdx.yml first and then enabled the TDX option in the bios" + +- name: check linux distro version for tdx + ansible.builtin.assert: + that: > + - (ansible_distribution == 'Ubuntu' and ansible_distribution_version == '22.04') + msg: + - "TDX is verified only on Ubuntu 22.04 with RA" diff --git a/roles/bootstrap/install_tdx_drivers/tasks/ubuntu.yml b/roles/bootstrap/install_tdx_drivers/tasks/ubuntu.yml new file mode 100644 index 00000000..89480f53 --- /dev/null +++ b/roles/bootstrap/install_tdx_drivers/tasks/ubuntu.yml @@ -0,0 +1,135 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +- name: install dependencies for Intel tdx + ansible.builtin.include_role: + name: install_dependencies + +# Intel tdx driver compilation and installation +- name: create folder for tdx packages + ansible.builtin.file: + path: "{{ intel_tdx_packages_dir }}" + state: directory + mode: 0755 + +- name: set the tdx(1.5) tag and kernel version by default + ansible.builtin.set_fact: + intel_tdx_tag: "{{ tdx_1_5_tag }}" + tdx_kernel_version: "{{ tdx_1_5_kernel_version }}" + +# override the fact for tdx 1.0 +- name: overwrite the tag and kernel version for tdx 1.0 + ansible.builtin.set_fact: + intel_tdx_tag: "{{ tdx_1_0_tag }}" + tdx_kernel_version: "{{ tdx_1_0_kernel_version }}" + when: + tdx_version == "1.0" + +- name: block to compile Intel TDX drvier + block: + - name: download Intel TDX source code + ansible.builtin.git: + repo: "{{ intel_tdx_repo_url }}" + dest: "{{ intel_tdx_download_dir }}" + version: "{{ intel_tdx_tag }}" + force: true + + - name: block to fix the compilation error for tdx 1.5 + block: + - name: fix the tdx migration module compilation error + ansible.builtin.lineinfile: + path: "{{ (intel_tdx_download_dir, 'build', 'ubuntu-22.04', 'intel-mvp-tdx-migration', 'build.sh') | path_join }}" + insertafter: " ./sh_script/preparation.sh" + line: " cp ./Cargo.lock ./deps/td-shim/" + - name: fix the tdx vtpm module compilation error + ansible.builtin.lineinfile: + path: "{{ (intel_tdx_download_dir, 'build', 'ubuntu-22.04', 'intel-mvp-vtpm-td', 'build.sh') | path_join }}" + insertafter: " source sh_script/pre-build.sh" + line: " cp ../../intel-mvp-tdx-migration/migtd/Cargo.lock ./deps/td-shim/" + when: + - tdx_version == "1.5" + + - name: cleanup the already built libraries and packages + ansible.builtin.command: >- + ./build-repo.sh cleanup + args: + chdir: "{{ (intel_tdx_download_dir, 'build', 'ubuntu-22.04') | path_join }}" + changed_when: true + + - name: compile the tdx source code + ansible.builtin.command: >- + ./build-repo.sh > /dev/null + args: + chdir: "{{ (intel_tdx_download_dir, 'build', 'ubuntu-22.04') | path_join }}" + changed_when: true + +- name: copy the comiled host/guest packages to {{ intel_tdx_packages_dir }} + block: + - name: copy host packages + ansible.builtin.copy: + src: "{{ (intel_tdx_download_dir, 'build', 'ubuntu-22.04', 'host_repo') | path_join }}" + dest: "{{ intel_tdx_packages_dir }}" + remote_src: yes + mode: '0644' + + - name: sync guest packages + ansible.builtin.copy: + src: "{{ (intel_tdx_download_dir, 'build', 'ubuntu-22.04', 'guest_repo') | path_join }}" + dest: "{{ intel_tdx_packages_dir }}" + remote_src: yes + mode: '0644' + +- name: install the host packages + # npm do not use the apt module due to it can not resolve the dependencies for multiple packages + # noqa command-instead-of-module + ansible.builtin.shell: apt-get install -y --allow-downgrades ./*.deb + args: + chdir: "{{ (intel_tdx_packages_dir, 'host_repo') | path_join }}" + changed_when: true + +- name: set tdx kernel as default kernel + block: + - name: fetch tdx kernel first entry + ansible.builtin.shell: >- + set -o pipefail && cat /boot/grub/grub.cfg | grep submenu | awk -F\' '{print $2}' + args: + executable: /bin/bash + register: kernel_fisrt_entry + failed_when: kernel_fisrt_entry.rc > 1 + changed_when: false + + - name: fetch tdx kernel second entry + ansible.builtin.shell: >- + set -o pipefail && cat /boot/grub/grub.cfg | grep menuentry | grep {{ tdx_kernel_version }} | grep -v recovery | awk -F\' '{print $2}' + args: + executable: /bin/bash + register: kernel_second_entry + failed_when: kernel_second_entry.rc > 1 + changed_when: false + + - name: Set tdx kernel(2-level entries) as default boot kernel + ansible.builtin.lineinfile: + path: /etc/default/grub + regexp: "^GRUB_DEFAULT" + line: GRUB_DEFAULT="{{ kernel_fisrt_entry.stdout }}>{{ kernel_second_entry.stdout }}" + when: kernel_fisrt_entry.stdout != "" + + - name: Set tdx kernel(1-level entry) as default boot kernel + ansible.builtin.lineinfile: + path: /etc/default/grub + regexp: "^GRUB_DEFAULT" + line: GRUB_DEFAULT="{{ kernel_second_entry.stdout }}" + when: kernel_fisrt_entry.stdout == "" diff --git a/roles/bootstrap/configure_intel_pstate/vars/main.yml b/roles/bootstrap/install_tdx_drivers/vars/main.yml similarity index 71% rename from roles/bootstrap/configure_intel_pstate/vars/main.yml rename to roles/bootstrap/install_tdx_drivers/vars/main.yml index 5d26956e..a420777f 100644 --- a/roles/bootstrap/configure_intel_pstate/vars/main.yml +++ b/roles/bootstrap/install_tdx_drivers/vars/main.yml @@ -14,19 +14,22 @@ ## limitations under the License. ## --- -intel_turbo_path: /sys/devices/system/cpu/intel_pstate/no_turbo -intel_pstate_marker: "# intel_pstate" -acceptable_intel_pstate_values: - - disable - - passive - - force - - no_hwp - - hwp_only - - support_acpi_ppc - - per_cpu_perf_limits - install_dependencies: Debian: - - cpuid - RedHat: - - cpuid + - build-essential + - fakeroot + - devscripts + - wget + - git + - equivs + - liblz4-tool + - sudo + - python-is-python3 + - pkg-config + - unzip + - curl + - xz-utils + - binutils + - cpio + - rpm2cpio + - python3-dev diff --git a/roles/bootstrap/reset_qat_option/tasks/main.yml b/roles/bootstrap/reset_qat_option/tasks/main.yml deleted file mode 100644 index 19710959..00000000 --- a/roles/bootstrap/reset_qat_option/tasks/main.yml +++ /dev/null @@ -1,32 +0,0 @@ -## -## Copyright (c) 2020-2023 Intel Corporation. -## -## Licensed under the Apache License, Version 2.0 (the "License"); -## you may not use this file except in compliance with the License. -## You may obtain a copy of the License at -## -## http://www.apache.org/licenses/LICENSE-2.0 -## -## Unless required by applicable law or agreed to in writing, software -## distributed under the License is distributed on an "AS IS" BASIS, -## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -## See the License for the specific language governing permissions and -## limitations under the License. -## ---- -# workaround for Redhat9.2 qat OOT(out-of-tree) not supported issue. -- name: block to force QAT InTree driver for RedHat9.2 OS - block: - - name: print warning message for users to force QAT InTree driver via reset the update_qat_drivers to false - ansible.builtin.debug: - msg="[warning:] RedHat9.2 only support intree driver currently, reset the update_qat_drivers to false" - - - name: reset the update_qat_drivers to false - ansible.builtin.set_fact: - update_qat_drivers: false - when: - - configure_qat | default(false) | bool - - update_qat_drivers | default(false) | bool - - ansible_os_family == "RedHat" - - ansible_distribution_version >= '9.2' - - configured_arch in ["spr", "emr"] diff --git a/roles/bootstrap/set_intel_flexran_kernel_flags/tasks/main.yml b/roles/bootstrap/set_intel_flexran_kernel_flags/tasks/main.yml index a01cbfc2..c292c972 100644 --- a/roles/bootstrap/set_intel_flexran_kernel_flags/tasks/main.yml +++ b/roles/bootstrap/set_intel_flexran_kernel_flags/tasks/main.yml @@ -41,18 +41,11 @@ - debug: msg="{{ generated_cmdline.stdout }}" -- name: set generic Intel FlexRAN kernel flags - set_fact: -# intel_flexran_cmdline: 'GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} intel_iommu=on iommu=pt" {{ intel_flexran_marker }}' - intel_flexran_cmdline: 'GRUB_CMDLINE_LINUX="{{ generated_cmdline.stdout }}" {{ intel_flexran_marker }}' -# intel_flexran_isol_cores: "{{ generated_cmdline.stdout | regex_search('isolcpus=*', '\\1') }}" # in ENV is $isolcpus - -- debug: msg="generic kernel cmdline is {{ intel_flexran_cmdline }}" - - name: set Intel FlexRAN kernel flags for Host-16c-single set_fact: - intel_flexran_cmdline: 'GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepages=30 hugepagesz=1G nmi_watchdog=0 softlockup_panic=0 intel_iommu=on iommu=pt rcu_nocbs=1-15,17-31 irqaffinity=0,16 isolcpus=managed_irq,domain,1-15,17-31 kthread_cpus=0,16 nohz_full=1-15,17-31 crashkernel=auto enforcing=0 quiet rcu_nocb_poll rhgb selinux=0 mce=off audit=0 pci=realloc pci=assign-busses rdt=l3cat skew_tick=1 nosoftlockup nohz=on" {{ intel_flexran_marker }}' # noqa yaml[line-length] - intel_flexran_isol_cores: "1-15,17-31" + intel_flexran_cmdline: >- + GRUB_CMDLINE_LINUX="{{ generated_cmdline.stdout }}" {{ intel_flexran_marker }} + intel_flexran_isol_cores: "2-15,18-31" intel_flexran_cpu_supported: true when: - ansible_processor_count == 1 @@ -61,17 +54,23 @@ - name: set Intel FlexRAN kernel flags for Host-20c-single set_fact: - intel_flexran_cmdline: 'GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepages=40 hugepagesz=1G nmi_watchdog=0 softlockup_panic=0 intel_iommu=on iommu=pt rcu_nocbs=1-19,21-39 irqaffinity=0,20 isolcpus=managed_irq,domain,1-19,21-39 kthread_cpus=0,20 nohz_full=1-19,21-39 crashkernel=auto enforcing=0 quiet rcu_nocb_poll rhgb selinux=0 mce=off audit=0 pci=realloc pci=assign-busses rdt=l3cat skew_tick=1 nosoftlockup nohz=on" {{ intel_flexran_marker }}' # noqa yaml[line-length] - intel_flexran_isol_cores: "1-19,21-39" + intel_flexran_cmdline: >- + GRUB_CMDLINE_LINUX="{{ generated_cmdline.stdout }}" {{ intel_flexran_marker }} + intel_flexran_isol_cores: "2-19,22-39" intel_flexran_cpu_supported: true when: - ansible_processor_count == 1 - ansible_processor_cores == 20 - intel_flexran_type == "host" +# For Host-32c-single, the isol_cores is less than recommended(1-30,33-62), for the purpose of leaving more cpus for common tasks - name: set Intel FlexRAN kernel flags for Host-32c-single set_fact: - intel_flexran_cmdline: 'GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepages=60 hugepagesz=1G nmi_watchdog=0 softlockup_panic=0 intel_iommu=on iommu=pt rcu_nocbs=4-31,36-63 irqaffinity=0-3,32-35 isolcpus=managed_irq,domain,4-31,36-63 kthread_cpus=0-3,32-35 nohz_full=4-31,36-63 crashkernel=auto enforcing=0 quiet rcu_nocb_poll rhgb selinux=0 mce=off audit=0 pci=realloc pci=assign-busses rdt=l3cat skew_tick=1 nosoftlockup nohz=on" {{ intel_flexran_marker }}' # noqa yaml[line-length] + intel_flexran_cmdline: >- + GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepages=60 hugepagesz=1G nmi_watchdog=0 softlockup_panic=0 intel_iommu=on iommu=pt + vfio_pci.enable_sriov=1 vfio_pci.disable_idle_d3=1 rcu_nocbs=4-31,36-63 irqaffinity=0-3,32-35 isolcpus=managed_irq,domain,4-31,36-63 + kthread_cpus=0-3,32-35 nohz_full=4-31,36-63 crashkernel=auto enforcing=0 quiet rcu_nocb_poll rhgb selinux=0 mce=off audit=0 + pci=realloc pci=assign-busses rdt=l3cat skew_tick=1 nosoftlockup nohz=on" {{ intel_flexran_marker }} intel_flexran_isol_cores: "4-31,36-63" intel_flexran_cpu_supported: true when: @@ -79,9 +78,14 @@ - ansible_processor_cores == 32 - intel_flexran_type == "host" +# For Host-32c-dual, the isol_cores is less than recommended(1-62,65-126), for the purpose of leaving more cpus for common tasks - name: set Intel FlexRAN kernel flags for Host-32c-dual set_fact: - intel_flexran_cmdline: 'GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepages=60 hugepagesz=1G nmi_watchdog=0 softlockup_panic=0 intel_iommu=on iommu=pt rcu_nocbs=4-59,68-123 irqaffinity=0-3,60-63,64-67,124-127 isolcpus=managed_irq,domain,4-59,68-123 kthread_cpus=0-3,60-63,64-67,124-127 nohz_full=4-59,68-123 crashkernel=auto enforcing=0 quiet rcu_nocb_poll rhgb selinux=0 mce=off audit=0 pci=realloc pci=assign-busses rdt=l3cat skew_tick=1 nosoftlockup nohz=on" {{ intel_flexran_marker }}' # noqa yaml[line-length] + intel_flexran_cmdline: >- + GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepages=60 hugepagesz=1G nmi_watchdog=0 softlockup_panic=0 intel_iommu=on iommu=pt + vfio_pci.enable_sriov=1 vfio_pci.disable_idle_d3=1 rcu_nocbs=4-59,68-123 irqaffinity=0-3,60-63,64-67,124-127 + isolcpus=managed_irq,domain,4-59,68-123 kthread_cpus=0-3,60-63,64-67,124-127 nohz_full=4-59,68-123 crashkernel=auto enforcing=0 + quiet rcu_nocb_poll rhgb selinux=0 mce=off audit=0 pci=realloc pci=assign-busses rdt=l3cat skew_tick=1 nosoftlockup nohz=on" {{ intel_flexran_marker }} intel_flexran_isol_cores: "4-59,68-123" intel_flexran_cpu_supported: true when: @@ -89,9 +93,14 @@ - ansible_processor_cores == 32 - intel_flexran_type == "host" +# For Host-52c-dual, the isol_cores is less than recommended, for the purpose of leaving more cpus for common tasks - name: set Intel FlexRAN kernel flags for Host-52c-dual set_fact: - intel_flexran_cmdline: 'GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepages=60 hugepagesz=1G nmi_watchdog=0 softlockup_panic=0 intel_iommu=on iommu=pt rcu_nocbs=4-99,108-203 irqaffinity=0-3,100-103,104-107,204-207 isolcpus=managed_irq,domain,4-99,108-203 kthread_cpus=0-3,100-103,104-107,204-207 nohz_full=4-99,108-203 crashkernel=auto enforcing=0 quiet rcu_nocb_poll rhgb selinux=0 mce=off audit=0 pci=realloc pci=assign-busses rdt=l3cat skew_tick=1 nosoftlockup nohz=on" {{ intel_flexran_marker }}' # noqa yaml[line-length] + intel_flexran_cmdline: >- + GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepages=60 hugepagesz=1G nmi_watchdog=0 softlockup_panic=0 intel_iommu=on iommu=pt + vfio_pci.enable_sriov=1 vfio_pci.disable_idle_d3=1 rcu_nocbs=4-99,108-203 irqaffinity=0-3,100-103,104-107,204-207 + isolcpus=managed_irq,domain,4-99,108-203 kthread_cpus=0-3,100-103,104-107,204-207 nohz_full=4-99,108-203 crashkernel=auto enforcing=0 + quiet rcu_nocb_poll rhgb selinux=0 mce=off audit=0 pci=realloc pci=assign-busses rdt=l3cat skew_tick=1 nosoftlockup nohz=on" {{ intel_flexran_marker }} intel_flexran_isol_cores: "4-99,108-203" intel_flexran_cpu_supported: true when: @@ -99,44 +108,31 @@ - ansible_processor_cores == 52 - intel_flexran_type == "host" +# For Host-56c-dual, the isol_cores is less than recommended, for the purpose of leaving more cpus for common tasks - name: set Intel FlexRAN kernel flags for Host-56c-dual set_fact: - intel_flexran_cmdline: 'GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepages=60 hugepagesz=1G nmi_watchdog=0 softlockup_panic=0 intel_iommu=on iommu=pt rcu_nocbs=4-107,116-219 irqaffinity=0-3,108-111,112-115,220-223 isolcpus=managed_irq,domain,4-107,116-219 kthread_cpus=0-3,108-111,112-115,220-223 nohz_full=4-107,116-219 crashkernel=auto enforcing=0 quiet rcu_nocb_poll rhgb selinux=0 mce=off audit=0 pci=realloc pci=assign-busses rdt=l3cat skew_tick=1 nosoftlockup nohz=on" {{ intel_flexran_marker }}' # noqa yaml[line-length] - intel_flexran_isol_cores: "4-99,108-203" + intel_flexran_cmdline: >- + GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepages=60 hugepagesz=1G nmi_watchdog=0 softlockup_panic=0 intel_iommu=on iommu=pt + vfio_pci.enable_sriov=1 vfio_pci.disable_idle_d3=1 rcu_nocbs=4-107,116-219 irqaffinity=0-3,108-111,112-115,220-223 + isolcpus=managed_irq,domain,4-107,116-219 kthread_cpus=0-3,108-111,112-115,220-223 nohz_full=4-107,116-219 crashkernel=auto enforcing=0 + quiet rcu_nocb_poll rhgb selinux=0 mce=off audit=0 pci=realloc pci=assign-busses rdt=l3cat skew_tick=1 nosoftlockup nohz=on" {{ intel_flexran_marker }} + intel_flexran_isol_cores: "4-107,116-219" intel_flexran_cpu_supported: true when: - ansible_processor_count == 2 - ansible_processor_cores == 56 - intel_flexran_type == "host" -- name: >- - set Intel FlexRAN kernel flags for Docker POD on Host-32c-single (6338N CPU) on ICX when kernel version is older than 5.15.0-1019RT. - See https://hub.docker.com/r/intel/flexran_vdu - set_fact: - intel_flexran_cmdline: >- - GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt usbcore.autosuspend=-1 selinux=0 enforcing=0 nmi_watchdog=0 crashkernel=auto softlockup_panic=0 - audit=0 cgroup_disable=memory mce=off hugepagesz=1G hugepages=60 hugepagesz=2M hugepages=0 default_hugepagesz=1G - kthread_cpus=0,28 irqaffinity=0,28" {{ intel_flexran_marker }} - intel_flexran_isol_cores: "1-27,29-55" - intel_flexran_cpu_supported: true - when: - - ansible_processor_count == 1 - - ansible_processor_cores == 32 - - intel_flexran_type == "pod" - - ansible_kernel < "5.15.0-1019-realtime" - - configured_arch == "icx" - # for 5.15.0-1019RT and later, cgroup_disable=memory is no longer neeeded. # RKE2 can be not installed with cgroup_disable=memory, so use 5.15.0-1019RT and later for FlexRAN deployment on RKE2. +# The isol_cores is different with written in wiki, as we think there is a mistake. - name: >- set Intel FlexRAN kernel flags for Docker POD on Host-32c-single (6338N CPU) on ICX when kernel version is 5.15.0-1019RT and later. - See https://hub.docker.com/r/intel/flexran_vdu + See wiki: https://hub.docker.com/r/intel/flexran_vdu set_fact: intel_flexran_cmdline: >- - GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt usbcore.autosuspend=-1 selinux=0 enforcing=0 nmi_watchdog=0 crashkernel=auto softlockup_panic=0 - audit=0 mce=off hugepagesz=1G hugepages=60 hugepagesz=2M hugepages=0 default_hugepagesz=1G kthread_cpus=0,28 irqaffinity=0,28" - {{ intel_flexran_marker }} - intel_flexran_isol_cores: "1-27,29-55" + GRUB_CMDLINE_LINUX="{{ generated_cmdline.stdout }}" {{ intel_flexran_marker }} + intel_flexran_isol_cores: "2-31,34-63" intel_flexran_cpu_supported: true when: - ansible_processor_count == 1 @@ -156,12 +152,12 @@ - name: >- set Intel FlexRAN kernel flags for Docker POD on SPR-EE MCC when kernel version is 5.15.0-1030RT and later. - See https://hub.docker.com/r/intel/flexran_l1_spree + See wiki: https://hub.docker.com/r/intel/flexran_l1_spree set_fact: intel_flexran_cmdline: >- - GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt vfio_pci.enable_sriov=1 vfio_pci.disable_idle_d3=1 usbcore.autosuspend=-1 selinux=0 enforcing=0 - nmi_watchdog=0 crashkernel=auto softlockup_panic=0 audit=0 cgroup_enable=memory mce=off hugepagesz=1G hugepages=60 hugepagesz=2M hugepages=0 - default_hugepagesz=1G kthread_cpus=0,31,32,63 irqaffinity=0,31,32,63" {{ intel_flexran_marker }} + GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt vfio_pci.enable_sriov=1 vfio_pci.disable_idle_d3=1 usbcore.autosuspend=-1 selinux=0 + enforcing=0 nmi_watchdog=0 crashkernel=auto softlockup_panic=0 audit=0 cgroup_enable=memory mce=off hugepagesz=1G hugepages=60 + hugepagesz=2M hugepages=0 default_hugepagesz=1G kthread_cpus=0,31,32,63 irqaffinity=0,31,32,63" {{ intel_flexran_marker }} intel_flexran_isol_cores: "1-30,33-62" intel_flexran_cpu_supported: true when: diff --git a/roles/bootstrap/set_sriov_kernel_flags/tasks/main.yml b/roles/bootstrap/set_sriov_kernel_flags/tasks/main.yml index 5b6f6e46..a91458a9 100644 --- a/roles/bootstrap/set_sriov_kernel_flags/tasks/main.yml +++ b/roles/bootstrap/set_sriov_kernel_flags/tasks/main.yml @@ -22,9 +22,9 @@ register: grub notify: - reboot server - when: not (iommu_enabled | default(false) | bool) and not (on_vms | default(false) | bool) + when: not iommu_enabled | default(false) | bool - name: setup sriov grub commandline parameters include_tasks: setup_sriov_kernel_flags.yml when: - - iommu_enabled | default(false) | bool or (on_vms is defined and on_vms) + - iommu_enabled | default(false) | bool or on_vms | default(false) | bool diff --git a/roles/bootstrap/set_sriov_kernel_flags/tasks/setup_sriov_kernel_flags.yml b/roles/bootstrap/set_sriov_kernel_flags/tasks/setup_sriov_kernel_flags.yml index 387dbfdf..1c4b2e2b 100644 --- a/roles/bootstrap/set_sriov_kernel_flags/tasks/setup_sriov_kernel_flags.yml +++ b/roles/bootstrap/set_sriov_kernel_flags/tasks/setup_sriov_kernel_flags.yml @@ -28,11 +28,11 @@ (ansible_distribution == "Ubuntu" and ansible_distribution_version >= "21.04") or (ansible_os_family == "RedHat" and ansible_distribution_version >= "8.4") -- name: set noiommu default kernel flags +- name: set iommu default kernel flags set_fact: iommu_cmdline: "" -- name: set iommu default kernel flags +- name: set iommu enabled kernel flags set_fact: iommu_cmdline: " intel_iommu=on iommu=pt" when: diff --git a/playbooks/intel/base_container.yml b/roles/bootstrap/set_tdx_kernel_flags/defaults/main.yml similarity index 90% rename from playbooks/intel/base_container.yml rename to roles/bootstrap/set_tdx_kernel_flags/defaults/main.yml index 3b7de268..7f8062cf 100644 --- a/playbooks/intel/base_container.yml +++ b/roles/bootstrap/set_tdx_kernel_flags/defaults/main.yml @@ -14,6 +14,4 @@ ## limitations under the License. ## --- -- hosts: k8s_cluster - roles: - - role: intel_base_container +tdx_marker: "# tdx marker" diff --git a/roles/bootstrap/configure_intel_pstate/tasks/setup_intel_pstate.yml b/roles/bootstrap/set_tdx_kernel_flags/tasks/main.yml similarity index 57% rename from roles/bootstrap/configure_intel_pstate/tasks/setup_intel_pstate.yml rename to roles/bootstrap/set_tdx_kernel_flags/tasks/main.yml index 97a793ec..31b9facd 100644 --- a/roles/bootstrap/configure_intel_pstate/tasks/setup_intel_pstate.yml +++ b/roles/bootstrap/set_tdx_kernel_flags/tasks/main.yml @@ -14,23 +14,23 @@ ## limitations under the License. ## --- -- name: validate that intel_pstate is defined correctly - fail: - msg: > - "intel_pstate accepts following options: - disable, passive, force, no_hwp, hwp_only, support_acpi_ppc, per_cpu_perf_limits. Provided: {{ intel_pstate }}" +- name: set tdx 1.5 kernel flags + set_fact: + tdx_cmdline: 'GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} numa_balancing=disable" {{ tdx_marker }}' when: - - intel_pstate not in acceptable_intel_pstate_values + - tdx_version == "1.5" -- name: set kernel flags required for Intel PState scaling driver +- name: set tdx 1.0 kernel flags set_fact: - intel_pstate_cmdline: 'GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} intel_pstate={{ intel_pstate }}" {{ intel_pstate_marker }}' + tdx_cmdline: 'GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} numa_balancing=disable ima_policy=tcb" {{ tdx_marker }}' + when: + - tdx_version == "1.0" -- name: set intel_pstate configuration in /etc/default/grub +- name: set tdx kernel flags in /etc/default/grub lineinfile: dest: /etc/default/grub - regexp: '^GRUB_CMDLINE_LINUX="\${GRUB_CMDLINE_LINUX}(.*?)" {{ intel_pstate_marker }}$' - line: '{{ intel_pstate_cmdline }}' + regexp: '^GRUB_CMDLINE_LINUX="\${GRUB_CMDLINE_LINUX}(.*?)" {{ tdx_marker }}$' + line: '{{ tdx_cmdline }}' state: present mode: 0664 notify: diff --git a/roles/bootstrap/update_nic_drivers/defaults/main.yml b/roles/bootstrap/update_nic_drivers/defaults/main.yml index f659c6c9..14184ccb 100644 --- a/roles/bootstrap/update_nic_drivers/defaults/main.yml +++ b/roles/bootstrap/update_nic_drivers/defaults/main.yml @@ -16,15 +16,15 @@ --- # i40e i40e_driver_name: i40e -i40e_driver_version: 2.22.18 +i40e_driver_version: 2.23.17 i40e_driver_url: https://sourceforge.net/projects/e1000/files/i40e%20stable/{{ i40e_driver_version }}/i40e-{{ i40e_driver_version }}.tar.gz -i40e_driver_checksum: sha1:0c94bd91014a0d81bd6b99fb41d0e4f1c12b09ff +i40e_driver_checksum: sha1:57904a541212174c20bacfbf109401c19c59c25c # ice ice_driver_name: ice -ice_driver_version: 1.11.14 +ice_driver_version: 1.12.7 ice_driver_url: https://sourceforge.net/projects/e1000/files/ice%20stable/{{ ice_driver_version }}/ice-{{ ice_driver_version }}.tar.gz -ice_driver_checksum: sha1:730cd04fcfd0ba1b33ba21aaf671d0e1654c999a +ice_driver_checksum: sha1:b286f3bdf48c2a355f6c1b0ed2a3d82bce21c6df # iavf iavf_driver_name: iavf diff --git a/roles/bootstrap/update_nic_drivers/tasks/ice.yml b/roles/bootstrap/update_nic_drivers/tasks/ice.yml index 1f2ac94c..92744580 100644 --- a/roles/bootstrap/update_nic_drivers/tasks/ice.yml +++ b/roles/bootstrap/update_nic_drivers/tasks/ice.yml @@ -27,17 +27,13 @@ set_fact: ice_driver_build_failed: false -# unloading before update is probably not necessay and does not work anyway when irdma is using ice -# - name: unload ice module -# modprobe: -# name: ice -# state: absent -# when: -# - ice_installed_version.stdout != ice_driver_version -# - mgmt_interface_driver.stdout != ice_driver_name -# - not update_kernel -# - (ansible_os_family == "RedHat" and ansible_distribution_version < "9.0") or -# (ansible_distribution == "Ubuntu" and ansible_distribution_version < "22.04") +- name: Determine if patch of driver needed + ansible.builtin.set_fact: + patch_ice_needed: >- + {{ + (intel_media_transport_library_enabled | default(false) and + intel_media_transport_library.patch_nic_driver | default(false)) | bool + }} - name: update ice driver block: @@ -63,6 +59,15 @@ become: yes register: ice_untar + - name: Patch ice driver to support IMTL + vars: + imtl_ice_driver_dir: "{{ (ice_untar.dest, ice_untar.files[0]) | path_join }}" + imtl_ice_version: "{{ ice_driver_version }}" + ansible.builtin.include_role: + name: imtl_install + tasks_from: ice_drv_patch.yml + when: patch_ice_needed + - name: build and install ice driver make: chdir: "{{ (ice_untar.dest, ice_untar.files[0], 'src') | path_join }}" @@ -92,7 +97,12 @@ - name: set ice driver build failed status set_fact: ice_driver_build_failed: true - when: ice_installed_version.stdout != ice_driver_version + + - name: Fail when patch of NIC driver is needed + ansible.builtin.fail: + msg: "Patch of ice driver failed." + when: patch_ice_needed + when: ice_installed_version.stdout != ice_driver_version or patch_ice_needed - name: reboot node after driver update become: yes diff --git a/roles/bootstrap/update_nic_firmware/defaults/main.yml b/roles/bootstrap/update_nic_firmware/defaults/main.yml index 67cc8b67..1f44f9cb 100644 --- a/roles/bootstrap/update_nic_firmware/defaults/main.yml +++ b/roles/bootstrap/update_nic_firmware/defaults/main.yml @@ -18,18 +18,18 @@ nvmupdate: # 700 Series i40e: - nvmupdate_pkg_url: "https://downloadmirror.intel.com/769287/700Series_NVMUpdatePackage_v9_20_Linux.tar.gz" - nvmupdate_pkg_checksum: "sha1:87F0BDA58BAAEE0ADF1FADBBCC485AF0A2F0777F" - required_fw_version: "9.20" + nvmupdate_pkg_url: "https://downloadmirror.intel.com/786060/700Series_NVMUpdatePackage_v9_30_Linux.tar.gz" + nvmupdate_pkg_checksum: "sha1:929987FAB30394C86AA1BBADDAA62BB7D4E8CC0E" + required_fw_version: "9.30" # min fw version for ddp was taken from: # https://www.intel.com/content/www/us/en/developer/articles/technical/dynamic-device-personalization-for-intel-ethernet-700-series.html min_ddp_loadable_fw_version: "6.01" min_updatable_fw_version: "5.02" # 800 Series (CVL) ice: - nvmupdate_pkg_url: "https://downloadmirror.intel.com/769278/E810_NVMUpdatePackage_v4_20_Linux.tar.gz" - nvmupdate_pkg_checksum: "sha1:36CE159E53E6060F2AC4E3419DB8A21E3D982A85" - required_fw_version: "4.20" + nvmupdate_pkg_url: "https://downloadmirror.intel.com/786047/E810_NVMUpdatePackage_v4_30_Linux.tar.gz" + nvmupdate_pkg_checksum: "sha1:993D79AC623B71C5378855738917495A0FA8FFB8" + required_fw_version: "4.30" # https://builders.intel.com/docs/networkbuilders/intel-ethernet-controller-800-series-device-personalization-ddp-for-telecommunications-workloads-technology-guide.pdf # document above does not specify any min fw version needed for ddp feature. So, min_ddp_loadable_fw is the same as min_updatable_fw min_ddp_loadable_fw_version: "0.70" diff --git a/roles/cadvisor_install/defaults/main.yaml b/roles/cadvisor_install/defaults/main.yaml index ee9407fb..ac507a55 100644 --- a/roles/cadvisor_install/defaults/main.yaml +++ b/roles/cadvisor_install/defaults/main.yaml @@ -14,16 +14,13 @@ ## limitations under the License. ## --- -cadvisor_application_name: "cadvisor" # cAdvisor Main application name -cadvisor_release_name: "cadvisor" # cAdvisor Helm Charts release name +cadvisor_git_repo_url: "https://github.com/google/cadvisor" +cadvisor_image: gcr.io/cadvisor/cadvisor +cadvisor_version: "v0.47.2" -cadvisor_image: gcr.io/cadvisor/cadvisor # cAdvisor Docker image -cadvisor_image_version: 0.44.0 # cAdvisor Version - -cadvisor_helm_repo_url: "https://ckotzbauer.github.io/helm-charts" # cAdvisor Helm Repo URL -cadvisor_helm_repo_name: "ckotzbauer" # cAdvisor Repo Name -cadvisor_helm_chart_ref: "ckotzbauer/cadvisor" # cAdvisor Helm Chart Reference -cadvisor_helm_chart_version: "2.2.4" # cAdvisor Helm Chart Version cadvisor_namespace: "cadvisor" # cAdvisor Namespace cadvisor_perf_config_filename: "perf-events.json" + +rbac_proxy_ssl_mount_path: /etc/ssl/rbac-proxy +rbac_proxy_ssl_secret_name: cadvisor-rbac-proxy-ssl diff --git a/roles/cadvisor_install/tasks/cleanup.yml b/roles/cadvisor_install/tasks/cleanup.yml index de980224..ab2b2eae 100644 --- a/roles/cadvisor_install/tasks/cleanup.yml +++ b/roles/cadvisor_install/tasks/cleanup.yml @@ -18,19 +18,21 @@ when: - inventory_hostname == groups['kube_control_plane'][0] block: - - name: Uninstall cAdvisor Helm Chart - kubernetes.core.helm: - name: "{{ cadvisor_release_name }}" - namespace: "{{ cadvisor_namespace }}" - state: absent - - name: Remove cAdvisor Helm Repo - kubernetes.core.helm_repository: - name: "{{ cadvisor_helm_repo_name }}" + - name: generate k8s manifest using kustomize + ansible.builtin.command: "/usr/local/bin/kubectl kustomize" + args: + chdir: "{{ (project_root_dir, 'cadvisor', 'deploy', 'kubernetes', 'overlays', 'cek') | path_join }}" + register: kustomize + changed_when: true + + - name: delete k8s resources + kubernetes.core.k8s: + definition: "{{ kustomize.stdout }}" state: absent tags: - cadvisor - name: Delete cAdvisor directory ansible.builtin.file: - path: "{{ (project_root_dir, 'charts', 'cadvisor') | path_join }}" + path: "{{ (project_root_dir, 'cadvisor') | path_join }}" state: absent diff --git a/roles/cadvisor_install/tasks/install.yml b/roles/cadvisor_install/tasks/install.yml index 0dc47595..5c108dd0 100644 --- a/roles/cadvisor_install/tasks/install.yml +++ b/roles/cadvisor_install/tasks/install.yml @@ -14,24 +14,35 @@ ## limitations under the License. ## --- -- name: template cAdvisor Helm chart values and push to controller node +- name: generate ssl certificate + include_role: + name: create_signed_k8s_certs + vars: + secret_name: "{{ rbac_proxy_ssl_secret_name }}" + service_name: cadvisor + key_pair_name: cadvisor-rbac-proxy + host_secrets_folder: "{{ rbac_proxy_ssl_mount_path }}" + k8s_namespace: "{{ cadvisor_namespace }}" + csr_cluster_name: "{{ cluster_name | default('cluster.local') }}" + +- name: template cAdvisor kustomize overlay values ansible.builtin.template: - src: "cadvisor_custom_values.yml.j2" - dest: "{{ (project_root_dir, 'charts', 'cadvisor', 'cadvisor-custom-values.yml') | path_join }}" - mode: 0644 + src: "{{ ( 'overlay', item) | path_join }}.j2" + dest: "{{ (project_root_dir, 'cadvisor', 'deploy', 'kubernetes', 'overlays', 'cek', item) | path_join }}" + mode: '0644' + with_items: + - "kustomization.yaml" + - "rbac-proxy.yaml" + - "daemonset.yaml" + - "service.yaml" -- name: Add Helm Repository - {{ cadvisor_helm_repo_url }} - kubernetes.core.helm_repository: - name: "{{ cadvisor_helm_repo_name }}" - url: "{{ cadvisor_helm_repo_url }}" - state: present +- name: generate k8s manifest using kustomize + ansible.builtin.command: "/usr/local/bin/kubectl kustomize" + args: + chdir: "{{ (project_root_dir, 'cadvisor', 'deploy', 'kubernetes', 'overlays', 'cek') | path_join }}" + register: kustomize + changed_when: true -- name: Deploy cAdvisor - kubernetes.core.helm: - name: "{{ cadvisor_release_name }}" - chart_ref: "{{ cadvisor_helm_chart_ref }}" - chart_version: "{{ cadvisor_helm_chart_version }}" - namespace: "{{ cadvisor_namespace }}" - create_namespace: true - values_files: "{{ (project_root_dir, 'charts', 'cadvisor', 'cadvisor-custom-values.yml') | path_join }}" - wait: true +- name: apply k8s resources + k8s: + definition: "{{ kustomize.stdout }}" diff --git a/roles/cadvisor_install/tasks/main.yml b/roles/cadvisor_install/tasks/main.yml index f9904aa1..fe7664c6 100644 --- a/roles/cadvisor_install/tasks/main.yml +++ b/roles/cadvisor_install/tasks/main.yml @@ -14,22 +14,44 @@ ## limitations under the License. ## --- -- name: create cAdvisor Helm chart directory +- name: clone cAdvisor git repo + ansible.builtin.git: + repo: "{{ cadvisor_git_repo_url }}" + dest: "{{ (project_root_dir, 'cadvisor') | path_join }}" + version: "{{ cadvisor_version }}" + +- name: create cAdvisor overlay directory ansible.builtin.file: - path: "{{ (project_root_dir, 'charts', 'cadvisor') | path_join }}" + path: "{{ (project_root_dir, 'cadvisor', 'deploy', 'kubernetes', 'overlays', 'cek') | path_join }}" state: directory - mode: 0755 + mode: '0644' - name: Check if perf events config enabled ansible.builtin.set_fact: cadvisor_perf_events: true when: cadvisor_sample_perf_events_enabled or cadvisor_pik_perf_events_enabled | default(false) -- name: Prepare perf events config for all nodes +- name: Prepare perf events config ansible.builtin.import_tasks: perf_events_config.yml - when: cadvisor_perf_events | default(false) + when: + - cadvisor_perf_events | default(false) + - inventory_hostname == groups['kube_control_plane'][0] + +# RHEL CRI-O install will setup /etc/containers/mounts.conf has mapping: "/usr/share/rhel/secrets:/run/secrets". +# This will tell crio engine to setup /var/run/secrets for *every* pod (/var/run is symbol link to /run) +# cAdvisor explicitly mounts host /var/run to pod with readonly permission which prevents /var/run/secrets folder to be created, so here we need a workaround. +- name: create /var/run/secrets for CRI workaround + ansible.builtin.file: + path: /var/run/secrets + state: directory + owner: root + group: root + mode: '0755' + when: + - container_runtime == "crio" + - ansible_os_family == 'RedHat' -- name: install cAdvisor Helm charts +- name: install cAdvisor ansible.builtin.import_tasks: install.yml when: - inventory_hostname == groups['kube_control_plane'][0] diff --git a/roles/cadvisor_install/tasks/perf_events_config.yml b/roles/cadvisor_install/tasks/perf_events_config.yml index 791cca22..9e75db4f 100644 --- a/roles/cadvisor_install/tasks/perf_events_config.yml +++ b/roles/cadvisor_install/tasks/perf_events_config.yml @@ -15,13 +15,13 @@ ## - name: create the perf_config folder for cAdvisor custom perf events ansible.builtin.file: - path: "{{ (project_root_dir, 'charts', 'cadvisor', 'perf_config') | path_join }}" + path: "{{ (project_root_dir, 'cadvisor', 'perf_config') | path_join }}" state: directory - mode: 0755 + mode: '0755' - name: load sample perf events json file ansible.builtin.set_fact: - sample_perf_conf_json: "{{ lookup('file', 'sample-perf-event.json' ) | from_json }}" + sample_perf_conf_json: "{{ lookup('file', 'sample-perf-event.json') | from_json }}" when: cadvisor_sample_perf_events_enabled - name: load perf events json file supplied for PIK @@ -34,10 +34,10 @@ sample_perf: "{{ sample_perf_conf_json | default({}) }}" pik_perf: "{{ pik_perf_conf_json | default({}) }}" ansible.builtin.set_fact: - perf_conf_json: "{{ sample_perf | combine(pik_perf, recursive=true, list_merge='append_rp') | to_json(indent=2) }}" + perf_conf_json: "{{ sample_perf | combine(pik_perf, recursive=true, list_merge='append_rp') | to_json }}" -- name: create perf events configuration json file - ansible.builtin.copy: - content: "{{ perf_conf_json }}" - dest: "{{ (project_root_dir, 'charts', 'cadvisor', 'perf_config', cadvisor_perf_config_filename) | path_join }}" - mode: 0644 +- name: template perf events ConfigMap + ansible.builtin.template: + src: "overlay/configmap.yaml.j2" + dest: "{{ (project_root_dir, 'cadvisor', 'deploy', 'kubernetes', 'overlays', 'cek', 'configmap.yaml') | path_join }}" + mode: '0644' diff --git a/roles/cadvisor_install/tasks/preflight.yml b/roles/cadvisor_install/tasks/preflight.yml index eeb86ab9..457545f9 100644 --- a/roles/cadvisor_install/tasks/preflight.yml +++ b/roles/cadvisor_install/tasks/preflight.yml @@ -22,5 +22,5 @@ cAdvisor perf events counting can only be enabled on BMRA. Please disable perf events counting in group_vars. when: - - inventory_hostname == groups["kube_control_plane"][0] + - inventory_hostname == (groups["kube_control_plane"][0] | default("")) - cadvisor_pik_perf_events_enabled | default(false) or cadvisor_sample_perf_events_enabled | default(false) diff --git a/roles/cadvisor_install/templates/cadvisor_custom_values.yml.j2 b/roles/cadvisor_install/templates/cadvisor_custom_values.yml.j2 deleted file mode 100644 index 81df240d..00000000 --- a/roles/cadvisor_install/templates/cadvisor_custom_values.yml.j2 +++ /dev/null @@ -1,102 +0,0 @@ -image: - repository: {{ cadvisor_image }} - tag: v{{ cadvisor_image_version }} - pullPolicy: IfNotPresent - - ## Reference to one or more secrets to be used when pulling images - ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ - ## - pullSecrets: [] - -container: - port: 8080 - additionalArgs: - - --housekeeping_interval=10s # kubernetes default args - - --max_housekeeping_interval=15s - - --event_storage_event_limit=default=0 - - --event_storage_age_limit=default=0 - - --disable_metrics=percpu,process,sched,tcp,udp # enable only diskIO, cpu, memory, network, disk - {% if cadvisor_perf_events | default(false) -%} - - --perf_events_config={{ ('/mnt/perf-config', cadvisor_perf_config_filename) | path_join }} - {% endif -%} - - --docker_only - hostPaths: - {% if cadvisor_perf_events | default(false) -%} - - name: custom-events - path: "{{ (project_root_dir, 'charts', 'cadvisor', 'perf_config') | path_join }}" - mount: "/mnt/perf-config" - {% endif -%} - - name: varrun - path: "/var/run" - - name: sys - path: "/sys" - - name: docker - path: "/var/lib/docker" - - name: disk - path: "/dev/disk" - -resources: {} - # We usually recommend not to specify default resources and to leave this as a conscious - # choice for the user. This also increases chances charts run on environments with little - # resources, such as Minikube. If you do want to specify resources, uncomment the following - # lines, adjust them as necessary, and remove the curly braces after 'resources:'. - # limits: - # cpu: 100m - # memory: 128Mi - # requests: - # cpu: 100m - # memory: 128Mi - -podAnnotations: {} - -# priorityClassName: system-cluster-critical -priorityClassName: {} - -# sometimes errors are encountered when using the cpu load reader without being on the host network -hostNetwork: false - -serviceAccount: - # Specifies whether a service account should be created - create: true - # The name of the service account to use. - # If not set and create is true, a name is generated using the fullname template - name: - -podSecurityPolicy: - create: false - privileged: false - -# Specifies whether a securityContext should be created. Required for privileged operations. -podSecurityContext: - create: true - privileged: true - -nodeSelector: {} - -tolerations: [] - -affinity: {} - -# This will create a ServiceMonitor Custom Resource indicating the prometheus operator what to scrape. -metrics: - enabled: false - # This will allow you to specify relabelings on the metrics before ingestion. E.g. to use the kubernetes monitoring - # mixin with this chart set metrics.enabled above to true and use: - # relabelings: - # - sourceLabels: - # - name - # targetLabel: container - # - sourceLabels: - # - container_label_io_kubernetes_pod_namespace - # targetLabel: namespace - # - sourceLabels: - # - container_label_io_kubernetes_pod_name - # targetLabel: pod - metricRelabelings: [] - # This will allow you to specify relabelings on the metrics before scraping. - # relabelings: - # - action: replace - # sourceLabels: - # - __meta_kubernetes_pod_node_name - # targetLabel: node - relabelings: [] diff --git a/roles/cadvisor_install/templates/overlay/configmap.yaml.j2 b/roles/cadvisor_install/templates/overlay/configmap.yaml.j2 new file mode 100644 index 00000000..635c4f0d --- /dev/null +++ b/roles/cadvisor_install/templates/overlay/configmap.yaml.j2 @@ -0,0 +1,8 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: perf-config + namespace: {{ cadvisor_namespace }} +data: + perf-config.json: |- + {{ perf_conf_json }} diff --git a/roles/cadvisor_install/templates/overlay/daemonset.yaml.j2 b/roles/cadvisor_install/templates/overlay/daemonset.yaml.j2 new file mode 100644 index 00000000..023d932e --- /dev/null +++ b/roles/cadvisor_install/templates/overlay/daemonset.yaml.j2 @@ -0,0 +1,48 @@ +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: cadvisor + namespace: {{ cadvisor_namespace }} +spec: + template: + spec: + automountServiceAccountToken: true + containers: + - name: cadvisor + image: {{ cadvisor_image }}:{{ cadvisor_version }} + resources: + requests: + memory: 800Mi + cpu: 400m + limits: + memory: 4000Mi + cpu: 800m + args: + - --housekeeping_interval=10s # kubernetes default args + - --max_housekeeping_interval=15s + - --event_storage_event_limit=default=0 + - --event_storage_age_limit=default=0 + - --disable_metrics=percpu,process,sched,tcp,udp # enable only diskIO, cpu, memory, network, disk + {% if cadvisor_perf_events | default(false) -%} + - --perf_events_config="/etc/config/perf-config.json" + {% endif -%} + - --docker_only + - --listen_ip=127.0.0.1 + {% if cadvisor_perf_events | default(false) -%} + securityContext: + privileged: true + {% endif -%} + volumeMounts: + {% if cadvisor_perf_events | default(false) -%} + - name: perf-config + mountPath: /etc/config/ + {% endif -%} + - name: var-run + mountPath: /var/run + readOnly: false + {% if cadvisor_perf_events | default(false) -%} + volumes: + - name: perf-config + configMap: + name: perf-config + {% endif -%} diff --git a/roles/cadvisor_install/templates/overlay/kustomization.yaml.j2 b/roles/cadvisor_install/templates/overlay/kustomization.yaml.j2 new file mode 100644 index 00000000..1db28b67 --- /dev/null +++ b/roles/cadvisor_install/templates/overlay/kustomization.yaml.j2 @@ -0,0 +1,11 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - service.yaml + - ../../base +{% if cadvisor_perf_events | default(false) %} + - configmap.yaml +{% endif %} +patches: + - path: daemonset.yaml + - path: rbac-proxy.yaml diff --git a/roles/cadvisor_install/templates/overlay/rbac-proxy.yaml.j2 b/roles/cadvisor_install/templates/overlay/rbac-proxy.yaml.j2 new file mode 100644 index 00000000..9d09f5f2 --- /dev/null +++ b/roles/cadvisor_install/templates/overlay/rbac-proxy.yaml.j2 @@ -0,0 +1,29 @@ +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: cadvisor + namespace: {{ cadvisor_namespace }} +spec: + template: + spec: + containers: + - name: rbac-proxy + image: "{{ kube_rbac_proxy_image_repo }}:{{ kube_rbac_proxy_image_tag }}" + imagePullPolicy: IfNotPresent + volumeMounts: + - name: ssl + mountPath: "{{ rbac_proxy_ssl_mount_path }}" + readOnly: true + ports: + - name: https + containerPort: 8443 + args: + - "--tls-cert-file={{ rbac_proxy_ssl_mount_path }}/{{ rbac_proxy_ssl_secret_name }}.cert" + - "--tls-private-key-file={{ rbac_proxy_ssl_mount_path }}/{{ rbac_proxy_ssl_secret_name }}.key" + - "--tls-cipher-suites={{ kube_rbac_proxy_tls_ciphers }}" + - "--secure-listen-address=0.0.0.0:8443" + - "--upstream=http://127.0.0.1:8080/" + volumes: + - name: ssl + secret: + secretName: "{{ rbac_proxy_ssl_secret_name }}" diff --git a/roles/cadvisor_install/templates/overlay/service.yaml.j2 b/roles/cadvisor_install/templates/overlay/service.yaml.j2 new file mode 100644 index 00000000..7ea0d970 --- /dev/null +++ b/roles/cadvisor_install/templates/overlay/service.yaml.j2 @@ -0,0 +1,13 @@ +apiVersion: v1 +kind: Service +metadata: + name: cadvisor + namespace: {{ cadvisor_namespace }} +spec: + selector: + app: cadvisor + ports: + - protocol: TCP + port: 8443 + targetPort: 8443 + name: https diff --git a/roles/calico_vpp_install/defaults/main.yml b/roles/calico_vpp_install/defaults/main.yml new file mode 100644 index 00000000..14660420 --- /dev/null +++ b/roles/calico_vpp_install/defaults/main.yml @@ -0,0 +1,36 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +calico_vpp_files_dir: "{{ (project_root_dir, 'calico-vpp-files') | path_join }}" + +k8s_calico_vpp_version: "v3.25.1" +# install operator to manage the installation, upgrade, and general lifecycle of a Calico cluster +k8s_calico_tigera_operator: "https://raw.githubusercontent.com/projectcalico/calico/{{ k8s_calico_vpp_version }}/manifests/tigera-operator.yaml" +k8s_calico_tigera_operator_checksum: "sha256:606a55409e98ede9aa13864348ab90093539eea61dafca9c92f040980ba07b58" +# custom install resources to configure Calico +k8s_calico_custom_resources: "https://raw.githubusercontent.com/projectcalico/vpp-dataplane/{{ k8s_calico_vpp_version }}/yaml/calico/installation-default.yaml" + +# install the calicoctl command line tool to manage Calico resources and perform administrative functions +k8s_calicoctl: "https://github.com/projectcalico/calico/releases/download/{{ k8s_calico_vpp_version }}/calicoctl-linux-amd64" +k8s_calicoctl_install_dest: "/usr/local/bin/calicoctl" + +# install calivppctl for calico vpp networking diagnose +k8s_calivppctl: "https://raw.githubusercontent.com/projectcalico/vpp-dataplane/{{ k8s_calico_vpp_version }}/test/scripts/vppdev.sh" +k8s_calivppctl_install_dest: "/usr/local/bin/calivppctl" + +k8s_calico_dataplane: "VPP" +k8s_calico_encapsulation: "VXLANCrossSubnet" +k8s_calicovpp_uplink_driver: "dpdk" diff --git a/roles/calico_vpp_install/tasks/calico_vpp_preflight.yml b/roles/calico_vpp_install/tasks/calico_vpp_preflight.yml new file mode 100644 index 00000000..aaa605e1 --- /dev/null +++ b/roles/calico_vpp_install/tasks/calico_vpp_preflight.yml @@ -0,0 +1,50 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +- name: check configuration of Calico VPP Dataplane required components + ansible.builtin.assert: + that: + - kube_network_plugin == "cni" + - calico_network_backend == "vxlan" + - not kube_network_plugin_multus + - hugepages_enabled | default(false) + - number_of_hugepages_1G >= 16 + fail_msg: | + Make sure that following variables are set correctly: + - kube_network_plugin: cni + - calico_network_backend: vxlan + - kube_network_plugin_multus: false + - hugepages_enabled: true + - number_of_hugepages_1G: 16 (at least 16) + success_msg: "Required Calico VPP Dataplane variables are set correctly" + +- ansible.builtin.debug: msg="the given IP in inventory.ini is {{ ip }}" + +- name: parse interfaces from given IP + set_fact: + calico_vpp_interface_name: "{{ item }}" + when: + - hostvars[inventory_hostname]['ansible_' + item].ipv4 is defined + - hostvars[inventory_hostname]['ansible_' + item].ipv4.address == ip + with_items: + - "{{ ansible_interfaces }}" + +- name: check the interface name is found + ansible.builtin.assert: + that: + - calico_vpp_interface_name is defined + fail_msg: "calico_vpp_interface_name is not found, the provided IP in inventory.ini is not correct, please check again" + success_msg: "the NIC interface {{ calico_vpp_interface_name }} is used for calico vpp" diff --git a/roles/calico_vpp_install/tasks/calicoctl.yml b/roles/calico_vpp_install/tasks/calicoctl.yml new file mode 100644 index 00000000..eb237c0a --- /dev/null +++ b/roles/calico_vpp_install/tasks/calicoctl.yml @@ -0,0 +1,25 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +# install calicoctl to manage Calico policies and configuration, as well as view detailed cluster status +- name: Download calicoctl + ansible.builtin.get_url: + url: "{{ k8s_calicoctl }}" + dest: "{{ k8s_calicoctl_install_dest }}" + mode: '0755' + register: calicoctl_download + until: calicoctl_download is not failed + retries: 5 diff --git a/roles/calico_vpp_install/tasks/calivppctl.yml b/roles/calico_vpp_install/tasks/calivppctl.yml new file mode 100644 index 00000000..08b351e8 --- /dev/null +++ b/roles/calico_vpp_install/tasks/calivppctl.yml @@ -0,0 +1,25 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +# install calico vpp network diagnose tools +- name: Download calivppctl + ansible.builtin.get_url: + url: "{{ k8s_calivppctl }}" + dest: "{{ k8s_calivppctl_install_dest }}" + mode: '0755' + register: calivppctl_download + until: calivppctl_download is not failed + retries: 5 diff --git a/roles/calico_vpp_install/tasks/main.yml b/roles/calico_vpp_install/tasks/main.yml new file mode 100644 index 00000000..2681bc1b --- /dev/null +++ b/roles/calico_vpp_install/tasks/main.yml @@ -0,0 +1,139 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +- name: preflight check for Calico VPP + include_tasks: calico_vpp_preflight.yml + +- name: create Calico VPP files directory + ansible.builtin.file: + path: "{{ calico_vpp_files_dir }}" + state: directory + mode: '0755' + +- name: download tigera-operator deployment file + ansible.builtin.get_url: + url: "{{ k8s_calico_tigera_operator }}" + checksum: "{{ k8s_calico_tigera_operator_checksum }}" + dest: "{{ calico_vpp_files_dir }}" + mode: '0755' + register: tigera_operator_download + until: tigera_operator_download is not failed + retries: 5 + +- name: install tigera-operator + kubernetes.core.k8s: + state: present + src: "{{ (calico_vpp_files_dir, 'tigera-operator.yaml') | path_join }}" + +- name: wait for tigera-operator ready + kubernetes.core.k8s_info: + kind: Deployment + name: tigera-operator + namespace: tigera-operator + wait: true + wait_condition: + reason: NewReplicaSetAvailable + type: Progressing + wait_timeout: 240 + +- name: generate calico and calico-vpp deployment file + ansible.builtin.template: + src: "{{ item.src }}" + dest: "{{ (calico_vpp_files_dir, item.dst) | path_join }}" + force: yes + mode: preserve + loop: + - {src: 'calico.yaml.j2', dst: 'calico.yaml'} + - {src: 'calico-vpp.yaml.j2', dst: 'calico-vpp.yaml'} + +# Pause 15 secs for waiting the calico basic part start firslty, and then calico vpp +- name: deploy calico + kubernetes.core.k8s: + state: present + src: "{{ (calico_vpp_files_dir, item) | path_join }}" + loop: + - 'calico.yaml' + - 'calico-vpp.yaml' + loop_control: + pause: 15 + +# API Server will be in shortly disconnected status during calico cni initialization, so wait for a while +- name: wait for calico cni to be fully initialized + pause: + seconds: 180 + +- name: wait for calico-vpp to be ready + kubernetes.core.k8s_info: + kind: DaemonSet + name: calico-vpp-node + namespace: calico-vpp-dataplane + register: ds_status + retries: 30 + delay: 10 + until: | + ds_status.failed or + ( + ds_status.resources | length > 0 and + ds_status.resources[0].status.numberReady > 0 + ) + +- name: wait for calico to be ready + block: + - name: wait all calico deployments to be ready + kubernetes.core.k8s_info: + kind: Deployment + name: "{{ item.name }}" + namespace: "{{ item.namespace }}" + wait: yes + wait_condition: + reason: NewReplicaSetAvailable + type: Progressing + wait_timeout: 240 + loop: + - {namespace: 'calico-system', name: 'calico-typha'} + - {namespace: 'calico-system', name: 'calico-kube-controllers'} + - {namespace: 'calico-apiserver', name: 'calico-apiserver'} + - name: wait all calico daemonsets to be ready + kubernetes.core.k8s_info: + kind: DaemonSet + name: "{{ item }}" + namespace: calico-system + with_items: + - calico-node + - csi-node-driver + register: ds_status + retries: 30 + delay: 10 + until: | + ds_status.failed or + ( + ds_status.resources | length > 0 and + ds_status.resources[0].status.numberReady > 0 + ) + +- name: restart kubelet service + ansible.builtin.systemd: + name: kubelet + state: restarted + enabled: yes + +- name: install calicoctl + ansible.builtin.include_tasks: + file: calicoctl.yml + +- name: install calivppctl + ansible.builtin.include_tasks: + file: calivppctl.yml diff --git a/roles/calico_vpp_install/templates/calico-vpp.yaml.j2 b/roles/calico_vpp_install/templates/calico-vpp.yaml.j2 new file mode 100644 index 00000000..98db4a7a --- /dev/null +++ b/roles/calico_vpp_install/templates/calico-vpp.yaml.j2 @@ -0,0 +1,335 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: calico-vpp-dataplane +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: calico-vpp-node-sa + namespace: calico-vpp-dataplane +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: calico-vpp-node-role +rules: +- apiGroups: + - "" + resources: + - pods + - nodes + - namespaces + verbs: + - get +- apiGroups: + - "" + resources: + - endpoints + - services + verbs: + - watch + - list + - get +- apiGroups: + - "" + resources: + - configmaps + verbs: + - get +- apiGroups: + - "" + resources: + - nodes/status + verbs: + - patch + - update +- apiGroups: + - networking.k8s.io + resources: + - networkpolicies + verbs: + - watch + - list +- apiGroups: + - "" + resources: + - pods + - namespaces + - serviceaccounts + verbs: + - list + - watch +- apiGroups: + - "" + resources: + - pods/status + verbs: + - patch +- apiGroups: + - crd.projectcalico.org + resources: + - globalfelixconfigs + - felixconfigurations + - bgppeers + - globalbgpconfigs + - bgpconfigurations + - ippools + - ipamblocks + - globalnetworkpolicies + - globalnetworksets + - networkpolicies + - networksets + - clusterinformations + - hostendpoints + - blockaffinities + verbs: + - get + - list + - watch +- apiGroups: + - "" + resources: + - nodes + verbs: + - get + - list + - watch +- apiGroups: + - crd.projectcalico.org + resources: + - blockaffinities + - ipamblocks + - ipamhandles + verbs: + - get + - list + - create + - update + - delete +- apiGroups: + - crd.projectcalico.org + resources: + - ipamconfigs + verbs: + - get +- apiGroups: + - crd.projectcalico.org + resources: + - blockaffinities + verbs: + - watch +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: calico-vpp-node +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: calico-vpp-node-role +subjects: +- kind: ServiceAccount + name: calico-vpp-node-sa + namespace: calico-vpp-dataplane +--- +apiVersion: v1 +data: + CALICOVPP_CONFIG_TEMPLATE: |- + unix { + nodaemon + full-coredump + cli-listen /var/run/vpp/cli.sock + pidfile /run/vpp/vpp.pid + exec /etc/vpp/startup.exec + } + api-trace { on } + cpu { + workers 0 + } + socksvr { + socket-name /var/run/vpp/vpp-api.sock + } + plugins { + plugin default { enable } + plugin dpdk_plugin.so { disable } + plugin calico_plugin.so { enable } + plugin ping_plugin.so { disable } + plugin dispatch_trace_plugin.so { enable } + } + buffers { + buffers-per-numa 131072 + } + CALICOVPP_INITIAL_CONFIG: |- + { + "vppStartupSleepSeconds": 1, + "corePattern": "/var/lib/vpp/vppcore.%e.%p" + } + CALICOVPP_INTERFACES: |- + { + "maxPodIfSpec": { + "rx": 10, "tx": 10, "rxqsz": 1024, "txqsz": 1024 + }, + "defaultPodIfSpec": { + "rx": 1, "tx":1, "isl3": true + }, + "vppHostTapSpec": { + "rx": 1, "tx":1, "rxqsz": 1024, "txqsz": 1024, "isl3": false + }, + "uplinkInterfaces": [ + { + "interfaceName": "{{ calico_vpp_interface_name }}", + "vppDriver": "{{ k8s_calicovpp_uplink_driver }}" + } + ] + } + SERVICE_PREFIX: {{ kube_service_addresses }} +kind: ConfigMap +metadata: + name: calico-vpp-config + namespace: calico-vpp-dataplane +--- +apiVersion: apps/v1 +kind: DaemonSet +metadata: + labels: + k8s-app: calico-vpp-node + name: calico-vpp-node + namespace: calico-vpp-dataplane +spec: + selector: + matchLabels: + k8s-app: calico-vpp-node + template: + metadata: + labels: + k8s-app: calico-vpp-node + spec: + containers: + - env: + - name: DATASTORE_TYPE + value: kubernetes + - name: WAIT_FOR_DATASTORE + value: "true" + - name: NODENAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + envFrom: + - configMapRef: + name: calico-vpp-config + image: docker.io/calicovpp/vpp:{{ k8s_calico_vpp_version }} + imagePullPolicy: IfNotPresent + name: vpp + resources: + limits: + hugepages-1Gi: 8Gi + requests: + cpu: 500m + memory: 8Gi + securityContext: + privileged: true + volumeMounts: + - mountPath: /lib/firmware + name: lib-firmware + - mountPath: /var/run/vpp + name: vpp-rundir + - mountPath: /var/lib/vpp + name: vpp-data + - mountPath: /etc/vpp + name: vpp-config + - mountPath: /dev + name: devices + - mountPath: /sys + name: hostsys + - mountPath: /run/netns/ + mountPropagation: Bidirectional + name: netns + - mountPath: /host + name: host-root + - env: + - name: DATASTORE_TYPE + value: kubernetes + - name: WAIT_FOR_DATASTORE + value: "true" + - name: NODENAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + - name: NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + envFrom: + - configMapRef: + name: calico-vpp-config + image: docker.io/calicovpp/agent:{{ k8s_calico_vpp_version }} + imagePullPolicy: IfNotPresent + name: agent + resources: + requests: + cpu: 250m + securityContext: + privileged: true + volumeMounts: + - mountPath: /var/run/calico + name: var-run-calico + readOnly: false + - mountPath: /var/lib/calico/felix-plugins + name: felix-plugins + readOnly: false + - mountPath: /var/run/vpp + name: vpp-rundir + - mountPath: /run/netns/ + mountPropagation: Bidirectional + name: netns + hostNetwork: true + hostPID: true + nodeSelector: + kubernetes.io/os: linux + priorityClassName: system-node-critical + serviceAccountName: calico-vpp-node-sa + terminationGracePeriodSeconds: 10 + tolerations: + - effect: NoSchedule + operator: Exists + - key: CriticalAddonsOnly + operator: Exists + - effect: NoExecute + operator: Exists + volumes: + - hostPath: + path: /lib/firmware + name: lib-firmware + - hostPath: + path: /var/run/vpp + name: vpp-rundir + - hostPath: + path: /var/lib/vpp + type: DirectoryOrCreate + name: vpp-data + - hostPath: + path: /etc/vpp + name: vpp-config + - hostPath: + path: /dev + name: devices + - hostPath: + path: /sys + name: hostsys + - hostPath: + path: /var/run/calico + name: var-run-calico + - hostPath: + path: /run/netns + name: netns + - hostPath: + path: /var/lib/calico/felix-plugins + name: felix-plugins + - hostPath: + path: / + name: host-root + updateStrategy: + rollingUpdate: + maxUnavailable: 1 + type: RollingUpdate diff --git a/roles/calico_vpp_install/templates/calico.yaml.j2 b/roles/calico_vpp_install/templates/calico.yaml.j2 new file mode 100644 index 00000000..d484bebc --- /dev/null +++ b/roles/calico_vpp_install/templates/calico.yaml.j2 @@ -0,0 +1,27 @@ +# This section includes base Calico installation configuration. +# For more information, see: https://projectcalico.docs.tigera.io/master/reference/installation/api#operator.tigera.io/v1.Installation +apiVersion: operator.tigera.io/v1 +kind: Installation +metadata: + name: default +spec: + # Configures Calico networking. + calicoNetwork: + linuxDataplane: {{ k8s_calico_dataplane }} + # Note: The ipPools section cannot be modified post-install. + ipPools: + - blockSize: 26 + cidr: {{ kube_pods_subnet }} + encapsulation: {{ k8s_calico_encapsulation }} + natOutgoing: Enabled + nodeSelector: all() + +--- + +# This section configures the Calico API server. +# For more information, see: https://projectcalico.docs.tigera.io/master/reference/installation/api#operator.tigera.io/v1.APIServer +apiVersion: operator.tigera.io/v1 +kind: APIServer +metadata: + name: default +spec: {} diff --git a/roles/check_machine_type/vars/main.yml b/roles/check_machine_type/vars/main.yml index 347f6549..5bb5013e 100644 --- a/roles/check_machine_type/vars/main.yml +++ b/roles/check_machine_type/vars/main.yml @@ -74,3 +74,5 @@ confirmed_spr_cpus: confirmed_emr_cpus: - "0000" + - "8592" + - "6548N" diff --git a/roles/cluster_defaults/defaults/main.yml b/roles/cluster_defaults/defaults/main.yml index b0f09bee..f96833f1 100644 --- a/roles/cluster_defaults/defaults/main.yml +++ b/roles/cluster_defaults/defaults/main.yml @@ -36,5 +36,5 @@ proxy_env: {} registry_containerd: "/var/lib/kubelet/config.json" kube_rbac_proxy_image_repo: "quay.io/brancz/kube-rbac-proxy" -kube_rbac_proxy_image_tag: "v0.14.1" +kube_rbac_proxy_image_tag: "v0.14.3" kube_rbac_proxy_tls_ciphers: "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305" # noqa yaml[line-length] diff --git a/roles/container_engine/containerd/defaults/main.yml b/roles/container_engine/containerd/defaults/main.yml index 5fc71cb8..c706b14a 100644 --- a/roles/container_engine/containerd/defaults/main.yml +++ b/roles/container_engine/containerd/defaults/main.yml @@ -14,8 +14,8 @@ ## limitations under the License. ## --- -containerd_version: 1.7.0 -containerd_archive_checksum: "b068b05d58025dc9f2fc336674cac0e377a478930f29b48e068f97c783a423f0" +containerd_version: 1.7.3 +containerd_archive_checksum: "de7f61aacba88ee647a7dcde1ca77672ec44ab9fb3e58ae90c0efc9b2d8f3068" containerd_download_url: "https://github.com/containerd/containerd/releases/download/v{{ containerd_version }}/containerd-{{ containerd_version }}-linux-amd64.tar.gz" # noqa yaml[line-length] containerd_bin_dir: "/usr/local/bin" @@ -30,7 +30,7 @@ containerd_archive: owner: "root" mode: "0755" -containerd_sandbox_image: "k8s.gcr.io/pause:3.6" +containerd_sandbox_image: "k8s.gcr.io/pause:3.9" containerd_storage_dir: "/var/lib/containerd" containerd_state_dir: "/run/containerd" diff --git a/roles/container_engine/crictl/defaults/main.yml b/roles/container_engine/crictl/defaults/main.yml index 1c17879e..53f16a25 100644 --- a/roles/container_engine/crictl/defaults/main.yml +++ b/roles/container_engine/crictl/defaults/main.yml @@ -13,8 +13,8 @@ ## See the License for the specific language governing permissions and ## limitations under the License. ## -crictl_version: "v1.26.0" -crictl_binary_checksum: "cda5e2143bf19f6b548110ffba0fe3565e03e8743fadd625fee3d62fc4134eed" +crictl_version: "v1.27.0" +crictl_binary_checksum: "d335d6e16c309fbc3ff1a29a7e49bb253b5c9b4b030990bf7c6b48687f985cee" crictl_repo_url: "https://github.com/kubernetes-sigs/cri-tools/releases/download/" crictl_download_url: "{{ crictl_repo_url }}{{ crictl_version }}/crictl-{{ crictl_version }}-{{ ansible_system | lower }}-amd64.tar.gz" diff --git a/roles/container_engine/crio/defaults/main.yml b/roles/container_engine/crio/defaults/main.yml index f71dedc6..7361d665 100644 --- a/roles/container_engine/crio/defaults/main.yml +++ b/roles/container_engine/crio/defaults/main.yml @@ -22,11 +22,11 @@ crio_conmon: "{{ ( bin_dir, 'conmon') | path_join }}" crio_enable_metrics: false crio_log_level: "info" crio_metrics_port: "9090" -crio_pause_image: "k8s.gcr.io/pause:3.3" +crio_pause_image: "k8s.gcr.io/pause:3.9" -crio_version: "v1.26.3" +crio_version: "v1.27.0" crio_download_url: "https://storage.googleapis.com/cri-o/artifacts/cri-o.amd64.{{ crio_version }}.tar.gz" -crio_archive_checksums: "942772081d9cd4bd0c07e466439b76a1ca95d3f10a7b53dc524d2946b2b17a71" +crio_archive_checksums: "8f99db9aeea00299cb3f28ee61646472014cac91930e4c7551c9153f8f720093" crio: version: "{{ crio_version }}" @@ -82,10 +82,6 @@ crio_runtimes: type: oci root: /run/runc -# Configure the cri-o pids limit, increase this for heavily multi-threaded workloads -# see https://github.com/cri-o/cri-o/issues/1921 -crio_pids_limit: 1024 - # Reserve 16M uids and gids for user namespaces (256 pods * 65536 uids/gids) # at the end of the uid/gid space crio_remap_enable: false diff --git a/roles/container_engine/crio/templates/crio.conf.j2 b/roles/container_engine/crio/templates/crio.conf.j2 index fd29b3fa..d513fcf5 100644 --- a/roles/container_engine/crio/templates/crio.conf.j2 +++ b/roles/container_engine/crio/templates/crio.conf.j2 @@ -201,9 +201,6 @@ default_mounts = [ # #default_mounts_file = "" -# Maximum number of processes allowed in a container. -pids_limit = {{ crio_pids_limit }} - # Maximum sized allowed for the container log file. Negative numbers indicate # that no size limit is imposed. If it is positive, it must be >= 8192 to # match/exceed conmon's read buffer. The file is truncated and re-opened so the diff --git a/roles/container_engine/docker/defaults/main.yml b/roles/container_engine/docker/defaults/main.yml index db1891f0..b14f7efd 100644 --- a/roles/container_engine/docker/defaults/main.yml +++ b/roles/container_engine/docker/defaults/main.yml @@ -16,7 +16,7 @@ --- docker_version: "20.10.20" docker_cli_version: "{{ docker_version }}" -containerd_version: "1.6.4" # Containerd version installed when docker runtime is used +containerd_version: "1.6.16" # Containerd version installed when docker runtime is used containerd_package: 'containerd.io' diff --git a/roles/container_engine/runc/defaults/main.yml b/roles/container_engine/runc/defaults/main.yml index 6acf0b7b..c6ef9beb 100644 --- a/roles/container_engine/runc/defaults/main.yml +++ b/roles/container_engine/runc/defaults/main.yml @@ -16,10 +16,10 @@ --- runc_bin_dir: "/usr/local/bin" -runc_version: v1.1.4 +runc_version: v1.1.8 runc_download_url: "https://github.com/opencontainers/runc/releases/download/{{ runc_version }}/runc.amd64" -runc_binary_checksum: "db772be63147a4e747b4fe286c7c16a2edc4a8458bd3092ea46aaee77750e8ce" +runc_binary_checksum: "1d05ed79854efc707841dfc7afbf3b86546fc1d0b3a204435ca921c14af8385b" runc_binary: dest: "{{ (runc_bin_dir, 'runc') | path_join }}" diff --git a/roles/container_registry/defaults/main.yml b/roles/container_registry/defaults/main.yml index 92fbe159..e848f8c2 100644 --- a/roles/container_registry/defaults/main.yml +++ b/roles/container_registry/defaults/main.yml @@ -23,9 +23,9 @@ registry_password: registry_size: 10Gi registry_image: "docker.io/library/registry" -registry_version: 2.8.1 +registry_version: 2.8.2 registry_nginx_image: "docker.io/library/nginx" -registry_nginx_version: 1.24.0 +registry_nginx_version: 1.25.2 docker_pip_pkg_version: 6.0.0 registry_tls_secret_name: container-registry-tls diff --git a/roles/elasticsearch_install/tasks/main.yml b/roles/elasticsearch_install/tasks/main.yml index 472b186f..1bd42470 100644 --- a/roles/elasticsearch_install/tasks/main.yml +++ b/roles/elasticsearch_install/tasks/main.yml @@ -90,6 +90,18 @@ - elasticsearch_storageclass.yml - elasticsearch_pv.yml + - name: Wait till the elasticsearch certificate is created + kubernetes.core.k8s_info: + kind: Certificate + name: elasticsearch-tls + namespace: monitoring + wait: yes + wait_condition: + type: Ready + status: "True" + wait_sleep: 10 + wait_timeout: 360 + - name: add elasticsearch chart repo kubernetes.core.helm_repository: name: "{{ elasticsearch_chart_name }}" @@ -104,3 +116,8 @@ values_files: "{{ (project_root_dir, 'elasticsearch', 'elasticsearch_values.yml') | path_join }}" wait: true timeout: 15m0s + + - name: create elasticsearch settings pod + kubernetes.core.k8s: + state: present + src: "{{ (project_root_dir, 'elasticsearch', 'elasticsearch_settings.yml') | path_join }}" diff --git a/roles/elasticsearch_install/templates/elasticsearch_settings.yml.j2 b/roles/elasticsearch_install/templates/elasticsearch_settings.yml.j2 new file mode 100644 index 00000000..a8b4b0fc --- /dev/null +++ b/roles/elasticsearch_install/templates/elasticsearch_settings.yml.j2 @@ -0,0 +1,86 @@ +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: elasticsearch-stack-settings-script + namespace: monitoring +data: + elastic-stack-settings.sh: | + #!/bin/sh + + apk add --no-cache curl jq + + sleep 300 + + UNAVAILABLE=true + + while $UNAVAILABLE; do + status=$(curl -s -k -XGET \ + --user "${ELASTIC_USERNAME}:${ELASTIC_PASS}" \ + https://elasticsearch-master.monitoring.svc.cluster.local:9200/_cluster/health | jq '.status' | tr -d '"') + if [[ "$status" == "green" || "$status" == "yellow" ]]; then + UNAVAILABLE=false + else + echo "Elasticsearch stack is not ready. Trying again after 30s..." + sleep 30 + fi + done + + result=$(curl -s -k -XPUT \ + --user "${ELASTIC_USERNAME}:${ELASTIC_PASS}" \ + -H "Content-Type: application/json" \ + https://elasticsearch-master.monitoring.svc.cluster.local:9200/_all/_settings?preserve_existing=true \ + -d '{"index.auto_expand_replicas": "0-all", "index.number_of_replicas": "1"}' | jq '.acknowledged') + + if [ "$result" == true ]; then + exit 0 + else + echo "Elasticstack replicas settings was not set properly." + exit 1 + fi +--- +apiVersion: v1 +kind: Pod +metadata: + name: elasticsearch-stack-settings + namespace: monitoring +spec: + restartPolicy: OnFailure + containers: + - name: elasticsearch-stack-settings + image: alpine:3.18 + command: ["/root/elastic-stack-settings.sh"] + env: +{% if "http_proxy" in proxy_env %} + - name: http_proxy + value: {{ proxy_env.http_proxy }} +{% endif %} +{% if "https_proxy" in proxy_env %} + - name: https_proxy + value: {{ proxy_env.https_proxy }} +{% endif %} +{% if "no_proxy" in proxy_env %} + - name: no_proxy + value: {{ proxy_env.no_proxy }} +{% endif %} + - name: ELASTIC_PASS + valueFrom: + secretKeyRef: + name: elasticsearch-master-credentials + key: password + - name: ELASTIC_USERNAME + valueFrom: + secretKeyRef: + name: elasticsearch-master-credentials + key: username + volumeMounts: + - name: elasticsearch-stack-settings-script + mountPath: "/root/" + volumes: + - name: elasticsearch-stack-settings-script + configMap: + name: elasticsearch-stack-settings-script + defaultMode: 0500 + items: + - key: elastic-stack-settings.sh + path: elastic-stack-settings.sh diff --git a/roles/ffmpeg_install/defaults/main.yml b/roles/ffmpeg_install/defaults/main.yml index 938fff2c..b98ec6c7 100644 --- a/roles/ffmpeg_install/defaults/main.yml +++ b/roles/ffmpeg_install/defaults/main.yml @@ -17,8 +17,8 @@ ffmpeg_path: "{{ (project_root_dir, 'ffmpeg') | path_join }}" ffmpeg_patch_path: "{{ (ffmpeg_path, 'ffmpeg_patch') | path_join }}" -# Define the FFmpeg version using the release tag or commit hash. If both are used, the commit hash is used. -# ffmpeg_version: "n4.2.9" -ffmpeg_commit_hash: "c3a7999" +# ffmpeg upstream base version +ffmpeg_commit_hash: "9b6d191" ffmpeg_git_url: "https://github.com/FFmpeg/FFmpeg.git" -ffmpeg_configure_options: "--enable-shared --enable-vaapi --enable-libvpl" +ffmpeg_configure_options_gpu: "--enable-shared --enable-vaapi --enable-libvpl" +ffmpeg_configure_options_cpu: "--enable-shared" diff --git a/roles/ffmpeg_install/tasks/ffmpeg_archive_patch.yml b/roles/ffmpeg_install/tasks/ffmpeg_archive_patch.yml index 3f3a8842..40240846 100644 --- a/roles/ffmpeg_install/tasks/ffmpeg_archive_patch.yml +++ b/roles/ffmpeg_install/tasks/ffmpeg_archive_patch.yml @@ -23,7 +23,7 @@ - name: download patch archive ansible.builtin.get_url: dest: '{{ ffmpeg_path }}' - sha256sum: '{{ patch_item.sha256 }}' + checksum: 'sha256:{{ patch_item.sha256 }}' url: "{{ patch_item.url }}" mode: '0640' register: patch_download diff --git a/roles/ffmpeg_install/tasks/ffmpeg_install.yml b/roles/ffmpeg_install/tasks/ffmpeg_install.yml index ed4974fa..4de06cfd 100644 --- a/roles/ffmpeg_install/tasks/ffmpeg_install.yml +++ b/roles/ffmpeg_install/tasks/ffmpeg_install.yml @@ -45,13 +45,25 @@ loop_control: loop_var: patch_item +- name: set ffmpeg configure option for gpu acceleration + ansible.builtin.set_fact: + ffmpeg_configure_options: "{{ ffmpeg_configure_options_gpu }}" + when: + - configure_gpu | default(false) | bool + +- name: set ffmpeg configure option for cpu only + ansible.builtin.set_fact: + ffmpeg_configure_options: "{{ ffmpeg_configure_options_cpu }}" + when: + - not configure_gpu | default(false) | bool + - name: configure FFmpeg source ansible.builtin.command: "{{ item }}" changed_when: false args: chdir: "{{ (ffmpeg_path, 'ffmpeg_src') | path_join }}" with_items: - - ./configure --prefix={{ ffmpeg_path }} --libdir=/usr/lib {{ ffmpeg_configure_options }} + - ./configure --prefix={{ ffmpeg_path }} {{ ffmpeg_configure_options }} - name: get number of CPUs ansible.builtin.command: nproc diff --git a/roles/imtl_install/defaults/main.yml b/roles/imtl_install/defaults/main.yml new file mode 100644 index 00000000..0767be20 --- /dev/null +++ b/roles/imtl_install/defaults/main.yml @@ -0,0 +1,31 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +imtl_git_url: "https://github.com/OpenVisualCloud/Media-Transport-Library" +imtl_version: "v23.08" + +# IMTL source has patches only for major.minor DPDK versions, strip patch version if present +imtl_stripped_dpdk_version: "{{ dpdk_version | regex_replace('^([0-9]+).([0-9]+).*$', '\\1.\\2') }}" + +imtl_root_dir: "{{ (project_root_dir, 'imtl') | path_join }}" +imtl_dpdk_patches_base_dir: "{{ imtl_root_dir }}/patches/dpdk/" +imtl_ice_patches_base_dir: "{{ imtl_root_dir }}/patches/ice_drv/" +imtl_deps_dir: "{{ (project_root_dir, 'imtl_deps') | path_join }}" + +# Minimum NIC FW version supported +imtl_min_fw_version_supported: "4.20" + +imtl_dpdk_patches_strip: 1 +imtl_ice_patches_strip: 1 diff --git a/roles/imtl_install/tasks/download.yml b/roles/imtl_install/tasks/download.yml new file mode 100644 index 00000000..2a38e4a5 --- /dev/null +++ b/roles/imtl_install/tasks/download.yml @@ -0,0 +1,21 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +- name: Clone Intel Media Transport Library repository + ansible.builtin.git: + repo: "{{ imtl_git_url }}" + dest: "{{ imtl_root_dir }}" + version: "{{ imtl_version }}" + force: true diff --git a/roles/imtl_install/tasks/dpdk_patch.yml b/roles/imtl_install/tasks/dpdk_patch.yml new file mode 100644 index 00000000..2022a25a --- /dev/null +++ b/roles/imtl_install/tasks/dpdk_patch.yml @@ -0,0 +1,127 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +- name: Fetch IMTL git repository + ansible.builtin.include_tasks: download.yml + +- name: Check patches dir exist for defined DPDK version + ansible.builtin.stat: + path: "{{ (imtl_dpdk_patches_base_dir, imtl_stripped_dpdk_version) | path_join }}" + get_checksum: false + get_mime: false + get_attributes: false + register: patch_dir + +- name: Select patch files if no patches available for defined version + when: not patch_dir.stat.exists + block: + - name: Get all versions of DPDK patches + ansible.builtin.find: + paths: "{{ imtl_dpdk_patches_base_dir }}" + file_type: directory + register: dpdk_patches_dirs + + - name: Set newest available version of patches + ansible.builtin.set_fact: + newest_patch_version: "{{ (dpdk_patches_dirs.files | map(attribute='path') | map('basename') | list | community.general.version_sort)[-1] }}" + + - name: Warn about using patches for different DPDK version + ansible.builtin.debug: + msg: | + Intel Media Library {{ imtl_version }} does not provide patches for DPDK v{{ dpdk_version }}. + Patches for highest available version of DPDK (v{{ newest_patch_version }}) used instead! + +- name: Set version of DPDK patches to use + ansible.builtin.set_fact: + imtl_dpdk_patches_dir: >- + {% if patch_dir.stat.exists -%} + {{ (imtl_dpdk_patches_base_dir, imtl_stripped_dpdk_version) | path_join }} + {%- else -%} + {{ (imtl_dpdk_patches_base_dir, newest_patch_version) | path_join }} + {%- endif %} + +- name: Handle DPDK patch symlink files + block: + - name: Identify links in DPDK patches + ansible.builtin.find: + paths: "{{ imtl_dpdk_patches_dir }}" + recurse: true + file_type: link + register: patch_link_list + + - name: Get link targets for each patch link + stat: + path: "{{ item }}" + loop: "{{ (patch_link_list.files | map(attribute='path') | list | sort) }}" + register: patch_links + + - name: replace links + copy: + remote_src: true + src: "{{ item.stat.lnk_source }}" + dest: "{{ item.stat.path }}" + mode: '0644' + loop: "{{ patch_links.results }}" + +- name: Handle patch files containing only link to other file + block: + - name: Find patch files that contain only link to another patch file + ansible.builtin.find: + paths: "{{ imtl_dpdk_patches_dir }}" + recurse: true + contains: '^\.\.\/\S*$' + read_whole_file: true + register: link_file_list + + - name: Find original patch files from links + ansible.builtin.slurp: + src: "{{ item.path }}" + register: link_points_list + loop: "{{ link_file_list.files }}" + when: link_file_list.matched > 0 + + - name: Replace identified link files with real patch files + ansible.builtin.copy: + remote_src: true + src: "{{ (item.source | dirname, item.content | b64decode) | path_join | realpath }}" + dest: "{{ item.source }}" + mode: '0644' + force: true + loop: "{{ link_points_list.results }}" + when: link_file_list.matched > 0 + +- name: Identify DPDK patches to be used + ansible.builtin.find: + paths: "{{ imtl_dpdk_patches_dir }}" + recurse: true + register: patch_file_list + +- block: + - name: Apply DPDK patches + vars: + # Sort patch files + patch_files: "{{ (patch_file_list.files | map(attribute='path') | list | sort) }}" + ansible.posix.patch: + remote_src: true + src: "{{ item }}" + basedir: "{{ dpdk_dir }}" + strip: "{{ imtl_dpdk_patches_strip }}" + state: present + loop: "{{ patch_files }}" + when: patch_file_list.matched > 0 + rescue: + - name: Report unsupported DPDK version + ansible.builtin.fail: + msg: "Patch of DPDK with files provided by iMTL failed. It is likely iMTL {{ imtl_version }} does not support DPDK v{{ dpdk_version }}." diff --git a/roles/imtl_install/tasks/ice_drv_patch.yml b/roles/imtl_install/tasks/ice_drv_patch.yml new file mode 100644 index 00000000..6d032d89 --- /dev/null +++ b/roles/imtl_install/tasks/ice_drv_patch.yml @@ -0,0 +1,80 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +- name: Fetch IMTL git repository + ansible.builtin.include_tasks: download.yml + +- name: Check patches dir exist for defined ice driver version + ansible.builtin.stat: + path: "{{ (imtl_ice_patches_base_dir, imtl_ice_version) | path_join }}" + get_checksum: false + get_mime: false + get_attributes: false + register: patch_dir + +- name: Select patch files if no patches available for defined version + when: not patch_dir.stat.exists + block: + - name: Get all versions of ICE patches + ansible.builtin.find: + paths: "{{ imtl_ice_patches_base_dir }}" + file_type: directory + register: ice_patches_dirs + + - name: Set newest available version of patches + ansible.builtin.set_fact: + newest_patch_version: "{{ (ice_patches_dirs.files | map(attribute='path') | map('basename') | list | community.general.version_sort)[-1] }}" + + - name: Warn about using patches for different ICE driver version + ansible.builtin.debug: + msg: | + Intel Media Library {{ imtl_version }} does not provide patches for ICE driver {{ imtl_ice_version }}. + Patches for highest available version of ICE ({{ newest_patch_version }}) used instead! + +- name: Set version of ICE patches to use + ansible.builtin.set_fact: + imtl_ice_patches_dir: >- + {% if patch_dir.stat.exists -%} + {{ (imtl_ice_patches_base_dir, imtl_ice_version) | path_join }} + {%- else -%} + {{ (imtl_ice_patches_base_dir, newest_patch_version) | path_join }} + {%- endif %} + +- name: Identify ice driver patches to be used + ansible.builtin.find: + paths: "{{ imtl_ice_patches_dir }}" + recurse: yes + # ignore patch of version + # ignore patch for RHEL9 & Kernel 5.14.0 + excludes: "*version-update-to-kahawai*,*RHEL9-fix-build-with-5.14.0*" + register: patch_file_list + +- block: + - name: Apply patches to ice driver + vars: + # Sort patch files + patch_files: "{{ (patch_file_list.files | map(attribute='path') | list | sort) }}" + ansible.posix.patch: + remote_src: true + src: "{{ item }}" + basedir: "{{ imtl_ice_driver_dir }}" + strip: "{{ imtl_ice_patches_strip }}" + state: present + loop: "{{ patch_files }}" + when: patch_file_list.matched > 0 + rescue: + - name: Report unsupported ICE version + ansible.builtin.fail: + msg: "Patch of ICE driver with files provided by iMTL failed. It is likely iMTL {{ imtl_version }} does not support ICE {{ imtl_ice_version }}." diff --git a/roles/intel_power_manager/tasks/enable_drivers.yml b/roles/imtl_install/tasks/main.yml similarity index 51% rename from roles/intel_power_manager/tasks/enable_drivers.yml rename to roles/imtl_install/tasks/main.yml index 2a486187..d96b5088 100644 --- a/roles/intel_power_manager/tasks/enable_drivers.yml +++ b/roles/imtl_install/tasks/main.yml @@ -13,17 +13,23 @@ ## See the License for the specific language governing permissions and ## limitations under the License. ## -- name: Uncore Frequency driver - when: hostvars[power_node]['uncore_frequency']['enabled'] | default(false) | bool - block: - # Ubuntu only - - name: obtain mandatory package for Uncore Frequency driver - ansible.builtin.apt: - name: "linux-generic-hwe-{{ ansible_distribution_version.split('.')[:2]|join('.') }}" - state: present - when: ansible_distribution == "Ubuntu" - - name: enable Uncore Frequency driver - community.general.modprobe: - name: intel_uncore_frequency - state: present +# IMTL Git Repo was already fetched during DPDK and ICE driver patching + +- name: Install dependencies + ansible.builtin.include_role: + name: install_dependencies + +- name: Install pip dependencies + ansible.builtin.pip: + name: "{{ install_pip_dependencies }}" + +- name: (RedHat) Build & Install additional dependencies + ansible.builtin.include_tasks: redhat_deps.yml + when: ansible_os_family == 'RedHat' + +- name: Build Intel Media Transport library + ansible.builtin.command: + chdir: "{{ imtl_root_dir }}" + cmd: "./build.sh" + changed_when: true # No reliable stdout/err to use diff --git a/roles/imtl_install/tasks/preflight.yml b/roles/imtl_install/tasks/preflight.yml new file mode 100644 index 00000000..bf832d1b --- /dev/null +++ b/roles/imtl_install/tasks/preflight.yml @@ -0,0 +1,78 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +- name: Check network card is supported + ansible.builtin.assert: + that: configured_nic == 'cvl' + fail_msg: + Intel Media Transport Library is supported only for Intel® E810 Series Network Cards. + +- name: Check that at least 1 NIC interface defined + ansible.builtin.assert: + that: dataplane_interfaces | default([]) | length != 0 + fail_msg: | + At least 1 nic interface should be defined in host_vars via dataplane_interfaces parameter. + +- name: Update of NIC drivers enabled when patch_drivers set to true + ansible.builtin.assert: + that: update_nic_drivers | default(false) + fail_msg: | + Automaed patch of ICE driver is done during ICE driver update + Please set update_nic_drivers option to 'true'. + when: intel_media_transport_library.patch_nic_driver | default(false) + +- name: Check iommu enabled + ansible.builtin.assert: + that: + - iommu_enabled | default(false) + fail_msg: | + Intel Media Trasport library needs iommu enabled. Please set iommu_enabled in host_vars to 'true'. + +- name: Check DPDK installation enabled + ansible.builtin.assert: + that: install_dpdk | default(false) + fail_msg: | + Build of Intel Media Transport library needs DPDK installed on the system. + Please set install_dpdk option to 'true'. + +# SR-IOV setup gets done everytime when DPDK & iommu enabled - no check needed + +- name: Check DPDK version support + ansible.builtin.assert: + that: + - dpdk_version | default('1.0') is ansible.builtin.version('23.07', '<') + - dpdk_version | default('1.0') is ansible.builtin.version('21.05', '>=') + fail_msg: | + Intel Media Transport Library {{ imtl_version }} does support only dpdk versions >= 21.05 and < 23.07. + +- name: Check target NIC FW version to be used for update + ansible.builtin.assert: + that: nvmupdate.ice.required_fw_version is version(imtl_min_fw_version_supported, '>=') + fail_msg: | + Use of Intel Media Transport library needs NIC FW version at least '{{ imtl_min_fw_version_supported }}'. + Please set version {{ imtl_min_fw_version_supported }} or higher to option nvmupdate.ice.required_fw_version for node {{ inventory_hostname }}. + when: update_nic_firmware | default(false) + +- name: Check NIC FW version + check_nic_firmware: + pci_id: "{{ item.bus_info }}" + min_fw_version: "{{ imtl_min_fw_version_supported }}" + fail_msg: | + Use of Intel Media Transport library needs NIC FW version at least '{{ imtl_min_fw_version_supported }}'. + Update of NIC firmware can be done via update_nic_firmware option set to 'true'. + loop: "{{ dataplane_interfaces }}" + when: + - not update_nic_firmware | default(false) + - dataplane_interfaces | default([]) | length != 0 diff --git a/roles/imtl_install/tasks/redhat_deps.yml b/roles/imtl_install/tasks/redhat_deps.yml new file mode 100644 index 00000000..65728395 --- /dev/null +++ b/roles/imtl_install/tasks/redhat_deps.yml @@ -0,0 +1,95 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +- name: Download dependencies from git + ansible.builtin.git: + repo: "{{ item.url }}" + dest: "{{ (imtl_deps_dir, item.name) | path_join }}" + version: "{{ item.version }}" + force: true + loop: + - name: jsonc + url: https://github.com/json-c/json-c.git + version: json-c-0.16 + - name: libpcap + url: https://github.com/the-tcpdump-group/libpcap.git + version: libpcap-1.9 + - name: gtest + url: https://github.com/google/googletest.git + version: v1.13.x + +- name: Build jsonc + vars: + build_dir: "{{ (imtl_deps_dir, 'jsonc/build') | path_join }}" + block: + - name: (jsonc) Create build dir + ansible.builtin.file: + path: "{{ build_dir }}" + state: directory + mode: 0750 + - name: (jsonc) Run cmake + ansible.builtin.command: + cmd: cmake ../ + chdir: "{{ build_dir }}" + changed_when: true # TDOD + - name: (jsonc) Run make + community.general.make: + chdir: "{{ build_dir }}" + - name: (jsonc) Run make install + become: true + community.general.make: + target: install + chdir: "{{ build_dir }}" + +- name: Build libpcap + vars: + build_dir: "{{ (imtl_deps_dir, 'libpcap') | path_join }}" + block: + - name: (libpcap) Run configure + ansible.builtin.command: + cmd: ./configure + chdir: "{{ build_dir }}" + changed_when: true # TDOD + - name: (libpcap) Run make + community.general.make: + chdir: "{{ build_dir }}" + - name: (libpcap) Run make install + become: true + community.general.make: + target: install + chdir: "{{ build_dir }}" + +- name: Build gtest + vars: + build_dir: "{{ (imtl_deps_dir, 'gtest/build') | path_join }}" + block: + - name: (gtest) Create build dir + ansible.builtin.file: + path: "{{ build_dir }}" + state: directory + mode: 0750 + - name: (gtest) Run cmake + ansible.builtin.command: + cmd: cmake ../ + chdir: "{{ build_dir }}" + changed_when: true # TDOD + - name: (gtest) Run make + community.general.make: + chdir: "{{ build_dir }}" + - name: (gtest) Run make install + become: true + community.general.make: + target: install + chdir: "{{ build_dir }}" diff --git a/roles/imtl_install/vars/main.yml b/roles/imtl_install/vars/main.yml new file mode 100644 index 00000000..cdc2144a --- /dev/null +++ b/roles/imtl_install/vars/main.yml @@ -0,0 +1,45 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +install_dependencies: + Debian: + - git + - gcc + - pkg-config + - libnuma-dev + - libjson-c-dev + - libpcap-dev + - libgtest-dev + - libssl-dev + - libsdl2-dev + - libsdl2-ttf-dev + RedHat: + - git + - gcc + - gcc-c++ + - pkg-config + - openssl-devel + - numactl-devel + - libasan + - SDL2-devel + # Following ones needed to build other dependencies from source + - cmake + - flex + - bison + +install_pip_dependencies: + - pyelftools + - ninja + - meson diff --git a/roles/install_ddp_pkgs/defaults/main.yml b/roles/install_ddp_pkgs/defaults/main.yml index e7d3eb5e..ee844dcf 100644 --- a/roles/install_ddp_pkgs/defaults/main.yml +++ b/roles/install_ddp_pkgs/defaults/main.yml @@ -13,27 +13,68 @@ ## See the License for the specific language governing permissions and ## limitations under the License. ## ---- +ddp_download_basedir: "{{ (project_root_dir, 'ddp_packages') | path_join }}" + ddp_pkgs: - - nic_driver: i40e - install_dir: /lib/firmware/intel/i40e/ddp - pkgurls: - - "https://downloadmirror.intel.com/27587/eng/gtp.zip" - - "https://downloadmirror.intel.com/28940/eng/mplsogreudp.zip" - - "https://downloadmirror.intel.com/28040/eng/ppp-oe-ol2tpv2.zip" - - "https://downloadmirror.intel.com/29446/eng/esp-ah.zip" - - "https://downloadmirror.intel.com/29780/eng/ecpri.zip" - - nic_driver: ice - install_dir: /lib/firmware/updates/intel/ice/ddp/ - pkgurls: - - "https://downloadmirror.intel.com/29892/eng/ice_comms-1.3.17.0.zip" - - "https://downloadmirror.intel.com/30028/eng/ice_comms-1.3.20.0.zip" - - "https://downloadmirror.intel.com/30237/eng/ice_comms-1.3.22.0.zip" - - "https://downloadmirror.intel.com/30335/eng/ice_comms-1.3.24.0.zip" - - "https://downloadmirror.intel.com/30467/eng/800%20series%20comms%20binary%20package%201.3.28.0.zip" - - "https://downloadmirror.intel.com/30590/eng/800%20series%20comms%20binary%20package%201.3.30.0.zip" - - "https://downloadmirror.intel.com/29889/eng/800%20series%20comms%20binary%20package%201.3.30.0_rev1.1.zip" - - "https://downloadmirror.intel.com/713853/800%20Series%20DDP%20Comms%20Package%201.3.31.0.zip" - - "https://downloadmirror.intel.com/727568/ice_comms-1.3.35.0.zip" - - "https://downloadmirror.intel.com/738733/800%20Series%20DDP%20Comms%20Package%201.3.37.0.zip" - - "https://downloadmirror.intel.com/772040/800%20Series%20DDP%20for%20Comms%20Package%201.3.40.0.zip" + i40e: + download_dir: "{{ (ddp_download_basedir, 'i40e') | path_join }}" + install_dir: "/lib/firmware/intel/i40e/ddp" + pkgs: + - name: gtp.zip + url: "https://downloadmirror.intel.com/27587/eng/gtp.zip" + checksum: cd179e95a988271bc8709687172875d40a9f4fea + - name: mplsogreudp.zip + url: "https://downloadmirror.intel.com/28940/eng/mplsogreudp.zip" + checksum: 11657a63fb5b6621ecbc4750c4559234cea497d9 + - name: ppp-oe-ol2tpv2.zip + url: "https://downloadmirror.intel.com/28040/eng/ppp-oe-ol2tpv2.zip" + checksum: 271f922ef7d6f5309c7cc0cacc5ed0375e058586 + - name: esp-ah.zip + url: "https://downloadmirror.intel.com/29446/eng/esp-ah.zip" + checksum: ca0c023afcd63dab52fb56e6bbefa557f540aa9a + - name: ecpri.zip + url: "https://downloadmirror.intel.com/29780/eng/ecpri.zip" + checksum: ed70dedc0178b67109e714cbc4b6dcff86aa800c + ice: + download_dir: "{{ (ddp_download_basedir, 'ice') | path_join }}" + install_dir: "/lib/firmware/updates/intel/ice/ddp/" + pkgs: + - name: ice_comms-1.3.17.0.zip + url: "https://downloadmirror.intel.com/29892/eng/ice_comms-1.3.17.0.zip" + checksum: a10d5df30d34559b6050c807e0baf44364b35130 + - name: ice_comms-1.3.20.0.zip + url: "https://downloadmirror.intel.com/30028/eng/ice_comms-1.3.20.0.zip" + checksum: 5208c100fee36ba47c90dec955d43e24c6039614 + - name: ice_comms-1.3.22.0.zip + url: "https://downloadmirror.intel.com/30237/eng/ice_comms-1.3.22.0.zip" + checksum: eb5ffa506a77d15facdcc8226b2f3b0a410d6458 + - name: ice_comms-1.3.24.0.zip + url: "https://downloadmirror.intel.com/30335/eng/ice_comms-1.3.24.0.zip" + checksum: 0717f03ab72c1531320ee84be78cdcb194d789ba + - name: ice_comms-1.3.28.0.zip + url: "https://downloadmirror.intel.com/30467/eng/800%20series%20comms%20binary%20package%201.3.28.0.zip" + checksum: 14d82b50949e942ab308f55e44d82baaee5978fe + archive: true + - name: ice_comms-1.3.30.0.zip + url: "https://downloadmirror.intel.com/29889/eng/800%20series%20comms%20binary%20package%201.3.30.0_rev1.1.zip" + checksum: 89eba571845bd80bfc11869cbcfce995edb9a2f9 + archive: true + - name: ice_comms-1.3.31.0.zip + url: "https://downloadmirror.intel.com/713853/800%20Series%20DDP%20Comms%20Package%201.3.31.0.zip" + checksum: eef7c13fd48d1ed8836751c155224ceb0a5cfd29 + archive: true + - name: ice_comms-1.3.35.0.zip + url: "https://downloadmirror.intel.com/727568/ice_comms-1.3.35.0.zip" + checksum: c61189b98bb116e05853f67ba21ca915416aef46 + - name: ice_comms-1.3.37.0.zip + url: "https://downloadmirror.intel.com/738733/800%20Series%20DDP%20Comms%20Package%201.3.37.0.zip" + checksum: 9da13684e8564d92f4364f4fb30f805e0d7c6558 + archive: true + - name: ice_comms-1.3.40.0.zip + url: "https://downloadmirror.intel.com/772040/800%20Series%20DDP%20for%20Comms%20Package%201.3.40.0.zip" + checksum: bfbefba6e444f2850d0d9015cebc207d44cf6327 + archive: true + - name: ice_comms-1.3.45.0.zip + url: "https://downloadmirror.intel.com/785846/738693_ice_comms-1.3.45.0.zip" + checksum: 322d187c08a8903a89dc4d885a9b4b05b8974c6f + archive: true diff --git a/roles/install_ddp_pkgs/tasks/download.yml b/roles/install_ddp_pkgs/tasks/download.yml new file mode 100644 index 00000000..ebd91437 --- /dev/null +++ b/roles/install_ddp_pkgs/tasks/download.yml @@ -0,0 +1,54 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +- name: Define DDP package directory + set_fact: + ddp_download_dir: "{{ ddp_download_dir_override | default(ddp_pkgs[nic_driver].download_dir) }}" + +- name: Create DDP package directory + ansible.builtin.file: + path: "{{ ddp_download_dir }}" + state: directory + mode: '0700' + +- name: Download DDP packages + ansible.builtin.get_url: + checksum: "sha1:{{ item.checksum }}" + url: "{{ item.url }}" + dest: "{{ ddp_download_dir }}" + mode: '0644' + loop: "{{ ddp_pkgs[nic_driver].pkgs }}" + register: fetched_pkgs + +- name: Unarchive fetched file when needed + ansible.builtin.unarchive: + src: "{{ item.dest }}" + dest: "{{ ddp_download_dir }}" + remote_src: true + loop: "{{ fetched_pkgs.results }}" + when: + - item.item.archive | default(false) + +- name: Find temporary files to remove + ansible.builtin.find: + paths: "{{ ddp_download_dir }}" + excludes: "{{ ddp_pkgs[nic_driver].pkgs | map(attribute='name') | list }}" + register: ddp_files_to_remove + +- name: Remove temporary files + ansible.builtin.file: + path: "{{ item }}" + state: absent + loop: "{{ ddp_files_to_remove.files | map(attribute='path') | list }}" diff --git a/roles/install_ddp_pkgs/tasks/install.yml b/roles/install_ddp_pkgs/tasks/install.yml new file mode 100644 index 00000000..35bee5f4 --- /dev/null +++ b/roles/install_ddp_pkgs/tasks/install.yml @@ -0,0 +1,58 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +- name: Create temp directory for extraction of packaes + ansible.builtin.tempfile: + state: directory + prefix: ddp + register: ddp_tempfile + +- name: Extract PKG files from DDP packages + ansible.builtin.unarchive: + src: "{{ (ddp_pkgs[nic_driver].download_dir, item.name) | path_join }}" + dest: "{{ ddp_tempfile.path }}" + remote_src: true + loop: "{{ ddp_pkgs[nic_driver].pkgs }}" + +- name: Create DDP package installation directory + become: yes + ansible.builtin.file: + path: "{{ ddp_pkgs[nic_driver].install_dir }}" + state: directory + mode: '0700' + owner: root + group: root + +- name: Find PKG files + ansible.builtin.find: + paths: "{{ ddp_tempfile.path }}" + patterns: '*.pkgo,*.pkg' + file_type: file + register: pkgfiles + +- name: Copy PKG files into installation directory + ansible.builtin.copy: + src: "{{ pkgfile.path }}" + dest: "{{ ddp_pkgs[nic_driver].install_dir }}" + remote_src: yes + mode: 0755 + loop: "{{ pkgfiles.files }}" + loop_control: + loop_var: pkgfile + +- name: Remove temp directory + ansible.builtin.file: + name: "{{ ddp_tempfile.path }}" + state: absent diff --git a/roles/install_ddp_pkgs/tasks/install_a_pkg.yml b/roles/install_ddp_pkgs/tasks/install_a_pkg.yml deleted file mode 100644 index 1a9c4141..00000000 --- a/roles/install_ddp_pkgs/tasks/install_a_pkg.yml +++ /dev/null @@ -1,96 +0,0 @@ -## -## Copyright (c) 2020-2023 Intel Corporation. -## -## Licensed under the Apache License, Version 2.0 (the "License"); -## you may not use this file except in compliance with the License. -## You may obtain a copy of the License at -## -## http://www.apache.org/licenses/LICENSE-2.0 -## -## Unless required by applicable law or agreed to in writing, software -## distributed under the License is distributed on an "AS IS" BASIS, -## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -## See the License for the specific language governing permissions and -## limitations under the License. -## ---- -- name: set temporary DDP profiles path - set_fact: - temp_ddp_path: "{{ tmp_placeholder }}/{{ pkgurl | regex_replace('^https?://.*/(.*).zip$', '\\1') }}" - -- name: create temp placeholder for each DDP package - become: yes - file: - path: "{{ temp_ddp_path }}" - state: directory - mode: 0700 - owner: root - group: root - -- name: download and unarchive DDP packages from list of URLs - unarchive: - src: "{{ pkgurl }}" - dest: "{{ temp_ddp_path }}" - remote_src: yes - mode: 0644 - register: ddppkg_download - until: ddppkg_download is not failed - retries: 5 - -- name: unarchive DDP package subfolder excluding from list of URLs - unarchive: - src: "{{ temp_ddp_path }}/ice_comms-1.3.28.0.zip" - dest: "{{ temp_ddp_path }}" - remote_src: yes - mode: 0644 - when: '"1.3.28.0" in pkgurl' - -- name: unarchive DDP package subfolder excluding from list of URLs - unarchive: - src: "{{ temp_ddp_path }}/ice_comms-1.3.30.0.zip" - dest: "{{ temp_ddp_path }}" - remote_src: yes - mode: 0644 - when: '"1.3.30.0" in pkgurl' - -- name: unarchive DDP package subfolder excluding from list of URLs - unarchive: - src: "{{ temp_ddp_path }}/ice_comms-1.3.31.0.zip" - dest: "{{ temp_ddp_path }}" - remote_src: yes - mode: 0644 - when: '"1.3.31.0" in pkgurl' - -- name: unarchive DDP package subfolder excluding from list of URLs - unarchive: - src: "{{ temp_ddp_path }}/ice_comms-1.3.37.0.zip" - dest: "{{ temp_ddp_path }}" - remote_src: yes - mode: 0644 - when: '"1.3.37.0" in pkgurl' - -- name: unarchive DDP package subfolder excluding from list of URLs - unarchive: - src: "{{ temp_ddp_path }}/ice_comms-1.3.40.0.zip" - dest: "{{ temp_ddp_path }}" - remote_src: yes - mode: 0644 - when: '"1.3.40.0" in pkgurl' - -- name: find PKG files - find: - paths: "{{ temp_ddp_path }}" - patterns: '*.pkgo,*.pkg' - file_type: file - recurse: yes - register: pkgfiles - -- name: copy DDP packages into installation directory - copy: - src: "{{ pkgfile.path }}" - dest: "{{ install_dir }}" - remote_src: yes - mode: 0755 - loop: "{{ pkgfiles.files }}" - loop_control: - loop_var: pkgfile diff --git a/roles/install_ddp_pkgs/tasks/main.yml b/roles/install_ddp_pkgs/tasks/main.yml index 6ce2c556..dc5d344b 100644 --- a/roles/install_ddp_pkgs/tasks/main.yml +++ b/roles/install_ddp_pkgs/tasks/main.yml @@ -13,58 +13,14 @@ ## See the License for the specific language governing permissions and ## limitations under the License. ## ---- -# This role installs Intel DDP packages for X700 and X800 series NICs. -# -# It depends on package variables defined in vars/main.yml -# The "ddp_pkgs" is map of a list of NIC specific package definition. -# -# Each list expected to provide "nic_driver": , "install_dir":, and "pkgurls":. -# -# latest list of available packages can be found here: -# https://downloadcenter.intel.com/search?keyword=Dynamic+Device+Personalization -# -# All packages will be downloaded, uncompressed and placed in NICs -# driver specific location so that these packages are available to be -# loaded onto a NIC adapter on demand. -# -# Example configurations: -# ddp_pkgs: -# - nic_driver: i40e -# install_dir: /lib/firmware/intel/i40e/ddp -# pkgurls: -# - "https://downloadmirror.intel.com/27587/eng/gtp.zip" -# - "https://downloadmirror.intel.com/28940/eng/mplsogreudp.zip" -# - "https://downloadmirror.intel.com/28040/eng/ppp-oe-ol2tpv2.zip" -# - nic_driver: ice -# install_dir: /lib/firmware/updates/intel/ice/ddp/ -# pkgurls: -# - "https://downloadmirror.intel.com/29889/eng/ice_comms-1.3.24.0.zip" +- name: Download listed DDP packages for each NIC driver + ansible.builtin.include_tasks: download.yml + loop: "{{ ddp_pkgs.keys() | list }}" + loop_control: + loop_var: nic_driver -- name: create temporary directory for DDP profiles - tempfile: - state: directory - prefix: ddp - register: tempdir_ddp - -# Download and install ddp pkgs listed given in vars/main.yml file -- name: download and install all listed DDP packages - block: - - name: calling install_pkgs.yml - include_tasks: install_pkgs.yml - vars: - install_dir: "{{ item.install_dir }}" - pkgurls: "{{ item.pkgurls }}" - tmp_placeholder: "{{ tempdir_ddp.path }}/{{ item.nic_driver }}" - with_items: "{{ ddp_pkgs }}" - when: - - ddp_pkgs|length > 0 - -# clean up -- name: remove temporary directory - become: yes - file: - path: "{{ tempdir_ddp.path }}" - state: absent +- name: Install DDP packages + ansible.builtin.include_tasks: install.yml + loop: "{{ ddp_pkgs.keys() | list }}" + loop_control: + loop_var: nic_driver diff --git a/roles/install_dependencies/tasks/main.yml b/roles/install_dependencies/tasks/main.yml index c31b4ffe..ae2dfcaa 100644 --- a/roles/install_dependencies/tasks/main.yml +++ b/roles/install_dependencies/tasks/main.yml @@ -27,8 +27,9 @@ when: ansible_os_family == 'Debian' - name: install packages - action: "{{ ansible_pkg_mgr }} \ - name={{ install_dependencies[ansible_os_family] }} state=present" + ansible.builtin.package: + name: "{{ install_dependencies[ansible_os_family] }}" + state: present register: pkg_mgr_results retries: "{{ number_of_retries | default(5) }}" until: pkg_mgr_results is success diff --git a/roles/install_dpdk/tasks/install_dpdk_meson.yml b/roles/install_dpdk/tasks/install_dpdk_meson.yml index dfae5e1c..b670daae 100644 --- a/roles/install_dpdk/tasks/install_dpdk_meson.yml +++ b/roles/install_dpdk/tasks/install_dpdk_meson.yml @@ -35,7 +35,7 @@ delay: 5 - name: meson build for ease of compiling and linking libraries enablement - command: "meson build" + command: "meson setup build" changed_when: true args: chdir: "{{ dpdk_dir }}" diff --git a/roles/install_dpdk/tasks/main.yml b/roles/install_dpdk/tasks/main.yml index ec9e3c11..8c3cc0a7 100644 --- a/roles/install_dpdk/tasks/main.yml +++ b/roles/install_dpdk/tasks/main.yml @@ -82,6 +82,12 @@ when: dpdk_local_patches_dir is defined +- name: Apply DPDK patches for media transport library (IMTL) + ansible.builtin.include_role: + name: imtl_install + tasks_from: dpdk_patch.yml + when: intel_media_transport_library_enabled | default(false) + - name: include install_dpdk task for DPDK <= 20.08 versions include_tasks: "install_dpdk_make.yml" when: dpdk_version is version("19.11", ">=") and dpdk_version is version("20.08", "<") diff --git a/roles/intel_base_container/defaults/main.yml b/roles/intel_base_container/defaults/main.yml index 0396e683..bb75f7cd 100644 --- a/roles/intel_base_container/defaults/main.yml +++ b/roles/intel_base_container/defaults/main.yml @@ -19,7 +19,7 @@ base_container_dockerfile_path: "{{ (base_container_path, 'dockerfile') | path_j base_container_test_path: "{{ (base_container_path, 'test') | path_join }}" base_container_sudo: true -aibox_base_container_version: 3.0 +aibox_base_container_version: 3.1 aibox_base_container_set: - { name: aibox-base, @@ -30,13 +30,18 @@ aibox_base_container_set: username: aibox, adduser: true, gpu_stack: true, + gpu_stack_version: default, oneapi_base: false, oneapi_ai: false, openvino: true, openvino_dev: false, + openvino_version: 2022.3.0, dlstreamer: false, + dlstreamer_version: default, ffmpeg: false, + ffmpeg_version: default, opencv: false, + opencv_version: default, dpdk: false, } - { name: aibox-base-devel, @@ -47,13 +52,62 @@ aibox_base_container_set: username: aibox, adduser: false, gpu_stack: false, + gpu_stack_version: default, oneapi_base: false, oneapi_ai: false, openvino: false, openvino_dev: true, + openvino_version: 2022.3.0, dlstreamer: false, + dlstreamer_version: default, ffmpeg: false, + ffmpeg_version: default, opencv: false, + opencv_version: default, + dpdk: false, + } + - { name: aibox-adv, + base: ubuntu, + base_version: 22.04, + filename: Dockerfile.adv, + buildname: build_adv.sh, + username: aibox, + adduser: true, + gpu_stack: true, + gpu_stack_version: default, + oneapi_base: false, + oneapi_ai: false, + openvino: true, + openvino_dev: false, + openvino_version: 2023.1.0, + dlstreamer: false, + dlstreamer_version: default, + ffmpeg: false, + ffmpeg_version: default, + opencv: false, + opencv_version: default, + dpdk: false, + } + - { name: aibox-adv-devel, + base: aibox-adv, + base_version: "{{ aibox_base_container_version }}", + filename: Dockerfile.base-adv-devel, + buildname: build_adv_devel.sh, + username: aibox, + adduser: false, + gpu_stack: false, + gpu_stack_version: default, + oneapi_base: false, + oneapi_ai: false, + openvino: false, + openvino_dev: true, + openvino_version: 2023.1.0, + dlstreamer: false, + dlstreamer_version: default, + ffmpeg: false, + ffmpeg_version: default, + opencv: false, + opencv_version: default, dpdk: false, } - { name: aibox-dlstreamer, @@ -64,13 +118,18 @@ aibox_base_container_set: username: aibox, adduser: false, gpu_stack: false, + gpu_stack_version: default, oneapi_base: false, oneapi_ai: false, openvino: false, openvino_dev: false, + openvino_version: default, dlstreamer: true, + dlstreamer_version: default, ffmpeg: false, + ffmpeg_version: default, opencv: false, + opencv_version: default, dpdk: false, } - { name: aibox-opencv-ffmpeg, @@ -81,13 +140,18 @@ aibox_base_container_set: username: aibox, adduser: false, gpu_stack: false, + gpu_stack_version: default, oneapi_base: false, oneapi_ai: false, openvino: false, + openvino_version: default, openvino_dev: false, dlstreamer: false, + dlstreamer_version: default, ffmpeg: true, + ffmpeg_version: default, opencv: true, + opencv_version: default, dpdk: false, } @@ -96,7 +160,7 @@ aibox_base_container_test_set: - { name: test-gpu, base: aibox-base, base_version: "{{ aibox_base_container_version }}", - filename: Dockerfile.test_gpu, + filename: Dockerfile.test-gpu, entryname: test_gpu_entry.sh, testname: test_gpu.sh, username: aibox @@ -104,23 +168,39 @@ aibox_base_container_test_set: - { name: test-openvino, base: aibox-base, base_version: "{{ aibox_base_container_version }}", - filename: Dockerfile.test_openvino, + filename: Dockerfile.test-openvino, entryname: test_openvino_entry.sh, testname: test_openvino.sh, username: aibox } - - { name: test-base-devel, + - { name: test-openvino-dev, base: aibox-base-devel, base_version: "{{ aibox_base_container_version }}", - filename: Dockerfile.test_base_devel, - entryname: test_base_devel_entry.sh, - testname: test_base_devel.sh, + filename: Dockerfile.test-openvino-dev, + entryname: test_openvino_dev_entry.sh, + testname: test_openvino_dev.sh, + username: aibox + } + - { name: test-openvino-adv, + base: aibox-adv, + base_version: "{{ aibox_base_container_version }}", + filename: Dockerfile.test-openvino-adv, + entryname: test_openvino_entry.sh, + testname: test_openvino_adv.sh, + username: aibox + } + - { name: test-openvino-adv-dev, + base: aibox-adv-devel, + base_version: "{{ aibox_base_container_version }}", + filename: Dockerfile.test-openvino-adv-dev, + entryname: test_openvino_dev_entry.sh, + testname: test_openvino_adv_dev.sh, username: aibox } - { name: test-dlstreamer, base: aibox-dlstreamer, base_version: "{{ aibox_base_container_version }}", - filename: Dockerfile.test_dlstreamer, + filename: Dockerfile.test-dlstreamer, entryname: test_dlstreamer_entry.sh, testname: test_dlstreamer.sh, username: aibox @@ -128,7 +208,7 @@ aibox_base_container_test_set: - { name: test-ffmpeg, base: aibox-opencv-ffmpeg, base_version: "{{ aibox_base_container_version }}", - filename: Dockerfile.test_ffmpeg, + filename: Dockerfile.test-ffmpeg, entryname: test_ffmpeg_entry.sh, testname: test_ffmpeg.sh, username: aibox @@ -136,7 +216,7 @@ aibox_base_container_test_set: - { name: test-opencv, base: aibox-opencv-ffmpeg, base_version: "{{ aibox_base_container_version }}", - filename: Dockerfile.test_opencv, + filename: Dockerfile.test-opencv, entryname: test_opencv_entry.sh, testname: test_opencv.sh, username: aibox diff --git a/roles/intel_base_container/files/install_dlstreamer.sh b/roles/intel_base_container/files/install_dlstreamer.sh index b9c28e4e..487e9c0d 100755 --- a/roles/intel_base_container/files/install_dlstreamer.sh +++ b/roles/intel_base_container/files/install_dlstreamer.sh @@ -1,14 +1,35 @@ #!/bin/bash +DLSTREAMER_VERSION="$1" +GPU_TYPE="$2" + +echo "dlstreamer version is : $DLSTREAMER_VERSION" +echo "gpu type is : $GPU_TYPE" + apt-get install -y software-properties-common curl -L -o /tmp/intel-sw-products.pub https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB gpg --dearmor < /tmp/intel-sw-products.pub > /usr/share/keyrings/intel-sw-products.gpg echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-sw-products.gpg] https://apt.repos.intel.com/openvino/2022 jammy main" >> /etc/apt/sources.list.d/intel-dlstreamer.list echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-sw-products.gpg] https://apt.repos.intel.com/oneapi all main" >> /etc/apt/sources.list.d/intel-oneapi.list + +rm -f /usr/share/keyrings/intel-graphics.gpg +curl -L -o /tmp/intel-graphics.key https://repositories.intel.com/graphics/intel-graphics.key +gpg --dearmor < /tmp/intel-graphics.key > /usr/share/keyrings/intel-graphics.gpg + +rm /etc/apt/sources.list.d/intel-gpu-jammy.list +if [ "${GPU_TYPE}" = "Flex" ]; then + echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.gpg] https://repositories.intel.com/graphics/ubuntu jammy flex" >> /etc/apt/sources.list.d/intel-gpu-jammy.list +elif [ "${GPU_TYPE}" = "Arc" ] || [ "${GPU_TYPE}" = "iGPU" ]; then + echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.gpg] https://repositories.intel.com/graphics/ubuntu jammy arc" >> /etc/apt/sources.list.d/intel-gpu-jammy.list +else + echo "Unknown GPU, no gpu stack workaround" +fi + apt-get update # Workaround as current Dlstreamer release depends on libva 2.17 +if [ "${GPU_TYPE}" == "Flex" ] || [ "${GPU_TYPE}" == "Arc" ] || [ "${GPU_TYPE}" == "iGPU" ]; then apt-mark unhold \ libva2 \ libva-drm2 \ @@ -41,6 +62,7 @@ apt-mark hold \ intel-media-va-driver-non-free \ libigfxcmrt7 \ libigfxcmrt-dev +fi # Install dlstreamer apt-get install -y \ diff --git a/roles/intel_base_container/files/install_ffmpeg.sh b/roles/intel_base_container/files/install_ffmpeg.sh index 8f48be10..3714729b 100755 --- a/roles/intel_base_container/files/install_ffmpeg.sh +++ b/roles/intel_base_container/files/install_ffmpeg.sh @@ -1,6 +1,10 @@ #!/bin/bash -NETWORK_TYPE="$1" +FFMPEG_VERSION="$1" +NETWORK_TYPE="$2" + +echo "ffmpeg version is : $FFMPEG_VERSION" +echo "network type is : $NETWORK_TYPE" apt-get update apt-get install -y \ @@ -36,8 +40,8 @@ apt-get install -y \ libvpl-dev \ FFMPEG_BUILD_DIR=ffmpeg_build -FFMPEG_UPSTREAM_VERSION=c3a7999 -FFMPEG_PATCH_VERSION=2023q1 +FFMPEG_UPSTREAM_VERSION=9b6d191 +FFMPEG_PATCH_VERSION=2023q2 FFMPEG_INSTALL_PREFIX=/usr/local mkdir -p $FFMPEG_BUILD_DIR @@ -70,6 +74,6 @@ cd ./ffmpeg_src || exit git config user.email "builder@intel.com" git config user.name "builder" git am ../cartwheel-patch/patches/*.patch -./configure --prefix=${FFMPEG_INSTALL_PREFIX} --libdir=/usr/lib --enable-shared --enable-vaapi --enable-libvpl +./configure --prefix=${FFMPEG_INSTALL_PREFIX} --enable-shared --enable-vaapi --enable-libvpl make -j8 make install diff --git a/roles/intel_base_container/files/install_gpu_stack.sh b/roles/intel_base_container/files/install_gpu_stack.sh index b147d5e8..1971f037 100755 --- a/roles/intel_base_container/files/install_gpu_stack.sh +++ b/roles/intel_base_container/files/install_gpu_stack.sh @@ -1,104 +1,108 @@ #!/bin/bash +GPU_STACK_VERSION="$1" +GPU_TYPE="$2" + +echo "gpu stack version : $GPU_STACK_VERSION" +echo "gpu type is : $GPU_TYPE" + # set repo according to gpu type + curl -L -o /tmp/intel-graphics.key https://repositories.intel.com/graphics/intel-graphics.key gpg --dearmor < /tmp/intel-graphics.key > /usr/share/keyrings/intel-graphics.gpg -echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.gpg] https://repositories.intel.com/graphics/ubuntu jammy arc" >> /etc/apt/sources.list.d/intel-gpu-jammy.list -apt-get update -GPU_STACK_VERSION=20230714 +if [ "${GPU_TYPE}" = "Flex" ]; then + rm -f /etc/apt/sources.list.d/intel-gpu-jammy.list + echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.gpg] https://repositories.intel.com/gpu/ubuntu jammy/production/2328 unified" >> /etc/apt/sources.list.d/intel-gpu-jammy.list + apt-get update -if (("${GPU_STACK_VERSION}" == 20230714)) -then - - echo "gpu stack version : 20230714" - # install umd and runtime packages + echo "install server gpu stack" + # install umd and runtime packages for server GPU apt-get install -y --allow-downgrades \ - intel-opencl-icd=23.17.26241.33-647~22.04 \ - intel-level-zero-gpu=1.3.26241.33-647~22.04 \ - level-zero=1.11.0-647~22.04 \ - intel-media-va-driver-non-free=23.2.1-647~22.04 \ - libmfx1=23.2.1-647~22.04 \ - libmfxgen1=23.2.1-647~22.04 \ - libvpl2=2023.2.1.0-647~22.04 \ - libegl-mesa0=23.2.0.20230414.1+2061~u22.04 \ - libegl1-mesa=23.2.0.20230414.1+2061~u22.04 \ - libegl1-mesa-dev=23.2.0.20230414.1+2061~u22.04 \ - libgbm1=23.2.0.20230414.1+2061~u22.04 \ - libgl1-mesa-dev=23.2.0.20230414.1+2061~u22.04 \ - libgl1-mesa-dri=23.2.0.20230414.1+2061~u22.04 \ - libglapi-mesa=23.2.0.20230414.1+2061~u22.04 \ - libgles2-mesa-dev=23.2.0.20230414.1+2061~u22.04 \ - libglx-mesa0=23.2.0.20230414.1+2061~u22.04 \ - libigdgmm12=22.3.5-647~22.04 \ - libxatracker2=23.2.0.20230414.1+2061~u22.04 \ - mesa-va-drivers=23.2.0.20230414.1+2061~u22.04 \ - mesa-vdpau-drivers=23.2.0.20230414.1+2061~u22.04 \ - mesa-vulkan-drivers=23.2.0.20230414.1+2061~u22.04 \ - va-driver-all=2.18.0.2-61~u22.04 + intel-opencl-icd=23.22.26516.29-682~22.04 \ + intel-level-zero-gpu=1.3.26516.29-682~22.04 \ + level-zero=1.11.0-649~22.04 \ + intel-media-va-driver-non-free=23.2.4-678~22.04 \ + libmfx1=23.2.2-678~22.04 \ + libmfxgen1=23.2.4-678~22.04 \ + libvpl2=2023.3.0.0-678~22.04 \ + libegl-mesa0=23.2.0.20230712.1-2073~22.04 \ + libegl1-mesa=23.2.0.20230712.1-2073~22.04 \ + libegl1-mesa-dev=23.2.0.20230712.1-2073~22.04 \ + libgbm1=23.2.0.20230712.1-2073~22.04 \ + libgl1-mesa-dev=23.2.0.20230712.1-2073~22.04 \ + libgl1-mesa-dri=23.2.0.20230712.1-2073~22.04 \ + libglapi-mesa=23.2.0.20230712.1-2073~22.04 \ + libgles2-mesa-dev=23.2.0.20230712.1-2073~22.04 \ + libglx-mesa0=23.2.0.20230712.1-2073~22.04 \ + libigdgmm12=22.3.7-678~22.04 \ + libxatracker2=23.2.0.20230712.1-2073~22.04 \ + mesa-va-drivers=23.2.0.20230712.1-2073~22.04 \ + mesa-vdpau-drivers=23.2.0.20230712.1-2073~22.04 \ + mesa-vulkan-drivers=23.2.0.20230712.1-2073~22.04 \ + va-driver-all=2.19.0.2-64~u22.04 - # install dev packages + # install dev packages for server GPU apt-get install -y --allow-downgrades \ - libigc1=1.0.13822.8-647~22.04 \ - libigc-dev=1.0.13822.8-647~22.04 \ - intel-igc-cm=1.0.176+i600~22.04 \ - libigdfcl1=1.0.13822.8-647~22.04 \ - libigdfcl-dev=1.0.13822.8-647~22.04 \ - libigfxcmrt7=23.2.1-647~22.04 \ - libigfxcmrt-dev=23.2.1-647~22.04 \ - level-zero-dev=1.11.0-647~22.04 - - # install xpum packages - # apt-get install -y --allow-downgrades \ - # xpu-smi=1.2.13-24~22.04 + libigc1=1.0.14062.15-682~22.04 \ + libigc-dev=1.0.14062.15-682~22.04 \ + intel-igc-cm=1.0.202-682~22.04 \ + libigdfcl1=1.0.14062.15-682~22.04 \ + libigdfcl-dev=1.0.14062.15-682~22.04 \ + libigfxcmrt7=23.2.4-678~22.04 \ + libigfxcmrt-dev=23.2.4-678~22.04 \ + level-zero-dev=1.11.0-649~22.04 \ + libvpl-dev=2023.3.0.0-678~22.04 -else +elif [[ "${GPU_TYPE}" = "Arc" || "${GPU_TYPE}" = "iGPU" ]]; then + rm -f /etc/apt/sources.list.d/intel-gpu-jammy.list + echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.gpg] https://repositories.intel.com/gpu/ubuntu jammy client" >> /etc/apt/sources.list.d/intel-gpu-jammy.list + apt-get update - echo "gpu stack version : 20230526" - # install umd and runtime packages + echo "install client gpu stack" + # install umd and runtime packages for client GPU apt-get install -y --allow-downgrades \ - intel-opencl-icd=23.13.26032.26-627~22.04 \ - intel-level-zero-gpu=1.3.26032.26-627~22.04 \ - level-zero=1.9.9-625~22.04 \ - intel-media-va-driver-non-free=23.1.6-622~22.04 \ - libmfx1=23.1.6-622~22.04 \ - libmfxgen1=23.1.5-622~22.04 \ - libvpl2=2023.1.3.0-622~22.04 \ - libegl-mesa0=23.2.0.20230414.1+2061~u22.04 \ - libegl1-mesa=23.2.0.20230414.1+2061~u22.04 \ - libegl1-mesa-dev=23.2.0.20230414.1+2061~u22.04 \ - libgbm1=23.2.0.20230414.1+2061~u22.04 \ - libgl1-mesa-dev=23.2.0.20230414.1+2061~u22.04 \ - libgl1-mesa-dri=23.2.0.20230414.1+2061~u22.04 \ - libglapi-mesa=23.2.0.20230414.1+2061~u22.04 \ - libgles2-mesa-dev=23.2.0.20230414.1+2061~u22.04 \ - libglx-mesa0=23.2.0.20230414.1+2061~u22.04 \ - libigdgmm12=22.3.5-622~22.04 \ - libxatracker2=23.2.0.20230414.1+2061~u22.04 \ - mesa-va-drivers=23.2.0.20230414.1+2061~u22.04 \ - mesa-vdpau-drivers=23.2.0.20230414.1+2061~u22.04 \ - mesa-vulkan-drivers=23.2.0.20230414.1+2061~u22.04 \ - va-driver-all=2.18.0.2-60~u22.04 + intel-opencl-icd=23.26.26690.36-704~22.04 \ + intel-level-zero-gpu=1.3.26690.36-704~22.04 \ + level-zero=1.12.0-693~22.04 \ + intel-media-va-driver-non-free=23.3.1-704~22.04 \ + libmfx1=23.2.2-704~22.04 \ + libmfxgen1=23.3.1-704~22.04 \ + libvpl2=2023.3.1.0-704~22.04 \ + libegl-mesa0=23.2.0.20230712.1-2073~22.04 \ + libegl1-mesa=23.2.0.20230712.1-2073~22.04 \ + libegl1-mesa-dev=23.2.0.20230712.1-2073~22.04 \ + libgbm1=23.2.0.20230712.1-2073~22.04 \ + libgl1-mesa-dev=23.2.0.20230712.1-2073~22.04 \ + libgl1-mesa-dri=23.2.0.20230712.1-2073~22.04 \ + libglapi-mesa=23.2.0.20230712.1-2073~22.04 \ + libgles2-mesa-dev=23.2.0.20230712.1-2073~22.04 \ + libglx-mesa0=23.2.0.20230712.1-2073~22.04 \ + libigdgmm12=22.3.8-687~22.04 \ + libxatracker2=23.2.0.20230712.1-2073~22.04 \ + mesa-va-drivers=23.2.0.20230712.1-2073~22.04 \ + mesa-vdpau-drivers=23.2.0.20230712.1-2073~22.04 \ + mesa-vulkan-drivers=23.2.0.20230712.1-2073~22.04 \ + va-driver-all=2.19.0.2-66~u22.04 - # install dev packages + # install dev packages for client GPU apt-get install -y --allow-downgrades \ - libigc1=1.0.13700.13-627~22.04 \ - libigc-dev=1.0.13700.13-627~22.04 \ - intel-igc-cm=1.0.176+i600~22.04 \ - libigdfcl1=1.0.13700.13-627~22.04 \ - libigdfcl-dev=1.0.13700.13-627~22.04 \ - libigfxcmrt7=23.1.6-622~22.04 \ - libigfxcmrt-dev=23.1.6-622~22.04 \ - level-zero-dev=1.9.9-625~22.04 \ - libvpl-dev=2023.1.3.0-622~22.04 + libigc1=1.0.14508.23-704~22.04 \ + libigc-dev=1.0.14508.23-704~22.04 \ + intel-igc-cm=1.0.206-704~22.04 \ + libigdfcl1=1.0.14508.23-704~22.04 \ + libigdfcl-dev=1.0.14508.23-704~22.04 \ + libigfxcmrt7=23.3.1-704~22.04 \ + libigfxcmrt-dev=23.3.1-704~22.04 \ + level-zero-dev=1.12.0-693~22.04 \ + libvpl-dev=2023.3.1.0-704~22.04 - # include xpum packages - # apt-get install -y --allow-downgrades \ - # xpu-smi=1.2.3-13~u22.04 +else + echo "Unknown GPU, no gpu stack will be installed" fi -# include test and tool packages +# install GPU type independent test and tool packages apt-get install -y --allow-downgrades \ hwinfo \ vainfo \ diff --git a/roles/intel_base_container/files/install_opencv.sh b/roles/intel_base_container/files/install_opencv.sh index 78c274a6..f7d7b3bd 100755 --- a/roles/intel_base_container/files/install_opencv.sh +++ b/roles/intel_base_container/files/install_opencv.sh @@ -1,6 +1,10 @@ #!/bin/bash -NETWORK_TYPE="$1" +OPENCV_VERSION="$1" +NETWORK_TYPE="$2" + +echo "opencv version is : $OPENCV_VERSION" +echo "network type is : $NETWORK_TYPE" apt-get update @@ -17,10 +21,12 @@ libmfx-dev \ libva-glx2 \ libgbm-dev \ -OPENCV_VERSION=4.7.0 +if [ "${OPENCV_VERSION}" = "default" ]; then + OPENCV_VERSION=4.7.0 +fi OPENCV_INSTALL_PREFIX=/usr/local -source /opt/intel/openvino_2022/setupvars.sh +source /opt/intel/openvino/setupvars.sh rm -rf ./opencv_src diff --git a/roles/intel_base_container/files/install_openvino.sh b/roles/intel_base_container/files/install_openvino.sh index 7bebb5e3..de97e525 100755 --- a/roles/intel_base_container/files/install_openvino.sh +++ b/roles/intel_base_container/files/install_openvino.sh @@ -1,14 +1,47 @@ #!/bin/bash -#curl -L -o openvino_2022.3.1.tgz \ -# https://storage.openvinotoolkit.org/repositories/openvino/packages/2022.3.1/linux/l_openvino_toolkit_ubuntu20_2022.3.1.9227.cf2c7da5689_x86_64.tgz +OPENVINO_VERSION="$1" -curl -L -o openvino_2022.3.0.tgz \ - https://storage.openvinotoolkit.org/repositories/openvino/packages/2022.3/linux/l_openvino_toolkit_ubuntu20_2022.3.0.9052.9752fafe8eb_x86_64.tgz -tar -xzvf openvino_2022.3.0.tgz -mv l_openvino_toolkit_ubuntu20_2022.3.0.9052.9752fafe8eb_x86_64 /opt/intel/openvino_2022.3.0 -ln -s /opt/intel/openvino_2022.3.0 /opt/intel/openvino_2022 -cd /opt/intel/openvino_2022 || exit -./install_dependencies/install_openvino_dependencies.sh +echo "openvino version is : $OPENVINO_VERSION" + +if [ "${OPENVINO_VERSION}" = "default" ]; then + # the latest openvino 2022.3 lts release is 2022.3.1 + # default use openvino 2022.3.0 for dlsreamer 2022.3.0 compatiblity + OPENVINO_VERSION=2022.3.0 +fi +echo "openvino version is : $OPENVINO_VERSION" + + +if [ "${OPENVINO_VERSION}" = "2023.1.0" ]; then + + curl -L -o openvino_2023.1.0.tgz \ + https://storage.openvinotoolkit.org/repositories/openvino/packages/2023.1/linux/l_openvino_toolkit_ubuntu22_2023.1.0.12185.47b736f63ed_x86_64.tgz + tar -xzvf openvino_2023.1.0.tgz + mv l_openvino_toolkit_ubuntu22_2023.1.0.12185.47b736f63ed_x86_64 /opt/intel/openvino_2023.1.0 + ln -s /opt/intel/openvino_2023.1.0 /opt/intel/openvino_2023 + ln -s /opt/intel/openvino_2023.1.0 /opt/intel/openvino + +elif [ "${OPENVINO_VERSION}" = "2023.0.0" ]; then + + curl -L -o openvino_2023.0.0.tgz \ + https://storage.openvinotoolkit.org/repositories/openvino/packages/2023.0/linux/l_openvino_toolkit_ubuntu22_2023.0.0.10926.b4452d56304_x86_64.tgz + tar -xzvf openvino_2023.0.0.tgz + mv l_openvino_toolkit_ubuntu22_2023.0.0.10926.b4452d56304_x86_64 /opt/intel/openvino_2023.0.0 + ln -s /opt/intel/openvino_2023.0.0 /opt/intel/openvino_2023 + ln -s /opt/intel/openvino_2023.0.0 /opt/intel/openvino + +else + + curl -L -o openvino_2022.3.0.tgz \ + https://storage.openvinotoolkit.org/repositories/openvino/packages/2022.3/linux/l_openvino_toolkit_ubuntu20_2022.3.0.9052.9752fafe8eb_x86_64.tgz + tar -xzvf openvino_2022.3.0.tgz + mv l_openvino_toolkit_ubuntu20_2022.3.0.9052.9752fafe8eb_x86_64 /opt/intel/openvino_2022.3.0 + ln -s /opt/intel/openvino_2022.3.0 /opt/intel/openvino_2022 + ln -s /opt/intel/openvino_2022.3.0 /opt/intel/openvino + +fi + +cd /opt/intel/openvino || exit +./install_dependencies/install_openvino_dependencies.sh -y apt-get install -y libtbb2 diff --git a/roles/intel_base_container/files/install_openvino_dev.sh b/roles/intel_base_container/files/install_openvino_dev.sh index 867275d0..507e9fc0 100755 --- a/roles/intel_base_container/files/install_openvino_dev.sh +++ b/roles/intel_base_container/files/install_openvino_dev.sh @@ -1,7 +1,35 @@ #!/bin/bash -python3 -m venv venv_openvino_2022 --prompt openvino_2022_dev -export PATH="$HOME/venv_openvino_2022/bin:$PATH" -python3 -m pip install --upgrade pip -pip install "openvino-dev[onnx,tensorflow,pytorch]==2022.3.0" +OPENVINO_VERSION="$1" +echo "openvino version is : $OPENVINO_VERSION" + +if [ "${OPENVINO_VERSION}" = "default" ]; then + # the latest openvino 2022.3 lts release is 2022.3.1 + # default use openvino 2022.3.0 for dlsreamer 2022.3.0 compatiblity + OPENVINO_VERSION=2022.3.0 +fi +echo "openvino version is : $OPENVINO_VERSION" + + +if [ "${OPENVINO_VERSION}" = "2023.1.0" ]; then + + python3 -m venv venv_openvino_2023 --prompt openvino_2023_dev + python3 -m pip install --upgrade pip + pip install "openvino-dev[onnx,tensorflow,pytorch]==2023.1.0" + ln -s "$HOME/venv_openvino_2023" "$HOME/venv_openvino" + +elif [ "${OPENVINO_VERSION}" = "2023.0.0" ]; then + + python3 -m venv venv_openvino_2023 --prompt openvino_2023_dev + python3 -m pip install --upgrade pip + pip install "openvino-dev[onnx,tensorflow,pytorch]==2023.0.0" + ln -s "$HOME/venv_openvino_2023" "$HOME/venv_openvino" + +else + + python3 -m venv venv_openvino_2022 --prompt openvino_2022_dev + python3 -m pip install --upgrade pip + pip install "openvino-dev[onnx,tensorflow,pytorch]==2022.3.0" + ln -s "$HOME/venv_openvino_2022" "$HOME/venv_openvino" +fi diff --git a/roles/intel_base_container/files/test_base_devel_entry.sh b/roles/intel_base_container/files/test_base_devel_entry.sh deleted file mode 100755 index c2de5a05..00000000 --- a/roles/intel_base_container/files/test_base_devel_entry.sh +++ /dev/null @@ -1,40 +0,0 @@ -#!/bin/bash - -source /opt/intel/openvino_2022/setupvars.sh - -test_data_path="/home/aibox/test_data/" -mtcnn_path="/home/aibox/data/models/public/mtcnn" -result="Failed" - -if [[ ! -d /home/aibox/data/models ]] -then - mkdir -p /home/aibox/data/models -fi - -# Download and convert mtcnn models, which is also the input for opencv_ffmpeg test. -if [[ -d /home/aibox/data/models/public/mtcnn ]] -then - rm -rf /home/aibox/data/models/public/mtcnn -fi - -timeout 120 omz_downloader --name mtcnn -o /home/aibox/data/models -cd ${mtcnn_path} || exit -check_download_md5=$(find . -name "*caffemodel" -o -name "*prototxt" | sort -k 2 | xargs md5sum) -if [[ $(grep -E "caffemodel|prototxt" ${test_data_path}/test_base_devel_md5.txt | sort -k 2) != "${check_download_md5}" ]] -then - echo "" - echo "[Warning] !!!The models download failed, please rerun the case." - exit 1 -fi - -omz_converter --name mtcnn -d /home/aibox/data/models -cd ${mtcnn_path} || exit -current_md5=$(find . -name "*bin" -o -name "*mapping" -o -name "*caffemodel" -o -name "*prototxt" | sort -k 2 | xargs md5sum) -echo "Current files md5:" -find . -name "*bin" -o -name "*mapping" -o -name "*caffemodel" -o -name "*prototxt" | sort -k 2 | xargs md5sum - -if [[ $(cat ${test_data_path}/test_base_devel_md5.txt) == "${current_md5}" ]] -then - result="Passed" -fi -echo ${result} | tee /home/aibox/data/test_base_devel_result diff --git a/roles/intel_base_container/files/test_dlstreamer_entry.sh b/roles/intel_base_container/files/test_dlstreamer_entry.sh index 8f75975f..f942d60e 100755 --- a/roles/intel_base_container/files/test_dlstreamer_entry.sh +++ b/roles/intel_base_container/files/test_dlstreamer_entry.sh @@ -1,79 +1,91 @@ #!/bin/bash -NETWORK_TYPE="$1" +TEST_POSTFIX="$1" +NETWORK_TYPE="$2" -source /opt/intel/openvino_2022/setupvars.sh +source /opt/intel/openvino/setupvars.sh source /opt/intel/dlstreamer/setupvars.sh export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/intel/oneapi/compiler/2023.2.1/linux/lib:/opt/intel/oneapi/compiler/2023.2.1/linux/compiler/lib/intel64_lin -DATA_PATH=$HOME/data - -if [[ ! -d "${DATA_PATH}"/models/intel/person-vehicle-bike-detection-2004/FP16 ]] +dlstreamer_data_path="$HOME"/data/dlstreamer_data_"$TEST_POSTFIX" +dlstreamer_output_path="$HOME"/data/dlstreamer_output_"$TEST_POSTFIX" +bin_file="${dlstreamer_data_path}"/models/intel/person-vehicle-bike-detection-2004/FP16/person-vehicle-bike-detection-2004.bin +if [[ ! -f ${bin_file} ]] || [[ $(md5sum "${bin_file}" | awk '{print $1}') != "650103e63235e8feae8a32f955cac855" ]] then - mkdir -p "${DATA_PATH}"/models/intel/person-vehicle-bike-detection-2004/FP16/ + if [[ -d "${dlstreamer_data_path}"/models ]] + then + rm -rf "${dlstreamer_data_path}"/models + fi + mkdir -p "${dlstreamer_data_path}"/models/intel/person-vehicle-bike-detection-2004/FP16/ curl -L https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.3/models_bin/1/person-vehicle-bike-detection-2004/FP16/person-vehicle-bike-detection-2004.xml \ - --output "${DATA_PATH}"/models/intel/person-vehicle-bike-detection-2004/FP16/person-vehicle-bike-detection-2004.xml + --output "${dlstreamer_data_path}"/models/intel/person-vehicle-bike-detection-2004/FP16/person-vehicle-bike-detection-2004.xml curl -L https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.3/models_bin/1/person-vehicle-bike-detection-2004/FP16/person-vehicle-bike-detection-2004.bin \ - --output "${DATA_PATH}"/models/intel/person-vehicle-bike-detection-2004/FP16/person-vehicle-bike-detection-2004.bin + --output "${dlstreamer_data_path}"/models/intel/person-vehicle-bike-detection-2004/FP16/person-vehicle-bike-detection-2004.bin if [ "${NETWORK_TYPE}" = "prc_network" ]; then echo "Download .proc file from gitmirror" curl -L https://raw.gitmirror.com/dlstreamer/dlstreamer/2022.3-release/samples/gstreamer/model_proc/intel/person-vehicle-bike-detection-2004.json \ - --output "${DATA_PATH}"/models/intel/person-vehicle-bike-detection-2004/person-vehicle-bike-detection-2004.json + --output "${dlstreamer_data_path}"/models/intel/person-vehicle-bike-detection-2004/person-vehicle-bike-detection-2004.json else echo "Download .proc file from github" curl -L https://raw.githubusercontent.com/dlstreamer/dlstreamer/2022.3-release/samples/gstreamer/model_proc/intel/person-vehicle-bike-detection-2004.json \ - --output "${DATA_PATH}"/models/intel/person-vehicle-bike-detection-2004/person-vehicle-bike-detection-2004.json + --output "${dlstreamer_data_path}"/models/intel/person-vehicle-bike-detection-2004/person-vehicle-bike-detection-2004.json fi fi -if [[ ! -f "${DATA_PATH}"/videos/car-detection.mp4 ]] +video_file="${dlstreamer_data_path}"/videos/car-detection.mp4 +if [[ ! -f ${video_file} ]] || [[ $(md5sum "${video_file}" | awk '{print $1}') != "e919de1193da5ceb8b0fd3cd998c2694" ]] then - mkdir -p "${DATA_PATH}"/videos/ + if [[ -d "${dlstreamer_data_path}"/videos ]] + then + rm -rf "${dlstreamer_data_path}"/videos + fi + mkdir -p "${dlstreamer_data_path}"/videos/ curl -L https://storage.openvinotoolkit.org/test_data/videos/car-detection.mp4 \ - --output "${DATA_PATH}"/videos/car-detection.mp4 + --output "${dlstreamer_data_path}"/videos/car-detection.mp4 fi -if [[ -d "${DATA_PATH}"/videos/dump_yuv ]] +output_mp4="${dlstreamer_output_path}"/output_person-vehicle-bike-detection-2004.mp4 +if [[ -d ${dlstreamer_output_path} ]] then - rm -rf "${DATA_PATH}"/videos/dump_yuv + rm -rf "${dlstreamer_output_path}" fi -if [[ -d "${DATA_PATH}"/videos/dump_jpg ]] -then - rm -rf "${DATA_PATH}"/videos/dump_jpg -fi -mkdir -p "${DATA_PATH}"/videos/dump_yuv -mkdir -p "${DATA_PATH}"/videos/dump_jpg - +mkdir -p "${dlstreamer_output_path}" -gst-launch-1.0 -e filesrc location="${DATA_PATH}"/videos/car-detection.mp4 ! \ +gst-launch-1.0 -e filesrc location="${dlstreamer_data_path}"/videos/car-detection.mp4 ! \ qtdemux ! h264parse ! vaapih264dec ! video/x-raw\(memory:VASurface\) ! \ gvadetect pre-process-backend=vaapi-surface-sharing pre-process-config=VAAPI_FAST_SCALE_LOAD_FACTOR=1 \ -model="${DATA_PATH}"/models/intel/person-vehicle-bike-detection-2004/FP16/person-vehicle-bike-detection-2004.xml \ -model-proc="${DATA_PATH}"/models/intel/person-vehicle-bike-detection-2004/person-vehicle-bike-detection-2004.json \ +model="${dlstreamer_data_path}"/models/intel/person-vehicle-bike-detection-2004/FP16/person-vehicle-bike-detection-2004.xml \ +model-proc="${dlstreamer_data_path}"/models/intel/person-vehicle-bike-detection-2004/person-vehicle-bike-detection-2004.json \ device=GPU ! \ meta_overlay ! \ gvafpscounter ! \ -vaapih264enc ! h264parse ! mp4mux ! filesink location="${DATA_PATH}"/videos/output_person-vehicle-bike-detection-2004.mp4 +vaapih264enc ! h264parse ! mp4mux ! filesink location="${output_mp4}" -gst-discoverer-1.0 "${DATA_PATH}"/videos/output_person-vehicle-bike-detection-2004.mp4 | tee "${DATA_PATH}"/dlstreamer_test_output.txt +gst-discoverer-1.0 "${output_mp4}" | tee "${dlstreamer_output_path}"/dlstreamer_test_output.txt result="Failed" pass_count=0 -if grep -n "Bitrate" "${DATA_PATH}"/dlstreamer_test_output.txt | awk -F: '{print $3}' +if grep -n "Bitrate" "${dlstreamer_output_path}"/dlstreamer_test_output.txt | awk -F: '{print $3}' then (( pass_count+=1 )) || true +else + result_fail_txt="Can not find Bitrate." fi -if grep -n "H.264" "${DATA_PATH}"/dlstreamer_test_output.txt +if grep -n "H.264" "${dlstreamer_output_path}"/dlstreamer_test_output.txt then (( pass_count+=1 )) || true +else + result_fail_txt="Can not find H.264. ${result_fail_txt}" fi if [[ ${pass_count} -eq 2 ]] then result="Passed" +else + result="Failed: ${result_fail_txt}" fi -echo "${result}" | tee "${DATA_PATH}"/test_dlstreamer_result +echo "${result}" | tee "$HOME"/data/test_dlstreamer_result_"$TEST_POSTFIX" diff --git a/roles/intel_base_container/files/test_ffmpeg_entry.sh b/roles/intel_base_container/files/test_ffmpeg_entry.sh index 07cd2523..df656f64 100755 --- a/roles/intel_base_container/files/test_ffmpeg_entry.sh +++ b/roles/intel_base_container/files/test_ffmpeg_entry.sh @@ -1,22 +1,27 @@ #!/bin/bash +TEST_POSTFIX="$1" + ffmpeg -decoders | grep -Ei "qsv|vaapi" ffmpeg -encoders | grep -Ei "qsv|vaapi" -if [[ -d /home/aibox/data/videos/ffmpeg ]] +ffmpeg_data_path="$HOME"/data/ffmpeg_data_"$TEST_POSTFIX" +ffmpeg_output_path="$HOME"/data/ffmpeg_output_"$TEST_POSTFIX" +if [[ ! -d $ffmpeg_data_path ]] then - rm -rf /home/aibox/data/videos/ffmpeg + mkdir -p "$ffmpeg_data_path" fi -mkdir -p /home/aibox/data/videos/ffmpeg -DATA_PATH=/home/aibox/data/videos/ffmpeg input_mp4_md5="e919de1193da5ceb8b0fd3cd998c2694" output_yuv_md5="8dccdfb5226e94732d83343cbaf25e8c" count=0 while [[ ${count} -le 5 ]] do - if [[ -f ${DATA_PATH}/input_car-detection.mp4 ]] && [[ $(md5sum ${DATA_PATH}/input_car-detection.mp4 | awk '{print $1}') == "${input_mp4_md5}" ]] + if [[ ! -f ${ffmpeg_data_path}/input_car-detection.mp4 ]] || [[ $(md5sum "${ffmpeg_data_path}"/input_car-detection.mp4 | awk '{print $1}') != "${input_mp4_md5}" ]] then + curl -L https://storage.openvinotoolkit.org/test_data/videos/car-detection.mp4 --output "${ffmpeg_data_path}"/input_car-detection.mp4 + (( count+=1 )) || true + else break fi if [[ ${count} == 5 ]] @@ -25,22 +30,25 @@ do echo "[Warning] !!!The mp4 file download failed, please rerun the case." exit 1 fi - curl -L https://storage.openvinotoolkit.org/test_data/videos/car-detection.mp4 --output ${DATA_PATH}/input_car-detection.mp4 - (( count+=1 )) || true done +if [[ -d ${ffmpeg_output_path} ]] +then + rm -rf "${ffmpeg_output_path}" +fi +mkdir -p "${ffmpeg_output_path}" + ffmpeg -v verbose -hwaccel qsv -init_hw_device qsv=qsv,child_device=/dev/dri/renderD128 -hwaccel_output_format nv12 \ - -hwaccel_flags allow_profile_mismatch -c:v h264_qsv -i ${DATA_PATH}/input_car-detection.mp4 -lavfi 'null' -c:v rawvideo -pix_fmt yuv420p \ - -fps_mode passthrough -autoscale 0 -y ${DATA_PATH}/ffmpeg_decode_output_car-detection.yuv + -hwaccel_flags allow_profile_mismatch -c:v h264_qsv -i "${ffmpeg_data_path}"/input_car-detection.mp4 -lavfi 'null' -c:v rawvideo -pix_fmt yuv420p \ + -fps_mode passthrough -autoscale 0 -y "${ffmpeg_output_path}"/ffmpeg_decode_output_car-detection.yuv ffmpeg -v verbose -hwaccel qsv -init_hw_device qsv=qsv,child_device=/dev/dri/renderD128 -hwaccel_output_format qsv \ - -f rawvideo -pix_fmt yuv420p -s:v 768x432 -r:v 30 -i ${DATA_PATH}/ffmpeg_decode_output_car-detection.yuv \ - -vf 'format=nv12,hwupload=extra_hw_frames=120' -an -c:v h264_qsv -profile:v main -g 30 -slices 1 -b:v 2000k -maxrate 2000k -y ${DATA_PATH}/ffmpeg_encode_output_car-detection.264 + -f rawvideo -pix_fmt yuv420p -s:v 768x432 -r:v 30 -i "${ffmpeg_output_path}"/ffmpeg_decode_output_car-detection.yuv \ + -vf 'format=nv12,hwupload=extra_hw_frames=120' -an -c:v h264_qsv -profile:v main -g 30 -slices 1 -b:v 2000k -maxrate 2000k -y "${ffmpeg_output_path}"/ffmpeg_encode_output_car-detection.264 -ffmpeg_path="/home/aibox/data/videos/ffmpeg" result="Failed" -cd ${ffmpeg_path} || exit +cd "${ffmpeg_output_path}" || exit current_yuv_md5=$(md5sum -- *yuv | awk '{print $1}') current_264_size=$(du ./*264 | awk '{print $1}' ) echo "***Current files md5:" @@ -50,5 +58,12 @@ ls -l if [[ "${current_yuv_md5}" == "${output_yuv_md5}" ]] && [[ ${current_264_size} -gt 2500 ]] then result="Passed" +else + if [[ ${current_264_size} -le 2500 ]] + then + result="Failed: The size of h264 file is abnormal." + else + result="Failed: The MD5 value of the yuv file is different from the base value." + fi fi -echo ${result} | tee /home/aibox/data/test_ffmpeg_result +echo "${result}" | tee "$HOME"/data/test_ffmpeg_result_"$TEST_POSTFIX" diff --git a/roles/intel_base_container/files/test_gpu_entry.sh b/roles/intel_base_container/files/test_gpu_entry.sh index 39bd7ffe..a61379aa 100755 --- a/roles/intel_base_container/files/test_gpu_entry.sh +++ b/roles/intel_base_container/files/test_gpu_entry.sh @@ -1,5 +1,7 @@ #!/bin/bash +TEST_POSTFIX="$1" + result="Failed" pass_count=0 echo "---------------------Check glxinfo -------------------" @@ -8,6 +10,7 @@ if glxinfo | grep OpenGL | grep "Intel" | grep "Graphics" then (( pass_count+=1 )) || true else + result_failed_txt="Get glxinfo: Failed." echo "glxinfo: Failed" fi @@ -17,6 +20,7 @@ if clinfo | grep Device | grep "Intel" | grep "Graphics" then (( pass_count+=1 )) || true else + result_failed_txt="Get clinfo: Failed. ${result_failed_txt}" echo "clinfo: Failed" fi @@ -26,6 +30,7 @@ if vulkaninfo | grep deviceName | grep "Intel" | grep "Graphics" then (( pass_count+=1 )) || true else + result_failed_txt="Get vulkaninfo: Failed. ${result_failed_txt}" echo "vulkaninfo: Failed" fi @@ -35,12 +40,15 @@ if vainfo | grep Driver | grep "Intel" | grep "Graphics" then (( pass_count+=1 )) || true else + result_failed_txt="Get vainfo: Failed. ${result_failed_txt}" echo "vainfo: Failed" fi if [[ ${pass_count} -eq 4 ]] then result="Passed" +else + result="Failed: ${result_failed_txt}" fi -echo ${result} | tee /home/aibox/data/test_gpu_result +echo "${result}" | tee "$HOME"/data/test_gpu_result_"$TEST_POSTFIX" diff --git a/roles/intel_base_container/files/test_opencv_entry.sh b/roles/intel_base_container/files/test_opencv_entry.sh index dba7bbce..91148a11 100755 --- a/roles/intel_base_container/files/test_opencv_entry.sh +++ b/roles/intel_base_container/files/test_opencv_entry.sh @@ -1,14 +1,38 @@ #!/bin/bash -git clone --depth 1 https://github.com/opencv/opencv_extra.git opencv_extra -export OPENCV_TEST_DATA_PATH=$HOME/opencv_extra/testdata/ -source /opt/intel/openvino_2022/setupvars.sh +TEST_POSTFIX="$1" + +opencv_data_path="$HOME"/data/opencv_data_"$TEST_POSTFIX" +opencv_output_path="$HOME"/data/opencv_output_"$TEST_POSTFIX" +if [[ ! -d ${opencv_data_path} ]] +then + mkdir -p "${opencv_data_path}" +fi +if [[ -d ${opencv_output_path} ]] +then + rm -rf "${opencv_output_path}" +fi +mkdir -p "${opencv_output_path}" + +cd "$opencv_data_path" || exit +if [[ -d opencv_extra ]] +then + cd opencv_extra || exit + git pull +else + git clone --depth 1 https://github.com/opencv/opencv_extra.git opencv_extra +fi + +export OPENCV_TEST_DATA_PATH=${opencv_data_path}/opencv_extra/testdata/ +source /opt/intel/openvino/setupvars.sh cd /opt/intel/nep/opencv_test/bin/ || exit -./opencv_perf_objdetect | tee /home/aibox/data/opencv_test_output.txt +./opencv_perf_objdetect | tee "${opencv_output_path}"/opencv_test_output.txt result="Failed" -if grep -n PASSED /home/aibox/data/opencv_test_output.txt |grep "81 tests" +if grep -n PASSED "${opencv_output_path}"/opencv_test_output.txt |grep "81 tests" then result="Passed" +else + result="Failed: Please check the result of opencv perf objdetect." fi -echo ${result} | tee /home/aibox/data/test_opencv_result +echo "${result}" | tee "$HOME"/data/test_opencv_result_"$TEST_POSTFIX" diff --git a/roles/intel_base_container/files/test_openvino_dev_entry.sh b/roles/intel_base_container/files/test_openvino_dev_entry.sh new file mode 100755 index 00000000..7d0aa979 --- /dev/null +++ b/roles/intel_base_container/files/test_openvino_dev_entry.sh @@ -0,0 +1,41 @@ +#!/bin/bash + +TEST_POSTFIX="$1" + +source /opt/intel/openvino/setupvars.sh + +openvino_dev_output_path="$HOME"/data/openvino_dev_output_"$TEST_POSTFIX" +mtcnn_path=${openvino_dev_output_path}/models/public/mtcnn +result="Failed" + +if [[ -d ${openvino_dev_output_path} ]] +then + rm -rf "${openvino_dev_output_path}" +fi +mkdir -p "${openvino_dev_output_path}" + +# Download and convert mtcnn models, which is also the input for opencv_ffmpeg test. +if [[ -d ${mtcnn_path} ]] +then + rm -rf "${mtcnn_path}" +fi + +timeout 120 omz_downloader --name mtcnn -o "${openvino_dev_output_path}"/models +cd "${mtcnn_path}" || exit +if [[ $(find . -name "*caffemodel" -size +10k -o -name "*prototxt" -size +1k | wc -l) -ne 6 ]] +then + echo "" + echo "[Warning] !!!The models download failed, please rerun the case." + exit 1 +fi + +omz_converter --name mtcnn -d "${openvino_dev_output_path}"/models +cd "${mtcnn_path}" || exit +if [[ $(find . -name "*bin" -size +10k | wc -l) -eq 6 ]] +then + result="Passed" +else + result="Failed: The converted file does not match expectations." + find . -name "*bin" -exec ls -l {} \; +fi +echo "${result}" | tee "$HOME"/data/test_openvino_dev_result_"$TEST_POSTFIX" diff --git a/roles/intel_base_container/files/test_openvino_entry.sh b/roles/intel_base_container/files/test_openvino_entry.sh index fea493c8..735fa239 100755 --- a/roles/intel_base_container/files/test_openvino_entry.sh +++ b/roles/intel_base_container/files/test_openvino_entry.sh @@ -1,9 +1,11 @@ #!/bin/bash -source /opt/intel/openvino_2022/setupvars.sh +TEST_POSTFIX="$1" + +source /opt/intel/openvino/setupvars.sh echo "---------------- Build benchmark_app ----------------" -cd /opt/intel/openvino_2022/samples/cpp || exit +cd /opt/intel/openvino/samples/cpp || exit ./build_samples.sh cd "$HOME"/openvino_cpp_samples_build/intel64/Release || exit @@ -13,49 +15,69 @@ echo "---------------- Check Avaliable OpenVINO devices ----------------" echo "---------------- Check OpenVINO inference ----------------" -DATA_PATH=$HOME/data -mkdir -p "${DATA_PATH}"/models/ +openvino_data_path="$HOME"/data/openvino_data_"$TEST_POSTFIX" +openvino_output_path="$HOME"/data/openvino_output_"$TEST_POSTFIX" +if [[ -d "${openvino_output_path}" ]] +then + rm -rf "${openvino_output_path}" +fi +mkdir -p "${openvino_output_path}" -if [[ ! -f "${DATA_PATH}"/models/intel/resnet50-binary-0001/FP16-INT1/resnet50-binary-0001.bin ]] +output_result="${openvino_output_path}"/openvino_test_output.txt +bin_file="${openvino_data_path}"/models/intel/resnet50-binary-0001/FP16-INT1/resnet50-binary-0001.bin +if [[ ! -f ${bin_file} ]] || [[ $(md5sum "${bin_file}" | awk '{print $1}') != "e2823d2a2548a8c03ae9920659175c58" ]] then + if [[ -d "${openvino_data_path}"/models ]] + then + rm -rf "${openvino_data_path}"/models + fi + mkdir -p "${openvino_data_path}"/models/intel/resnet50-binary-0001/FP16-INT1/ echo "---------------- Download ResNet50 model ----------------" - mkdir -p "${DATA_PATH}"/models/intel/resnet50-binary-0001/FP16-INT1/ - curl -L https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.3/models_bin/1/resnet50-binary-0001/FP16-INT1/resnet50-binary-0001.bin \ - --output "${DATA_PATH}"/models/intel/resnet50-binary-0001/FP16-INT1/resnet50-binary-0001.bin - + --output "${openvino_data_path}"/models/intel/resnet50-binary-0001/FP16-INT1/resnet50-binary-0001.bin curl -L https://storage.openvinotoolkit.org/repositories/open_model_zoo/2022.3/models_bin/1/resnet50-binary-0001/FP16-INT1/resnet50-binary-0001.xml \ - --output "${DATA_PATH}"/models/intel/resnet50-binary-0001/FP16-INT1/resnet50-binary-0001.xml - + --output "${openvino_data_path}"/models/intel/resnet50-binary-0001/FP16-INT1/resnet50-binary-0001.xml fi -./benchmark_app -m "${DATA_PATH}"/models/intel/resnet50-binary-0001/FP16-INT1/resnet50-binary-0001.xml -d GPU | tee /home/aibox/data/openvino_test_output.txt + +./benchmark_app -m "${openvino_data_path}"/models/intel/resnet50-binary-0001/FP16-INT1/resnet50-binary-0001.xml -d GPU | tee "${output_result}" result="Failed" pass_count=0 -if [[ $(grep -n "Throughput:" /home/aibox/data/openvino_test_output.txt | awk '{if($(NF-1)>0) {print "Passed"} else {print "Failed"}}') == "Passed" ]] +if [[ $(grep -n "Throughput:" "${output_result}" | awk '{if($(NF-1)>0) {print "Passed"} else {print "Failed"}}') == "Passed" ]] then (( pass_count+=1 )) || true +else + result_failed_txt="No Throughput value." fi -if [[ $(grep -n "Median:" /home/aibox/data/openvino_test_output.txt | awk '{if($(NF-1)>0) {print "Passed"} else {print "Failed"}}') == "Passed" ]] +if [[ $(grep -n "Median:" "${output_result}" | awk '{if($(NF-1)>0) {print "Passed"} else {print "Failed"}}') == "Passed" ]] then (( pass_count+=1 )) || true +else + result_failed_txt="No Median value. ${result_failed_txt}" fi -if [[ $(grep -n "Average:" /home/aibox/data/openvino_test_output.txt | awk '{if($(NF-1)>0) {print "Passed"} else {print "Failed"}}') == "Passed" ]] +if [[ $(grep -n "Average:" "${output_result}" | awk '{if($(NF-1)>0) {print "Passed"} else {print "Failed"}}') == "Passed" ]] then (( pass_count+=1 )) || true +else + result_failed_txt="No Average value. ${result_failed_txt}" fi -if [[ $(grep -n "Min:" /home/aibox/data/openvino_test_output.txt | awk '{if($(NF-1)>0) {print "Passed"} else {print "Failed"}}') == "Passed" ]] +if [[ $(grep -n "Min:" "${output_result}" | awk '{if($(NF-1)>0) {print "Passed"} else {print "Failed"}}') == "Passed" ]] then (( pass_count+=1 )) || true +else + result_failed_txt="No Min value. ${result_failed_txt}" fi -if [[ $(grep -n "Max:" /home/aibox/data/openvino_test_output.txt | awk '{if($(NF-1)>0) {print "Passed"} else {print "Failed"}}') == "Passed" ]] +if [[ $(grep -n "Max:" "${output_result}" | awk '{if($(NF-1)>0) {print "Passed"} else {print "Failed"}}') == "Passed" ]] then (( pass_count+=1 )) || true +else + result_failed_txt="No Max value. ${result_failed_txt}" fi if [[ ${pass_count} -eq 5 ]] then result="Passed" +else + result="Failed: ${result_failed_txt}" fi -echo ${result} | tee /home/aibox/data/test_openvino_result - +echo "${result}" | tee "${HOME}"/data/test_openvino_result_"$TEST_POSTFIX" diff --git a/roles/intel_base_container/tasks/main.yml b/roles/intel_base_container/tasks/main.yml index 99727a28..0f6bd836 100644 --- a/roles/intel_base_container/tasks/main.yml +++ b/roles/intel_base_container/tasks/main.yml @@ -44,13 +44,18 @@ container_user: "{{ item.username }}" container_adduser: "{{ item.adduser }}" container_gpu_stack: "{{ item.gpu_stack }}" + container_gpu_stack_version: "{{ item.gpu_stack_version }}" container_oneapi_base: "{{ item.oneapi_base }}" container_oneapi_ai: "{{ item.oneapi_ai }}" container_openvino: "{{ item.openvino }}" container_openvino_dev: "{{ item.openvino_dev }}" + container_openvino_version: "{{ item.openvino_version }}" container_dlstreamer: "{{ item.dlstreamer }}" + container_dlstreamer_version: "{{ item.dlstreamer_version }}" container_ffmpeg: "{{ item.ffmpeg }}" + container_ffmpeg_version: "{{ item.ffmpeg_version }}" container_opencv: "{{ item.opencv }}" + container_opencv_version: "{{ item.opencv_version }}" container_dpdk: "{{ item.dpdk }}" ansible.builtin.template: src: "templates/Dockerfile.j2" @@ -125,6 +130,6 @@ - 'test_dlstreamer_entry.sh' - 'test_ffmpeg_entry.sh' - 'test_opencv_entry.sh' - - 'test_base_devel_entry.sh' + - 'test_openvino_dev_entry.sh' - 'test_data' become: "{{ base_container_sudo }}" diff --git a/roles/intel_base_container/templates/Dockerfile.j2 b/roles/intel_base_container/templates/Dockerfile.j2 index f579ac5b..ff5d0eb7 100644 --- a/roles/intel_base_container/templates/Dockerfile.j2 +++ b/roles/intel_base_container/templates/Dockerfile.j2 @@ -21,7 +21,7 @@ RUN mkdir -p /opt/intel/nep COPY install_gpu_stack.sh /opt/intel/nep/ WORKDIR /opt/intel/nep # hadolint ignore=DL3059 -RUN chmod +x install_gpu_stack.sh && ./install_gpu_stack.sh +RUN chmod +x install_gpu_stack.sh && ./install_gpu_stack.sh {{ container_gpu_stack_version }} {{ gpu_type }} {% endif %} {% if container_oneapi_base %} @@ -40,7 +40,7 @@ RUN mkdir -p /opt/intel/nep COPY install_openvino.sh /opt/intel/nep/ WORKDIR /opt/intel/nep # hadolint ignore=DL3059 -RUN chmod +x install_openvino.sh && ./install_openvino.sh +RUN chmod +x install_openvino.sh && ./install_openvino.sh {{ container_openvino_version }} {% endif %} @@ -50,7 +50,7 @@ RUN mkdir -p /opt/intel/nep COPY install_dlstreamer.sh /opt/intel/nep/ WORKDIR /opt/intel/nep # hadolint ignore=DL3059 -RUN chmod +x install_dlstreamer.sh && ./install_dlstreamer.sh +RUN chmod +x install_dlstreamer.sh && ./install_dlstreamer.sh {{ container_dlstreamer_version }} {{ gpu_type }} {% endif %} @@ -63,10 +63,10 @@ WORKDIR /opt/intel/nep RUN chmod +x install_ffmpeg.sh {% if prc_network %} # hadolint ignore=DL3059 -RUN ./install_ffmpeg.sh prc_network +RUN ./install_ffmpeg.sh {{ container_ffmpeg_version }} prc_network {% else %} # hadolint ignore=DL3059 -RUN ./install_ffmpeg.sh +RUN ./install_ffmpeg.sh {{ container_ffmpeg_version }} {% endif %} {% endif %} @@ -80,10 +80,10 @@ WORKDIR /opt/intel/nep RUN chmod +x install_opencv.sh {% if prc_network %} # hadolint ignore=DL3059 -RUN ./install_opencv.sh prc_network +RUN ./install_opencv.sh {{ container_opencv_version }} prc_network {% else %} # hadolint ignore=DL3059 -RUN ./install_opencv.sh +RUN ./install_opencv.sh {{ container_opencv_version }} {% endif %} {% endif %} @@ -105,16 +105,19 @@ RUN useradd --create-home --no-log-init --shell /bin/bash aibox \ {% if container_user != 'root' %} USER {{container_user}} ENV HOME=/home/{{container_user}} -ENV PATH=${PATH}:/usr/local/bin +ENV PATH=${HOME}/.local/bin:${HOME}/bin:${PATH}:/usr/local/bin ENV LD_LIBRARY_PATH=${LD_LIBRAY_PATH}:/usr/local/lib WORKDIR /home/{{container_user}} {% endif %} +{% if prc_network %} +RUN pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple +{% endif %} {% if container_openvino_dev %} COPY --chown={{container_user}}:{{container_user}} install_openvino_dev.sh . -RUN chmod +x install_openvino_dev.sh && ./install_openvino_dev.sh -ENV PATH=${HOME}/venv_openvino_2022/bin:${PATH} +RUN chmod +x install_openvino_dev.sh && ./install_openvino_dev.sh {{ container_openvino_version }} +ENV PATH=${HOME}/venv_openvino/bin:${PATH} {% endif %} HEALTHCHECK NONE diff --git a/roles/intel_base_container/templates/Dockerfile_test.j2 b/roles/intel_base_container/templates/Dockerfile_test.j2 index 7e48134f..21c51676 100644 --- a/roles/intel_base_container/templates/Dockerfile_test.j2 +++ b/roles/intel_base_container/templates/Dockerfile_test.j2 @@ -13,7 +13,7 @@ RUN chmod +x {{container_entry}} HEALTHCHECK NONE {% if prc_network %} -ENTRYPOINT ["./{{container_entry}}", "prc_network"] +ENTRYPOINT ["./{{container_entry}}", "{{ container_base }}", "prc_network"] {% else %} -ENTRYPOINT ["./{{container_entry}}"] +ENTRYPOINT ["./{{container_entry}}", "{{ container_base }}"] {% endif %} diff --git a/roles/intel_csl_excat/defaults/main.yml b/roles/intel_csl_excat/defaults/main.yml new file mode 100644 index 00000000..8fc21bb8 --- /dev/null +++ b/roles/intel_csl_excat/defaults/main.yml @@ -0,0 +1,27 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +# This var is not intended to be customized by user +# Define respective field in the group_vars/all.yml instead +csl_excat_tar_staging_location: "/tmp/csl-excat-v3.0.0.tar.gz" +csl_excat_image_checksum: "acc5842f51acaa6aea6b7f3dc875b94e0930dc362827512535bb1673c53f87ef" +csl_excat_version: "v0.1.0" +csl_excat_admission_name: "csl-excat-admission" +csl_excat_dp_name: "csl-excat-deviceplugin" +csl_excat_admission_image: "{{ csl_excat_admission_name }}.tar" +csl_excat_dp_image: "{{ csl_excat_dp_name }}.tar" +csl_excat_admission_container: "localhost/{{ csl_excat_admission_name }}:{{ csl_excat_version }}" +csl_excat_dp_container: "localhost/{{ csl_excat_dp_name }}:{{ csl_excat_version }}" diff --git a/roles/intel_csl_excat/files/rdt-config.yaml b/roles/intel_csl_excat/files/rdt-config.yaml new file mode 100644 index 00000000..94607a6d --- /dev/null +++ b/roles/intel_csl_excat/files/rdt-config.yaml @@ -0,0 +1,34 @@ +# Common options +options: + l2: + # Set to false if L2 CAT must be available (Default is true). + optional: true + l3: + # Set to false if L3 CAT must be available (Default is true). + optional: true + mb: + # Set to false if MBA must be available (Default is true). + optional: true +# Partition definitions +partitions: + pdef: + l2Allocation: "50%" + l3Allocation: "50%" + classes: + system/default: + l2schema: "100%" + l3schema: "100%" + p0: + l2Allocation: "25%" + l3Allocation: "25%" + classes: + c0: + l2schema: "100%" + l3schema: "100%" + p1: + l2Allocation: "25%" + l3Allocation: "25%" + classes: + c1: + l2schema: "100%" + l3schema: "100%" diff --git a/roles/intel_csl_excat/files/sys-fs-resctrl.mount b/roles/intel_csl_excat/files/sys-fs-resctrl.mount new file mode 100644 index 00000000..9e8bd622 --- /dev/null +++ b/roles/intel_csl_excat/files/sys-fs-resctrl.mount @@ -0,0 +1,12 @@ +[Unit] +Description=/sys/fs/resctrl automount + +[Mount] +What=none +Where=/sys/fs/resctrl +Type=resctrl +Options=defaults + +[Install] +WantedBy=multi-user.target + diff --git a/roles/intel_csl_excat/tasks/cleanup.yml b/roles/intel_csl_excat/tasks/cleanup.yml new file mode 100644 index 00000000..e31d2c4b --- /dev/null +++ b/roles/intel_csl_excat/tasks/cleanup.yml @@ -0,0 +1,38 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +- name: delete csl_excat helm + kubernetes.core.helm: + name: "csl-excat" + state: absent + namespace: "csl-excat" + wait: true + timeout: 4m0s + when: inventory_hostname == groups['kube_control_plane'][0] + tags: + - intel_csl_excat + +- name: clear csl-excat namespace + kubernetes.core.k8s: + name: "csl-excat" + api_version: v1 + kind: Namespace + state: absent + wait: true + wait_timeout: 240 + when: inventory_hostname == groups['kube_control_plane'][0] + tags: + - intel_csl_excat diff --git a/roles/intel_csl_excat/tasks/main.yml b/roles/intel_csl_excat/tasks/main.yml new file mode 100644 index 00000000..e3471bd6 --- /dev/null +++ b/roles/intel_csl_excat/tasks/main.yml @@ -0,0 +1,182 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +- name: copy csl_excat contents to kube node + when: excat_dp_enabled | default(false) | bool or inventory_hostname in groups['kube_control_plane'] + block: + - name: create csl directory + ansible.builtin.file: + path: "{{ (project_root_dir, 'csl') | path_join }}" + state: directory + mode: '0644' + + - name: extract the tar ball + ansible.builtin.unarchive: + src: "{{ csl_excat_tar_staging_location }}" + dest: "{{ (project_root_dir, 'csl') | path_join }}" + list_files: yes + register: csl_contents + + - name: get csl_excat folder + set_fact: + csl_home: "{{ (project_root_dir, 'csl', csl_contents.files[0]) | path_join }}" + +- name: load local csl_excat admission container image + containers.podman.podman_load: + input: "{{ (csl_home, 'images', csl_excat_admission_image) | path_join }}" + when: + - inventory_hostname in groups['kube_control_plane'] + - container_runtime == "crio" + +- name: load local csl_excat dp container image + containers.podman.podman_load: + input: "{{ (csl_home, 'images', csl_excat_dp_image) | path_join }}" + when: + - excat_dp_enabled | default(false) | bool + - container_runtime == "crio" + +- name: load local csl_excat admission container image + ansible.builtin.command: + cmd: "ctr -n k8s.io images import {{ csl_excat_admission_image }}" + chdir: "{{ (csl_home, 'images') | path_join }}" + changed_when: true + when: + - inventory_hostname in groups['kube_control_plane'] + - container_runtime == "containerd" + +- name: load local csl_excat dp container image + ansible.builtin.command: + cmd: "ctr -n k8s.io images import {{ csl_excat_dp_image }}" + chdir: "{{ (csl_home, 'images') | path_join }}" + changed_when: true + when: + - excat_dp_enabled | default(false) | bool + - container_runtime == "containerd" + +- name: generate csr for csl_excat + ansible.builtin.command: + cmd: "./gencerts.sh certs {{ csl_excat_admission_name }} csl-excat 365" + chdir: "{{ (csl_home, 'deployments/helm') | path_join }}" + when: inventory_hostname == groups['kube_control_plane'][0] + +- name: mark kube node who supports excat + kubernetes.core.k8s: + state: present + definition: + apiVersion: v1 + kind: Node + metadata: + name: "{{ item }}" + labels: + 'excat': 'yes' + loop: "{{ groups['kube_node'] }}" + when: + - hostvars[item]['excat_dp_enabled'] | default(false) | bool + - inventory_hostname == groups['kube_control_plane'][0] + +- name: prepare RDT resources + when: excat_dp_enabled | default(false) | bool + block: + - name: install resctrl automount service + ansible.builtin.copy: + src: sys-fs-resctrl.mount + dest: /etc/systemd/system + owner: root + group: root + mode: '0644' + + - name: install rdt-config.yaml + ansible.builtin.copy: + src: rdt-config.yaml + dest: /etc + owner: root + group: root + mode: '0644' + + - name: start automount service + ansible.builtin.systemd: + name: sys-fs-resctrl.mount + state: started + enabled: yes + + - name: enable crio RDT awareness + community.general.ini_file: + dest: "/etc/crio/crio.conf" + section: "crio.runtime" + option: 'rdt_config_file = "/etc/rdt-config.yaml"' + allow_no_value: yes + mode: '0644' + backup: no + when: container_runtime == "crio" + + - name: enable containerd RDT awareness + ansible.builtin.blockinfile: + path: /etc/containerd/config.toml + mode: '0644' + block: | + #### + [plugins."io.containerd.service.v1.tasks-service"] + rdt_config_file = "/etc/rdt-config.yaml" + when: container_runtime == "containerd" + + - name: reload systemd + ansible.builtin.systemd: + daemon_reload: true + + - name: reload crio + ansible.builtin.service: + name: crio + state: restarted + enabled: yes + when: container_runtime == "crio" + + - name: reload containerd + ansible.builtin.service: + name: containerd + state: restarted + enabled: yes + when: container_runtime == "containerd" + +# install helm chart on controller[0] only +- name: prepare csl_excat helm chart + when: inventory_hostname == groups['kube_control_plane'][0] + block: + - name: patch node-role after kubernetes 1.24 + ansible.builtin.replace: + path: "{{ (csl_home, 'deployments/helm', 'values.yaml') | path_join }}" + regexp: 'node-role.kubernetes.io/master' + replace: 'node-role.kubernetes.io/control-plane' + when: kube_version is version('v1.24', '>=') + + - name: patch node-role after kubernetes 1.27 + ansible.builtin.blockinfile: + path: "{{ (csl_home, 'deployments/helm', 'values.yaml') | path_join }}" + insertafter: "effect: NoSchedule" + block: | + #### + - key: node-role.kubernetes.io/master + effect: NoSchedule + when: kube_version is version('v1.27', '>=') + + - name: install csl_excat helm chart + kubernetes.core.helm: + chart_ref: "{{ (csl_home, 'deployments/helm') | path_join }}" + release_name: "csl-excat" + release_namespace: "csl-excat" + create_namespace: true + values_files: "{{ (csl_home, 'deployments/helm', 'values.yaml') | path_join }}" + wait: true + timeout: 2m0s diff --git a/roles/intel_csl_excat/tasks/preflight.yml b/roles/intel_csl_excat/tasks/preflight.yml new file mode 100644 index 00000000..d2d302f8 --- /dev/null +++ b/roles/intel_csl_excat/tasks/preflight.yml @@ -0,0 +1,44 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +- name: check csl_excat container runtime requirements + assert: + that: + - container_runtime == "containerd" or container_runtime == "crio" + fail_msg: "Intel CSL EXCAT feature requires container_runtime set to 'containerd' or 'crio'" + success_msg: "Intel CSL EXCAT container runtime uequirements verified" + +- name: check if csl_excat tar ball exists + delegate_to: localhost + become: false + stat: + path: "{{ csl_excat_tar_staging_location }}" + checksum_algorithm: sha256 + register: provided_csl + +- name: assert if no tar ball + assert: + that: "provided_csl.stat.checksum | default('')" + msg: + - File {{ csl_excat_tar_staging_location }} on localhost does not exist. + - Please refer to group_vars/all.yml and search intel_csl_excat for how to get this tar ball. + +- name: check the csl_excat image integrity + assert: + that: "provided_csl.stat.checksum | default('') == '{{ csl_excat_image_checksum }}'" + msg: + - File {{ csl_excat_tar_staging_location }} on localhost is NOT the expected one. + - Please provide the correct file. diff --git a/roles/intel_dp_operator/tasks/main.yml b/roles/intel_dp_operator/tasks/main.yml index bffb452f..d35b523d 100644 --- a/roles/intel_dp_operator/tasks/main.yml +++ b/roles/intel_dp_operator/tasks/main.yml @@ -63,7 +63,7 @@ namespace: "{{ intel_dp_namespace }}" wait: yes wait_condition: - reason: MinimumReplicasAvailable - type: Available + reason: NewReplicaSetAvailable + type: Progressing wait_timeout: 240 when: inventory_hostname == groups['kube_control_plane'][0] diff --git a/roles/intel_eci/tasks/eci_host.yml b/roles/intel_eci/tasks/eci_host.yml new file mode 100644 index 00000000..33351db9 --- /dev/null +++ b/roles/intel_eci/tasks/eci_host.yml @@ -0,0 +1,41 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +- name: Apply CAT + block: + - name: Install CAT tools + ansible.builtin.apt: + name: "intel-cmt-cat" + state: present + + - name: Configure Cache Allocation Technology (CAT) in VM Host + ansible.builtin.shell: | # noqa command-instead-of-shell + pqos -R + pqos -e '{{ cat_define }}' + pqos -a '{{ cat_affinity }}' + register: shell_result + changed_when: true + when: cat_enable + +- name: Wait for VM + ansible.builtin.debug: + msg: "After install eci rt kernel, host will reboot. To avoid accessing VM before it starts, just wait for a moment." + +- name: Wait for VM to become available + ansible.builtin.wait_for: + host: "{{ item.name }}" + port: 22 + timeout: 180 + with_items: "{{ vms }}" diff --git a/roles/intel_eci/tasks/handle-RT.yml b/roles/intel_eci/tasks/handle-RT.yml new file mode 100644 index 00000000..fc31db9d --- /dev/null +++ b/roles/intel_eci/tasks/handle-RT.yml @@ -0,0 +1,45 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +- name: Gather the package facts + ansible.builtin.package_facts: + manager: auto + +- name: patch RT kernel cmdline if excat_dp is on + ansible.builtin.replace: + path: "{{ item }}" + regexp: 'rdt=!l3cat,!l2cat' + replace: ' ' + loop: + - /etc/grub.d/08_eci-tcc + - /boot/grub/grub.cfg + when: + - excat_dp_enabled | default(false) + - "'tcc-tools-grub' in ansible_facts.packages" + +- name: reboot into RT kernel + ansible.builtin.reboot: + +- name: re-gather o/s facts + ansible.builtin.setup: + filter: + - 'ansible_kernel' + +- name: check RT kernel + ansible.builtin.assert: + that: "'intel-ese-standard-lts-rt' in ansible_kernel" + fail_msg: "System failed to boot the RT kernel. Detected '{{ ansible_kernel }}' kernel" + success_msg: "Assertion passed. Kernel is now '{{ ansible_kernel }}'" diff --git a/roles/intel_eci/tasks/main.yml b/roles/intel_eci/tasks/main.yml index e687307b..27a42a19 100644 --- a/roles/intel_eci/tasks/main.yml +++ b/roles/intel_eci/tasks/main.yml @@ -14,7 +14,34 @@ ## limitations under the License. ## --- -- debug: msg="Entering Intel ECI Role" +- ansible.builtin.debug: msg="Entering Intel ECI Role" + +- block: + - name: Get ECI Package from Command line + ansible.builtin.debug: + var: eci_package + + - name: Check Command line ECI packages + ansible.builtin.debug: + msg: eci package name invalid. + when: "eci_package not in eci_packages" + + # Overwrite intel_eci + - name: Set value + ansible.builtin.set_fact: + intel_eci: "{{ intel_eci | combine({ item : true}) }}" + when: eci_package == item + loop: "{{ eci_packages }}" + + - name: Set value + ansible.builtin.set_fact: + intel_eci: "{{ intel_eci | combine({ item : false}) }}" + when: eci_package != item + loop: "{{ eci_packages }}" + when: eci_package is defined + +- ansible.builtin.debug: + var: intel_eci - name: add the ECI APT key ansible.builtin.apt_key: @@ -47,19 +74,8 @@ include_role: name: install_dependencies -- name: reboot into RT kernel - ansible.builtin.reboot: - -- name: re-gather o/s facts - ansible.builtin.setup: - filter: - - 'ansible_kernel' - -- name: check RT kernel - ansible.builtin.assert: - that: "'intel-ese-standard-lts-rt' in ansible_kernel" - fail_msg: "System failed to boot the RT kernel. Detected '{{ ansible_kernel }}' kernel" - success_msg: "Assertion passed. Kernel is now '{{ ansible_kernel }}'" +- name: handle RT kernel + include_tasks: handle-RT.yml - name: install ECI meta-packages ansible.builtin.apt: @@ -69,6 +85,16 @@ with_items: "{{ intel_eci | dict2items }}" when: item.value +# need handle RT kernel again due to above meta-package installation +- name: handle RT kernel + include_tasks: handle-RT.yml + +- name: Install OpenPLC + include_tasks: openplc.yml + when: + - intel_eci | dict2items | map(attribute='value') | select('==', true) | list | length != 0 + - ethercat_mac | default("") != "" + - name: deploy Codesys OPC UA Client block: - name: install packages for Codesys OPC UA Client @@ -88,17 +114,18 @@ - codesys_runtime.rc != 0 - '"Codesys preparation complete" not in codesys_runtime.stdout' + - name: gather service facts + ansible.builtin.service_facts: + - name: restart docker service (codesys_native script killed it) ansible.builtin.service: name: docker state: restarted - - - name: gather service facts - ansible.builtin.service_facts: + when: ansible_facts.services.docker.state | default("") != "" # sudo systemctl status codesyscontrol - name: print codesyscontrol status - debug: + ansible.builtin.debug: var: ansible_facts.services.codesyscontrol.state - name: check codesyscontrol status @@ -107,7 +134,7 @@ success_msg: "Assertion passed. The codesyscontrol service is active (running)" fail_msg: "The codesyscontrol service is in {{ ansible_facts.services.codesyscontrol.state }} state (not running)" - - debug: msg="Intel ECI with Codesys OPC UA Client is ready on target '{{ inventory_hostname }}'" + - ansible.builtin.debug: msg="Intel ECI with Codesys OPC UA Client is ready on target '{{ inventory_hostname }}'" when: opcua_framework.codesys_opcua_client | bool - name: deploy Standalone OPC UA Server @@ -156,5 +183,5 @@ success_msg: "Assertion passed. The OPC UA Server is running as 'opcsvr' process with PID {{ opcsvr_pid.stdout }}" fail_msg: "The OPC UA Server failed to start. No 'opcsvr' process is running" - - debug: msg="Intel ECI with Standalone OPC UA Server is ready on target '{{ inventory_hostname }}'" + - ansible.builtin.debug: msg="Intel ECI with Standalone OPC UA Server is ready on target '{{ inventory_hostname }}'" when: opcua_framework.standalone_opcua_server | bool diff --git a/roles/intel_eci/tasks/openplc.yml b/roles/intel_eci/tasks/openplc.yml new file mode 100644 index 00000000..16683146 --- /dev/null +++ b/roles/intel_eci/tasks/openplc.yml @@ -0,0 +1,125 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +- name: Check if Ethercat Has Been Installed + ansible.builtin.command: /etc/init.d/ethercat status + changed_when: false + register: first_check_ethercat_status + ignore_errors: yes + +- name: Configure Ethercat + block: + - name: Make Sure Ethercat Exists + ansible.builtin.stat: + path: /etc/sysconfig/ethercat + register: file_status + + - name: Assert File Exists + ansible.builtin.assert: + that: file_status.stat.exists + fail_msg: "File does not exist!" + + - name: Get BDF + ansible.builtin.shell: set -o pipefail && + ethtool -i "$( + ip link | grep -B 1 "{{ ethercat_mac }}" | head -n 1 | awk '{print $2}' | sed 's/.$//' + )" | grep bus-info | awk '{print $2}' + args: + executable: /bin/bash + register: bdf + changed_when: false + failed_when: (bdf.stderr | length > 0) and ('No such device' not in bdf.stderr) + + - name: Check the BDF + assert: + that: + - bdf.rc == 0 + fail_msg: >- + "Can't get the BDF of ethercat NIC. Please check configuration of ethercat_mac parameter in host_vars file for VM - {{ inventory_hostname }} + whether you filled in the right mac address of NIC to be used by ethercat inside VM. Please check that BDF for that NIC + was passthrough to VM - {{ inventory_hostname }} (vms section of host_vars file for vm_host)" + + - name: Show passthrough device information + ansible.builtin.debug: + msg: "BDF: {{ bdf.stdout }} ; Mac Address: {{ ethercat_mac }}" + + - name: Replace BDF + ansible.builtin.lineinfile: + path: /etc/sysconfig/ethercat + regexp: '^REBIND_NICS=' + line: "REBIND_NICS={{ bdf.stdout }}" + + - name: Replace Mac Address + ansible.builtin.lineinfile: + path: /etc/sysconfig/ethercat + regexp: '^MASTER0_DEVICE=' + line: "MASTER0_DEVICE={{ ethercat_mac }}" + + - name: Replace Device Modules + ansible.builtin.lineinfile: + path: /etc/sysconfig/ethercat + regexp: '^DEVICE_MODULES=' + line: 'DEVICE_MODULES="igb"' + + - name: Start Ethercat + ansible.builtin.command: /etc/init.d/ethercat start + changed_when: true + + - name: Get Ethercat Status + ansible.builtin.command: /etc/init.d/ethercat status + register: ethercat_status + changed_when: false + + - name: Check Ethercat Status + ansible.builtin.assert: + that: "'running' in ethercat_status.stdout" + fail_msg: "Ethercat Start Failed. Status: {{ ethercat_status.stdout }}" + success_msg: "Ethercat Start Successfully." + when: "'running' not in first_check_ethercat_status.stdout" + + +- name: Check if Openplc Runtime Has Been Installed + ansible.builtin.systemd: + name: openplc.service + register: first_check_openplc_status + +- name: Install Openplc Runtime + block: + - name: Change file permissions + ansible.builtin.file: + path: /opt/plcopen/openplc-runtime/install_openplc_runtime.sh + mode: u+x + + - name: Execute Install Script + ansible.builtin.command: /opt/plcopen/openplc-runtime/install_openplc_runtime.sh + register: shell_result + failed_when: shell_result.rc != 0 + changed_when: true + + - name: Start openplc Service + ansible.builtin.systemd: + state: started + name: openplc.service + + - name: Check if Openplc Runtime Start + ansible.builtin.systemd: + name: openplc.service + register: service_status + + - name: Check Openplc Runtime installed + ansible.builtin.assert: + that: "service_status.status.ActiveState == 'active'" + fail_msg: "Failed to install openplc runtime. You can check the last 20 lines log to debug" + when: first_check_openplc_status.status.ActiveState == "inactive" diff --git a/roles/intel_eci/vars/main.yml b/roles/intel_eci/vars/main.yml index ecbd4c5c..4cb1d23d 100644 --- a/roles/intel_eci/vars/main.yml +++ b/roles/intel_eci/vars/main.yml @@ -18,3 +18,4 @@ install_dependencies: Debian: - eci-customizations - linux-intel-rt +eci_packages: ['eci-process-automation', 'eci-manufacturing-equipment', 'eci-discrete-manufacturing'] diff --git a/roles/intel_ethernet_operator/defaults/main.yml b/roles/intel_ethernet_operator/defaults/main.yml index 8b014be7..6bd8140b 100644 --- a/roles/intel_ethernet_operator/defaults/main.yml +++ b/roles/intel_ethernet_operator/defaults/main.yml @@ -15,27 +15,35 @@ ## --- intel_ethernet_operator_git: "https://github.com/intel/intel-ethernet-operator.git" -intel_ethernet_operator_git_ref: "v22.11" +intel_ethernet_operator_git_ref: "v23.08" # expected format (v)X.Y(.Z) intel_ethernet_operator_dir: "{{ (project_root_dir, 'intel-ethernet-operator') | path_join }}" +# Build of IEO requires version supplied to follow semantic: MAJOR.MINOR.PATCH consiting only of 0-9 digits +intel_ethernet_operator_build_version: >- + {{ + intel_ethernet_operator_git_ref | regex_replace('^v(.*)$', '\1') + | split('.') | map('int') | list | join('.') + | regex_replace('^([0-9]+).([0-9]+)$', '\1.\2.0') + }} + uft_git: "https://github.com/intel/UFT.git" uft_git_ref: "v22.11" uft_dir: "{{ (project_root_dir, 'uft') | path_join }}" uft_image: "uft" uft_image_ver: "{{ uft_git_ref }}" -dpdk_tag: "v22.07" +dpdk_tag: "v22.11" flow_config_resource_name: "cvl_uft_admin" flow_config_sriov_network_name: "sriov-cvl-dcf" intel_ethernet_operator_namespace: "intel-ethernet-operator" -intel_ethernet_operator_img_ver: "0.0.1" intel_ethernet_operator_make_tls: "false" intel_ethernet_operator_target_platform: "K8S" -intel_ethernet_operator_catalog_image: "{{ registry_local_address }}/intel-ethernet-operator-catalog:v{{ intel_ethernet_operator_img_ver }}" +intel_ethernet_operator_catalog_image: "{{ registry_local_address }}/intel-ethernet-operator-catalog:v{{ intel_ethernet_operator_build_version }}" -intel_ethernet_operator_cache_webserver_image: "{{ registry_local_address }}/intel-ethernet-operator-cache-webserver:v1.0" +intel_ethernet_operator_cache_webserver_image: >- + "{{ registry_local_address }}/intel-ethernet-operator-cache-webserver:{{ intel_ethernet_operator_build_version }}" intel_ethernet_operator_cache_webserver_name: "fwddp-cache-webserver" intel_ethernet_operator_files_dir: "{{ (project_root_dir, 'intel-ethernet-operator-files') | path_join }}" @@ -45,33 +53,5 @@ intel_ethernet_operator_ddp_files_dir: "{{ (intel_ethernet_operator_files_dir, ' intel_ethernet_operator_flow_config_files_dir: "{{ (intel_ethernet_operator_files_dir, 'flowconfig') | path_join }}" intel_ethernet_operator_flow_config_rules_dir: "{{ (intel_ethernet_operator_flow_config_files_dir, 'flow_config_rules') | path_join }}" -intel_ethernet_operator_fw_url: "https://downloadmirror.intel.com/759036/E810_NVMUpdatePackage_v4_10_Linux.tar.gz" -intel_ethernet_operator_fw_sum: "64598337c8730ee00239d728e17bdcb180bb8154" - -intel_ethernet_operator_ddp_urls: - 'ice_comms-1.3.17.0.pkg': https://downloadmirror.intel.com/29892/eng/ice_comms-1.3.17.0.zip - 'ice_comms-1.3.20.0.pkg': https://downloadmirror.intel.com/30028/eng/ice_comms-1.3.20.0.zip - 'ice_comms-1.3.22.0.pkg': https://downloadmirror.intel.com/30237/eng/ice_comms-1.3.22.0.zip - 'ice_comms-1.3.24.0.pkg': https://downloadmirror.intel.com/30335/eng/ice_comms-1.3.24.0.zip - 'ice_comms-1.3.28.0.pkg': https://downloadmirror.intel.com/30467/eng/800%20series%20comms%20binary%20package%201.3.28.0.zip - 'ice_comms-1.3.30.0.pkg': https://downloadmirror.intel.com/30590/eng/800%20series%20comms%20binary%20package%201.3.30.0.zip - 'ice_comms-1.3.30.0_rev1.1.pkg': https://downloadmirror.intel.com/29889/eng/800%20series%20comms%20binary%20package%201.3.30.0_rev1.1.zip - 'ice_comms-1.3.31.0.pkg': https://downloadmirror.intel.com/713853/800%20Series%20DDP%20Comms%20Package%201.3.31.0.zip - 'ice_comms-1.3.35.0.pkg': https://downloadmirror.intel.com/727568/ice_comms-1.3.35.0.zip - 'ice_comms-1.3.37.0.pkg': https://downloadmirror.intel.com/738733/800%20Series%20DDP%20Comms%20Package%201.3.37.0.zip - 'ice_comms-1.3.40.0.pkg': https://downloadmirror.intel.com/772040/800%20Series%20DDP%20for%20Comms%20Package%201.3.40.0.zip - - -# SHA-1 sums of DDP packages -intel_ethernet_operator_ddp_sums: - 'ice_comms-1.3.17.0.pkg': a10d5df30d34559b6050c807e0baf44364b35130 - 'ice_comms-1.3.20.0.pkg': 5208c100fee36ba47c90dec955d43e24c6039614 - 'ice_comms-1.3.22.0.pkg': eb5ffa506a77d15facdcc8226b2f3b0a410d6458 - 'ice_comms-1.3.24.0.pkg': 0717f03ab72c1531320ee84be78cdcb194d789ba - 'ice_comms-1.3.28.0.pkg': dc9ebe605ae1b151cbe67aa2588a0050992461ee - 'ice_comms-1.3.30.0.pkg': cd076e008972005d93be651286dd2955a17681d4 - 'ice_comms-1.3.30.0_rev1.1.pkg': 89eba571845bd80bfc11869cbcfce995edb9a2f9 - 'ice_comms-1.3.31.0.pkg': 5dbe3ae8d2ada5b78de05da150e5df5befb3bf75 - 'ice_comms-1.3.35.0.pkg': c61189b98bb116e05853f67ba21ca915416aef46 - 'ice_comms-1.3.37.0.pkg': e73d24bdf6b3c8fe46b52ccc31ee534034b0b3e0 - 'ice_comms-1.3.40.0.pkg': 8fcc3eab682f2a023ae49831669e42e63bf05f8f +intel_ethernet_operator_fw_url: "https://downloadmirror.intel.com/786047/E810_NVMUpdatePackage_v4_30_Linux.tar.gz" +intel_ethernet_operator_fw_sum: "993d79ac623b71c5378855738917495a0fa8ffb8" diff --git a/roles/intel_ethernet_operator/tasks/cache_server.yml b/roles/intel_ethernet_operator/tasks/cache_server.yml index fb134ad4..0fafe256 100644 --- a/roles/intel_ethernet_operator/tasks/cache_server.yml +++ b/roles/intel_ethernet_operator/tasks/cache_server.yml @@ -39,61 +39,31 @@ ansible.builtin.set_fact: is_fw_update_enabled: "{{ is_fw_update | json_query('results[*].msg') | replace('\"', '') | replace('\\n', '') }}" -- block: - - block: - - name: download DDP archives - command: "wget {{ item.value }}" # noqa command-instead-of-module - changed_when: true - args: - chdir: "{{ intel_ethernet_operator_packages_dir }}" - loop: "{{ lookup('ansible.builtin.dict', intel_ethernet_operator_ddp_urls) }}" - - - name: find DDP archives - ansible.builtin.find: - paths: "{{ intel_ethernet_operator_packages_dir }}" - use_regex: yes - patterns: "800" - register: ddp_archives - - - name: unarchive DDP packages - ansible.builtin.unarchive: - src: "{{ item }}" - dest: "{{ intel_ethernet_operator_packages_dir }}" - remote_src: true - loop: "{{ ddp_archives.files | map(attribute='path') | list }}" - - - name: remove DDP archives - ansible.builtin.file: - path: "{{ item }}" - state: absent - loop: "{{ ddp_archives.files | map(attribute='path') | list }}" - - - name: find DDP files to remove - ansible.builtin.find: - paths: "{{ intel_ethernet_operator_packages_dir }}" - patterns: "*.pdf" - register: ddp_pdf - - - name: remove DDP files - ansible.builtin.file: - path: "{{ item }}" - state: absent - loop: "{{ ddp_pdf.files | map(attribute='path') | list }}" +- name: Prepare Cache server + when: '"True" in is_ddp_update_enabled or "True" in is_fw_update_enabled' + block: + - name: Download DDP packages + vars: + nic_driver: ice + ddp_download_dir_override: "{{ intel_ethernet_operator_packages_dir }}" + ansible.builtin.include_role: + name: install_ddp_pkgs + tasks_from: download.yml when: '"True" in is_ddp_update_enabled' - - name: download FW package + - name: Download FW package ansible.builtin.get_url: url: "{{ intel_ethernet_operator_fw_url }}" dest: "{{ intel_ethernet_operator_packages_dir }}/E810_NVMUpdatePackage_v4_10_Linux.tar.gz" mode: '0644' when: '"True" in is_fw_update_enabled' - - name: create Dockerfile + - name: Create Dockerfile ansible.builtin.copy: dest: "{{ (intel_ethernet_operator_files_dir, 'Dockerfile') | path_join }}" mode: '0644' content: | - FROM nginx:1.23.2 + FROM nginx:1.24.0 COPY packages /usr/share/nginx/html - block: @@ -107,7 +77,6 @@ ansible.builtin.command: docker push {{ intel_ethernet_operator_cache_webserver_image }} changed_when: true when: - - inventory_hostname == groups['kube_control_plane'][0] - '"docker" in container_runtime' - block: @@ -121,7 +90,6 @@ ansible.builtin.command: podman push {{ intel_ethernet_operator_cache_webserver_image }} changed_when: true when: - - inventory_hostname == groups['kube_control_plane'][0] - '"docker" not in container_runtime' - name: populate cache webserver yaml files and push to controller node @@ -165,4 +133,3 @@ -o jsonpath='{.spec.clusterIP}' changed_when: false register: cache_server_clusterip - when: '"True" in is_ddp_update_enabled or "True" in is_fw_update_enabled' diff --git a/roles/intel_ethernet_operator/tasks/ddp.yml b/roles/intel_ethernet_operator/tasks/ddp.yml index 2706fc93..076fabee 100644 --- a/roles/intel_ethernet_operator/tasks/ddp.yml +++ b/roles/intel_ethernet_operator/tasks/ddp.yml @@ -14,6 +14,12 @@ ## limitations under the License. ## --- +- name: Find DDP packages atributes + ansible.builtin.find: + path: "{{ intel_ethernet_operator_packages_dir }}" + get_checksum: true + register: ddp_checksums + - name: DDP update when: hostvars[node_name]['intel_ethernet_operator']['ddp_update'] |d(false) block: @@ -59,7 +65,7 @@ until: ddp_update_api_info.status == 200 retries: 60 delay: 5 - + always: # check for status even if reboot process not started or failed so more information is gathered about status of update - name: Check status of DDP update kubernetes.core.k8s_info: kind: EthernetNodeConfig @@ -85,14 +91,6 @@ src: "{{ (intel_ethernet_operator_ddp_files_dir, node_name + '-ddp-update.yml') | path_join }}" state: absent - - name: Remove EthernetNodeConfig after update - kubernetes.core.k8s: - name: "{{ hostvars[node_name]['ansible_hostname'] }}" - kind: EthernetNodeConfig - namespace: "{{ intel_ethernet_operator_namespace }}" - state: absent - wait: true - - name: Reload nic modules block: - name: register mgmt driver diff --git a/roles/intel_ethernet_operator/tasks/ethernet_operator.yml b/roles/intel_ethernet_operator/tasks/ethernet_operator.yml index acb20f15..eda1c982 100644 --- a/roles/intel_ethernet_operator/tasks/ethernet_operator.yml +++ b/roles/intel_ethernet_operator/tasks/ethernet_operator.yml @@ -26,31 +26,22 @@ register: gopath changed_when: false -# Workaoround to get IEO daemon image build working -- name: patch Intel Ethernet Operator Daemon Dockerfile - ansible.builtin.replace: - path: "{{ intel_ethernet_operator_dir }}/Dockerfile.daemon" - regexp: 'libbpf\.so\.0' - replace: 'libbpf.so.1' - -# Workaround for IEO labeler build issue -- name: patch IEO Labeler to fix curl issue - lineinfile: - path: "{{ intel_ethernet_operator_dir }}/Dockerfile.labeler" - regexp: "^RUN apk update && apk add --no-cache curl.*" - line: "RUN apk update && apk add --no-cache curl" - - name: build Intel Ethernet Operator + vars: + # in case of RHEL 9.2 & podman, SYS_CHROOT is missing from default caps. + # no need to limit only to RHEL OS, no impact to Ubuntu + podman_build_args: "--cap-add SYS_CHROOT" community.general.make: target: "{{ item }}" chdir: "{{ intel_ethernet_operator_dir }}" params: - VERSION: "{{ intel_ethernet_operator_img_ver }}" + VERSION: "{{ intel_ethernet_operator_build_version }}" IMAGE_REGISTRY: "{{ registry_local_address }}" IMGTOOL: "{{ 'docker' if container_runtime == 'docker' else 'podman' }}" TLS_VERIFY: "{{ intel_ethernet_operator_make_tls }}" TARGET_PLATFORM: "{{ intel_ethernet_operator_target_platform }}" UFT_IMAGE: "{{ uft_image }}:{{ uft_image_ver }}" + DOCKERARGS: "{{ podman_build_args if container_runtime in ['crio', 'containerd'] else omit }}" register: ieo_build_status retries: 5 delay: 120 diff --git a/roles/intel_ethernet_operator/templates/cache-server-svc.yml.j2 b/roles/intel_ethernet_operator/templates/cache-server-svc.yml.j2 index 933cf977..124b69a5 100644 --- a/roles/intel_ethernet_operator/templates/cache-server-svc.yml.j2 +++ b/roles/intel_ethernet_operator/templates/cache-server-svc.yml.j2 @@ -5,6 +5,7 @@ metadata: namespace: {{ intel_ethernet_operator_namespace }} labels: app: {{ intel_ethernet_operator_cache_webserver_name }} + version: {{ intel_ethernet_operator_git_ref }} spec: selector: app: {{ intel_ethernet_operator_cache_webserver_name }} diff --git a/roles/intel_ethernet_operator/templates/cache-server.yml.j2 b/roles/intel_ethernet_operator/templates/cache-server.yml.j2 index 3831692f..49a01c4f 100644 --- a/roles/intel_ethernet_operator/templates/cache-server.yml.j2 +++ b/roles/intel_ethernet_operator/templates/cache-server.yml.j2 @@ -5,6 +5,7 @@ metadata: namespace: {{ intel_ethernet_operator_namespace }} labels: app: {{ intel_ethernet_operator_cache_webserver_name }} + version: {{ intel_ethernet_operator_git_ref }} spec: selector: matchLabels: diff --git a/roles/intel_ethernet_operator/templates/catalog.yml.j2 b/roles/intel_ethernet_operator/templates/catalog.yml.j2 index 909a2d72..cbdb14a9 100644 --- a/roles/intel_ethernet_operator/templates/catalog.yml.j2 +++ b/roles/intel_ethernet_operator/templates/catalog.yml.j2 @@ -3,6 +3,9 @@ kind: CatalogSource metadata: name: intel-ethernet-operators namespace: {{ intel_ethernet_operator_namespace }} + labels: + app: intel-ethernet-operator + version: {{ intel_ethernet_operator_git_ref }} spec: sourceType: grpc image: {{ intel_ethernet_operator_catalog_image }} diff --git a/roles/intel_ethernet_operator/templates/ddp-update.yml.j2 b/roles/intel_ethernet_operator/templates/ddp-update.yml.j2 index 4a50a68e..bd73257a 100644 --- a/roles/intel_ethernet_operator/templates/ddp-update.yml.j2 +++ b/roles/intel_ethernet_operator/templates/ddp-update.yml.j2 @@ -1,20 +1,23 @@ {%- for iface in hostvars[node_name]['dataplane_interfaces'] %} {%- if iface.ddp_profile is defined and iface.bus_info.endswith(':00.0') and iface.pf_driver == "ice" %} -{%- set intel_ethernet_operator_ddp_package = iface.ddp_profile %} -{%- set intel_ethernet_operator_ddp_package_sum = intel_ethernet_operator_ddp_sums[iface.ddp_profile]%} +{%- set ddp_package = (iface.ddp_profile | regex_replace('.pkg', '.zip')) %} +{%- set ddp_package_checksum = (ddp_checksums.files | selectattr('path', 'search', ddp_package) | map(attribute='checksum') | list | first ) %} --- apiVersion: ethernet.intel.com/v1 kind: EthernetClusterConfig metadata: name: {{ hostvars[node_name]['ansible_hostname'] }}-{{ iface.name }}-ddp-config namespace: "{{ intel_ethernet_operator_namespace }}" + labels: + app: intel-ethernet-operator + version: {{ intel_ethernet_operator_git_ref }} spec: nodeSelectors: kubernetes.io/hostname: {{ hostvars[node_name]['ansible_hostname'] }} deviceSelector: pciAddress: "0000:{{ iface.bus_info }}" deviceConfig: - ddpURL: "http://{{ cache_server_clusterip.stdout }}/{{ intel_ethernet_operator_ddp_package | regex_replace('.pkg', '.zip') }}" - ddpChecksum: "{{ intel_ethernet_operator_ddp_package_sum }}" + ddpURL: "http://{{ cache_server_clusterip.stdout }}/{{ ddp_package }}" + ddpChecksum: "{{ ddp_package_checksum }}" {% endif %} {% endfor %} diff --git a/roles/intel_ethernet_operator/templates/firmware-update.yml.j2 b/roles/intel_ethernet_operator/templates/firmware-update.yml.j2 index b21fdbb2..b4f306d6 100644 --- a/roles/intel_ethernet_operator/templates/firmware-update.yml.j2 +++ b/roles/intel_ethernet_operator/templates/firmware-update.yml.j2 @@ -6,13 +6,16 @@ kind: EthernetClusterConfig metadata: name: {{ hostvars[node_name]['ansible_hostname'] }}-{{ iface.name }}-fw-config namespace: "{{ intel_ethernet_operator_namespace }}" + labels: + app: intel-ethernet-operator + version: {{ intel_ethernet_operator_git_ref }} spec: nodeSelectors: kubernetes.io/hostname: {{ hostvars[node_name]['ansible_hostname'] }} deviceSelector: pciAddress: "0000:{{ iface.bus_info }}" deviceConfig: - fwURL: "http://{{ cache_server_clusterip.stdout }}/E810_NVMUpdatePackage_v4_10_Linux.tar.gz" + fwURL: "http://{{ cache_server_clusterip.stdout }}/E810_NVMUpdatePackage_v4_30_Linux.tar.gz" fwChecksum: "{{ intel_ethernet_operator_fw_sum }}" {% endif %} {% endfor %} diff --git a/roles/intel_ethernet_operator/templates/operator-group.yml.j2 b/roles/intel_ethernet_operator/templates/operator-group.yml.j2 index e2110815..7b13b922 100644 --- a/roles/intel_ethernet_operator/templates/operator-group.yml.j2 +++ b/roles/intel_ethernet_operator/templates/operator-group.yml.j2 @@ -3,6 +3,9 @@ kind: OperatorGroup metadata: name: intel-ethernet-operator namespace: {{ intel_ethernet_operator_namespace }} + labels: + app: intel-ethernet-operator + version: {{ intel_ethernet_operator_git_ref }} spec: targetNamespaces: - {{ intel_ethernet_operator_namespace }} diff --git a/roles/intel_ethernet_operator/templates/subscription.yml.j2 b/roles/intel_ethernet_operator/templates/subscription.yml.j2 index 9d672caf..8babe8fb 100644 --- a/roles/intel_ethernet_operator/templates/subscription.yml.j2 +++ b/roles/intel_ethernet_operator/templates/subscription.yml.j2 @@ -3,6 +3,9 @@ kind: Subscription metadata: name: intel-ethernet-subscription namespace: {{ intel_ethernet_operator_namespace }} + labels: + app: intel-ethernet-operator + version: {{ intel_ethernet_operator_git_ref }} spec: {% if http_proxy is defined or https_proxy is defined %} config: diff --git a/roles/intel_flexran/defaults/main.yml b/roles/intel_flexran/defaults/main.yml index 60176ffa..14135fa3 100644 --- a/roles/intel_flexran/defaults/main.yml +++ b/roles/intel_flexran/defaults/main.yml @@ -23,17 +23,15 @@ # - include_role: dpdk # when: dpdk_dir is not defined # to allow tagged execution - +dpdk_dir: "{{ (project_root_dir, 'dpdk-' + dpdk_version) | path_join }}" # Intel FlexRAN # intel_flexran_repo: "Intel’s Developer Zone Portal aka RDC" # intel_flexran_token: "not public. pkg access requires NDA. see docs/flexran_guide.md" intel_flexran_staging_location: "/tmp/flexran/" # a directory on localhost (ansible host) -intel_flexran_ver: "23.03" # "22.03" (RA22.06) "22.07" (RA22.08) "22.07.3" (RA22.11) "22.11" (RA23.02) "23.03" (RA23.07) +intel_flexran_ver: "23.07" # "22.03" (RA22.06) "22.07" (RA22.08) "22.07.3" (RA22.11) "22.11" (RA23.02) "23.03" (RA23.07) "23.07" (RA23.10) intel_flexran_pod_version_icx_sp: "22.07" # (RA23.07) -intel_flexran_pod_version_spr_ee: "23.03" # (RA23.07) +intel_flexran_pod_version_spr_ee: "23.07" # "23.03"(RA23.07) intel_flexran_namespace: "default" -# intel_flexran_tarball: "FlexRAN-22.03.tar.gz" -# intel_flexran_tar_chk: "65e59ac1295ef392f54b80047db2efe458962fc78e5d84c5d54703439a364cda" # SHA256 intel_flexran_dir: "{{ (project_root_dir, 'intel-flexran') | path_join }}" intel_flexran_files_dir: "{{ (project_root_dir, 'intel-flexran-files') | path_join }}" # for FEC ACC CRs, kernel cmdline, etc intel_flexran_dpdk_ver: "22.11.1" # "21.11" for FlexRAN 22.03, 22.07, 22.07.3, 22.11 @@ -43,9 +41,7 @@ intel_flexran_dpdk_zip: "dpdk_patch-{{ intel_flexran_ver }}.patch.zip" intel_flexran_dpdk_zip_chk: "dab1a0c3a0530be9904d62d3c3f4f88166b73360dcc11402500070237000f468" # SHA256 for dpdk_patch-22.07.3.patch.zip intel_flexran_dpdk_patch: "dpdk_patch-{{ intel_flexran_ver }}.patch" -# intel_flexran_dpdk_patch_chk: "bd136f939609545d70e4d6a8dca83a1550d6a28a0b6b70fdfb10d8283922f4c5" # SHA256 for dpdk_patch-22.07.3.patch -# intel_flexran_dpdk_patch_chk: "d7943817a04d58ee3a78a36b2cbbaa18f45df15402f6566e904648820e657ad7" # SHA256 for dpdk_patch-22.11.patch -intel_flexran_dpdk_patch_chk: "91b4d1911568f59d349adc0152d0d60fb4c5fd927373d021f7de188ba20f1267" # SHA256 for dpdk_patch-23.03.patch +intel_flexran_dpdk_patch_chk: "fd80a13c454cd7930ada3704306d29bf8c3acfef2630b7d410f6c215d32028ff" # SHA256 for dpdk_patch-23.07.patch intel_flexran_patch: "FlexRAN-R{{ intel_flexran_ver }}.zip" intel_flexran_patch_chk: "1089d1bd3d86fe2f2198c497fa26e6f9322fd867f5f6ece087190499ff427593" # SHA256 for FlexRAN-R22.07.3.zip @@ -53,3 +49,6 @@ intel_flexran_patch_chk: "1089d1bd3d86fe2f2198c497fa26e6f9322fd867f5f6ece0871904 intel_pfbb_version: "v23.03" inih_version: "r44" + +rt_test_repo: "https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git" +rt_test_version: "v2.5" diff --git a/roles/intel_flexran/files/kernel_cmdline_gen.sh b/roles/intel_flexran/files/kernel_cmdline_gen.sh index 520aec32..b9a2844b 100644 --- a/roles/intel_flexran/files/kernel_cmdline_gen.sh +++ b/roles/intel_flexran/files/kernel_cmdline_gen.sh @@ -2,37 +2,38 @@ threads_per_core=$(lscpu | grep "Thread(s) per core" | awk -F ':' '{print $2}' | xargs) cores_per_socket=$(lscpu | grep "Core(s) per socket" | awk -F ':' '{print $2}' | xargs) -numa=$(lscpu | grep "NUMA node(s)" |awk -F ':' '{print $2}' | xargs) +socket=$(lscpu | grep "Socket(s)" |awk -F ':' '{print $2}' | xargs) -# if cores_per_socket < 2, which means no cores available to isolate for realtime app. -# and isolcpus will results in 1-0, keep it here for test purpose. -# On socket 0, core 0 and its sibling thread core will be kept for housekeeping +# On socket 0, core 0-1 and its sibling thread core will be kept for housekeeping. +# On socket 1, the first two cores and its sibling will be kept for housekeeping # all the other cores will be isolated -# No cores isolated from socket 1. -# NUMA node0 CPU(s): 0-27,56-83 -# NUMA node1 CPU(s): 28-55,84-111 -if [ "$numa" == "1" ] ; then - if [ "$threads_per_core" == "2" ] ; then - isolcpus="1-$(( cores_per_socket - 1 )),$(( cores_per_socket + 1 ))-$(( cores_per_socket * 2 - 1 ))" - housekeeping="0,$cores_per_socket" +# Set isolcpus and housekeeping +if [ "$socket" == "1" ] ; then + if [ "$threads_per_core" == "2" ] ; then + isolcpus="2-$(( cores_per_socket - 1 )),$(( cores_per_socket + 2 ))-$(( cores_per_socket * 2 - 1 ))" + housekeeping="0-1,$(( cores_per_socket ))-$(( cores_per_socket + 1 ))" else - isolcpus="1-$(( cores_per_socket - 1 ))" - housekeeping="0" + isolcpus="2-$(( cores_per_socket - 1 ))" + housekeeping="0-1" fi -elif [ "$numa" == "2" ]; then +elif [ "$socket" == "2" ]; then if [ "$threads_per_core" == "2" ] ; then - isolcpus="1-$(( cores_per_socket - 1 )),$(( cores_per_socket * 2 + 1 ))-$(( cores_per_socket * 3 - 1 ))" - housekeeping="0,$(( cores_per_socket * 2 )),$(( cores_per_socket ))-$(( cores_per_socket * 2 - 1 )),$(( cores_per_socket * 3 ))-$(( cores_per_socket * 4 - 1 ))" - + isolcpus="2-$(( cores_per_socket - 1 )),$(( cores_per_socket + 2 ))-$(( cores_per_socket * 2 - 1 )),$(( cores_per_socket * 2 + 2 ))-$(( cores_per_socket * 3 - 1 )),$(( cores_per_socket * 3 + 2 ))-$(( cores_per_socket * 4 - 1 ))" + housekeeping="0-1,$(( cores_per_socket ))-$(( cores_per_socket + 1 )),$(( cores_per_socket * 2 ))-$(( cores_per_socket * 2 + 1 )),$(( cores_per_socket * 3 ))-$(( cores_per_socket * 3 + 1 ))" else - - isolcpus="1-$(( cores_per_socket - 1 ))" - housekeeping="0,$(( cores_per_socket ))-$(( cores_per_socket * 2 - 1 ))" + isolcpus="2-$(( cores_per_socket - 1 )),$(( cores_per_socket + 2 ))-$(( cores_per_socket * 2 - 1 ))" + housekeeping="0-1,$(( cores_per_socket ))-$(( cores_per_socket + 1 ))" fi fi -flexran_kernel_cmdline="intel_iommu=on iommu=pt usbcore.autosuspend=-1 selinux=0 enforcing=0 nmi_watchdog=0 crashkernel=auto softlockup_panic=0 audit=0 cgroup_disable=memory tsc=nowatchdog intel_pstate=disable mce=off hugepagesz=1G hugepages=40 hugepagesz=2M hugepages=0 default_hugepagesz=1G kthread_cpus=$housekeeping irqaffinity=$housekeeping nohz=on nosoftlockup nohz_full=$isolcpus rcu_nocbs=$isolcpus rcu_nocb_poll skew_tick=1 isolcpus=$isolcpus" +# Set hugepage size +if [ "$cores_per_socket" -lt "32" ] ; then + pagesize="40" +else + pagesize="60" +fi +flexran_kernel_cmdline="hugepagesz=1G hugepages=$pagesize hugepagesz=2M hugepages=0 default_hugepagesz=1G nmi_watchdog=0 softlockup_panic=0 intel_iommu=on iommu=pt vfio_pci.enable_sriov=1 vfio_pci.disable_idle_d3=1 rcu_nocbs=$isolcpus irqaffinity=$housekeeping isolcpus=managed_irq,domain,$isolcpus kthread_cpus=$housekeeping nohz_full=$isolcpus crashkernel=auto enforcing=0 quiet rcu_nocb_poll rhgb selinux=0 mce=off audit=0 pci=realloc pci=assign-busses rdt=l3cat skew_tick=1 nosoftlockup nohz=on" echo "$flexran_kernel_cmdline" diff --git a/roles/intel_flexran/tasks/bind_fec.yml b/roles/intel_flexran/tasks/bind_fec.yml index 92dcb71f..8d4decd2 100644 --- a/roles/intel_flexran/tasks/bind_fec.yml +++ b/roles/intel_flexran/tasks/bind_fec.yml @@ -63,12 +63,17 @@ - debug: msg="fec_acc pciid is {{ fec_acc | regex_replace('^ip-', '') }}" -- name: bind FEC ACC physical device +- name: bind FEC ACC physical device to igb_uio driver ansible.builtin.command: "{{ (intel_flexran_dpdk_dir, 'usertools', 'dpdk-devbind.py -b igb_uio ' + fec_acc) | path_join }}" -# ansible.builtin.command: date # "dpdk-devbind.py -b igb_uio {{ fec_acc }}" -# chdir: "{{ (intel_flexran_dpdk_dir, 'usertools') | path_join }}" register: fec_bind_set changed_when: fec_bind_set.rc == 0 + when: fec_dev != "acc200" or intel_flexran_type != "pod" + +- name: bind FEC ACC physical device to vfio-pci driver + ansible.builtin.command: "{{ (intel_flexran_dpdk_dir, 'usertools', 'dpdk-devbind.py -b vfio-pci ' + fec_acc) | path_join }}" + register: fec_bind_set + changed_when: fec_bind_set.rc == 0 + when: fec_dev == "acc200" and intel_flexran_type == "pod" - name: show DPDK devices binding status ansible.builtin.command: "{{ (intel_flexran_dpdk_dir, 'usertools', 'dpdk-devbind.py -s') | path_join }}" @@ -84,7 +89,7 @@ name: vfio-pci state: present - - name: reset FEC VFs + - name: reset FEC VFs for acc100 # ansible.builtin.copy: # dest: "{{ ('/sys/bus/pci/devices/', fec_acc, 'max_vfs') | path_join }}" # content: "0" @@ -98,8 +103,18 @@ register: set_fec_max_vfs changed_when: set_fec_max_vfs.rc == 0 failed_when: set_fec_max_vfs.rc != 0 + when: fec_dev == "acc100" - - name: instate one FEC VF + - name: reset FEC VFs for acc200 + ansible.builtin.shell: "echo 0 > {{ ('/sys/bus/pci/devices/', fec_acc, 'sriov_numvfs') | path_join }}" # noqa command-instead-of-shell + args: + executable: /bin/bash + register: set_fec_max_vfs + changed_when: set_fec_max_vfs.rc == 0 + failed_when: set_fec_max_vfs.rc != 0 + when: fec_dev == "acc200" + + - name: instate one FEC VF for acc100 # ansible.builtin.copy: # dest: "{{ ('/sys/bus/pci/devices/', fec_acc, 'max_vfs') | path_join }}" # content: "1" @@ -110,6 +125,16 @@ register: set_fec_max_vfs changed_when: set_fec_max_vfs.rc == 0 failed_when: set_fec_max_vfs.rc != 0 + when: fec_dev == "acc100" + + - name: instate one FEC VF for acc200 + ansible.builtin.shell: "echo 1 > {{ ('/sys/bus/pci/devices/', fec_acc, 'sriov_numvfs') | path_join }}" + args: + executable: /bin/bash + register: set_fec_max_vfs + changed_when: set_fec_max_vfs.rc == 0 + failed_when: set_fec_max_vfs.rc != 0 + when: fec_dev == "acc200" - name: probe for FEC VF shell: "set -o pipefail && lspci | grep -i acc | grep -i -E \"0d5d|57c1\"" # noqa command-instead-of-shell diff --git a/roles/intel_flexran/tasks/fec_acc.yml b/roles/intel_flexran/tasks/fec_acc.yml index ee6bffc2..a072f1dd 100644 --- a/roles/intel_flexran/tasks/fec_acc.yml +++ b/roles/intel_flexran/tasks/fec_acc.yml @@ -58,14 +58,30 @@ - name: configure ACC200 FEC device with pf_bb tool for FlexRAN in Docker POD block: + - name: set FEC device uuid + ansible.builtin.set_fact: + uuid: "00112233-4455-6677-8899-aabbccddeeff" - name: configure ACC200 FEC device with pf_bb tool for FlexRAN in Docker POD ansible.builtin.command: - cmd: "./pf_bb_config acc200 -c acc200/acc200_config_vf_5g.cfg" # select .cfg file as desired + cmd: "./pf_bb_config acc200 -v {{ uuid }} -c acc200/acc200_config_vf_5g.cfg" # select .cfg file as desired chdir: "{{ (intel_flexran_dir, 'source/pf-bb-config') | path_join }}" register: pf_bb_config changed_when: false - - debug: msg="{{ pf_bb_config.stdout }}" + - name: get the first FEC VF + ansible.builtin.shell: "set -o pipefail && lspci | grep -i acc | grep -i 57c1 | awk '{print $1}' | head -1" + args: + executable: /bin/bash + register: fec_first_vf + changed_when: false + - ansible.builtin.debug: msg="fec_first_vf is at {{ fec_first_vf.stdout }}" + - name: run basic bbdev test + ansible.builtin.command: >- + ./test-bbdev.py -e="-c 0xff0 -a 0000:{{ fec_first_vf.stdout }} + --vfio-vf-token={{ uuid }}" -t 6 -n 100 -b 80 -l 1 -c validation -v ./ldpc_dec_default.data + args: + chdir: "{{ dpdk_dir }}/app/test-bbdev" + changed_when: false when: - fec_dev == "acc200" - intel_flexran_type == "pod" diff --git a/roles/intel_flexran/tasks/flexran_preflight.yml b/roles/intel_flexran/tasks/flexran_preflight.yml index 04214ca8..be939424 100644 --- a/roles/intel_flexran/tasks/flexran_preflight.yml +++ b/roles/intel_flexran/tasks/flexran_preflight.yml @@ -275,22 +275,25 @@ when: - configured_arch == "spr" - # FlexRAN pod for SPR-EE MCC do not support core pinning. So need to disable k8s cpu manager. - - name: check k8s cpu manager is disabled for FlexRAN Pod on SPR-EE MCC + # check runtime for FlexRAN in POD on SPR + # needs containerd runtime to support non-root user to use FEC in pod + - debug: msg="Container runtime is set to {{ container_runtime }}" + - name: check runtime for FlexRAN in POD on SPR ansible.builtin.assert: - that: native_cpu_manager_enabled == false - fail_msg: "Please set 'native_cpu_manager_enabled' to 'false' in group vars to run FlexRAN pod on SPR-EE MCC" + that: container_runtime == "containerd" + fail_msg: "Deploying Intel FlexRAN as a POD on SPR needs containerd runtime to support non-root user to use FEC in pod. Please correct the group_vars configuration" + success_msg: "Assertion passed. Intel FlexRAN as a POD on SPR is supported and can be deployed on '{{ container_runtime }}' runtime" when: - configured_arch == "spr" - - native_cpu_manager_enabled is defined - # check runtime for FlexRAN in Docker POD - - debug: msg="Container runtime is set to {{ container_runtime }}" - - name: check runtime for FlexRAN in Docker POD + # check runtime for FlexRAN in POD on ICX + - name: check runtime for FlexRAN in Docker POD on ICX ansible.builtin.assert: that: container_runtime in ['docker', 'containerd'] - fail_msg: "Deploying Intel FlexRAN as a Docker POD is supported only for docker/containerd runtime. Please correct the group_vars configuration" - success_msg: "Assertion passed. Intel FlexRAN as a Docker POD is supported and can be deployed on '{{ container_runtime }}' runtime" + fail_msg: "Deploying Intel FlexRAN as a Docker POD on ICX is supported only for docker/containerd runtime. Please correct the group_vars configuration" + success_msg: "Assertion passed. Intel FlexRAN as a Docker POD on ICX is supported and can be deployed on '{{ container_runtime }}' runtime" + when: + - configured_arch == "icx" # check SRIOV for FlexRAN in Docker POD - debug: msg="SRIOV Operator is {{ sriov_network_operator_enabled | default(false) }} or undefined, and SRIOV DP is set to {{ sriov_net_dp_enabled }}" diff --git a/roles/intel_flexran/tasks/main.yml b/roles/intel_flexran/tasks/main.yml index 7c17130c..a07103f3 100644 --- a/roles/intel_flexran/tasks/main.yml +++ b/roles/intel_flexran/tasks/main.yml @@ -52,6 +52,24 @@ include_tasks: power.yml when: inventory_hostname == groups['kube_node'][0] +- name: add rt_test to test the real-time kernel performance + block: + - name: clone rt_test repository + ansible.builtin.git: + repo: "{{ rt_test_repo }}" + version: "{{ rt_test_version }}" + dest: "{{ (project_root_dir, 'rt_tests') | path_join }}" + force: yes + - name: build rt_test + make: + chdir: "{{ (project_root_dir, 'rt_tests') | path_join }}" + - name: run basic rt_test to test the real-time enviroment + ansible.builtin.command: > + taskset -c 0-16 ./cyclictest -m -p95 -h 15 -a 1-16 -t 10 -D 5s --mainaffinity=0 -q + args: + chdir: "{{ (project_root_dir, 'rt_tests') | path_join }}" + changed_when: false + - name: deploy Intel FlexRAN include_tasks: flexran.yml when: intel_flexran_type == "host" diff --git a/roles/intel_flexran/tasks/pod.yml b/roles/intel_flexran/tasks/pod.yml index c4611b63..1b9bf9ed 100644 --- a/roles/intel_flexran/tasks/pod.yml +++ b/roles/intel_flexran/tasks/pod.yml @@ -14,6 +14,23 @@ ## limitations under the License. ## --- +- name: Enable non root user to start Flexran pod on SPR-EE + block: + - name: enable device_ownership_from_security_context in /etc/containerd/config.toml + ansible.builtin.lineinfile: + path: /etc/containerd/config.toml + insertafter: 'enable_unprivileged_icmp = false' + line: " device_ownership_from_security_context = true" + - name: containerd | restart containerd + ansible.builtin.systemd: + name: containerd + state: restarted + enabled: yes + daemon-reload: yes + when: + - configured_arch == "spr" + - container_runtime == "containerd" + - name: generate FlexRAN Docker POD files template: src: "{{ item.src }}" @@ -23,8 +40,8 @@ loop: - {src: 'intel_flexran_pod_timer_mode_icx_sp.yaml.j2', dst: 'intel_flexran_pod_timer_mode_icx_sp.yaml'} - {src: 'intel_flexran_pod_xran_mode_icx_sp.yaml.j2', dst: 'intel_flexran_pod_xran_mode_icx_sp.yaml'} - - {src: 'intel_flexran_pod_timer_mode_spr_ee.yaml.j2', dst: 'intel_flexran_pod_timer_mode_spr_ee.yaml'} - - {src: 'intel_flexran_pod_xran_mode_spr_ee.yaml.j2', dst: 'intel_flexran_pod_xran_mode_spr_ee.yaml'} + - {src: 'intel_flexran_pod_timer_mode_spr_ee_non_root.yaml.j2', dst: 'intel_flexran_pod_timer_mode_spr_ee_non_root.yaml'} + - {src: 'intel_flexran_pod_xran_mode_spr_ee_non_root.yaml.j2', dst: 'intel_flexran_pod_xran_mode_spr_ee_non_root.yaml'} - name: create namespace for FlexRAN as Docker POD k8s: @@ -56,18 +73,18 @@ - intel_flexran_mode == "xran" - configured_arch == "icx" -- name: run FlexRAN as Docker POD in Timer test mode on SPR-EE +- name: run FlexRAN as POD in Timer test mode on SPR-EE k8s: state: present - src: "{{ (intel_flexran_files_dir, 'intel_flexran_pod_timer_mode_spr_ee.yaml') | path_join }}" + src: "{{ (intel_flexran_files_dir, 'intel_flexran_pod_timer_mode_spr_ee_non_root.yaml') | path_join }}" when: - intel_flexran_mode == "timer" - configured_arch == "spr" -- name: run FlexRAN as Docker POD in XRAN test mode on SPR-EE +- name: run FlexRAN as POD in XRAN test mode on SPR-EE k8s: state: present - src: "{{ (intel_flexran_files_dir, 'intel_flexran_pod_xran_mode_spr_ee.yaml') | path_join }}" + src: "{{ (intel_flexran_files_dir, 'intel_flexran_pod_xran_mode_spr_ee_non_root.yaml') | path_join }}" when: - intel_flexran_mode == "xran" - configured_arch == "spr" diff --git a/roles/intel_flexran/templates/intel_flexran_pod_timer_mode_spr_ee.yaml.j2 b/roles/intel_flexran/templates/intel_flexran_pod_timer_mode_spr_ee_non_root.yaml.j2 similarity index 58% rename from roles/intel_flexran/templates/intel_flexran_pod_timer_mode_spr_ee.yaml.j2 rename to roles/intel_flexran/templates/intel_flexran_pod_timer_mode_spr_ee_non_root.yaml.j2 index ccaac612..9f7ea7f0 100644 --- a/roles/intel_flexran/templates/intel_flexran_pod_timer_mode_spr_ee.yaml.j2 +++ b/roles/intel_flexran/templates/intel_flexran_pod_timer_mode_spr_ee_non_root.yaml.j2 @@ -6,66 +6,82 @@ metadata: name: flexran-dockerimage-release namespace: "{{ intel_flexran_namespace }}" spec: + securityContext: + fsGroup: 1250 nodeSelector: testnode: worker1 containers: - securityContext: - privileged: false + runAsNonRoot: true + runAsUser: 1250 capabilities: add: - IPC_LOCK - SYS_NICE command: [ "/bin/bash", "-c", "--" ] - args: ["sh docker_entry.sh -m timer ; cd /home/flexran/bin/nr5g/gnb/l1/; ./l1.sh -e ; top"] + args: ["sh docker_entry.sh -m timer ; cd /home/flexran/flexran/bin/nr5g/gnb/l1/; ./l1.sh -e ; top"] tty: true stdin: true env: - name: LD_LIBRARY_PATH value: /opt/oneapi/lib/intel64 - image: intel/flexran_l1_spree:v{{ intel_flexran_pod_version_spr_ee }} + image: docker.io/intel/flexran_l1_spree:v{{ intel_flexran_pod_version_spr_ee }} name: flexran-l1app resources: requests: - memory: "16Gi" + memory: "32Gi" +{% if native_cpu_manager_enabled == true %} + cpu: 24 +{% endif %} intel.com/intel_fec_5g: '1' - hugepages-1Gi: 8Gi + hugepages-1Gi: 16Gi limits: - memory: "16Gi" + memory: "32Gi" +{% if native_cpu_manager_enabled == true %} + cpu: 24 +{% endif %} intel.com/intel_fec_5g: '1' - hugepages-1Gi: 8Gi + hugepages-1Gi: 16Gi volumeMounts: - name: hugepage mountPath: /hugepages - name: varrun - mountPath: /var/run/dpdk + mountPath: /tmp/dpdk readOnly: false - securityContext: - privileged: false + runAsNonRoot: true + runAsUser: 1250 capabilities: add: - IPC_LOCK - SYS_NICE command: [ "/bin/bash", "-c", "--" ] - args: ["sleep 10; sh docker_entry.sh -m timer ; cd /home/flexran/bin/nr5g/gnb/testmac/; ./l2.sh --testfile=spr-sp-eec/sprsp.cfg; top"] + args: ["sleep 10; sh docker_entry.sh -m timer ; cd /home/flexran/flexran/bin/nr5g/gnb/testmac/; ./l2.sh --testfile=spr-sp-mcc/sprsp.cfg; top"] tty: true stdin: true env: - name: LD_LIBRARY_PATH value: /opt/oneapi/lib/intel64 - image: intel/flexran_l1_spree:v{{ intel_flexran_pod_version_spr_ee }} + image: docker.io/intel/flexran_l1_spree:v{{ intel_flexran_pod_version_spr_ee }} name: flexran-testmac resources: requests: - memory: "6Gi" - hugepages-1Gi: 4Gi + memory: "12Gi" +{% if native_cpu_manager_enabled == true %} + cpu: 16 +{% endif %} + hugepages-1Gi: 8Gi limits: - memory: "6Gi" - hugepages-1Gi: 4Gi + memory: "12Gi" +{% if native_cpu_manager_enabled == true %} + cpu: 16 +{% endif %} + hugepages-1Gi: 8Gi volumeMounts: - name: hugepage mountPath: /hugepages - name: varrun - mountPath: /var/run/dpdk + mountPath: /tmp/dpdk readOnly: false volumes: - name: hugepage diff --git a/roles/intel_flexran/templates/intel_flexran_pod_xran_mode_spr_ee.yaml.j2 b/roles/intel_flexran/templates/intel_flexran_pod_xran_mode_spr_ee_non_root.yaml.j2 similarity index 74% rename from roles/intel_flexran/templates/intel_flexran_pod_xran_mode_spr_ee.yaml.j2 rename to roles/intel_flexran/templates/intel_flexran_pod_xran_mode_spr_ee_non_root.yaml.j2 index 63559e8d..57c344e2 100644 --- a/roles/intel_flexran/templates/intel_flexran_pod_xran_mode_spr_ee.yaml.j2 +++ b/roles/intel_flexran/templates/intel_flexran_pod_xran_mode_spr_ee_non_root.yaml.j2 @@ -6,11 +6,14 @@ metadata: name: flexran-vdu namespace: "{{ intel_flexran_namespace }}" spec: + securityContext: + fsGroup: 1250 nodeSelector: testnode: worker1 containers: - securityContext: - privileged: false + runAsNonRoot: true + runAsUser: 1250 capabilities: add: - IPC_LOCK @@ -22,16 +25,22 @@ spec: env: - name: LD_LIBRARY_PATH value: /opt/oneapi/lib/intel64 - image: intel/flexran_l1_spree:v{{ intel_flexran_pod_version_spr_ee }} + image: docker.io/intel/flexran_l1_spree:v{{ intel_flexran_pod_version_spr_ee }} name: flexran-vdu resources: requests: memory: "24Gi" +{% if native_cpu_manager_enabled == true %} + cpu: 24 +{% endif %} intel.com/intel_fec_5g: '1' intel.com/intel_sriov_odu: '4' hugepages-1Gi: 16Gi limits: memory: "24Gi" +{% if native_cpu_manager_enabled == true %} + cpu: 24 +{% endif %} intel.com/intel_fec_5g: '1' intel.com/intel_sriov_odu: '4' hugepages-1Gi: 16Gi @@ -39,7 +48,7 @@ spec: - name: hugepage mountPath: /hugepages - name: varrun - mountPath: /var/run/dpdk + mountPath: /tmp/dpdk readOnly: false volumes: - name: hugepage @@ -56,11 +65,14 @@ metadata: name: flexran-vru namespace: "{{ intel_flexran_namespace }}" spec: + securityContext: + fsGroup: 1250 nodeSelector: testnode: worker1 containers: - securityContext: - privileged: false + runAsNonRoot: true + runAsUser: 1250 capabilities: add: - IPC_LOCK @@ -72,22 +84,28 @@ spec: env: - name: LD_LIBRARY_PATH value: /opt/oneapi/lib/intel64 - image: intel/flexran_l1_spree:v{{ intel_flexran_pod_version_spr_ee }} + image: docker.io/intel/flexran_l1_spree:v{{ intel_flexran_pod_version_spr_ee }} name: flexran-oru resources: requests: memory: "24Gi" +{% if native_cpu_manager_enabled == true %} + cpu: 16 +{% endif %} intel.com/intel_sriov_oru: '4' hugepages-1Gi: 16Gi limits: memory: "24Gi" +{% if native_cpu_manager_enabled == true %} + cpu: 16 +{% endif %} intel.com/intel_sriov_oru: '4' hugepages-1Gi: 16Gi volumeMounts: - name: hugepage mountPath: /hugepages - name: varrun - mountPath: /var/run/dpdk + mountPath: /tmp/dpdk readOnly: false volumes: - name: hugepage diff --git a/roles/intel_media_analytics/templates/Dockerfile.j2 b/roles/intel_media_analytics/templates/Dockerfile.j2 index ac29e443..3ba3f012 100755 --- a/roles/intel_media_analytics/templates/Dockerfile.j2 +++ b/roles/intel_media_analytics/templates/Dockerfile.j2 @@ -20,6 +20,11 @@ ARG https_proxy USER root +RUN curl https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB -O GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB \ + && apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB \ + && apt-get install -y --no-install-recommends gnupg \ + && echo "deb https://apt.repos.intel.com/openvino/2022 focal main" | sudo tee /etc/apt/sources.list.d/intel-openvino-2022.list + RUN apt-get update && apt-get install -y --no-install-recommends \ apt-utils \ git \ @@ -59,3 +64,5 @@ RUN chmod +x /home/dlstreamer/run_vehicle_detection_attribute.sh # USER dlstreamer # USER root CMD ["./run_vehicle_detection_attribute.sh"] + +HEALTHCHECK NONE diff --git a/roles/intel_oneapi_install/defaults/main.yml b/roles/intel_oneapi_install/defaults/main.yml index 502284e2..c5f21b56 100644 --- a/roles/intel_oneapi_install/defaults/main.yml +++ b/roles/intel_oneapi_install/defaults/main.yml @@ -19,16 +19,16 @@ supported_intel_oneapi_kits: default_intel_oneapi_kit: basekit -oneapi_basekit_version: 2023.1.0 # reference for versions playbook -oneapi_ai_version: 2023.1.1 # reference for versions playbook +oneapi_basekit_version: 2023.2.0 # reference for versions playbook +oneapi_ai_version: 2023.2.0 # reference for versions playbook intel_oneapi_checksum: - basekit: "aa874c08c985095c710f849af7e3d1f0cfecf561398056d391aae2528c40ea12994b17e244646597be4e55cb762490e1" # SHA384 - ai_analytics: "720f9f0bc10c92a8591142881e5f87bf2a254a550fdc948182af620f9c0f83f2c3987b71d25747c6e80f53cdb2c4646a" # SHA384 + basekit: "fd15ab63e4401adda819b158751c83e5127a05ee834fdbc48f32a558166644e12f2e884e72b9b78c03360d8ef12c17c3" # SHA384 + ai_analytics: "27f53a06d5a458bc85a71a0ce4458c0c3e7543bf0db7d7f1cca046be261dd94afe905d4e4607cfe6536770f1b55d755b" # SHA384 intel_oneapi_url: - basekit: "https://registrationcenter-download.intel.com/akdlm/IRC_NAS/7deeaac4-f605-4bcf-a81b-ea7531577c61/l_BaseKit_p_2023.1.0.46401_offline.sh" - ai_analytics: "https://registrationcenter-download.intel.com/akdlm/IRC_NAS/ef4efa4d-9b83-4994-a122-d5a4d2dec84c/l_AIKit_p_2023.1.1.48862_offline.sh" + basekit: "https://registrationcenter-download.intel.com/akdlm/IRC_NAS/992857b9-624c-45de-9701-f6445d845359/l_BaseKit_p_2023.2.0.49397_offline.sh" + ai_analytics: "https://registrationcenter-download.intel.com/akdlm/IRC_NAS/af4bc50d-898e-45a4-8f7d-378448ba294a/l_AIKit_p_2023.2.0.48997_offline.sh" intel_oneapi_components: basekit: diff --git a/roles/intel_power_manager/defaults/main.yml b/roles/intel_power_manager/defaults/main.yml deleted file mode 100644 index b982bdd1..00000000 --- a/roles/intel_power_manager/defaults/main.yml +++ /dev/null @@ -1,24 +0,0 @@ -## -## Copyright (c) 2020-2023 Intel Corporation. -## -## Licensed under the Apache License, Version 2.0 (the "License"); -## you may not use this file except in compliance with the License. -## You may obtain a copy of the License at -## -## http://www.apache.org/licenses/LICENSE-2.0 -## -## Unless required by applicable law or agreed to in writing, software -## distributed under the License is distributed on an "AS IS" BASIS, -## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -## See the License for the specific language governing permissions and -## limitations under the License. -## ---- -intel_power_manager_git_url: "https://github.com/intel/kubernetes-power-manager.git" -intel_power_manager_git_ref: "v2.2.0" # project is consistent with git ref and image version -intel_power_manager_dir: "{{ (project_root_dir, 'intel-power-manager') | path_join }}" -intel_power_manager_namespace: "intel-power" -intel_power_operator_image: "docker.io/intel/power-operator" -intel_power_operator_image_local: "{{ registry_local_address }}/intel-power-operator" -intel_power_node_agent_image: "intel/power-node-agent" -intel_power_node_agent_image_local: "{{ registry_local_address }}/intel-power-node-agent" diff --git a/roles/intel_power_manager/templates/global_shared_profile.yaml.j2 b/roles/intel_power_manager/templates/global_shared_profile.yaml.j2 deleted file mode 100644 index 22a8d06a..00000000 --- a/roles/intel_power_manager/templates/global_shared_profile.yaml.j2 +++ /dev/null @@ -1,12 +0,0 @@ ---- -apiVersion: "power.intel.com/v1" -kind: PowerProfile -metadata: - name: shared-global - namespace: {{ intel_power_manager_namespace }} -spec: - name: "shared-global" - max: {{ intel_power_manager.global_max_frequency }} - min: {{ intel_power_manager.global_min_frequency }} - epp: "power" - governor: {{ intel_power_manager.global_pstate_governor }} diff --git a/roles/intel_sriov_fec_operator/defaults/main.yml b/roles/intel_sriov_fec_operator/defaults/main.yml index 1fb98272..c1aa0be0 100644 --- a/roles/intel_sriov_fec_operator/defaults/main.yml +++ b/roles/intel_sriov_fec_operator/defaults/main.yml @@ -22,9 +22,9 @@ # Intel Smart Edge Open (SEO) SRIOV-FEC Operator intel_sriov_fec_operator_git: "https://github.com/smart-edge-open/sriov-fec-operator.git" -intel_sriov_fec_operator_git_ref: "sriov-fec-operator-23.05" +intel_sriov_fec_operator_git_ref: "sriov-fec-operator-23.34" intel_sriov_fec_operator_dir: "{{ (project_root_dir, 'intel-sriov-fec-operator') | path_join }}" -intel_sriov_fec_operator_img_ver: "2.6.1" +intel_sriov_fec_operator_img_ver: "2.7.1" intel_sriov_fec_operator_tool: "{{ 'docker' if container_runtime == 'docker' else 'podman' }}" intel_sriov_fec_operator_make_tls: "false" # intel_sriov_fec_operator_target_platform: "K8S" @@ -39,7 +39,7 @@ fec_acc_dev: "{{ fec_acc }}" # Operator Package Manager (OPM) opm_url: "https://github.com/operator-framework/operator-registry/releases/download/{{ opm_ver }}/linux-amd64-opm" -opm_ver: "v1.26.3" -opm_chk: "605b48722e7c387ac96c7bc9b1c410044cfc1e9e9f22323e4e7d74c8888798a5" +opm_ver: "v1.28.0" +opm_chk: "e18e5abc8febb63c9dc76db0f33475553d98495465bd2dca81c39dcdbc875c08" opm_dir: "/usr/local/bin/" opm_cmd: "opm" diff --git a/roles/intel_sriov_fec_operator/tasks/check_sriov_fec_operator.yml b/roles/intel_sriov_fec_operator/tasks/check_sriov_fec_operator.yml index 70d10414..1bc74d09 100644 --- a/roles/intel_sriov_fec_operator/tasks/check_sriov_fec_operator.yml +++ b/roles/intel_sriov_fec_operator/tasks/check_sriov_fec_operator.yml @@ -54,3 +54,34 @@ failed_when: false - debug: msg={{ fec_operator_devs.stdout }} + +- name: check vfio-token feature is valid + block: + - name: get vfio-token value + kubernetes.core.k8s_info: + api_version: v1 + kind: secret + name: vfio-token + namespace: "{{ intel_sriov_fec_operator_namespace }}" + register: vfio_token + no_log: True + changed_when: false + - name: get the first FEC VF + ansible.builtin.shell: "set -o pipefail && lspci | grep -i acc | grep -i 57c1 | awk '{print $1}' | head -1" + args: + executable: /bin/bash + register: fec_first_vf + changed_when: false + - ansible.builtin.debug: msg="fec_first_vf is at {{ fec_first_vf.stdout }}" + - name: run basic bbdev test + ansible.builtin.command: >- + ./test-bbdev.py -e="-c 0xff0 -a 0000:{{ fec_first_vf.stdout }} + --vfio-vf-token={{ vfio_token | json_query('resources[].data.VFIO_TOKEN') | b64decode }}" + -t 6 -n 100 -b 80 -l 1 -c validation -v ./ldpc_dec_default.data + args: + chdir: "{{ dpdk_dir }}/app/test-bbdev" + no_log: True + changed_when: false + when: + - fec_acc == "acc200" + - intel_flexran_type == "pod" diff --git a/roles/intel_sriov_fec_operator/tasks/sriov_fec_operator.yml b/roles/intel_sriov_fec_operator/tasks/sriov_fec_operator.yml index 72f7424e..474c00d6 100644 --- a/roles/intel_sriov_fec_operator/tasks/sriov_fec_operator.yml +++ b/roles/intel_sriov_fec_operator/tasks/sriov_fec_operator.yml @@ -44,13 +44,6 @@ when: - container_runtime == "containerd" -# Workaoround to get SRIOV-FEC Operator daemon image build working -- name: patch Intel Smart Edge Open (SEO) SRIOV-FEC Operator Dockerfile - ansible.builtin.replace: - path: "{{ intel_sriov_fec_operator_dir }}/Dockerfile.daemon" - regexp: 'libbpf\.so\.0' - replace: 'libbpf.so.1' - - name: build Intel Smart Edge Open (SEO) SRIOV-FEC Operator make: target: "{{ item }}" @@ -64,6 +57,9 @@ loop: - build_all - build_index + register: sriov_fec_operator_build + retries: 5 + until: sriov_fec_operator_build is success - name: create SRIOV-FEC Operator files directory on controller file: diff --git a/roles/intel_sriov_fec_operator/templates/acc200-cr.yaml.j2 b/roles/intel_sriov_fec_operator/templates/acc200-cr.yaml.j2 index 6b18d29d..5296dec9 100644 --- a/roles/intel_sriov_fec_operator/templates/acc200-cr.yaml.j2 +++ b/roles/intel_sriov_fec_operator/templates/acc200-cr.yaml.j2 @@ -10,7 +10,11 @@ spec: acceleratorSelector: pciAddress: "{{ fec_acc_dev }}" physicalFunction: +{% if intel_flexran_type == "host" %} pfDriver: pci-pf-stub +{% elif intel_flexran_type == "pod" %} + pfDriver: vfio-pci +{% endif %} vfDriver: vfio-pci vfAmount: 16 bbDevConfig: diff --git a/roles/intel_xpumanager/defaults/main.yml b/roles/intel_xpumanager/defaults/main.yml new file mode 100644 index 00000000..902bceaa --- /dev/null +++ b/roles/intel_xpumanager/defaults/main.yml @@ -0,0 +1,28 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +xpumanager_version: "v1.2.13" +xpumanager_image: "intel/xpumanager" +xpumanager_namespace: "intel-xpumanager" + +rbac_proxy_ssl_mount_path: /etc/ssl/rbac-proxy +rbac_proxy_ssl_secret_name: xpumanager-rbac-proxy-ssl + +xpumanager_path: "{{ (project_root_dir, 'xpumanager') | path_join }}" + +container_cert_path: "{{ (project_root_dir, 'cert') | path_join }}" +xpumanager_key_path: "{{ (container_cert_path, 'xpum_key.pem') | path_join }}" +xpumanager_cert_path: "{{ (container_cert_path, 'xpum_cert.pem') | path_join }}" diff --git a/roles/intel_xpumanager/tasks/main.yml b/roles/intel_xpumanager/tasks/main.yml new file mode 100644 index 00000000..58f9acc9 --- /dev/null +++ b/roles/intel_xpumanager/tasks/main.yml @@ -0,0 +1,125 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +- block: + - name: create xpumanager namespace + kubernetes.core.k8s: + name: "{{ xpumanager_namespace }}" + api_version: v1 + kind: Namespace + state: present + + - name: generate ssl certificate + include_role: + name: create_signed_k8s_certs + vars: + secret_name: "{{ rbac_proxy_ssl_secret_name }}" + service_name: xpumanager + key_pair_name: xpumanager-rbac-proxy + host_secrets_folder: "{{ rbac_proxy_ssl_mount_path }}" + k8s_namespace: "{{ xpumanager_namespace }}" + csr_cluster_name: "{{ cluster_name | default('cluster.local') }}" + + - name: deploy xpumanager + kubernetes.core.k8s: + state: present + template: "{{ item }}" + loop: + - "xpumanager_service.yml.j2" + - "xpumanager_daemonset.yml.j2" + - "xpumanager_servicemonitor.yml.j2" + - "prometheus_role.yml.j2" + - "prometheus_rolebinding.yml.j2" + when: + - kubernetes | default(false) | bool + + +- block: + - name: create xpumanager folder + ansible.builtin.file: + path: "{{ item }}" + state: directory + mode: '0755' + with_items: + - "{{ xpumanager_path }}" + + - name: generate xpumanager private key (RSA, 4096 bits) + community.crypto.openssl_privatekey: + path: "{{ xpumanager_key_path }}" + + - name: generate xpumanager certificates signing request + community.crypto.openssl_csr_pipe: + privatekey_path: "{{ xpumanager_key_path }}" + common_name: "xpumanager" + subject_alt_name: + - "DNS:xpumanager" + - "IP:127.0.0.1" + register: csr + + - name: create self-signed certificate from CSR + community.crypto.x509_certificate: + path: "{{ xpumanager_cert_path }}" + csr_content: "{{ csr.csr }}" + privatekey_path: "{{ xpumanager_key_path }}" + provider: selfsigned + + - name: check whether gpu is ready by checking dri folder + ansible.builtin.stat: + path: /dev/dri + register: dri_folder + + - name: set xpumanager state to start if gpu ready + set_fact: + xpumanager_state: started + when: + - dri_folder.stat.exists + + - name: set xpumanager state to present if gpu not ready + set_fact: + xpumanager_state: present + when: + - not dri_folder.stat.exists + + - name: create or start xpumanager container + docker_container: + name: xpumanager + image: "{{ xpumanager_image }}:{{ xpumanager_version }}" + state: "{{ xpumanager_state }}" + restart_policy: always + user: root + cap_drop: + - all + capabilities: + - sys_admin + devices: + - "/dev/dri:/dev/dri" + ports: + - "127.0.0.1:29999:29999" + networks: + - name: telemetry + aliases: + - xpumanager + volumes: + - "{{ xpumanager_cert_path }}:/usr/lib/xpum/rest/conf/cert.pem" + - "{{ xpumanager_key_path }}:/usr/lib/xpum/rest/conf/key.pem" + env: + XPUM_EXPORTER_NO_AUTH: "1" + XPUM_EXPORTER_ONLY: "1" + XPUM_METRICS: "0-38" + + when: + - not kubernetes | default(false) | bool + - container_runtime == 'docker' diff --git a/roles/intel_xpumanager/tasks/xpumanager_cleanup.yml b/roles/intel_xpumanager/tasks/xpumanager_cleanup.yml new file mode 100644 index 00000000..420d2b21 --- /dev/null +++ b/roles/intel_xpumanager/tasks/xpumanager_cleanup.yml @@ -0,0 +1,55 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +- block: + - name: uninstall xpumanager + kubernetes.core.k8s: + state: absent + template: "{{ item }}" + loop: + - "xpumanager_service.yml.j2" + - "xpumanager_daemonset.yml.j2" + - "xpumanager_servicemonitor.yml.j2" + - "prometheus_role.yml.j2" + - "prometheus_rolebinding.yml.j2" + failed_when: false + + - name: delete xpumanager namespace + kubernetes.core.k8s: + name: "{{ xpumanager_namespace }}" + api_version: v1 + kind: Namespace + state: absent + failed_when: false + when: + - kubernetes | default(false) | bool + +- block: + - name: stop and remove xpumanager container + docker_container: + name: xpumanager + state: absent + + - name: delete xpumanager folder + ansible.builtin.file: + path: "{{ item }}" + state: absent + with_items: + - "{{ xpumanager_path }}" + + when: + - not kubernetes | default(false) | bool + - container_runtime == 'docker' diff --git a/roles/intel_xpumanager/tasks/xpumanager_preflight.yml b/roles/intel_xpumanager/tasks/xpumanager_preflight.yml new file mode 100644 index 00000000..6e336e6f --- /dev/null +++ b/roles/intel_xpumanager/tasks/xpumanager_preflight.yml @@ -0,0 +1,66 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +- name: preflight xpumanager for k8s env + block: + - name: check if Observability stack is enabled + ansible.builtin.assert: + that: + - prometheus_enabled | d(false) + - telegraf_enabled | d(false) + - jaeger_operator | d(false) + - opentelemetry_enabled | d(false) + - elasticsearch_enabled | d(false) + - kibana_enabled | d(false) + msg: | + Incorrect configuration !! + XPUManager requires Onservability stack. + Please enable prometheus_enabled, telegraf_enabled, jaeger_operator, opentelemetry_enabled, elasticsearch_enabled, kibana_enabled in group_vars + + - name: check if GPU Device Plugin is enabled and configured + ansible.builtin.assert: + that: + - gpu_dp_enabled | d(false) + - gpu_dp_monitor_resources | d(false) + msg: | + Incorrect configuration !! + XPUManager requires GPU Device Plugin and resource monitoring to be enabled. + Please set gpu_dp_enabled and gpu_dp_monitor_resources to true + + - name: check if cert-manager is enabled + ansible.builtin.assert: + that: + - cert_manager_enabled | d(false) + msg: | + Incorrect configuration !! + XPUManager requires Cert Manager to be enabled. + Please set cert_manager_enabled to true + when: + - kubernetes | default(false) | bool + + +- name: preflight xpumanager for none k8s env + block: + - name: check if Observability stack is enabled + ansible.builtin.assert: + that: + - prometheus_enabled | d(false) + msg: | + Incorrect configuration !! + XPUManager requires Onservability stack. + Please enable prometheus_enabled in group_vars + when: + - not kubernetes | default(false) | bool diff --git a/roles/intel_xpumanager/templates/prometheus_role.yml.j2 b/roles/intel_xpumanager/templates/prometheus_role.yml.j2 new file mode 100644 index 00000000..ea8864f3 --- /dev/null +++ b/roles/intel_xpumanager/templates/prometheus_role.yml.j2 @@ -0,0 +1,23 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: xpumanager-prometheus + namespace: "{{ xpumanager_namespace }}" +rules: +- apiGroups: [""] + resources: + - nodes + - nodes/metrics + - services + - endpoints + - pods + verbs: ["get", "list", "watch"] +- apiGroups: [""] + resources: + - configmaps + verbs: ["get"] +- apiGroups: + - networking.k8s.io + resources: + - ingresses + verbs: ["get", "list", "watch"] diff --git a/roles/intel_xpumanager/templates/prometheus_rolebinding.yml.j2 b/roles/intel_xpumanager/templates/prometheus_rolebinding.yml.j2 new file mode 100644 index 00000000..16843c34 --- /dev/null +++ b/roles/intel_xpumanager/templates/prometheus_rolebinding.yml.j2 @@ -0,0 +1,13 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: xpumanager-prometheus-rolebinding + namespace: "{{ xpumanager_namespace }}" +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: xpumanager-prometheus +subjects: +- namespace: monitoring + kind: ServiceAccount + name: prometheus-k8s diff --git a/roles/intel_xpumanager/templates/xpumanager_daemonset.yml.j2 b/roles/intel_xpumanager/templates/xpumanager_daemonset.yml.j2 new file mode 100644 index 00000000..b60161e1 --- /dev/null +++ b/roles/intel_xpumanager/templates/xpumanager_daemonset.yml.j2 @@ -0,0 +1,163 @@ +--- +apiVersion: apps/v1 +kind: DaemonSet +metadata: + labels: + app: intel-xpumanager + name: intel-xpumanager + namespace: "{{ xpumanager_namespace }}" +spec: + selector: + matchLabels: + app: intel-xpumanager + template: + metadata: + labels: + app: intel-xpumanager + spec: + # hostNetwork should be set true for getting xelink metrics + hostNetwork: true + containers: + - name: xpumd + image: "{{ xpumanager_image }}:{{ xpumanager_version }}" + imagePullPolicy: IfNotPresent + command: [ "/usr/bin/xpumd" ] + env: + - name: SPDLOG_LEVEL + value: info + - name: XPUM_REST_NO_TLS + value: "1" + - name: XPUM_EXPORTER_NO_AUTH + value: "1" + - name: XPUM_EXPORTER_ONLY + value: "1" + - name: XPUM_METRICS + value: 0-38 + resources: + limits: + gpu.intel.com/i915_monitoring: 1 + securityContext: + privileged: true + readOnlyRootFilesystem: true + runAsUser: 0 + capabilities: + drop: [ "ALL" ] + add: [ "SYS_ADMIN", "SYS_RAWIO" ] + volumeMounts: + # for getting pod resources + - mountPath: /var/lib/kubelet/pod-resources + name: kubeletpodres + # for PCIe read/write data collection + - mountPath: /pcm/sys/firmware/acpi/tables/MCFG:ro + name: mcfg + # for PCIe read/write data collection + - mountPath: /pcm/proc/bus/pci/ + name: pci + # for PCIe read/write data collection + - mountPath: /pcm/proc/sys/kernel/nmi_watchdog + name: nmiwatchdog + - name: sockdir + mountPath: /tmp + - name: python-exporter + # - socket location for "xpumd" communication + # - GPU device file names for "dev_file" label + volumeMounts: + - name: sockdir + mountPath: /tmp + - name: devdri + mountPath: /dev/dri + readOnly: true + imagePullPolicy: IfNotPresent + image: "{{ xpumanager_image }}:{{ xpumanager_version }}" + # needs same user as "xpumd" to be able to access its socket + securityContext: + privileged: true + readOnlyRootFilesystem: true + runAsUser: 0 + capabilities: + drop: [ "ALL" ] + env: + - name: NODE_NAME + valueFrom: + fieldRef: + apiVersion: v1 + fieldPath: spec.nodeName + - name: XPUM_EXPORTER_NO_AUTH + value: "1" + # Run only Prometheus exporter, not rest XPUM + - name: XPUM_EXPORTER_ONLY + value: "1" + # so that Gunicorn finds the Python files + workingDir: /usr/lib/xpum/rest + # There should be only single outstanding Prometheus request + # being handled at the time + manual debugging calls as this + # is cluster internal, so it does not need to scale as much + # as Gunicorn defaults do. For all settings, see: + # https://docs.gunicorn.org/en/stable/settings.html + command: [ + "gunicorn", + "--bind", "127.0.0.1:29999", + "--worker-connections", "64", + "--worker-class", "gthread", + "--workers", "1", + "--threads", "4", + "xpum_rest_main:main()" + ] + startupProbe: + httpGet: + path: /metrics + port: 29999 + failureThreshold: 10 + periodSeconds: 10 + livenessProbe: + httpGet: + path: /healtz + port: 29999 + initialDelaySeconds: 60 + periodSeconds: 10 + - name: rbac-proxy + image: "{{ kube_rbac_proxy_image_repo }}:{{ kube_rbac_proxy_image_tag }}" + imagePullPolicy: IfNotPresent + volumeMounts: + - name: ssl + mountPath: "{{ rbac_proxy_ssl_mount_path }}" + readOnly: true + ports: + - containerPort: 8443 + name: metrics + protocol: TCP + args: + - "--tls-cert-file={{ rbac_proxy_ssl_mount_path }}/{{ rbac_proxy_ssl_secret_name }}.cert" + - "--tls-private-key-file={{ rbac_proxy_ssl_mount_path }}/{{ rbac_proxy_ssl_secret_name }}.key" + - "--tls-cipher-suites={{ kube_rbac_proxy_tls_ciphers }}" + - "--secure-listen-address=0.0.0.0:8443" + - "--upstream=http://127.0.0.1:29999/" + nodeSelector: + intel.feature.node.kubernetes.io/gpu: "true" + restartPolicy: Always + volumes: + - hostPath: + path: /var/lib/kubelet/pod-resources + type: "" + name: kubeletpodres + - hostPath: + path: /sys/firmware/acpi/tables/MCFG + type: "" + name: mcfg + - hostPath: + path: /proc/bus/pci/ + type: "" + name: pci + - hostPath: + path: /proc/sys/kernel/nmi_watchdog + type: "" + name: nmiwatchdog + - emptyDir: + medium: Memory + name: sockdir + - hostPath: + path: /dev/dri + name: devdri + - name: ssl + secret: + secretName: "{{ rbac_proxy_ssl_secret_name }}" diff --git a/roles/intel_xpumanager/templates/xpumanager_service.yml.j2 b/roles/intel_xpumanager/templates/xpumanager_service.yml.j2 new file mode 100644 index 00000000..bb7610c8 --- /dev/null +++ b/roles/intel_xpumanager/templates/xpumanager_service.yml.j2 @@ -0,0 +1,18 @@ +--- +apiVersion: v1 +kind: Service +metadata: + name: intel-xpumanager + namespace: "{{ xpumanager_namespace }}" + labels: + app: intel-xpumanager +spec: + type: ClusterIP + selector: + app: intel-xpumanager + sessionAffinity: None + ports: + - name: metrics + port: 8443 + targetPort: 8443 + protocol: TCP diff --git a/roles/intel_xpumanager/templates/xpumanager_servicemonitor.yml.j2 b/roles/intel_xpumanager/templates/xpumanager_servicemonitor.yml.j2 new file mode 100644 index 00000000..3a4cc816 --- /dev/null +++ b/roles/intel_xpumanager/templates/xpumanager_servicemonitor.yml.j2 @@ -0,0 +1,23 @@ +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + name: intel-xpumanager + namespace: monitoring + labels: + app: intel-xpumanager +spec: + selector: + matchLabels: + app: intel-xpumanager + namespaceSelector: + matchNames: + - "{{ xpumanager_namespace }}" + endpoints: + - port: metrics + path: /metrics + interval: 5s + scheme: https + bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token + tlsConfig: + caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + insecureSkipVerify: true diff --git a/roles/intent_driven_orchestration/defaults/main.yml b/roles/intent_driven_orchestration/defaults/main.yml new file mode 100644 index 00000000..7437cc4a --- /dev/null +++ b/roles/intent_driven_orchestration/defaults/main.yml @@ -0,0 +1,47 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +ido_dir: "{{ (project_root_dir, 'intent-driven-orchestration') | path_join }}" + +ido_git_url: "https://github.com/intel/intent-driven-orchestration.git" +ido_git_version: "v0.2.0" + +ido_enabled: "{{ ido.enabled | d(false) }}" +ido_demo_workload: "{{ ido.demo_workload | d(false) }}" + +ido_plugins: + - name: "scaleout" + image: "scaleout:{{ ido_git_version }}" + directory: "plugins/scale_out" + manifest: "scaleout-actuator-plugin.yaml" + - name: "rmpod" + image: "rmpod:{{ ido_git_version }}" + directory: "plugins/rm_pod" + manifest: "rmpod-actuator-plugin.yaml" + - name: "rdt" + image: "rdt:{{ ido_git_version }}" + directory: "plugins/rdt" + manifest: "rdt-actuator-plugin.yaml" + - name: "cpuscale" + image: "cpuscale:{{ ido_git_version }}" + directory: "plugins/cpu_scale" + manifest: "cpu-scale-actuator-plugin.yaml" + +ido_planner: + - name: "planner" + image: "planner:{{ ido_git_version }}" + directory: "artefacts/deploy" + manifest: "manifest.yaml" diff --git a/roles/intent_driven_orchestration/files/example_deployment.yaml b/roles/intent_driven_orchestration/files/example_deployment.yaml new file mode 100644 index 00000000..1b651b3e --- /dev/null +++ b/roles/intent_driven_orchestration/files/example_deployment.yaml @@ -0,0 +1,35 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ido-example-deployment + labels: + app: sample-function +spec: + replicas: 1 + selector: + matchLabels: + app: sample-function + template: + metadata: + labels: + app: sample-function + annotations: + linkerd.io/inject: enabled + spec: + containers: + - name: sample-function + image: testfunction/rust_function:0.1 + env: + - name: WORKERS + value: "2" + securityContext: + capabilities: + drop: ['ALL'] + seccompProfile: + type: RuntimeDefault + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 10001 + runAsGroup: 10001 + restartPolicy: Always diff --git a/roles/intent_driven_orchestration/tasks/cleanup.yml b/roles/intent_driven_orchestration/tasks/cleanup.yml new file mode 100644 index 00000000..fc17a5cb --- /dev/null +++ b/roles/intent_driven_orchestration/tasks/cleanup.yml @@ -0,0 +1,86 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +- name: Cleanup IDO example workload and intent + when: + - inventory_hostname == groups["kube_control_plane"][0] + - ido_enabled + - ido_demo_workload + tags: + - intent-driven-orchestration + block: + - name: Cleanup IDO example workload + kubernetes.core.k8s: + state: absent + namespace: default + src: "{{ (ido_dir, 'artefacts', 'examples', 'example_deployment.yaml') | path_join }}" + failed_when: false + + - name: Cleanup IDO example intent + kubernetes.core.k8s: + state: absent + namespace: default + src: "{{ (ido_dir, 'artefacts', 'examples', 'example_intent.yaml') | path_join }}" + failed_when: false + +- name: Cleanup Linkerd Viz and IDO + when: + - inventory_hostname == groups["kube_control_plane"][0] + - ido_enabled + tags: + - intent-driven-orchestration + block: + - name: Cleanup IDO default KPIProfiles + kubernetes.core.k8s: + state: absent + namespace: default + src: "{{ (ido_dir, 'artefacts', 'examples', 'default_profiles.yaml') | path_join }}" + failed_when: false + + - name: Cleanup IDO namespace and content + kubernetes.core.k8s: + state: absent + name: ido + kind: Namespace + +# Below task uses 'kubectl' due to issues with 'kubernetes.core.k8s' module for CRD manifest + - name: Cleanup CRDs for IDO + ansible.builtin.command: "kubectl delete -f {{ (ido_dir, 'artefacts', 'intents_crds_v1alpha1.yaml') | path_join }}" + register: crd_result + changed_when: "'deleted' in crd_result.stdout" + failed_when: false + + - name: Remove IDO directory + ansible.builtin.file: + state: absent + path: "{{ ido_dir }}" + + - name: Remove access to linkerd-viz namespace + kubernetes.core.k8s: + state: absent + template: "linkerd-viz-auth.yaml.j2" + + - name: Generate linkerd-viz uninstall manifest + ansible.builtin.command: linkerd viz uninstall + register: linkerd_viz_uninstall_manifest + changed_when: false + failed_when: false + + - name: Remove linkerd-viz + kubernetes.core.k8s: + state: absent + definition: "{{ linkerd_viz_uninstall_manifest.stdout }}" + when: linkerd_viz_uninstall_manifest.rc == 0 diff --git a/roles/intent_driven_orchestration/tasks/install_ido.yml b/roles/intent_driven_orchestration/tasks/install_ido.yml new file mode 100644 index 00000000..6131c2f5 --- /dev/null +++ b/roles/intent_driven_orchestration/tasks/install_ido.yml @@ -0,0 +1,156 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +- name: Install linkerd-viz + include_tasks: linkerd_viz_install.yml + +- name: Create IDO directory + ansible.builtin.file: + path: "{{ ido_dir }}" + state: directory + mode: '0755' + +- name: Clone IDO repository + ansible.builtin.git: + repo: "{{ ido_git_url }}" + dest: "{{ ido_dir }}" + version: "{{ ido_git_version }}" + force: true + +- name: Update local address of images + ansible.builtin.replace: + path: "{{ (ido_dir, item.directory, item.manifest) | path_join }}" + regexp: '127\.0\.0\.1\:5000' + replace: "{{ registry_local_address }}" + with_items: + - "{{ ido_planner }}" + - "{{ ido_plugins }}" + +- name: Update version of images + ansible.builtin.replace: + path: "{{ (ido_dir, item.directory, item.manifest) | path_join }}" + regexp: '0\.2\.0' + replace: "{{ ido_git_version }}" + with_items: + - "{{ ido_planner }}" + - "{{ ido_plugins }}" + +- name: Update telemetry endpoint of planner + ansible.builtin.replace: + path: "{{ (ido_dir, item.directory, item.manifest) | path_join }}" + regexp: 'prometheus\-service\.telemetry\:9090' + replace: "prometheus-k8s.monitoring:9090" + loop: "{{ ido_planner }}" + +- name: Build and push images + vars: + buildtool: "{{ 'docker' if container_runtime == 'docker' else 'podman' }}" + block: + - name: Build plugin images + ansible.builtin.command: '{{ buildtool }} build -f {{ (item.directory, "Dockerfile") | path_join }} -t {{ registry_local_address }}/{{ item.image }} .' + args: + chdir: "{{ ido_dir }}" + changed_when: true + loop: "{{ ido_plugins }}" + + - name: Build planner image + ansible.builtin.command: '{{ buildtool }} build -f Dockerfile -t {{ registry_local_address }}/{{ item.image }} .' + args: + chdir: "{{ ido_dir }}" + changed_when: true + loop: "{{ ido_planner }}" + + - name: Push images + ansible.builtin.command: '{{ buildtool }} push {{ registry_local_address }}/{{ item.image }}' + changed_when: true + with_items: + - "{{ ido_planner }}" + - "{{ ido_plugins }}" + +# Below task uses 'kubectl' due to issues with 'kubernetes.core.k8s' module for CRD manifest +- name: Install CRDs for IDO + ansible.builtin.command: "kubectl apply -f {{ (ido_dir, 'artefacts', 'intents_crds_v1alpha1.yaml') | path_join }}" + register: crd_result + changed_when: "'created' in crd_result.stdout" + +- name: Create IDO namespace + kubernetes.core.k8s: + state: present + name: ido + kind: Namespace + +- name: Deploy IDO Planner + kubernetes.core.k8s: + state: present + namespace: ido + src: "{{ (ido_dir, item.directory, item.manifest) | path_join }}" + loop: "{{ ido_planner }}" + +- name: Wait for IDO Planner to run + ansible.builtin.include_role: + name: wait_for_kubernetes_ready + +- name: Deploy IDO Plugins + kubernetes.core.k8s: + state: present + namespace: ido + src: "{{ (ido_dir, item.directory, item.manifest) | path_join }}" + loop: "{{ ido_plugins }}" + +- name: Wait for IDO Plugins to run + ansible.builtin.include_role: + name: wait_for_kubernetes_ready + +- name: Deploy IDO default KPIProfiles + kubernetes.core.k8s: + state: present + namespace: default + src: "{{ (ido_dir, 'artefacts', 'examples', 'default_profiles.yaml') | path_join }}" + +- name: Update and deploy IDO example workload and intent + block: + - name: Update IDO example intent + ansible.builtin.replace: + path: "{{ (ido_dir, 'artefacts', 'examples', 'example_intent.yaml') | path_join }}" + regexp: "{{ item.existing }}" + replace: "{{ item.updated }}" + with_items: + - { existing: 'my\-function', updated: 'ido-example' } + - { existing: 'function\-deployment', updated: 'ido-example-deployment' } + + - name: Deploy IDO example workload + kubernetes.core.k8s: + state: present + namespace: default + definition: "{{ lookup('file', '../files/example_deployment.yaml') | from_yaml }}" + + - name: Wait for IDO example workload to start + kubernetes.core.k8s_info: + kind: Deployment + namespace: default + name: ido-example-deployment + wait: true + wait_condition: + reason: MinimumReplicasAvailable + type: Available + wait_timeout: 300 + + - name: Deploy IDO example intent + kubernetes.core.k8s: + state: present + namespace: default + src: "{{ (ido_dir, 'artefacts', 'examples', 'example_intent.yaml') | path_join }}" + when: ido_demo_workload diff --git a/roles/intent_driven_orchestration/tasks/linkerd_viz_install.yml b/roles/intent_driven_orchestration/tasks/linkerd_viz_install.yml new file mode 100644 index 00000000..bb00492c --- /dev/null +++ b/roles/intent_driven_orchestration/tasks/linkerd_viz_install.yml @@ -0,0 +1,49 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +- name: Generate linkerd-viz manifest + ansible.builtin.command: linkerd viz install + register: linkerd_viz_manifest + changed_when: false + failed_when: + - linkerd_viz_manifest.rc != 0 + +- name: Deploy linkerd-viz manifest + kubernetes.core.k8s: + state: present + definition: "{{ linkerd_viz_manifest.stdout }}" + +- name: Wait for linkerd-viz to start + kubernetes.core.k8s_info: + kind: Deployment + namespace: linkerd-viz + wait: true + wait_condition: + reason: MinimumReplicasAvailable + type: Available + wait_timeout: 300 + +- name: Check status of linkerd-viz + ansible.builtin.command: linkerd viz check --proxy + changed_when: false + register: linkerd_viz_check + failed_when: + - linkerd_viz_check.rc != 0 + +- name: Allow access to linkerd-viz namespace + kubernetes.core.k8s: + state: present + template: "linkerd-viz-auth.yaml.j2" diff --git a/roles/intent_driven_orchestration/tasks/main.yml b/roles/intent_driven_orchestration/tasks/main.yml new file mode 100644 index 00000000..30493d25 --- /dev/null +++ b/roles/intent_driven_orchestration/tasks/main.yml @@ -0,0 +1,22 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +- name: Install IDO + include_role: + name: intent_driven_orchestration + tasks_from: install_ido + when: + - inventory_hostname == groups["kube_control_plane"][0] diff --git a/roles/intent_driven_orchestration/templates/linkerd-viz-auth.yaml.j2 b/roles/intent_driven_orchestration/templates/linkerd-viz-auth.yaml.j2 new file mode 100644 index 00000000..4774ee4d --- /dev/null +++ b/roles/intent_driven_orchestration/templates/linkerd-viz-auth.yaml.j2 @@ -0,0 +1,14 @@ +--- +apiVersion: policy.linkerd.io/v1beta1 +kind: ServerAuthorization +metadata: + name: allow-pod-subnet + namespace: linkerd-viz +spec: + client: + unauthenticated: true + networks: + - cidr: {{ kube_pods_subnet }} + server: + selector: + matchLabels: {} diff --git a/roles/minio_install/tasks/setup_loopdevices.yml b/roles/ipu/acc/defaults/main.yml similarity index 78% rename from roles/minio_install/tasks/setup_loopdevices.yml rename to roles/ipu/acc/defaults/main.yml index 4029e2ff..76ca1b90 100644 --- a/roles/minio_install/tasks/setup_loopdevices.yml +++ b/roles/ipu/acc/defaults/main.yml @@ -14,7 +14,9 @@ ## limitations under the License. ## --- -- name: setup loop devices for block devices files - command: >- - losetup --find --show /tmp/diskimage{{ ansible_loop.index }} - changed_when: true +p4sde_install_dir: "/opt/p4/p4sde" +p4cp_install_dir: "/opt/p4/p4-cp-nws" +active_p4_dir: "/root" +stratum_es2k_dir: "/usr/share/stratum/es2k" + +p4sde_bin_dir: "/opt/p4/p4sde/bin" diff --git a/roles/ipu/acc/tasks/main.yml b/roles/ipu/acc/tasks/main.yml index f9ae5e5a..465f12ce 100644 --- a/roles/ipu/acc/tasks/main.yml +++ b/roles/ipu/acc/tasks/main.yml @@ -14,8 +14,228 @@ ## limitations under the License. ## --- +- name: check if /etc/redhat-release exists + stat: + path: "/etc/redhat-release" + register: redhat_release + +- name: get ACC version - redhat + ansible.builtin.command: + cmd: "cat /etc/redhat-release" + changed_when: false + when: redhat_release.stat.exists + - name: get ACC version ansible.builtin.command: cmd: "cat /etc/issue" changed_when: false - register: acc_version + when: not redhat_release.stat.exists + +- name: extract p4 tarball + ansible.builtin.unarchive: + src: /opt/p4.tar.gz + dest: /opt/ + remote_src: true + +- name: copy P4 programs from ansible host + ansible.builtin.copy: + src: "{{ (ipu_tmp_dir, ipu_imc_p4_tarball) | path_join }}" + dest: "/tmp" + mode: '0644' + +- name: unarchive P4 programs tarball + ansible.builtin.unarchive: + src: "{{ ('/tmp', ipu_imc_p4_tarball) | path_join }}" + dest: "/tmp" + remote_src: true + mode: '0755' + +- name: copy active P4 program package on acc + ansible.builtin.copy: + src: "{{ ('/tmp', 'p4-programs', 'artifacts', active_p4_program) | path_join }}" + dest: "{{ active_p4_dir }}" + remote_src: true + mode: '0644' + +- name: create tofino.bin file + ansible.builtin.file: + path: "{{ active_p4_dir }}/{{ active_p4_program }}/tofino.bin" + state: touch + owner: root + group: root + mode: '0644' + +- name: set /etc/environment for P4 + ansible.builtin.lineinfile: + path: /etc/environment + state: present + regexp: '^{{ item.key }}' + line: '{{ item.key }}={{ item.value }}' + create: true + owner: root + group: root + mode: '0644' + with_items: + - { key: SDE_INSTALL, value: "{{ p4sde_install_dir }}" } + - { key: P4CP_INSTALL, value: "{{ p4cp_install_dir }}" } + - { key: DEPEND_INSTALL, value: '$P4CP_INSTALL' } + become: true + +- name: set .bashrc for P4 + ansible.builtin.lineinfile: + path: "{{ ansible_env.HOME }}/.bashrc" + state: present + regexp: '^{{ item.key }}' + line: '{{ item.key }}={{ item.value }}' + create: true + owner: "{{ ansible_user | default(ansible_user_id) }}" + group: "{{ ansible_user | default(ansible_user_id) }}" + mode: '0644' + with_items: + - { key: SDE_INSTALL, value: "{{ p4sde_install_dir }}" } + - { key: P4CP_INSTALL, value: "{{ p4cp_install_dir }}" } + - { key: DEPEND_INSTALL, value: '$P4CP_INSTALL' } + become: true + +- name: set /etc/environment for P4 continue + ansible.builtin.lineinfile: + path: /etc/environment + state: present + regexp: '^{{ item }}' + line: '{{ item }}' + create: true + owner: root + group: root + mode: '0644' + with_items: + - "alias ll='ls -l'" + - ". $P4CP_INSTALL/sbin/setup_env.sh $P4CP_INSTALL $SDE_INSTALL $DEPEND_INSTALL > /dev/null" + become: true + +- name: set .bashrc for P4 continue + ansible.builtin.lineinfile: + path: "{{ ansible_env.HOME }}/.bashrc" + state: present + regexp: '^{{ item }}' + line: '{{ item }}' + create: true + owner: "{{ ansible_user | default(ansible_user_id) }}" + group: "{{ ansible_user | default(ansible_user_id) }}" + mode: '0644' + with_items: + - "alias ll='ls -l'" + - ". $P4CP_INSTALL/sbin/setup_env.sh $P4CP_INSTALL $SDE_INSTALL $DEPEND_INSTALL > /dev/null" + become: true + +- name: reset ssh connection to take changed environment + ansible.builtin.meta: reset_connection + +- name: WA for issue in script + ansible.builtin.replace: + path: "{{ p4cp_install_dir }}/sbin/copy_config_files.sh" + regexp: '^sudo mkdir /usr/share/target_sys/$' + replace: 'sudo mkdir -p /usr/share/target_sys/' + +- name: copy config files + ansible.builtin.shell: + cmd: "$P4CP_INSTALL/sbin/copy_config_files.sh $P4CP_INSTALL $SDE_INSTALL" + changed_when: true + +- name: generate certs + ansible.builtin.shell: + cmd: "COMMON_NAME=localhost ./generate-certs.sh" + chdir: "{{ stratum_es2k_dir }}/" + args: + executable: /bin/bash + changed_when: true + +- name: copy generated certs + ansible.builtin.copy: + src: "{{ stratum_es2k_dir }}/certs" + dest: "/usr/share/stratum/" + remote_src: true + mode: '0755' + +- name: get bdf from acc + ansible.builtin.shell: + cmd: "set -o pipefail && lspci | grep 1453 | cut -f1 -d' '" + args: + executable: /bin/bash + register: acc_bdf + changed_when: false + failed_when: acc_bdf.stdout | length == 0 + +- name: bind bdf to vfio-pci driver + ansible.builtin.command: + cmd: "./dpdk-devbind.py -b vfio-pci {{ acc_bdf.stdout }}" + chdir: "{{ p4sde_bin_dir }}" + register: devbind_result + changed_when: true + +- name: set hugepages + ansible.builtin.command: + cmd: "./dpdk-hugepages.py -p 2M -r 2G" + chdir: "{{ p4sde_bin_dir }}" + register: hugepages_result + changed_when: true + +- name: get IOMMU group for bdf from acc + ansible.builtin.shell: + cmd: "set -o pipefail && lspci -vv -s {{ acc_bdf.stdout }} | grep 'IOMMU group:' | cut -f2 -d':' | tr -d ' '" + args: + executable: /bin/bash + register: acc_bdf_group + changed_when: false + failed_when: acc_bdf_group.stdout | length == 0 + +- name: prepare es2k_skip_p4.conf for active P4 program + ansible.builtin.template: + src: "es2k_skip_p4.conf.j2" + dest: "{{ stratum_es2k_dir }}/es2k_skip_p4.conf" + force: true + backup: true + mode: '0644' + +- name: build tdi pipeline + ansible.builtin.shell: + cmd: >- + "$P4CP_INSTALL/bin/tdi_pipeline_builder" + "--p4c_conf_file={{ stratum_es2k_dir }}/es2k_skip_p4.conf" + "--bf_pipeline_config_binary_file={{ active_p4_dir }}/{{ active_p4_program }}/{{ active_p4_program }}.pb.bin" + args: + executable: /bin/bash + changed_when: true + +- name: kill old infrap4d + ansible.builtin.shell: + cmd: "set -o pipefail && p4d_pid=$(ps -ef | grep 'infrap4d' | grep -v 'grep' | awk '{print $2;}'); if [ ! -z \"$p4d_pid\" ]; then kill $p4d_pid; fi" + args: + executable: /bin/bash + changed_when: true + +- name: start infrap4d + ansible.builtin.command: + cmd: "infrap4d" + changed_when: true + +- name: wait for infrap4d port + ansible.builtin.wait_for: + port: 9559 + host: 'localhost' + delay: 1 + timeout: 15 + +- name: set pipe to br0 + ansible.builtin.command: + cmd: "p4rt-ctl set-pipe br0 {{ active_p4_program }}.pb.bin p4Info.txt" + chdir: "{{ active_p4_dir }}/{{ active_p4_program }}" + changed_when: true + +- name: set rules to br0 + ansible.builtin.command: + cmd: "p4rt-ctl add-entry br0 {{ item }}" + chdir: "{{ active_p4_dir }}/{{ active_p4_program }}" + changed_when: true + with_items: + - "my_control.e_fwd \"hdrs.mac[vmeta.common.depth].da=0x000000000333,hdrs.mac[vmeta.common.depth].sa=0x9ebace98d9d3,action=my_control.send(1)\"" + - "my_control.i_fwd \"hdrs.mac[vmeta.common.depth].da=0x000000000343,hdrs.mac[vmeta.common.depth].sa=0x9ebace98d9d3,action=my_control.send(26)\"" diff --git a/roles/ipu/acc/templates/es2k_skip_p4.conf.j2 b/roles/ipu/acc/templates/es2k_skip_p4.conf.j2 new file mode 100644 index 00000000..4fdf058e --- /dev/null +++ b/roles/ipu/acc/templates/es2k_skip_p4.conf.j2 @@ -0,0 +1,41 @@ +{ + "chip_list": [ + { + "id": "asic-0", + "chip_family": "mev", + "instance": 0, + "pcie_bdf": "0000:{{ acc_bdf.stdout }}", + "iommu_grp_num": {{ acc_bdf_group.stdout }} + } + ], + "instance": 0, + "cfgqs-idx": "0-15", + "p4_devices": [ + { + "device-id": 0, + "fixed_functions" : [], + "eal-args": "--lcores=1-2 -a {{ acc_bdf.stdout }},vport=[0-1] -- -i --rxq=1 --txq=1 --hairpinq=1 --hairpin-mode=0x0", + "p4_programs": [ + { + "program-name": "{{ active_p4_program }}", + "bfrt-config": "{{ active_p4_dir }}/{{ active_p4_program }}/bf-rt.json", + "p4_pipelines": [ + { + "p4_pipeline_name": "main", + "context": "{{ active_p4_dir }}/{{ active_p4_program }}/context.json", + "config": "{{ active_p4_dir }}/{{ active_p4_program }}/tofino.bin", + "pipe_scope": [ + 0, + 1, + 2, + 3 + ], + "path": "{{ active_p4_dir }}/{{ active_p4_program }}" + } + ] + } + ], + "agent0": "lib/libpltfm_mgr.so" + } + ] +} diff --git a/roles/ipu/common/defaults/main.yml b/roles/ipu/common/defaults/main.yml index 673a62aa..ebc7b4a5 100644 --- a/roles/ipu/common/defaults/main.yml +++ b/roles/ipu/common/defaults/main.yml @@ -23,16 +23,24 @@ ipu_1gbe_link_interface_ip: "100.0.0.1/24" ipu_flavor: "release" ipu_build: "ci" -ipu_build_number: "4988" +ipu_build_number: "6330" # IPU host or IPU linkp based on variable ipu_1gbe_connected_to_linkp -ipu_ssd_image_tarball: "mev-hw-b0-{{ ipu_build }}-ts.{{ ipu_flavor }}.{{ ipu_build_number }}-mev-rl.tgz" +ipu_ssd_image_tarball: "hw-ssd.{{ ipu_build_number }}.tgz" # IPU linkp -ipu_nvm_image_tarball: "mev-hw-b0-{{ ipu_build }}-ts.{{ ipu_flavor }}.{{ ipu_build_number }}-imc.tgz" +ipu_nvm_image_tarball: "hw-flash.{{ ipu_build_number }}.tgz" ipu_eth_programmer_version: "2.0.1" ipu_eth_programmer_zip: "EthProgrammer-{{ ipu_eth_programmer_version }}.zip" +# IPU IMC +ipu_imc_p4_tarball: "hw-p4-programs.{{ ipu_build_number }}.tgz" +ipu_imc_persistent_dir: "/work" +ipu_imc_script_dir: "{{ (ipu_imc_persistent_dir, 'scripts') | path_join }}" +ipu_imc_package_dir: "/etc/dpcp/package" +ipu_imc_cfg_dir: "/etc/dpcp/cfg" +active_p4_program: "l2-fwd_lem" + ssh_options: "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" imc_ssh_port: "22" imc_user: "root" diff --git a/roles/ipu/common/tasks/main.yml b/roles/ipu/common/tasks/main.yml index e78ae5f8..27eaaecc 100644 --- a/roles/ipu/common/tasks/main.yml +++ b/roles/ipu/common/tasks/main.yml @@ -22,34 +22,38 @@ ansible.builtin.assert: that: - (ansible_distribution == "Rocky" and ansible_distribution_version == "9.1") or + (ansible_distribution == "Rocky" and ansible_distribution_version == "9.2") or (ansible_distribution == "Fedora") fail_msg: - "Current OS - {{ ansible_distribution }} {{ ansible_distribution_version }} - is not supported for IPU" - - "Supported OSes are Rocky 9.1 and Fedora" + - "Supported OSes are Rocky 9.1, Rocky 9.2 and Fedora" + when: + - inventory_hostname in groups['ipu_host'] or + inventory_hostname in groups['ipu_linkp'] - name: select the right connection host qroup ansible.builtin.set_fact: - connection_host_group: "{% if ipu_1gbe_connected_to_linkp %}ipu_linkp{% else %}ipu_host{% endif %}" + connection_host_group: "{% if ipu_1gbe_connected_to_linkp | bool %}ipu_linkp{% else %}ipu_host{% endif %}" - name: setup 1GbE conncetion when: - inventory_hostname in groups[connection_host_group] block: - name: check IP address - ansible.builtin.shell: "set -o pipefail && ip a show eno2 |grep \"inet \"" + ansible.builtin.shell: "set -o pipefail && ip a show {{ ipu_1gbe_link_interface }} | grep \"inet \"" args: executable: /bin/bash register: ip_status changed_when: false failed_when: '" does not exist." in ip_status.stderr' - - name: assign IP address to interface for 1GbE link from IPU - ansible.builtin.command: - cmd: "ip a add {{ ipu_1gbe_link_interface_ip }} dev {{ ipu_1gbe_link_interface }}" + - name: nmcli assign IP address to interface for 1GbE link from IPU + community.general.nmcli: + conn_name: "{{ ipu_1gbe_link_interface }}" + ifname: "{{ ipu_1gbe_link_interface }}" + type: ethernet + ip4: "{{ ipu_1gbe_link_interface_ip }}" + gw4: "{{ ipu_1gbe_link_interface_ip | ansible.utils.ipaddr('address') }}" + state: present when: - ipu_1gbe_link_interface_ip not in ip_status.stdout - - - name: bring 1GbE link interface up - ansible.builtin.command: - cmd: "ip link set dev {{ ipu_1gbe_link_interface }} up" - changed_when: true diff --git a/roles/ipu/flash_ipu_nvm/defaults/main.yml b/roles/ipu/flash_ipu_nvm/defaults/main.yml index 4b1fee80..8fbbfa61 100644 --- a/roles/ipu/flash_ipu_nvm/defaults/main.yml +++ b/roles/ipu/flash_ipu_nvm/defaults/main.yml @@ -15,8 +15,10 @@ ## --- eth_programmer_dir: "{{ (ipu_project_root_dir, 'net6') | path_join }}" -ipu_nvm_image_file: "{{ (ipu_project_root_dir, 'imc', 'images', 'anvm-image', 'release', 'image_256M', 'anvm-image.bin') | path_join }}" +ipu_nvm_image_file: "{{ (ipu_project_root_dir, 'flash', 'irot', 'nvm-image-recovery_50008.bin') | path_join }}" # USB addresses of USB1 and USB3 to be unbind from ftdi_sio driver -usb_addresses: - - '1-3:1.1' - - '1-3:1.3' +# Those addresses can change on different platforms, so we provide reasonable patterns to select them +# Address example: '1-3:1.1' and '1-3:1.3' +usb_address_patterns: + - '.-.:..1' + - '.-.:..3' diff --git a/roles/ipu/flash_ipu_nvm/tasks/main.yml b/roles/ipu/flash_ipu_nvm/tasks/main.yml index d055c2b4..6dc0aab1 100644 --- a/roles/ipu/flash_ipu_nvm/tasks/main.yml +++ b/roles/ipu/flash_ipu_nvm/tasks/main.yml @@ -18,14 +18,14 @@ ansible.builtin.copy: src: "{{ (ipu_tmp_dir, ipu_nvm_image_tarball) | path_join }}" dest: "{{ (ipu_project_root_dir, ipu_nvm_image_tarball) | path_join }}" - mode: 0644 + mode: '0644' - name: unarchive nvm image ansible.builtin.unarchive: src: "{{ (ipu_project_root_dir, ipu_nvm_image_tarball) | path_join }}" dest: "{{ ipu_project_root_dir }}" - remote_src: yes - mode: 0755 + remote_src: true + mode: '0755' - name: list USB device binding ansible.builtin.find: @@ -33,12 +33,28 @@ file_type: "link" register: ftdi_sio_out +- name: set USB device addresses + ansible.builtin.command: + cmd: "basename {{ ftdi_sio_out.files|selectattr('path', 'search', item)|map(attribute='path')|replace('[','')|replace(']','') }}" + with_items: "{{ usb_address_patterns }}" + register: selected_usb_addresses + when: + - ftdi_sio_out.files|selectattr('path', 'search', item)|map(attribute='path')|replace('[','')|replace(']','') + +- name: show selected USB device addresses to be unbind + ansible.builtin.debug: + msg: "USB: {{ item.stdout }}" + with_items: "{{ selected_usb_addresses.results }}" + when: + - not item.skipped | default(false) + - name: unbind USB1 and USB3 from ftdi_sio driver - ansible.builtin.shell: "set -o pipefail && echo '{{ item }}' > /sys/bus/usb/drivers/ftdi_sio/unbind" + ansible.builtin.shell: "set -o pipefail && echo '{{ item.stdout }}' > {{ ftdi_sio_driver_dir }}/unbind" args: executable: /bin/bash - with_items: "{{ usb_addresses }}" - when: ftdi_sio_out.files|selectattr("path", "search", item)|list|length == 1 + with_items: "{{ selected_usb_addresses.results }}" + when: + - not item.skipped | default(false) - name: add executable permission for EthProgrammer ansible.builtin.file: @@ -58,28 +74,75 @@ environment: DOTNET_SYSTEM_GLOBALIZATION_INVARIANT: 1 -# ipmi ansible module does not support power cycle -- name: power off IPU host - ansible.builtin.command: - cmd: "ipmitool -I lan -H {{ ipmi_ip }} -U {{ ipmi_user }} -P '{{ ipma_password }}' chassis power off" - changed_when: true - -- name: wait before power on again - ansible.builtin.pause: - prompt: "Waiting before power on..." - seconds: 10 +- name: IPU images are flashed + ansible.builtin.debug: + msg: "IPU images are flashed - power cycle / reboot IPU host machine" -- name: power on IPU host - ansible.builtin.command: - cmd: "ipmitool -I lan -H {{ ipmi_ip }} -U {{ ipmi_user }} -P '{{ ipma_password }}' chassis power on" - changed_when: true +- name: block for powercycle via ipmitool + when: + - ipmi_ip is defined + - ipmi_user is defined + - ipma_password is defined + block: + - name: install ipmitool if needed + ansible.builtin.package: + name: + - ipmitool + state: present + delegate_to: localhost + become: true + when: not ipu_1gbe_connected_to_linkp | bool + + - name: Set IPU host address + ansible.builtin.set_fact: + ipu_host_address: >- + {% if hostvars[groups['ipu_host'][0]]['ip'] is defined %}{{ hostvars[groups['ipu_host'][0]]['ip'] }}{% else %} + {{ hostvars[groups['ipu_host'][0]]['ansible_host'] }}{% endif %} + + # inventory_hostname value bellow means that we want to delegate to current host. It skips delegation. + # So, tasks are delegated to localhost only when condition is met + - name: set node to be used for delegate_to + ansible.builtin.set_fact: + node_used_for_delegate_to: "{% if not ipu_1gbe_connected_to_linkp | bool %}localhost{% else %}{{ inventory_hostname }}{% endif %}" + + # ipmi ansible module does not support power cycle + - name: power off IPU host + ansible.builtin.command: + cmd: "ipmitool -I lan -H {{ ipmi_ip }} -U {{ ipmi_user }} -P '{{ ipma_password }}' chassis power off" + changed_when: true + delegate_to: "{{ node_used_for_delegate_to }}" + + - name: wait before power on again + ansible.builtin.pause: + prompt: "Waiting before power on..." + seconds: 10 + delegate_to: "{{ node_used_for_delegate_to }}" + + - name: power on IPU host + ansible.builtin.command: + cmd: "ipmitool -I lan -H {{ ipmi_ip }} -U {{ ipmi_user }} -P '{{ ipma_password }}' chassis power on" + changed_when: true + delegate_to: "{{ node_used_for_delegate_to }}" + + - name: wait for ssh connection to IPU host + ansible.builtin.wait_for: + port: 22 + host: "{{ ipu_host_address | trim }}" + search_regex: OpenSSH + delay: 1 + timeout: 1200 + delegate_to: "{{ node_used_for_delegate_to }}" -- name: wait for ssh connection to IPU host - ansible.builtin.wait_for: - port: 22 - host: "{{ hostvars[groups['ipu_host'][0]]['ansible_default_ipv4']['address'] }}" - search_regex: OpenSSH - delay: 1 +- name: block for reboot when ipmi is not available + when: + - ipmi_ip is not defined or ipmi_user is not defined or ipma_password is not defined + block: + - name: reboot ipu host + ansible.builtin.reboot: + reboot_timeout: 1200 + delegate_to: "{{ item }}" + become: true + with_items: "{{ groups['ipu_host'] }}" - name: wait for ssh connection to IPU-IMC ansible.builtin.wait_for: @@ -94,14 +157,22 @@ changed_when: false register: imc_hostname +- name: Set IPU linkp address + ansible.builtin.set_fact: + ipu_linkp_address: >- + {% if hostvars[inventory_hostname]['ip'] is defined %}{{ hostvars[inventory_hostname]['ip'] }}{% else %} + {{ hostvars[inventory_hostname]['ansible_host'] }}{% endif %} + - name: Update /etc/hosts with ipu_linkp ansible.builtin.blockinfile: path: /etc/hosts block: | - {{ hostvars[inventory_hostname]['ansible_default_ipv4']['address'] }} {{ inventory_hostname }} + {{ ipu_linkp_address | trim }} {{ inventory_hostname }} marker: "# {mark} ANSIBLE MANAGED BLOCK {{ inventory_hostname }}" delegate_to: localhost become: false + when: + - ipu_linkp_address | trim | ansible.utils.ipaddr('bool') - name: Set login_user fact ansible.builtin.set_fact: @@ -143,7 +214,7 @@ src: "{{ inventory_file }}" dest: "{{ ipu_inventory_name }}" mode: '0644' - force: yes + force: true owner: "{{ local_login_user }}" group: "{{ local_login_user }}" run_once: true @@ -159,7 +230,7 @@ StrictHostKeyChecking no UserKnownHostsFile /dev/null marker: "# {mark} ANSIBLE MANAGED BLOCK {{ imc_hostname.stdout }}" - create: yes + create: true owner: "{{ local_login_user }}" group: "{{ local_login_user }}" mode: '0644' @@ -192,8 +263,8 @@ {{ imc_hostname.stdout }} ansible_host={{ imc_static_ip }} ip={{ imc_static_ip }} ansible_user={{ login_user }} ansible_ssh_user={{ ansible_user }} ansible_ssh_password='{{ ansible_password }}' inventory_dir={{ inventory_dir }} ansible_ssh_common_args='{{ ssh_options }} -o ProxyCommand="ssh -p 22 -W %h:%p {{ ssh_options }} -q {{ inventory_hostname }}"' - no_extra_spaces: yes - allow_no_value: yes + no_extra_spaces: true + allow_no_value: true mode: '0644' state: present backup: no @@ -205,8 +276,8 @@ dest: "{{ ipu_inventory_name }}" section: "ipu_imc" option: "{{ imc_hostname.stdout }}" - no_extra_spaces: yes - allow_no_value: yes + no_extra_spaces: true + allow_no_value: true mode: '0644' state: present backup: no @@ -282,7 +353,7 @@ StrictHostKeyChecking no UserKnownHostsFile /dev/null marker: "# {mark} ANSIBLE MANAGED BLOCK {{ acc_hostname.stdout }}" - create: yes + create: true owner: "{{ local_login_user }}" group: "{{ local_login_user }}" mode: '0644' @@ -318,8 +389,8 @@ ansible_ssh_user={{ ansible_user }} ansible_ssh_password='{{ ansible_password }}' inventory_dir={{ inventory_dir }} ansible_ssh_common_args='{{ ssh_options }} -o ProxyCommand="ssh -W %h:%p {{ ssh_options }} -o ProxyCommand=\"ssh -W {{ imc_static_ip }}:22 {{ ssh_options }} {{ inventory_hostname }}\" {{ imc_static_ip }}"' - no_extra_spaces: yes - allow_no_value: yes + no_extra_spaces: true + allow_no_value: true mode: '0644' state: present backup: no @@ -331,8 +402,8 @@ dest: "{{ ipu_inventory_name }}" section: "ipu_acc" option: "{{ acc_hostname.stdout }}" - no_extra_spaces: yes - allow_no_value: yes + no_extra_spaces: true + allow_no_value: true mode: '0644' state: present backup: no diff --git a/roles/ipu/flash_ipu_ssd/defaults/main.yml b/roles/ipu/flash_ipu_ssd/defaults/main.yml index 67f5cece..e0a92c61 100644 --- a/roles/ipu/flash_ipu_ssd/defaults/main.yml +++ b/roles/ipu/flash_ipu_ssd/defaults/main.yml @@ -16,6 +16,6 @@ --- nvme1_dev: "/dev/nvme0n1" dd_prefix: "dd bs=16M" -ipu_ssd_image_file: "{{ (ipu_project_root_dir, 'mev', 'images', 'nvme-image-mev.bin') | path_join }}" +ipu_ssd_image_file: "{{ (ipu_project_root_dir, 'ssd', 'nvme-image-mev.bin') | path_join }}" dd_src_cmd: "{{ dd_prefix }} if={{ ipu_ssd_image_file }}" dd_dst_cmd: "{{ dd_prefix }} of={{ nvme1_dev }}" diff --git a/roles/ipu/flash_ipu_ssd/tasks/main.yml b/roles/ipu/flash_ipu_ssd/tasks/main.yml index 6dfebfde..1646e978 100644 --- a/roles/ipu/flash_ipu_ssd/tasks/main.yml +++ b/roles/ipu/flash_ipu_ssd/tasks/main.yml @@ -22,20 +22,20 @@ ansible.builtin.file: path: "{{ ipu_project_root_dir }}" state: directory - mode: 0755 + mode: '0755' - name: copy ssd image from ansible host ansible.builtin.copy: src: "{{ (ipu_tmp_dir, ipu_ssd_image_tarball) | path_join }}" dest: "{{ (ipu_project_root_dir, ipu_ssd_image_tarball) | path_join }}" - mode: 0644 + mode: '0644' - name: unarchive ssd image ansible.builtin.unarchive: src: "{{ (ipu_project_root_dir, ipu_ssd_image_tarball) | path_join }}" dest: "{{ ipu_project_root_dir }}" - remote_src: yes - mode: 0755 + remote_src: true + mode: '0755' - name: check if ssd image is available ansible.builtin.stat: @@ -47,14 +47,6 @@ msg: "ssd image {{ ipu_ssd_image_file }} does not exist" when: not ssd_image_stats.stat.exists -- name: syslogmode DEV - ansible.builtin.shell: "set -o pipefail && ssh -p {{ imc_ssh_port }} {{ ssh_options }} {{ imc_user }}@{{ imc_static_ip }} /etc/init.d/syslogmode DEV" - args: - executable: /bin/bash - register: syslogmode_out - changed_when: "'DEV mode, syslog disabled' in syslogmode_out.stdout" - failed_when: syslogmode_out.rc != 0 - - name: umount loop0 ansible.builtin.shell: "set -o pipefail && ssh -p {{ imc_ssh_port }} {{ ssh_options }} {{ imc_user }}@{{ imc_static_ip }} umount /dev/loop0" args: @@ -63,7 +55,7 @@ changed_when: loop0_out.stderr | length == 0 failed_when: - loop0_out.rc != 0 - - "'t unmount /dev/loop0' not in loop0_out.stderr" + - "'umount: /dev/loop0: not mounted' not in loop0_out.stderr" - "'Invalid argument' not in loop0_out.stderr" - name: umount nvme0n1 @@ -74,7 +66,8 @@ changed_when: nvme0n1_out.stderr | length == 0 failed_when: - nvme0n1_out.rc != 0 - - "'t unmount /dev/nvme0n1' not in nvme0n1_out.stderr" + - "'umount: /dev/nvme0n1: not mounted' not in nvme0n1_out.stderr" + - "': not mounted' not in nvme0n1_out.stderr" - "'Invalid argument' not in nvme0n1_out.stderr" - name: kill tgtd diff --git a/roles/ipu/imc/tasks/main.yml b/roles/ipu/imc/tasks/main.yml index fc76a0c5..c292330a 100644 --- a/roles/ipu/imc/tasks/main.yml +++ b/roles/ipu/imc/tasks/main.yml @@ -18,4 +18,49 @@ ansible.builtin.command: cmd: "cat /etc/issue" changed_when: false - register: imc_version + +- name: get IPU board ID + ansible.builtin.command: + cmd: "ls -lah /etc/hwconf/active" + changed_when: false + +- name: copy P4 programs from ansible host + ansible.builtin.copy: + src: "{{ (ipu_tmp_dir, ipu_imc_p4_tarball) | path_join }}" + dest: "{{ (ipu_imc_persistent_dir, ipu_imc_p4_tarball) | path_join }}" + mode: '0644' + +- name: unarchive P4 programs tarball + ansible.builtin.unarchive: + src: "{{ (ipu_imc_persistent_dir, ipu_imc_p4_tarball) | path_join }}" + dest: "{{ ipu_imc_persistent_dir }}" + remote_src: true + owner: root + group: root + mode: '0755' + +- name: copy active P4 program package + ansible.builtin.copy: + src: "{{ (ipu_imc_persistent_dir, 'p4-programs', 'artifacts', active_p4_program, active_p4_program) | path_join }}.pkg" + dest: "{{ ipu_imc_script_dir }}" + remote_src: true + mode: '0644' + +- name: prepare load_custom_pkg.sh script for active P4 program + ansible.builtin.template: + src: "{{ active_p4_program }}.sh.j2" + dest: "{{ (ipu_imc_script_dir, 'load_custom_pkg.sh') | path_join }}" + force: true + backup: true + mode: '0755' + +- name: reboot IMC to reload P4 program + ansible.builtin.reboot: + +- name: wait for ssh connection from IPU-IMC to IPU-ACC after P4 reload + ansible.builtin.wait_for: + port: 22 + host: '{{ acc_static_ip }}' + search_regex: OpenSSH + delay: 1 + timeout: 3600 diff --git a/roles/ipu/imc/templates/l2-fwd_lem.sh.j2 b/roles/ipu/imc/templates/l2-fwd_lem.sh.j2 new file mode 100644 index 00000000..44c27c95 --- /dev/null +++ b/roles/ipu/imc/templates/l2-fwd_lem.sh.j2 @@ -0,0 +1,16 @@ +#!/bin/sh +CP_INIT_CFG={{ ipu_imc_cfg_dir }}/cp_init_use_case_2.cfg +ln -sf {{ ipu_imc_cfg_dir }}/cp_init_use_case_2.cfg /mnt/imc/cfg/active_cp_init.cfg +echo "Checking for custom package..." +if [ -e {{ active_p4_program }}.pkg ]; then + echo "Custom package {{ active_p4_program }}.pkg found. Overriding default package" + cp {{ active_p4_program }}.pkg {{ ipu_imc_package_dir }} + rm -rf {{ ipu_imc_package_dir }}/default_pkg.pkg + ln -s {{ ipu_imc_package_dir }}/{{ active_p4_program }}.pkg {{ ipu_imc_package_dir }}/default_pkg.pkg + sed -i 's/sem_num_pages = 25;/sem_num_pages = 1;/g' $CP_INIT_CFG + sed -i 's/lem_num_pages = 1;/lem_num_pages = 52;/g' $CP_INIT_CFG +else + echo "No custom package found. Continuing with default package" +fi + +echo "{{ acc_static_ip }} ipu-acc" >> /etc/hosts diff --git a/roles/ipu/prepare_ipu_linkp/tasks/main.yml b/roles/ipu/prepare_ipu_linkp/tasks/main.yml index bcb7bb03..1843a99e 100644 --- a/roles/ipu/prepare_ipu_linkp/tasks/main.yml +++ b/roles/ipu/prepare_ipu_linkp/tasks/main.yml @@ -28,7 +28,7 @@ pu parity N pu stopbits 1 pu rtscts No - mode: 0644 + mode: '0644' - name: create ACC minicom file ansible.builtin.copy: @@ -40,26 +40,26 @@ pu parity N pu stopbits 1 pu rtscts No - mode: 0644 + mode: '0644' - name: Create IPU project directory ansible.builtin.file: path: "{{ ipu_project_root_dir }}" state: directory - mode: 0755 + mode: '0755' - name: copy EthProgrammer from ansible host ansible.builtin.copy: src: "{{ (ipu_tmp_dir, ipu_eth_programmer_zip) | path_join }}" dest: "{{ (ipu_project_root_dir, ipu_eth_programmer_zip) | path_join }}" - mode: 0644 + mode: '0644' - name: unarchive EthProgrammer ansible.builtin.unarchive: src: "{{ (ipu_project_root_dir, ipu_eth_programmer_zip) | path_join }}" dest: "{{ ipu_project_root_dir }}" - remote_src: yes - mode: 0755 + remote_src: true + mode: '0755' register: unzip_result failed_when: (unzip_result.extract_results.rc != 0) and ('appears to use backslashes as path separators' not in unzip_result.extract_results.err) diff --git a/roles/ipu/prepare_ipu_linkp/vars/main.yml b/roles/ipu/prepare_ipu_linkp/vars/main.yml index c8f87e02..50e78fd1 100644 --- a/roles/ipu/prepare_ipu_linkp/vars/main.yml +++ b/roles/ipu/prepare_ipu_linkp/vars/main.yml @@ -20,3 +20,5 @@ install_dependencies: - minicom - ipmitool - dotnet-sdk-6.0 + - vim + - scapy diff --git a/roles/istio_service_mesh/files/istio-netfilter.conf b/roles/istio_service_mesh/files/istio-netfilter.conf new file mode 100644 index 00000000..7e86dbfa --- /dev/null +++ b/roles/istio_service_mesh/files/istio-netfilter.conf @@ -0,0 +1,7 @@ +br_netfilter +nf_nat +xt_REDIRECT +xt_owner +iptable_nat +iptable_mangle +iptable_filter diff --git a/roles/istio_service_mesh/tasks/cleanup.yml b/roles/istio_service_mesh/tasks/cleanup.yml index bdb2941d..2805b29e 100644 --- a/roles/istio_service_mesh/tasks/cleanup.yml +++ b/roles/istio_service_mesh/tasks/cleanup.yml @@ -92,6 +92,12 @@ state: absent failed_when: false +- name: remove istio netconf modules file + file: + path: "/etc/modules-load.d/istio-netfilter.conf" + state: absent + failed_when: false + - name: wait for istio-system namespace removal command: | kubectl get ns {{ istio_service_mesh.istio_namespace }} diff --git a/roles/istio_service_mesh/tasks/main.yml b/roles/istio_service_mesh/tasks/main.yml index 90cedc22..ca5cb5bb 100644 --- a/roles/istio_service_mesh/tasks/main.yml +++ b/roles/istio_service_mesh/tasks/main.yml @@ -64,19 +64,28 @@ - inventory_hostname == groups['kube_control_plane'][0] - name: fix the iptable-restore error by https://github.com/istio/istio/issues/23009 - modprobe: - name: "{{ item }}" - state: present - with_items: - - "br_netfilter" - - "nf_nat" - - "xt_REDIRECT" - - "xt_owner" - - "iptable_nat" - - "iptable_mangle" - - "iptable_filter" + block: + - name: load modules for iptables + modprobe: + name: "{{ item }}" + state: present + with_items: + - "br_netfilter" + - "nf_nat" + - "xt_REDIRECT" + - "xt_owner" + - "iptable_nat" + - "iptable_mangle" + - "iptable_filter" + - name: make module loading for iptables persistent + copy: + src: "istio-netfilter.conf" + dest: "/etc/modules-load.d/istio-netfilter.conf" + owner: root + mode: '0644' when: - - ansible_distribution in ['RedHat', 'Rocky'] and inventory_hostname != groups['kube_control_plane'][0] + - ansible_distribution in ['RedHat', 'Rocky'] + - inventory_hostname in groups['kube_node'] - name: deploy TLS splicing and bumping include_tasks: tls-splicing-and-bumping.yml diff --git a/roles/istio_service_mesh/vars/main.yml b/roles/istio_service_mesh/vars/main.yml index bda58119..6c521acc 100644 --- a/roles/istio_service_mesh/vars/main.yml +++ b/roles/istio_service_mesh/vars/main.yml @@ -16,11 +16,11 @@ istio_service_mesh_defaults: enabled: false image: istio/istioctl - version: 1.18.1 + version: 1.19.0 intel_preview: enabled: false image: intel/istioctl - version: 1.18.0-intel.0 + version: 1.19.0-intel.0 context: '' filename: [] namespace: '' diff --git a/roles/jaeger_install/defaults/main.yml b/roles/jaeger_install/defaults/main.yml index 18dff597..478cf405 100644 --- a/roles/jaeger_install/defaults/main.yml +++ b/roles/jaeger_install/defaults/main.yml @@ -13,7 +13,7 @@ ## See the License for the specific language governing permissions and ## limitations under the License. ## -jaeger_version: v1.44.0 +jaeger_version: v1.49.0 jaeger_crd_url: https://github.com/jaegertracing/jaeger-operator/releases/download/{{ jaeger_version }}/jaeger-operator.yaml jaeger_annotations_key_to_remove: 'sidecar.jaegertracing.io/inject' jaeger_query_remove: >- diff --git a/roles/kmra_install/charts/kmra-apphsm/templates/kmra-apphsm-config-configmap.yaml b/roles/kmra_install/charts/kmra-apphsm/templates/kmra-apphsm-config-configmap.yaml index e5c7c137..436e68cc 100644 --- a/roles/kmra_install/charts/kmra-apphsm/templates/kmra-apphsm-config-configmap.yaml +++ b/roles/kmra_install/charts/kmra-apphsm/templates/kmra-apphsm-config-configmap.yaml @@ -37,6 +37,7 @@ data: "token_name": {{ $key.token_name | quote }}, "pin": {{ $key.pin | quote }}, "key_name": {{ $key.key_name | quote }}, + "key_path": {{ $key.key_path | quote }}, "certificate_file": "{{ $key.token_cert }}" } {{- $_ := set $local "first" false -}} diff --git a/roles/kmra_install/charts/kmra-apphsm/templates/kmra-apphsm-deployment.yml b/roles/kmra_install/charts/kmra-apphsm/templates/kmra-apphsm-deployment.yml index ce082ef5..ef1eb015 100644 --- a/roles/kmra_install/charts/kmra-apphsm/templates/kmra-apphsm-deployment.yml +++ b/roles/kmra_install/charts/kmra-apphsm/templates/kmra-apphsm-deployment.yml @@ -65,12 +65,28 @@ spec: readOnly: true # we have to mount these keys one by one, since apphsm doesn't like symbol links - name: custom-config - mountPath: /opt/intel/custom_tls/server.key - subPath: server.key + mountPath: /opt/intel/custom_tls/server_cu.key + subPath: server_cu.key readOnly: true - name: custom-config - mountPath: /opt/intel/custom_tls/server.crt - subPath: server.crt + mountPath: /opt/intel/custom_tls/server_cu.crt + subPath: server_cu.crt + readOnly: true + - name: custom-config + mountPath: /opt/intel/custom_tls/server_du.key + subPath: server_du.key + readOnly: true + - name: custom-config + mountPath: /opt/intel/custom_tls/server_du.crt + subPath: server_du.crt + readOnly: true + - name: custom-config + mountPath: /opt/intel/custom_tls/server_ric.key + subPath: server_cu.key + readOnly: true + - name: custom-config + mountPath: /opt/intel/custom_tls/server_ric.crt + subPath: server_cu.crt readOnly: true - name: custom-config mountPath: /opt/intel/custom_tls/client.key diff --git a/roles/kmra_install/charts/kmra-apphsm/templates/kmra-apphsm-env-configmap.yml b/roles/kmra_install/charts/kmra-apphsm/templates/kmra-apphsm-env-configmap.yml index 5116abd9..c8377fb7 100644 --- a/roles/kmra_install/charts/kmra-apphsm/templates/kmra-apphsm-env-configmap.yml +++ b/roles/kmra_install/charts/kmra-apphsm/templates/kmra-apphsm-env-configmap.yml @@ -8,4 +8,4 @@ data: http_proxy: {{ .Values.http_proxy | default "" | quote }} https_proxy: {{ .Values.https_proxy | default "" | quote }} no_proxy: {{ .Values.no_proxy | default "" | quote }} - + ECDSA_KEYS: {{ .Values.apphsm.ecdsa_keys | default "false" | upper | quote }} diff --git a/roles/kmra_install/charts/kmra-ctk/templates/kmra-ctk-loadkey-configmap.yml b/roles/kmra_install/charts/kmra-ctk/templates/kmra-ctk-loadkey-configmap.yml index d999663e..e47551e4 100644 --- a/roles/kmra_install/charts/kmra-ctk/templates/kmra-ctk-loadkey-configmap.yml +++ b/roles/kmra_install/charts/kmra-ctk/templates/kmra-ctk-loadkey-configmap.yml @@ -20,3 +20,4 @@ data: DEFAULT_CLIENT_TOKEN_ID: {{ .Values.ctk_loadkey.default_client_token_id | quote }} PKCS11_PROXY_TLS_PSK_FILE: {{ .Values.ctk_loadkey.pkcs11_proxy_tls_psk_file | quote }} PKCS11_DAEMON_SOCKET: "tls://{{ .Values.ctk_loadkey.pkcs11_daemon_socket_hostname }}:{{ .Values.ctk_loadkey.pkcs11_daemon_socket_port }}" + ECDSA_KEYS: {{ .Values.ctk_loadkey.ecdsa_keys | default "false" | upper | quote }} diff --git a/roles/kmra_install/charts/kmra-oran-netopeer2-client/templates/kmra-oran-netopeer2-client-deployment.yml b/roles/kmra_install/charts/kmra-oran-netopeer2-client/templates/kmra-oran-netopeer2-client-deployment.yml index 22a814aa..236159e0 100644 --- a/roles/kmra_install/charts/kmra-oran-netopeer2-client/templates/kmra-oran-netopeer2-client-deployment.yml +++ b/roles/kmra_install/charts/kmra-oran-netopeer2-client/templates/kmra-oran-netopeer2-client-deployment.yml @@ -132,9 +132,6 @@ spec: - name: p11-proxy-tls-psk configMap: name: {{ .Release.Name }}-p11-proxy-tls-psk-conf - - name: oran-env - configMap: - name: {{ .Release.Name }}-oran-env - name: sysrepo-config configMap: - name: oran-sysrepo-config + name: {{ .Release.Name }}-oran-sysrepo-config diff --git a/roles/kmra_install/charts/kmra-oran-netopeer2-client/templates/kmra-oran-netopeer2-client-env-configmap.yml b/roles/kmra_install/charts/kmra-oran-netopeer2-client/templates/kmra-oran-netopeer2-client-env-configmap.yml index 787e0d81..2ed2a1a8 100644 --- a/roles/kmra_install/charts/kmra-oran-netopeer2-client/templates/kmra-oran-netopeer2-client-env-configmap.yml +++ b/roles/kmra_install/charts/kmra-oran-netopeer2-client/templates/kmra-oran-netopeer2-client-env-configmap.yml @@ -19,3 +19,4 @@ data: DEFAULT_SO_PIN: {{ .Values.oran_netopeer2_client.default_so_pin | quote }} DEFAULT_CLIENT_TOKEN_ID: {{ .Values.oran_netopeer2_client.default_client_token_id | quote }} PKCS11_PROXY_TLS_PSK_FILE: {{ .Values.oran_netopeer2_client.pkcs11_proxy_tls_psk_file | quote }} + ECDSA_KEYS: {{ .Values.oran_netopeer2_client.ecdsa_keys | default "false" | upper | quote }} diff --git a/roles/kmra_install/charts/kmra-oran-netopeer2-server/templates/kmra-oran-netopeer2-server-deployment.yml b/roles/kmra_install/charts/kmra-oran-netopeer2-server/templates/kmra-oran-netopeer2-server-deployment.yml index 09ac5a70..dc07566f 100644 --- a/roles/kmra_install/charts/kmra-oran-netopeer2-server/templates/kmra-oran-netopeer2-server-deployment.yml +++ b/roles/kmra_install/charts/kmra-oran-netopeer2-server/templates/kmra-oran-netopeer2-server-deployment.yml @@ -135,9 +135,6 @@ spec: - name: p11-proxy-tls-psk configMap: name: {{ .Release.Name }}-p11-proxy-tls-psk-conf - - name: oran-env - configMap: - name: {{ .Release.Name }}-oran-env - name: sysrepo-config configMap: - name: oran-sysrepo-config + name: {{ .Release.Name }}-oran-sysrepo-config diff --git a/roles/kmra_install/charts/kmra-oran-netopeer2-server/templates/kmra-oran-netopeer2-server-env-configmap.yml b/roles/kmra_install/charts/kmra-oran-netopeer2-server/templates/kmra-oran-netopeer2-server-env-configmap.yml index 00b8c2ae..0d82a3eb 100644 --- a/roles/kmra_install/charts/kmra-oran-netopeer2-server/templates/kmra-oran-netopeer2-server-env-configmap.yml +++ b/roles/kmra_install/charts/kmra-oran-netopeer2-server/templates/kmra-oran-netopeer2-server-env-configmap.yml @@ -19,3 +19,4 @@ data: DEFAULT_SO_PIN: {{ .Values.oran_netopeer2_server.default_so_pin | quote }} DEFAULT_CLIENT_TOKEN_ID: {{ .Values.oran_netopeer2_server.default_client_token_id | quote }} PKCS11_PROXY_TLS_PSK_FILE: {{ .Values.oran_netopeer2_server.pkcs11_proxy_tls_psk_file | quote }} + ECDSA_KEYS: {{ .Values.oran_netopeer2_server.ecdsa_keys | default "false" | upper | quote }} diff --git a/roles/kmra_install/defaults/main/main.yml b/roles/kmra_install/defaults/main/main.yml index ece1fc24..efcc47c3 100644 --- a/roles/kmra_install/defaults/main/main.yml +++ b/roles/kmra_install/defaults/main/main.yml @@ -23,7 +23,7 @@ kmra_defaults: OU: "root" CN: "localhost" certs_validity_period_days: 365 - image_tag: "v2.3" # can be overriden in each component section if needed + image_tag: "v2.4" # can be overriden in each component section if needed apphsm: enabled: false release_name: "kmra-apphsm" @@ -35,7 +35,7 @@ kmra_defaults: sbx_image_repo: "{{ registry_local_address }}" sbx_image_name: "apphsm" sbx_image_staging_location: "/tmp/apphsm/apphsm.sbx.tar" - sbx_image_checksum: "b32757e1263c6de52903ba31583e3d61274044f30d2270541c3d85c523224f02" + sbx_image_checksum: "8eb70e7e0b9db7e58ed8cd91f9512b726a3a6288775841daf65fadf89802e208" # sbx_image_tag: "" init_image_repo: "docker.io" init_image_name: "busybox" @@ -61,7 +61,6 @@ kmra_defaults: pin: "1234" key_name: "client_key_priv" token_cert: "testcert1.crt" - token_key: "testkey1.pem" crt_subj: O: "SampleOrganisation" CN: "localhost" @@ -70,7 +69,6 @@ kmra_defaults: pin: "1234" key_name: "key_2" token_cert: "testcert2.crt" - token_key: "testkey2.pem" crt_subj: O: "SampleOrganisation" CN: "localhost" @@ -79,10 +77,10 @@ kmra_defaults: pin: "1234" key_name: "key_3" token_cert: "testcert3.crt" - token_key: "testkey3.pem" crt_subj: O: "SampleOrganisation" CN: "localhost" + ecdsa_keys: "false" pccs: enabled: false api_key: "ffffffffffffffffffffffffffffffff" diff --git a/roles/kmra_install/defaults/main/oran.yml b/roles/kmra_install/defaults/main/oran.yml index ee311912..644da4c0 100644 --- a/roles/kmra_install/defaults/main/oran.yml +++ b/roles/kmra_install/defaults/main/oran.yml @@ -18,33 +18,51 @@ # Define respective field in the group_vars/all.yml instead kmra_oran: apphsm: - oran_netopeer2_sample_tls_url: "https://raw.githubusercontent.com/CESNET/netopeer2/v2.1.62/example_configuration/" oran_netopeer2_cert_user_id: "ctk_loadkey_user_id_01234" enabled: true app_keys: - - id: "unique_id_1234s" - token_name: "token_server" + - id: "unique_id_1234s_cu" + token_name: "token_server_cu" pin: "1234" - key_name: "/opt/intel/custom_tls/server.key" - token_cert: "/opt/intel/custom_tls/server.crt" - token_key: "/opt/intel/custom_tls/server.key" + key_name: "server_cu.key" + key_path: "/opt/intel/custom_tls/server_cu.key" + token_cert: "/opt/intel/custom_tls/server_cu.crt" + crt_subj: + O: "SampleOrganisation" + CN: "localhost" + - id: "unique_id_1234s_du" + token_name: "token_server_du" + pin: "1234" + key_name: "server_du.key" + key_path: "/opt/intel/custom_tls/server_du.key" + token_cert: "/opt/intel/custom_tls/server_du.crt" + crt_subj: + O: "SampleOrganisation" + CN: "localhost" + - id: "unique_id_1234s_ric" + token_name: "token_server_ric" + pin: "1234" + key_name: "server_ric.key" + key_path: "/opt/intel/custom_tls/server_ric.key" + token_cert: "/opt/intel/custom_tls/server_ric.crt" crt_subj: O: "SampleOrganisation" CN: "localhost" - id: "unique_id_1234c" token_name: "token_client" pin: "1234" - key_name: "/opt/intel/custom_tls/client.key" + key_name: "client.key" + key_path: "/opt/intel/custom_tls/client.key" token_cert: "/opt/intel/custom_tls/client.crt" - token_key: "/opt/intel/custom_tls/client.key" crt_subj: O: "SampleOrganisation" CN: "localhost" + ecdsa_keys: "true" oran: enabled: true local_build: true oran_image_staging_location: "/tmp/oran/oran.tar" - oran_image_checksum: "553b91005e19f5dfee1b3052b69146f027022d51784988f6521341092253fa3b" + oran_image_checksum: "3c17855ee876a2db3b8ea5b8dd0dfa0ac68b52b2a243c24704bee9678b0cf0c8" sw_provider_name: "oranprovider" sw_provider_crt_subj: O: "SampleOrganisation" diff --git a/roles/kmra_install/files/oran/Dockerfile b/roles/kmra_install/files/oran/Dockerfile index 699b5adc..8bc94560 100644 --- a/roles/kmra_install/files/oran/Dockerfile +++ b/roles/kmra_install/files/oran/Dockerfile @@ -1,3 +1,18 @@ +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## + FROM ubuntu:22.04 as builder RUN apt-get update -y && \ @@ -18,12 +33,12 @@ WORKDIR /workspace COPY openssl.cnf.j2 openssl.cnf.j2 -RUN python3 -m pip install --no-cache-dir j2cli==0.3.10 ENV openssl_install_path=/usr/ ENV p11_module_path=/usr/local/lib/libpkcs11-proxy.so -RUN j2 openssl.cnf.j2 > /workspace/openssl.cnf +RUN python3 -m pip install --no-cache-dir j2cli==0.3.10 && \ + j2 openssl.cnf.j2 > /workspace/openssl.cnf && \ + git clone https://github.com/SUNET/pkcs11-proxy.git /workspace/pkcs11-proxy -RUN git clone https://github.com/SUNET/pkcs11-proxy.git /workspace/pkcs11-proxy COPY 0001-Fix-slotID.patch /workspace WORKDIR /workspace/pkcs11-proxy @@ -57,18 +72,20 @@ RUN apt-get update && \ rm -rf /var/log/*log /var/lib/apt/lists/* /var/log/apt/* /var/lib/dpkg/*-old /var/cache/debconf/*-old WORKDIR /oran/NetConfServer -RUN git clone https://github.com/CESNET/libyang.git +RUN git clone -b v2.1.111 --depth 1 https://github.com/CESNET/libyang.git WORKDIR /oran/NetConfServer/libyang/build RUN cmake -DENABLE_BUILD_TESTS=OFF .. && make && make install WORKDIR /oran/NetConfServer/libyang -RUN git clone https://github.com/sysrepo/sysrepo.git +RUN git clone -b v2.2.105 --depth 1 https://github.com/sysrepo/sysrepo.git WORKDIR /oran/NetConfServer/libyang/sysrepo/build -RUN cmake -DSHM_DIR=/tmp/shm -DREPO_PATH=/tmp/sysrepo -DENABLE_TESTS=OFF .. && make && make install -RUN ldconfig +RUN cmake -DSHM_DIR=/tmp/shm -DREPO_PATH=/tmp/sysrepo -DENABLE_TESTS=OFF .. && \ + make && \ + make install && \ + ldconfig WORKDIR /oran/NetConfServer/libyang/sysrepo -RUN git clone https://github.com/CESNET/libnetconf2.git +RUN git clone -b v2.1.37 --depth 1 https://github.com/CESNET/libnetconf2.git WORKDIR /oran/NetConfServer/libyang/sysrepo/libnetconf2 COPY 0003-libnetconf2-add-pkcs11-support.patch . RUN git apply 0003-libnetconf2-add-pkcs11-support.patch @@ -76,7 +93,7 @@ WORKDIR /oran/NetConfServer/libyang/sysrepo/libnetconf2/build RUN cmake -DENABLE_TLS=ON -DENABLE_SSH=ON -DENABLE_DNSSEC=OFF .. && make && make install WORKDIR /oran/NetConfServer/libyang/sysrepo/libnetconf2 -RUN git clone https://github.com/CESNET/netopeer2.git +RUN git clone -b v2.1.71 --depth 1 https://github.com/CESNET/netopeer2.git WORKDIR /oran/NetConfServer/libyang/sysrepo/libnetconf2/netopeer2 COPY 0004-netopeer2-comms-fix.patch . RUN git apply 0004-netopeer2-comms-fix.patch @@ -98,6 +115,8 @@ COPY --from=builder /workspace/pkcs11-proxy/lib* /usr/local/lib/ ENV LD_LIBRARY_PATH="/usr/lib:/usr/local/lib" +RUN ln -s /usr/lib/x86_64-linux-gnu/engines-3/ /usr/lib/engines-1.1 + COPY oran_commands.sh /workspace/oran_commands.sh RUN chmod u+x /workspace/oran_commands.sh && \ mkdir /opt/intel/ && cp -R /tmp/sysrepo /opt/intel/ && \ diff --git a/roles/kmra_install/files/oran/openssl.cnf.j2 b/roles/kmra_install/files/oran/openssl.cnf.j2 index 9c2b342b..8801edcb 100644 --- a/roles/kmra_install/files/oran/openssl.cnf.j2 +++ b/roles/kmra_install/files/oran/openssl.cnf.j2 @@ -107,7 +107,7 @@ emailAddress = optional #################################################################### [ req ] -default_bits = 2048 +default_bits = 3072 default_keyfile = privkey.pem distinguished_name = req_distinguished_name attributes = req_attributes @@ -335,7 +335,7 @@ signer_cert = $dir/tsacert.pem # The TSA signing certificate certs = $dir/cacert.pem # Certificate chain to include in reply # (optional) signer_key = $dir/private/tsakey.pem # The TSA private key (optional) -signer_digest = sha256 # Signing digest to use. (Optional) +signer_digest = sha384 # Signing digest to use. (Optional) default_policy = tsa_policy1 # Policy if request did not specify it # (optional) other_policies = tsa_policy2, tsa_policy3 # acceptable policies (optional) @@ -362,4 +362,4 @@ pkcs11 = pkcs11_section engine_id = pkcs11 dynamic_path = {{ openssl_install_path }}/lib/engines-1.1/libpkcs11.so MODULE_PATH = {{ p11_module_path }} -init = 1 \ No newline at end of file +init = 1 diff --git a/roles/kmra_install/files/oran/oran_commands.sh b/roles/kmra_install/files/oran/oran_commands.sh index ae5d897e..f8ce8992 100644 --- a/roles/kmra_install/files/oran/oran_commands.sh +++ b/roles/kmra_install/files/oran/oran_commands.sh @@ -39,14 +39,7 @@ sysrepocfg --edit=tls_truststore.xml --format=xml --datastore=running --module=i sysrepocfg --edit=tls_listen.xml --format=xml --datastore=running --module=ietf-netconf-server -v3 sysrepocfg --copy-from=running --datastore=startup -echo -e "\033[0;31m------------------------------------------------------------" -echo -e "KMRA (Key Management Reference Application) is a proof-of-concept" -echo -e "software not suitable for production usage. Please note that the enclave" -echo -e "is signed with a test signing key. A production enclave should go through" -echo -e "the process of signing an enclave as explained in the section Enclave" -echo -e "Signing Tool in the Intel(R) SGX Developer Reference for Linux* OS" -echo -e "(https://download.01.org/intel-sgx/latest/linux-latest/docs/)" -echo -e "---------------------------------------------------------------\033[0m" +sysrepocfg --export --format json if [[ ${NETOPEER_TYPE} == "server" ]]; then echo "Starting netopeer2-server..." diff --git a/roles/kmra_install/tasks/cleanup.yml b/roles/kmra_install/tasks/cleanup.yml index 73ae2f24..ddf31214 100644 --- a/roles/kmra_install/tasks/cleanup.yml +++ b/roles/kmra_install/tasks/cleanup.yml @@ -43,17 +43,24 @@ wait: true timeout: 4m0s loop: - - "{{ kmra_oran.oran_netopeer2_server.release_name }}" + - "{{ kmra_oran.oran_netopeer2_server.release_name }}-cu" + - "{{ kmra_oran.oran_netopeer2_server.release_name }}-du" + - "{{ kmra_oran.oran_netopeer2_server.release_name }}-ric" - "{{ kmra_oran.oran_netopeer2_client.release_name }}" - name: delete oran configmap kubernetes.core.k8s: api_version: v1 - name: oran-sysrepo-config + name: "{{ item }}-oran-sysrepo-config" kind: ConfigMap namespace: "{{ cosign_enforce_namespace }}" state: absent failed_when: false + loop: + - "{{ kmra_oran.oran_netopeer2_server.release_name }}-cu" + - "{{ kmra_oran.oran_netopeer2_server.release_name }}-du" + - "{{ kmra_oran.oran_netopeer2_server.release_name }}-ric" + - "{{ kmra_oran.oran_netopeer2_client.release_name }}" - name: delete apphsm configmap kubernetes.core.k8s: @@ -84,6 +91,8 @@ - "{{ kmra.pccs.helm_values_file }}" - "{{ kmra.apphsm.helm_values_file }}" - "{{ kmra.ctk_loadkey_demo.helm_values_file }}" - - "{{ kmra_oran.oran_netopeer2_server.helm_values_file }}" + - "{{ kmra_oran.oran_netopeer2_server.helm_values_file }}-cu" + - "{{ kmra_oran.oran_netopeer2_server.helm_values_file }}-du" + - "{{ kmra_oran.oran_netopeer2_server.helm_values_file }}-ric" - "{{ kmra_oran.oran_netopeer2_client.helm_values_file }}" failed_when: false diff --git a/roles/kmra_install/tasks/create_cosign_tls_secrets.yml b/roles/kmra_install/tasks/create_cosign_tls_secrets.yml index c3d7174d..7efa01e4 100644 --- a/roles/kmra_install/tasks/create_cosign_tls_secrets.yml +++ b/roles/kmra_install/tasks/create_cosign_tls_secrets.yml @@ -22,7 +22,8 @@ - name: generate CA root crt ansible.builtin.command: >- - openssl req -nodes -x509 -newkey rsa:2048 + openssl req -SHA384 -nodes -x509 -newkey rsa:3072 + -days {{ kmra.certs_validity_period_days }} -keyout {{ (mtls_tmp_dir.path, 'ca.key') | path_join }} -out {{ (mtls_tmp_dir.path, 'ca.crt') | path_join }} -subj "/O={{ kmra.ca_root_crt_subj.O }}/OU={{ kmra.ca_root_crt_subj.OU }}/CN={{ kmra.ca_root_crt_subj.CN }}" @@ -30,7 +31,7 @@ - name: generate csr for the cosign key ansible.builtin.command: >- - openssl req -nodes -newkey rsa:2048 + openssl req -SHA384 -nodes -newkey rsa:3072 -keyout {{ (mtls_tmp_dir.path, item.name) | path_join }}.key -out {{ (mtls_tmp_dir.path, item.name) | path_join }}.csr -subj "/O={{ item.subj.O | default('') }}/OU={{ item.subj.OU | default('') }}/CN={{ item.subj.CN | default('') }}" @@ -40,7 +41,7 @@ - name: generate cert for the cosign key ansible.builtin.shell: >- set -o pipefail && - openssl x509 -req -in {{ (mtls_tmp_dir.path, item.name) | path_join }}.csr + openssl x509 -SHA384 -req -in {{ (mtls_tmp_dir.path, item.name) | path_join }}.csr -days {{ kmra.certs_validity_period_days }} -CA {{ (mtls_tmp_dir.path, 'ca.crt') | path_join }} -CAkey {{ (mtls_tmp_dir.path, 'ca.key') | path_join }} diff --git a/roles/kmra_install/tasks/create_custom_tls_configmap.yml b/roles/kmra_install/tasks/create_custom_tls_configmap.yml index 6e7f6588..20371088 100644 --- a/roles/kmra_install/tasks/create_custom_tls_configmap.yml +++ b/roles/kmra_install/tasks/create_custom_tls_configmap.yml @@ -20,16 +20,87 @@ suffix: kmra-custom-tls register: tls_tmp_dir -- name: download netopeer2 server and client keys-certs - ansible.builtin.get_url: - url: "{{ (kmra.apphsm.oran_netopeer2_sample_tls_url, 'tls_certs', item) | path_join }}" - dest: "{{ (tls_tmp_dir.path, item) | path_join }}" - mode: 0644 - loop: - - "server.key" - - "server.crt" - - "client.key" - - "client.crt" +- name: generate custom RSA keys + block: + - name: generate netopeer2 CA root crt + ansible.builtin.command: >- + openssl req -SHA384 -nodes -x509 -newkey rsa:3072 + -days {{ kmra.certs_validity_period_days }} + -keyout {{ (tls_tmp_dir.path, 'ca.key') | path_join }} + -out {{ (tls_tmp_dir.path, 'ca.crt') | path_join }} + -subj "/O={{ kmra.ca_root_crt_subj.O }}/OU={{ kmra.ca_root_crt_subj.OU }}/CN={{ kmra.ca_root_crt_subj.CN }}" + changed_when: true + + - name: generate new keys and csrs + ansible.builtin.command: >- + openssl req -SHA384 -nodes -newkey rsa:3072 + -keyout {{ (tls_tmp_dir.path, item.token_name) | path_join }}.key + -out {{ (tls_tmp_dir.path, item.token_name) | path_join }}.csr + -subj "/O={{ item.crt_subj.O | default('') }}/OU={{ item.crt_subj.OU | default('') }}/CN={{ item.crt_subj.CN | default('') }}" + loop: "{{ kmra.apphsm.app_keys }}" + changed_when: true + + - name: generate crts for the new keys + ansible.builtin.shell: >- + set -o pipefail && + openssl x509 -SHA384 -req -in {{ (tls_tmp_dir.path, item.token_name) | path_join }}.csr + -days {{ kmra.certs_validity_period_days }} + -CA {{ (tls_tmp_dir.path, 'ca.crt') | path_join }} + -CAkey {{ (tls_tmp_dir.path, 'ca.key') | path_join }} + {{ '-extfile <(printf "subjectAltName=DNS:' + item.crt_subj.CN + '")' + if item.crt_subj.CN | default('') | length > 0 }} + -CAcreateserial -CAserial {{ (tls_tmp_dir.path, 'ca.srl' ) | path_join }} + -out {{ (tls_tmp_dir.path, item.token_name) | path_join }}.crt + args: + executable: /bin/bash + loop: "{{ kmra.apphsm.app_keys }}" + changed_when: true + when: kmra.apphsm.ecdsa_keys != "true" + +- name: generate custom ECDSA keys + block: + - name: generate netopeer2 CA root crt + ansible.builtin.command: >- + openssl req -SHA384 -nodes -x509 -newkey ec -pkeyopt ec_paramgen_curve:secp384r1 + -days {{ kmra.certs_validity_period_days }} + -keyout {{ (tls_tmp_dir.path, 'ca.key') | path_join }} + -out {{ (tls_tmp_dir.path, 'ca.crt') | path_join }} + -subj "/O={{ kmra.ca_root_crt_subj.O }}/OU={{ kmra.ca_root_crt_subj.OU }}/CN={{ kmra.ca_root_crt_subj.CN }}" + changed_when: true + + - name: generate new keys and csrs + ansible.builtin.command: >- + openssl req -SHA384 -nodes -newkey ec -pkeyopt ec_paramgen_curve:secp384r1 + -keyout {{ (tls_tmp_dir.path, item.token_name) | path_join }}.ekey + -out {{ (tls_tmp_dir.path, item.token_name) | path_join }}.csr + -subj "/O={{ item.crt_subj.O | default('') }}/OU={{ item.crt_subj.OU | default('') }}/CN={{ item.crt_subj.CN | default('') }}" + loop: "{{ kmra.apphsm.app_keys }}" + changed_when: true + + - name: convert keys to pkcs8 format + ansible.builtin.command: >- + openssl pkcs8 -topk8 --inform PEM --outform PEM --nocrypt + -in {{ (tls_tmp_dir.path, item.token_name) | path_join }}.ekey + -out {{ (tls_tmp_dir.path, item.token_name) | path_join }}.key + loop: "{{ kmra.apphsm.app_keys }}" + changed_when: true + + - name: generate crts for the new keys + ansible.builtin.shell: >- + set -o pipefail && + openssl x509 -req -SHA384 -in {{ (tls_tmp_dir.path, item.token_name) | path_join }}.csr + -days {{ kmra.certs_validity_period_days }} + -CA {{ (tls_tmp_dir.path, 'ca.crt') | path_join }} + -CAkey {{ (tls_tmp_dir.path, 'ca.key') | path_join }} + {{ '-extfile <(printf "subjectAltName=DNS:' + item.crt_subj.CN + '")' + if item.crt_subj.CN | default('') | length > 0 }} + -CAcreateserial -CAserial {{ (tls_tmp_dir.path, 'ca.srl' ) | path_join }} + -out {{ (tls_tmp_dir.path, item.token_name) | path_join }}.crt + args: + executable: /bin/bash + loop: "{{ kmra.apphsm.app_keys }}" + changed_when: true + when: kmra.apphsm.ecdsa_keys == "true" - name: generate a list of all secrets files ansible.builtin.find: @@ -61,10 +132,14 @@ name: kmra-apphsm-custom-config namespace: "{{ kmra.namespace }}" data: - server.key: "{{ custom_tls_serverkey }}" - server.crt: "{{ custom_tls_servercrt }}" - client.key: "{{ custom_tls_clientkey }}" - client.crt: "{{ custom_tls_clientcrt }}" + server_cu.key: "{{ custom_tls_token_server_cukey }}" + server_cu.crt: "{{ custom_tls_token_server_cucrt }}" + server_du.key: "{{ custom_tls_token_server_dukey }}" + server_du.crt: "{{ custom_tls_token_server_ducrt }}" + server_ric.key: "{{ custom_tls_token_server_rickey }}" + server_ric.crt: "{{ custom_tls_token_server_riccrt }}" + client.key: "{{ custom_tls_token_clientkey }}" + client.crt: "{{ custom_tls_token_clientcrt }}" - name: clean up tmp directory ansible.builtin.file: @@ -77,22 +152,66 @@ suffix: oran-sysrepo register: sys_tmp_dir -- name: download netopeer2 sysrepo config - ansible.builtin.get_url: - url: "{{ (kmra.apphsm.oran_netopeer2_sample_tls_url, item) | path_join }}" - dest: "{{ (sys_tmp_dir.path, item) | path_join }}" - mode: 0644 - loop: - - "tls_keystore.xml" - - "tls_listen.xml" - - "tls_truststore.xml" +- name: create sysrepo for cu, du, ric separately - tls_truststore & tls_listen + block: + - name: get ca crt base64 + ansible.builtin.shell: >- + set -o pipefail && + echo -n "{{ custom_tls_cacrt }}" | openssl x509 -outform der | base64 -w0 + register: ca_crt_str + args: + executable: /bin/bash + changed_when: true -- name: hide the private key as we use pkcs11 to load it from ctk_loader - ansible.builtin.replace: - path: "{{ (sys_tmp_dir.path, 'tls_keystore.xml') | path_join }}" - regexp: '.+' - replace: "{{ ('pkcs11:token=token_server;object=client_key_priv;pin-value=' + \ - kmra.oran_netopeer2_server.default_user_pin + ';') | b64encode }}" + - name: get client crt base64 + ansible.builtin.shell: >- + set -o pipefail && + echo -n "{{ custom_tls_token_clientcrt }}" | openssl x509 -outform der | base64 -w0 + register: client_crt_str + args: + executable: /bin/bash + changed_when: true + + - name: populate tls_truststore.xml + ansible.builtin.template: + src: "tls_truststore.xml.j2" + dest: "{{ (sys_tmp_dir.path, 'tls_truststore.xml') | path_join }}" + mode: '0644' + + - name: get client finger + ansible.builtin.shell: >- + set -o pipefail && + echo -n "{{ custom_tls_cacrt }}" | openssl x509 -noout -fingerprint | cut -b '18-' + register: finger_str + args: + executable: /bin/bash + changed_when: true + + - name: populate tls_listen.xml + ansible.builtin.template: + src: "tls_listen.xml.j2" + dest: "{{ (sys_tmp_dir.path, 'tls_listen.xml') | path_join }}" + mode: '0644' + +- name: create sysrepo for cu, du, ric separately - tls_keystore + include_tasks: create_sysrepo_keystore.yml + no_log: true + loop: + - { + key: "{{ custom_tls_token_server_cukey }}", + crt: "{{ custom_tls_token_server_cucrt }}", + xml: 'tls_keystore_cu.xml' + } + - { + key: "{{ custom_tls_token_server_dukey }}", + crt: "{{ custom_tls_token_server_ducrt }}", + xml: 'tls_keystore_du.xml' + } + - { + key: "{{ custom_tls_token_server_rickey }}", + crt: "{{ custom_tls_token_server_riccrt }}", + xml: 'tls_keystore_ric.xml' + } - name: generate a list of all sysrepo files ansible.builtin.find: @@ -121,12 +240,31 @@ apiVersion: v1 kind: ConfigMap metadata: - name: oran-sysrepo-config + name: "{{ item.key }}-oran-sysrepo-config" namespace: "{{ cosign_enforce_namespace }}" data: - tls_keystore.xml: "{{ sysrepo_tls_keystorexml }}" + tls_keystore.xml: "{{ item.xml }}" tls_listen.xml: "{{ sysrepo_tls_listenxml }}" tls_truststore.xml: "{{ sysrepo_tls_truststorexml }}" + no_log: true + loop: + - { + key: "{{ kmra.oran_netopeer2_server.release_name }}-cu", + xml: "{{ sysrepo_tls_keystore_cuxml }}" + } + - { + key: "{{ kmra.oran_netopeer2_server.release_name }}-du", + xml: "{{ sysrepo_tls_keystore_duxml }}" + } + - { + key: "{{ kmra.oran_netopeer2_server.release_name }}-ric", + xml: "{{ sysrepo_tls_keystore_ricxml }}" + } + # just reuse cu keystore for client sysrepo format check only + - { + key: "{{ kmra.oran_netopeer2_client.release_name }}", + xml: "{{ sysrepo_tls_keystore_cuxml }}" + } - name: clean up tmp directory ansible.builtin.file: diff --git a/roles/kmra_install/tasks/create_sysrepo_keystore.yml b/roles/kmra_install/tasks/create_sysrepo_keystore.yml new file mode 100644 index 00000000..00c3d38f --- /dev/null +++ b/roles/kmra_install/tasks/create_sysrepo_keystore.yml @@ -0,0 +1,41 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +- name: create sysrepo for cu, du, ric separately - tls_keystore + block: + - name: get server pubkey base64 + ansible.builtin.shell: >- + set -o pipefail && + echo -n "{{ item.key }}" | openssl {% if kmra.apphsm.ecdsa_keys != "true" %}rsa{% else %}ec{% endif %} -outform der -pubout | base64 -w0 + register: key_str + args: + executable: /bin/bash + changed_when: true + + - name: get server crt base64 + ansible.builtin.shell: >- + set -o pipefail && + echo -n "{{ item.crt }}" | openssl x509 -outform der | base64 -w0 + register: crt_str + args: + executable: /bin/bash + changed_when: true + + - name: populate tls_truststore.xml + ansible.builtin.template: + src: "tls_keystore.xml.j2" + dest: "{{ (sys_tmp_dir.path, item.xml) | path_join }}" + mode: '0644' diff --git a/roles/kmra_install/tasks/create_tls_secrets.yml b/roles/kmra_install/tasks/create_tls_secrets.yml index 28b8c3b6..b86e3191 100644 --- a/roles/kmra_install/tasks/create_tls_secrets.yml +++ b/roles/kmra_install/tasks/create_tls_secrets.yml @@ -21,16 +21,17 @@ register: mtls_tmp_dir - name: generate CA root crt - command: >- - openssl req -nodes -x509 -newkey rsa:2048 + ansible.builtin.command: >- + openssl req -SHA384 -nodes -x509 -newkey rsa:3072 + -days {{ kmra.certs_validity_period_days }} -keyout {{ (mtls_tmp_dir.path, 'ca.key') | path_join }} -out {{ (mtls_tmp_dir.path, 'ca.crt') | path_join }} -subj "/O={{ kmra.ca_root_crt_subj.O }}/OU={{ kmra.ca_root_crt_subj.OU }}/CN={{ kmra.ca_root_crt_subj.CN }}" changed_when: true - name: generate csr for the kmra app - command: >- - openssl req -nodes -newkey rsa:2048 + ansible.builtin.command: >- + openssl req -SHA384 -nodes -newkey rsa:3072 -keyout {{ (mtls_tmp_dir.path, item.name) | path_join }}.key -out {{ (mtls_tmp_dir.path, item.name) | path_join }}.csr -subj "/O={{ item.subj.O | default('') }}/OU={{ item.subj.OU | default('') }}/CN={{ item.subj.CN | default('') }}" @@ -40,9 +41,9 @@ - item.deploy - name: generate cert for the kmra app - bmra - shell: >- + ansible.builtin.shell: >- set -o pipefail && - openssl x509 -req -in {{ (mtls_tmp_dir.path, item.name) | path_join }}.csr + openssl x509 -SHA384 -req -in {{ (mtls_tmp_dir.path, item.name) | path_join }}.csr -days {{ kmra.certs_validity_period_days }} -CA {{ (mtls_tmp_dir.path, 'ca.crt') | path_join }} -CAkey {{ (mtls_tmp_dir.path, 'ca.key') | path_join }} @@ -61,9 +62,9 @@ - not vm_enabled | default(false) - name: generate cert for the kmra app - vmra - shell: >- + ansible.builtin.shell: >- set -o pipefail && - openssl x509 -req -in {{ (mtls_tmp_dir.path, item.name) | path_join }}.csr + openssl x509 -SHA384 -req -in {{ (mtls_tmp_dir.path, item.name) | path_join }}.csr -CA {{ (mtls_tmp_dir.path, 'ca.crt') | path_join }} -CAkey {{ (mtls_tmp_dir.path, 'ca.key') | path_join }} {{ '-extfile <(printf "subjectAltName=DNS:' + item.subj.CN + '")' diff --git a/roles/kmra_install/tasks/main.yml b/roles/kmra_install/tasks/main.yml index 2ed31c31..83743e86 100644 --- a/roles/kmra_install/tasks/main.yml +++ b/roles/kmra_install/tasks/main.yml @@ -137,7 +137,19 @@ deploy: "{{ kmra.ctk_loadkey_demo.enabled | default(false) }}" } - { - name: "{{ kmra.oran_netopeer2_server.release_name | default('') }}", + name: "{{ kmra.oran_netopeer2_server.release_name | default('') }}-cu", + namespace: "{{ cosign_enforce_namespace | default('default') }}", + subj: "{{ kmra.oran_netopeer2_server.crt_subj | default('') }}", + deploy: "{{ kmra.oran_netopeer2_server.enabled | default(false) }}", + } + - { + name: "{{ kmra.oran_netopeer2_server.release_name | default('') }}-du", + namespace: "{{ cosign_enforce_namespace | default('default') }}", + subj: "{{ kmra.oran_netopeer2_server.crt_subj | default('') }}", + deploy: "{{ kmra.oran_netopeer2_server.enabled | default(false) }}", + } + - { + name: "{{ kmra.oran_netopeer2_server.release_name | default('') }}-ric", namespace: "{{ cosign_enforce_namespace | default('default') }}", subj: "{{ kmra.oran_netopeer2_server.crt_subj | default('') }}", deploy: "{{ kmra.oran_netopeer2_server.enabled | default(false) }}", @@ -211,28 +223,6 @@ dest: "{{ (kmra.ctk_loadkey_demo.chart_path, 'templates', 'kmra-ctk-loadkey-rbac-cluster-role.yml') | path_join }}", deploy: "{{ kmra.ctk_loadkey_demo.enabled | default(false) }}" } - - { - src: "kmra-oran-netopeer2-server-values.yaml.j2", - dest: "{{ (project_root_dir, 'charts', 'kmra-oran-netopeer2-server-values.yml') | path_join }}", - deploy: "{{ kmra.oran_netopeer2_server.enabled | default(false) }}" - } - - { - src: "kmra-oran-netopeer2-server-rbac-cluster-role.yml.j2", - dest: "{{ (kmra.oran_netopeer2_server.chart_path | default(project_root_dir), \ - 'templates', 'kmra-oran-netopeer2-server-rbac-cluster-role.yml') | path_join }}", - deploy: "{{ kmra.oran_netopeer2_server.enabled | default(false) }}" - } - - { - src: "kmra-oran-netopeer2-client-values.yaml.j2", - dest: "{{ (project_root_dir, 'charts', 'kmra-oran-netopeer2-client-values.yml') | path_join }}", - deploy: "{{ kmra.oran_netopeer2_client.enabled | default(false) }}" - } - - { - src: "kmra-oran-netopeer2-client-rbac-cluster-role.yml.j2", - dest: "{{ (kmra.oran_netopeer2_client.chart_path | default(project_root_dir), \ - 'templates', 'kmra-oran-netopeer2-client-rbac-cluster-role.yml') | path_join }}", - deploy: "{{ kmra.oran_netopeer2_client.enabled | default(false) }}" - } when: - item.deploy @@ -339,18 +329,23 @@ - name: prepare image for oran case include_tasks: prepare_oran_image.yml - - name: populate oran helm charts values templates and push to controller node + - name: populate oran helm charts values templates and push to controller node - servers + ansible.builtin.template: + src: "kmra-oran-netopeer2-server-values.yaml.j2" + dest: "{{ kmra.oran_netopeer2_server.helm_values_file }}-{{ item }}" + force: yes + mode: preserve + loop: ['cu', 'du', 'ric'] + when: + - kmra.oran_netopeer2_server.enabled | default(false) | bool + + - name: populate oran helm charts values templates and push to controller node - others ansible.builtin.template: src: "{{ item.src }}" dest: "{{ item.dest }}" force: yes mode: preserve loop: - - { - src: "kmra-oran-netopeer2-server-values.yaml.j2", - dest: "{{ (project_root_dir, 'charts', 'kmra-oran-netopeer2-server-values.yml') | path_join }}", - deploy: "{{ kmra.oran_netopeer2_server.enabled | default(false) }}" - } - { src: "kmra-oran-netopeer2-server-rbac-cluster-role.yml.j2", dest: "{{ (kmra.oran_netopeer2_server.chart_path, 'templates', 'kmra-oran-netopeer2-server-rbac-cluster-role.yml') | path_join }}", @@ -372,11 +367,12 @@ - name: install KMRA oran netopeer2 server helm chart kubernetes.core.helm: chart_ref: "{{ kmra.oran_netopeer2_server.chart_path }}" - release_name: "{{ kmra.oran_netopeer2_server.release_name }}" + release_name: "{{ kmra.oran_netopeer2_server.release_name }}-{{ item }}" release_namespace: "{{ cosign_enforce_namespace }}" - values_files: "{{ kmra.oran_netopeer2_server.helm_values_file }}" + values_files: "{{ kmra.oran_netopeer2_server.helm_values_file }}-{{ item }}" wait: true timeout: 4m0s + loop: ['cu', 'du', 'ric'] when: - kmra.oran_netopeer2_server.enabled | default(false) diff --git a/roles/kmra_install/templates/kmra-apphsm-values.yaml.j2 b/roles/kmra_install/templates/kmra-apphsm-values.yaml.j2 index bd964473..58b14f16 100644 --- a/roles/kmra_install/templates/kmra-apphsm-values.yaml.j2 +++ b/roles/kmra_install/templates/kmra-apphsm-values.yaml.j2 @@ -44,3 +44,4 @@ apphsm: default_so_pin: "{{ kmra.apphsm.default_so_pin }}" nonce_lifetime: "{{ kmra.apphsm.nonce_lifetime }}" keys: {{ kmra.apphsm.app_keys }} + ecdsa_keys: "{{ kmra.apphsm.ecdsa_keys }}" diff --git a/roles/kmra_install/templates/kmra-ctk-values.yaml.j2 b/roles/kmra_install/templates/kmra-ctk-values.yaml.j2 index 50bf6f46..d30e7c6b 100644 --- a/roles/kmra_install/templates/kmra-ctk-values.yaml.j2 +++ b/roles/kmra_install/templates/kmra-ctk-values.yaml.j2 @@ -47,3 +47,4 @@ ctk_loadkey: pkcs11_proxy_tls_psk_file: "{{ kmra.ctk_loadkey_demo.pkcs11_proxy_tls_psk_file }}" pkcs11_daemon_socket_hostname: "{{ kmra.ctk_loadkey_demo.pkcs11_daemon_socket_hostname }}" pkcs11_daemon_socket_port: "{{ kmra.ctk_loadkey_demo.pkcs11_daemon_socket_port }}" + ecdsa_keys: "{{ kmra.apphsm.ecdsa_keys }}" diff --git a/roles/kmra_install/templates/kmra-oran-netopeer2-client-values.yaml.j2 b/roles/kmra_install/templates/kmra-oran-netopeer2-client-values.yaml.j2 index 51d96e1d..7b9ff379 100644 --- a/roles/kmra_install/templates/kmra-oran-netopeer2-client-values.yaml.j2 +++ b/roles/kmra_install/templates/kmra-oran-netopeer2-client-values.yaml.j2 @@ -30,7 +30,7 @@ oran_netopeer2_client: tag: "{{ kmra.oran_netopeer2_client.oran_image_tag | default(kmra.image_tag) }}" pullPolicy: IfNotPresent netopeer2_server_port: "{{ kmra.oran_netopeer2_client.oran_netopeer2_server_port }}" - netopeer2_server_name: "{{ kmra.oran_netopeer2_client.oran_netopeer2_server_hostname | default('oran_netopeer2_server')}}" + netopeer2_server_name: "{{ kmra.oran_netopeer2_client.oran_netopeer2_server_hostname | default('oran_netopeer2_server')}}-cu" netopeer2_server_domain: {{ cluster_name | default('cluster.local') }} pullSecret: "{{ container_registry_secret }}" pccs_port: "{{ kmra.pccs.upstream_port }}" @@ -50,3 +50,4 @@ oran_netopeer2_client: pkcs11_proxy_tls_psk_file: "{{ kmra.oran_netopeer2_client.pkcs11_proxy_tls_psk_file }}" pkcs11_daemon_socket_hostname: "{{ kmra.oran_netopeer2_client.pkcs11_daemon_socket_hostname }}" pkcs11_daemon_socket_port: "{{ kmra.oran_netopeer2_client.pkcs11_daemon_socket_port }}" + ecdsa_keys: "{{ kmra.apphsm.ecdsa_keys }}" diff --git a/roles/kmra_install/templates/kmra-oran-netopeer2-server-values.yaml.j2 b/roles/kmra_install/templates/kmra-oran-netopeer2-server-values.yaml.j2 index ab9eaae4..b0055d09 100644 --- a/roles/kmra_install/templates/kmra-oran-netopeer2-server-values.yaml.j2 +++ b/roles/kmra_install/templates/kmra-oran-netopeer2-server-values.yaml.j2 @@ -24,7 +24,7 @@ oran_netopeer2_server: pullPolicy: IfNotPresent oran: type: server - hostname: "{{ kmra.oran_netopeer2_server.oran_netopeer2_server_hostname }}" + hostname: "{{ kmra.oran_netopeer2_server.oran_netopeer2_server_hostname }}-{{ item }}" servicePort: "{{ kmra.oran_netopeer2_server.oran_netopeer2_server_port }}" image: repo: "{{ kmra.oran_netopeer2_server.oran_image_repo }}" @@ -39,9 +39,9 @@ oran_netopeer2_server: sgx_prv_gid: "{{ hostvars[groups['kube_node'][0]]['getent_group']['sgx_prv'][1] | default('1002')}}" sgx_gid: "{{ hostvars[groups['kube_node'][0]]['getent_group']['sgx'][1] | default('107')}}" use_secure_cert: "{{ kmra.oran_netopeer2_server.use_secure_cert | quote }}" - client_token: "{{ kmra.oran_netopeer2_server.client_token }}" + client_token: "{{ kmra.oran_netopeer2_server.client_token }}_{{ item }}" client_key_label: "{{kmra.oran_netopeer2_server.client_key_label }}" - test_unique_uid: "{{ kmra.oran_netopeer2_server.test_unique_uid }}" + test_unique_uid: "{{ kmra.oran_netopeer2_server.test_unique_uid }}_{{ item }}" default_user_pin: "{{ kmra.oran_netopeer2_server.default_user_pin }}" default_so_pin: "{{ kmra.oran_netopeer2_server.default_so_pin }}" default_client_token_id: "{{ kmra.oran_netopeer2_server.default_client_token_id }}" @@ -49,3 +49,4 @@ oran_netopeer2_server: pkcs11_proxy_tls_psk_file: "{{ kmra.oran_netopeer2_server.pkcs11_proxy_tls_psk_file }}" pkcs11_daemon_socket_hostname: "{{ kmra.oran_netopeer2_server.pkcs11_daemon_socket_hostname }}" pkcs11_daemon_socket_port: "{{ kmra.oran_netopeer2_server.pkcs11_daemon_socket_port }}" + ecdsa_keys: "{{ kmra.apphsm.ecdsa_keys }}" diff --git a/roles/kmra_install/templates/tls_keystore.xml.j2 b/roles/kmra_install/templates/tls_keystore.xml.j2 new file mode 100644 index 00000000..12d89269 --- /dev/null +++ b/roles/kmra_install/templates/tls_keystore.xml.j2 @@ -0,0 +1,16 @@ + + + + serverkey + rsa2048 + {{ key_str.stdout }} + cGtjczExOnRva2VuPXRva2VuX3NlcnZlcjtvYmplY3Q9Y2xpZW50X2tleV9wcml2O3Bpbi12YWx1ZT0xMjM0Ow== + + + servercert + {{ crt_str.stdout }} + + + + + diff --git a/roles/kmra_install/templates/tls_listen.xml.j2 b/roles/kmra_install/templates/tls_listen.xml.j2 new file mode 100644 index 00000000..b492ef9a --- /dev/null +++ b/roles/kmra_install/templates/tls_listen.xml.j2 @@ -0,0 +1,38 @@ + + + + default-tls + + + 0.0.0.0 + + 1 + 10 + 5 + + + + + + serverkey + servercert + + + + + cacerts + clientcerts + + + 1 + 02:{{ finger_str.stdout }} + x509c2n:specified + tls-test + + + + + + + + diff --git a/roles/kmra_install/templates/tls_truststore.xml.j2 b/roles/kmra_install/templates/tls_truststore.xml.j2 new file mode 100644 index 00000000..bb3330b0 --- /dev/null +++ b/roles/kmra_install/templates/tls_truststore.xml.j2 @@ -0,0 +1,16 @@ + + + clientcerts + + clientcert + {{ client_crt_str.stdout }} + + + + cacerts + + cacert + {{ ca_crt_str.stdout }} + + + diff --git a/roles/kubernetes_power_manager/defaults/main.yml b/roles/kubernetes_power_manager/defaults/main.yml new file mode 100644 index 00000000..2a9435e6 --- /dev/null +++ b/roles/kubernetes_power_manager/defaults/main.yml @@ -0,0 +1,24 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +--- +kubernetes_power_manager_git_url: "https://github.com/intel/kubernetes-power-manager.git" +kubernetes_power_manager_git_ref: "v2.3.1" # project is consistent with git ref and image version +kubernetes_power_manager_dir: "{{ (project_root_dir, 'kubernetes-power-manager') | path_join }}" +kubernetes_power_manager_namespace: "intel-power" +kubernetes_power_operator_image: "docker.io/intel/power-operator" +kubernetes_power_operator_image_local: "{{ registry_local_address }}/intel-power-operator" +kubernetes_power_node_agent_image: "intel/power-node-agent" +kubernetes_power_node_agent_image_local: "{{ registry_local_address }}/intel-power-node-agent" diff --git a/roles/intel_power_manager/tasks/deploy_features.yml b/roles/kubernetes_power_manager/tasks/deploy_features.yml similarity index 84% rename from roles/intel_power_manager/tasks/deploy_features.yml rename to roles/kubernetes_power_manager/tasks/deploy_features.yml index 82c905e4..e23b2315 100644 --- a/roles/intel_power_manager/tasks/deploy_features.yml +++ b/roles/kubernetes_power_manager/tasks/deploy_features.yml @@ -13,8 +13,6 @@ ## See the License for the specific language governing permissions and ## limitations under the License. ## - -# Uncore Frequency - name: prepare and deploy Uncore Frequency when: hostvars[power_node]['uncore_frequency']['enabled'] | default(false) | bool block: @@ -28,7 +26,7 @@ - name: populate Uncore Frequency template ansible.builtin.template: src: uncore_frequency.yaml.j2 - dest: "{{ (intel_power_manager_dir, 'uncore_frequency_' + power_node + '.yaml') | path_join }}" + dest: "{{ (kubernetes_power_manager_dir, 'uncore_frequency_' + power_node + '.yaml') | path_join }}" force: yes mode: preserve when: inventory_hostname == groups['kube_control_plane'][0] @@ -36,7 +34,7 @@ - name: apply Uncore Frequency kubernetes.core.k8s: state: present - src: "{{ (intel_power_manager_dir, 'uncore_frequency_' + power_node + '.yaml') | path_join }}" + src: "{{ (kubernetes_power_manager_dir, 'uncore_frequency_' + power_node + '.yaml') | path_join }}" when: inventory_hostname == groups['kube_control_plane'][0] # C-States @@ -51,7 +49,7 @@ - name: populate C-States template ansible.builtin.template: src: cstates.yaml.j2 - dest: "{{ (intel_power_manager_dir, 'cstates_' + power_node + '.yaml') | path_join }}" + dest: "{{ (kubernetes_power_manager_dir, 'cstates_' + power_node + '.yaml') | path_join }}" force: yes mode: preserve when: inventory_hostname == groups['kube_control_plane'][0] @@ -59,5 +57,5 @@ - name: apply C-States kubernetes.core.k8s: state: present - src: "{{ (intel_power_manager_dir, 'cstates_' + power_node + '.yaml') | path_join }}" + src: "{{ (kubernetes_power_manager_dir, 'cstates_' + power_node + '.yaml') | path_join }}" when: inventory_hostname == groups['kube_control_plane'][0] diff --git a/roles/intel_power_manager/tasks/deploy_sample_pods.yml b/roles/kubernetes_power_manager/tasks/deploy_sample_pods.yml similarity index 82% rename from roles/intel_power_manager/tasks/deploy_sample_pods.yml rename to roles/kubernetes_power_manager/tasks/deploy_sample_pods.yml index 79ede989..be3dcdd6 100644 --- a/roles/intel_power_manager/tasks/deploy_sample_pods.yml +++ b/roles/kubernetes_power_manager/tasks/deploy_sample_pods.yml @@ -18,20 +18,20 @@ ansible.builtin.file: state: directory mode: 0755 - path: "{{ (intel_power_manager_dir, 'sample_power_pods') | path_join }}" + path: "{{ (kubernetes_power_manager_dir, 'sample_power_pods') | path_join }}" when: inventory_hostname == groups['kube_control_plane'][0] # this task will generate yaml files for each PowerProfile from -# power_profiles list for each node in intel_power_manager.power_nodes list +# power_profiles list for each node in kubernetes_power_manager.power_nodes list - name: generate templates for each available profile for the node ansible.builtin.include_tasks: power_pod_template_helper.yml - loop: "{{ intel_power_manager.power_nodes }}" + loop: "{{ kubernetes_power_manager.power_nodes }}" loop_control: loop_var: power_node - name: get yaml files to deploy ansible.builtin.find: - path: "{{ (intel_power_manager_dir, 'sample_power_pods') | path_join }}" + path: "{{ (kubernetes_power_manager_dir, 'sample_power_pods') | path_join }}" file_type: file patterns: "*.yaml" register: sample_pod_files diff --git a/roles/intel_power_manager/tasks/deploy_shared_resources.yml b/roles/kubernetes_power_manager/tasks/deploy_shared_resources.yml similarity index 78% rename from roles/intel_power_manager/tasks/deploy_shared_resources.yml rename to roles/kubernetes_power_manager/tasks/deploy_shared_resources.yml index b2bc16ef..89e1dd4b 100644 --- a/roles/intel_power_manager/tasks/deploy_shared_resources.yml +++ b/roles/kubernetes_power_manager/tasks/deploy_shared_resources.yml @@ -19,7 +19,7 @@ - name: make sure that directory for node-specific Shared Power Profiles exists ansible.builtin.file: state: directory - path: "{{ (intel_power_manager_dir, 'local_shared_power_profiles') | path_join }}" + path: "{{ (kubernetes_power_manager_dir, 'local_shared_power_profiles') | path_join }}" mode: 0755 - name: obtain variables needed for deployment of node-specific Shared Power Profile @@ -32,14 +32,14 @@ - name: populate template for node-specific Shared Power Profile ansible.builtin.template: src: local_shared_profile.yaml.j2 - dest: "{{ (intel_power_manager_dir, 'local_shared_power_profiles', node_name + '_local_shared_profile.yaml') | path_join }}" + dest: "{{ (kubernetes_power_manager_dir, 'local_shared_power_profiles', node_name + '_local_shared_profile.yaml') | path_join }}" mode: preserve force: yes - name: deploy node-specific Shared Power Profile kubernetes.core.k8s: state: present - src: "{{ (intel_power_manager_dir, 'local_shared_power_profiles', node_name + '_local_shared_profile.yaml') | path_join }}" + src: "{{ (kubernetes_power_manager_dir, 'local_shared_power_profiles', node_name + '_local_shared_profile.yaml') | path_join }}" when: hostvars[node_name]['local_shared_profile']['enabled'] - name: prepare and deploy node-specific Shared Power Workload @@ -47,7 +47,7 @@ - name: make sure that directory for node-specific Shared Power Workloads exists ansible.builtin.file: state: directory - path: "{{ (intel_power_manager_dir, 'shared_power_workloads') | path_join }}" + path: "{{ (kubernetes_power_manager_dir, 'shared_power_workloads') | path_join }}" mode: 0755 - name: obtain variables needed for deployment of Shared Power Workloads @@ -59,12 +59,12 @@ - name: populate template for Shared Power Workload ansible.builtin.template: src: shared_workload.yaml.j2 - dest: "{{ (intel_power_manager_dir, 'shared_power_workloads', node_name + '_shared_workload.yaml') | path_join }}" + dest: "{{ (kubernetes_power_manager_dir, 'shared_power_workloads', node_name + '_shared_workload.yaml') | path_join }}" mode: preserve force: yes - name: deploy node-specific Shared Power Workload kubernetes.core.k8s: state: present - src: "{{ (intel_power_manager_dir, 'shared_power_workloads', node_name + '_shared_workload.yaml') | path_join }}" + src: "{{ (kubernetes_power_manager_dir, 'shared_power_workloads', node_name + '_shared_workload.yaml') | path_join }}" when: hostvars[node_name]['shared_workload']['enabled'] diff --git a/roles/kubernetes_power_manager/tasks/kpm_preflight.yml b/roles/kubernetes_power_manager/tasks/kpm_preflight.yml new file mode 100644 index 00000000..c7bf5fcb --- /dev/null +++ b/roles/kubernetes_power_manager/tasks/kpm_preflight.yml @@ -0,0 +1,76 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +- name: check if Power Nodes are specified + ansible.builtin.assert: + that: + - kubernetes_power_manager.power_nodes | length > 0 + fail_msg: "Kubernetes Power Manager is enabled, but Power Nodes are not specified in group vars." + +- name: check if Power Nodes are available in inventory + ansible.builtin.assert: + that: + - item in groups['kube_node'] + fail_msg: "Kubernetes Power Manager Power Nodes have to be present in inventory. '{{ item }}' is not there: {{ groups['kube_node'] }}" + loop: "{{ kubernetes_power_manager.power_nodes }}" + +- name: check if Kubernetes Power Manager is enabled, the ISST features should be disabled + ansible.builtin.assert: + that: + - not (sst_bf_configuration_enabled is defined and sst_bf_configuration_enabled or + sst_cp_configuration_enabled is defined and sst_cp_configuration_enabled or + sst_tf_configuration_enabled is defined and sst_tf_configuration_enabled or + sst_pp_configuration_enabled is defined and sst_pp_configuration_enabled) + fail_msg: + - "Currently Kubernetes Power Manager and Intel SST features are mutually exclusive." + - "Please disable ISST (SST-BF, SST-CP, SST-TF and SST-PP) in host vars." + +- name: check if Kubernetes Power Manager is build locally on containerd/cri-o runtime + ansible.builtin.assert: + that: kubernetes_power_manager.build_image_locally + fail_msg: + - "Currently Kubernetes Power Manager must be build locally on containerd and cri-o runtime" + - "Please set build_image_locally as true in Kubernetes Power Manager settings in group vars" + when: container_runtime in ["crio", "containerd"] + +- name: check global scaling governor + ansible.builtin.assert: + that: kubernetes_power_manager.global_governor in {{ available_governors }} + fail_msg: + - "{{ kubernetes_power_manager.global_governor }} governor is not supported" + when: + - global_shared_profile_enabled is defined and + global_shared_profile_enabled and + kubernetes_power_manager.global_governor is defined + +- name: check local scaling governor + ansible.builtin.assert: + that: local_shared_profile.local_governor in {{ available_governors }} + fail_msg: + - "{{ local_shared_profile.local_governor }} governor is not supported" + when: + - local_shared_profile is defined and + local_shared_profile and + local_shared_profile.local_governor is defined + +- name: check scaling driver + ansible.builtin.assert: + that: frequency_scaling_driver is defined and frequency_scaling_driver == "acpi_cpufreq" + fail_msg: + - "Governors {{ acpi_only_governors }} are only available with acpi_cpufreq scaling driver. + Please change scaling driver in host vars." + when: + - kubernetes_power_manager.global_governor is defined and kubernetes_power_manager.global_governor in ["userspace", "schedutil"] or + local_shared_profile.local_governor is defined and local_shared_profile.local_governor in ["userspace", "schedutil"] diff --git a/roles/intel_power_manager/tasks/main.yml b/roles/kubernetes_power_manager/tasks/main.yml similarity index 79% rename from roles/intel_power_manager/tasks/main.yml rename to roles/kubernetes_power_manager/tasks/main.yml index 024bc77a..98571bb0 100644 --- a/roles/intel_power_manager/tasks/main.yml +++ b/roles/kubernetes_power_manager/tasks/main.yml @@ -20,27 +20,21 @@ - name: add labels for Power Nodes ansible.builtin.command: kubectl label nodes {{ hostvars[item]['ansible_hostname'] }} intel.power.node=true --overwrite - loop: "{{ intel_power_manager.power_nodes }}" + loop: "{{ kubernetes_power_manager.power_nodes }}" when: inventory_hostname == groups['kube_control_plane'][0] -- name: enable power manager drivers - ansible.builtin.include_tasks: enable_drivers.yml - loop: "{{ intel_power_manager.power_nodes }}" - loop_control: - loop_var: power_node - - name: prepare Intel Kubernetes Power Manager ansible.builtin.include_tasks: power_manager.yml - name: deploy example power pods ansible.builtin.include_tasks: deploy_sample_pods.yml when: - - intel_power_manager.deploy_example_pods + - kubernetes_power_manager.deploy_example_pods - inventory_hostname == groups['kube_control_plane'][0] - name: deploy power manager features ansible.builtin.include_tasks: deploy_features.yml - loop: "{{ intel_power_manager.power_nodes }}" + loop: "{{ kubernetes_power_manager.power_nodes }}" loop_control: loop_var: power_node @@ -50,16 +44,16 @@ - name: populate Global Shared Profile template to the controller node ansible.builtin.template: src: global_shared_profile.yaml.j2 - dest: "{{ (intel_power_manager_dir, 'global_shared_profile.yaml') | path_join }}" + dest: "{{ (kubernetes_power_manager_dir, 'global_shared_profile.yaml') | path_join }}" force: yes mode: preserve - name: deploy Global Shared Profile kubernetes.core.k8s: - src: "{{ (intel_power_manager_dir, 'global_shared_profile.yaml') | path_join }}" + src: "{{ (kubernetes_power_manager_dir, 'global_shared_profile.yaml') | path_join }}" state: present when: - - intel_power_manager.global_shared_profile_enabled + - kubernetes_power_manager.global_shared_profile_enabled - inventory_hostname == groups['kube_control_plane'][0] - name: prepare and deploy Local Shared Power Profiles/Workloads diff --git a/roles/intel_power_manager/tasks/power_manager.yml b/roles/kubernetes_power_manager/tasks/power_manager.yml similarity index 73% rename from roles/intel_power_manager/tasks/power_manager.yml rename to roles/kubernetes_power_manager/tasks/power_manager.yml index 0c288524..24b18ac8 100644 --- a/roles/intel_power_manager/tasks/power_manager.yml +++ b/roles/kubernetes_power_manager/tasks/power_manager.yml @@ -16,9 +16,9 @@ --- - name: clone Intel Kubernetes Power Manager repository ansible.builtin.git: - repo: "{{ intel_power_manager_git_url }}" - version: "{{ intel_power_manager_git_ref }}" - dest: "{{ intel_power_manager_dir }}" + repo: "{{ kubernetes_power_manager_git_url }}" + version: "{{ kubernetes_power_manager_git_ref }}" + dest: "{{ kubernetes_power_manager_dir }}" force: yes when: - inventory_hostname == groups['kube_control_plane'][0] @@ -27,11 +27,11 @@ # PowerProfile. The yaml file needs to be patched before building image to provide correct source for it. - name: patch image to use local registry ansible.builtin.lineinfile: - path: "{{ intel_power_manager_dir }}/build/manifests/power-node-agent-ds.yaml" - regexp: "^.*image: {{ intel_power_node_agent_image }}:{{ intel_power_manager_git_ref }}" - line: " - image: {{ intel_power_node_agent_image_local }}:{{ intel_power_manager_git_ref }}" + path: "{{ kubernetes_power_manager_dir }}/build/manifests/power-node-agent-ds.yaml" + regexp: "^.*image: {{ kubernetes_power_node_agent_image }}:{{ kubernetes_power_manager_git_ref }}" + line: " - image: {{ kubernetes_power_node_agent_image_local }}:{{ kubernetes_power_manager_git_ref }}" when: - - intel_power_manager.build_image_locally | default(false) | bool + - kubernetes_power_manager.build_image_locally | default(false) | bool - inventory_hostname == groups['kube_control_plane'][0] - name: count cpu quota @@ -41,7 +41,7 @@ - name: patch cpu quota ansible.builtin.lineinfile: - path: "{{ intel_power_manager_dir }}/build/manifests/power-node-agent-ds.yaml" + path: "{{ kubernetes_power_manager_dir }}/build/manifests/power-node-agent-ds.yaml" regexp: "^.*cpu: 100m" line: " cpu: {{ cpu_quota }}m" loop: [1, 2] @@ -54,7 +54,7 @@ - name: patch memory quota ansible.builtin.lineinfile: - path: "{{ intel_power_manager_dir }}/build/manifests/power-node-agent-ds.yaml" + path: "{{ kubernetes_power_manager_dir }}/build/manifests/power-node-agent-ds.yaml" regexp: "^.*memory: 64Mi" line: " memory: {{ memory_quota }}Mi" loop: [1, 2] @@ -64,20 +64,20 @@ - name: prepare images for Intel Kubernetes Power Manager when: - container_runtime == "docker" - - intel_power_manager.build_image_locally | default(false) | bool + - kubernetes_power_manager.build_image_locally | default(false) | bool - inventory_hostname == groups['kube_control_plane'][0] block: - name: build images for Intel Kubernetes Power Manager - ansible.builtin.command: docker build -f build/{{ item.file }} -t {{ registry_local_address }}/{{ item.name }}:{{ intel_power_manager_git_ref }} . + ansible.builtin.command: docker build -f build/{{ item.file }} -t {{ registry_local_address }}/{{ item.name }}:{{ kubernetes_power_manager_git_ref }} . changed_when: true args: - chdir: "{{ intel_power_manager_dir }}" + chdir: "{{ kubernetes_power_manager_dir }}" with_items: - {file: Dockerfile, name: intel-power-operator} - {file: Dockerfile.nodeagent, name: intel-power-node-agent} - name: push Intel Kubernetes Power Manager images to local registry - ansible.builtin.command: docker push {{ registry_local_address }}/{{ item }}:{{ intel_power_manager_git_ref }} + ansible.builtin.command: docker push {{ registry_local_address }}/{{ item }}:{{ kubernetes_power_manager_git_ref }} changed_when: true with_items: - intel-power-operator @@ -87,20 +87,20 @@ - name: prepare images for Intel Kubernetes Power Manager when: - container_runtime in ["crio", "containerd"] - - intel_power_manager.build_image_locally | default(false) | bool + - kubernetes_power_manager.build_image_locally | default(false) | bool - inventory_hostname == groups['kube_control_plane'][0] block: - name: build and tag images for Intel Kubernetes Power Manager - ansible.builtin.command: podman build -f build/{{ item.file }} -t {{ registry_local_address }}/{{ item.name }}:{{ intel_power_manager_git_ref }} . + ansible.builtin.command: podman build -f build/{{ item.file }} -t {{ registry_local_address }}/{{ item.name }}:{{ kubernetes_power_manager_git_ref }} . changed_when: true args: - chdir: "{{ intel_power_manager_dir }}" + chdir: "{{ kubernetes_power_manager_dir }}" with_items: - {file: Dockerfile, name: intel-power-operator} - {file: Dockerfile.nodeagent, name: intel-power-node-agent} - name: push Intel Kubernetes Power Manager images to local registry - ansible.builtin.command: podman push {{ registry_local_address }}/{{ item }}:{{ intel_power_manager_git_ref }} + ansible.builtin.command: podman push {{ registry_local_address }}/{{ item }}:{{ kubernetes_power_manager_git_ref }} changed_when: true with_items: - intel-power-operator @@ -111,7 +111,7 @@ block: - name: create Intel Power Manager namespace kubernetes.core.k8s: - name: "{{ intel_power_manager_namespace }}" + name: "{{ kubernetes_power_manager_namespace }}" kind: Namespace state: present definition: @@ -122,36 +122,36 @@ - name: apply k8s prerequisites kubernetes.core.k8s: state: present - src: "{{ (intel_power_manager_dir, 'config', 'rbac', 'rbac.yaml') | path_join }}" + src: "{{ (kubernetes_power_manager_dir, 'config', 'rbac', 'rbac.yaml') | path_join }}" # WA: go mod tidy is needed, until upstream issue is fixed. - name: run go mod tidy ansible.builtin.command: "go mod tidy -v" args: - chdir: "{{ intel_power_manager_dir }}" + chdir: "{{ kubernetes_power_manager_dir }}" changed_when: true - name: create and install Intel Power Manager CRDs community.general.make: - chdir: "{{ intel_power_manager_dir }}" + chdir: "{{ kubernetes_power_manager_dir }}" - name: populate Intel Kubernetes Power Manager Controller Manager template ansible.builtin.template: src: controller_manager.yaml.j2 - dest: "{{ (intel_power_manager_dir, 'controller_manager.yaml') | path_join }}" + dest: "{{ (kubernetes_power_manager_dir, 'controller_manager.yaml') | path_join }}" force: yes mode: preserve - name: deploy Intel Kubernetes Power Manager Controller Manager kubernetes.core.k8s: state: present - src: "{{ (intel_power_manager_dir, 'controller_manager.yaml') | path_join }}" + src: "{{ (kubernetes_power_manager_dir, 'controller_manager.yaml') | path_join }}" - name: wait for Power Manager to be up and running kubernetes.core.k8s_info: kind: Deployment name: controller-manager - namespace: "{{ intel_power_manager_namespace }}" + namespace: "{{ kubernetes_power_manager_namespace }}" wait: yes wait_condition: type: Available @@ -161,20 +161,20 @@ - name: combine power profiles from each power node set_fact: combined_profiles: "{{ combined_profiles + hostvars[item]['power_profiles'] }}" - loop: "{{ intel_power_manager.power_nodes }}" + loop: "{{ kubernetes_power_manager.power_nodes }}" when: inventory_hostname == groups['kube_control_plane'][0] - name: populate Power Config template ansible.builtin.template: src: power_config.yaml.j2 - dest: "{{ (intel_power_manager_dir, 'power_config.yaml') | path_join }}" + dest: "{{ (kubernetes_power_manager_dir, 'power_config.yaml') | path_join }}" force: yes mode: preserve - name: apply Power Config kubernetes.core.k8s: state: present - src: "{{ (intel_power_manager_dir, 'power_config.yaml') | path_join }}" + src: "{{ (kubernetes_power_manager_dir, 'power_config.yaml') | path_join }}" - name: check that all pods are running ansible.builtin.include_role: diff --git a/roles/intel_power_manager/tasks/power_pod_template_helper.yml b/roles/kubernetes_power_manager/tasks/power_pod_template_helper.yml similarity index 87% rename from roles/intel_power_manager/tasks/power_pod_template_helper.yml rename to roles/kubernetes_power_manager/tasks/power_pod_template_helper.yml index 85eb6b74..08bf0a59 100644 --- a/roles/intel_power_manager/tasks/power_pod_template_helper.yml +++ b/roles/kubernetes_power_manager/tasks/power_pod_template_helper.yml @@ -17,7 +17,7 @@ - name: populate sample power pods templates ansible.builtin.template: src: "sample_power_pod.yaml.j2" - dest: "{{ (intel_power_manager_dir, 'sample_power_pods', power_profile_name + '_power_pod_' + power_node + '.yaml') | path_join }}" + dest: "{{ (kubernetes_power_manager_dir, 'sample_power_pods', power_profile_name + '_power_pod_' + power_node + '.yaml') | path_join }}" force: yes mode: preserve loop: "{{ hostvars[power_node]['power_profiles'] }}" diff --git a/roles/intel_power_manager/templates/controller_manager.yaml.j2 b/roles/kubernetes_power_manager/templates/controller_manager.yaml.j2 similarity index 71% rename from roles/intel_power_manager/templates/controller_manager.yaml.j2 rename to roles/kubernetes_power_manager/templates/controller_manager.yaml.j2 index e9674497..2578cb7e 100644 --- a/roles/intel_power_manager/templates/controller_manager.yaml.j2 +++ b/roles/kubernetes_power_manager/templates/controller_manager.yaml.j2 @@ -3,7 +3,7 @@ apiVersion: apps/v1 kind: Deployment metadata: name: controller-manager - namespace: {{ intel_power_manager_namespace }} + namespace: {{ kubernetes_power_manager_namespace }} labels: control-plane: controller-manager spec: @@ -23,10 +23,10 @@ spec: args: - --enable-leader-election imagePullPolicy: IfNotPresent - {% if intel_power_manager.build_image_locally -%} - image: {{ intel_power_operator_image_local }}:{{ intel_power_manager_git_ref }} + {% if kubernetes_power_manager.build_image_locally -%} + image: {{ kubernetes_power_operator_image_local }}:{{ kubernetes_power_manager_git_ref }} {% else -%} - image: {{ intel_power_operator_image }}:{{ intel_power_manager_git_ref }} + image: {{ kubernetes_power_operator_image }}:{{ kubernetes_power_manager_git_ref }} {% endif -%} securityContext: allowPrivilegeEscalation: false @@ -36,10 +36,10 @@ spec: resources: limits: cpu: "{{ ( 100 + ( 100 * multiplier | float ) | int | abs ) }}m" - memory: "{{ ( 30 + ( 30 * multiplier | float ) | int | abs ) }}Mi" + memory: "{{ ( 200 + ( 200 * multiplier | float ) | int | abs ) }}Mi" requests: cpu: "{{ ( 100 + ( 100 * multiplier | float ) | int | abs ) }}m" - memory: "{{ ( 30 + ( 30 * multiplier | float ) | int | abs ) }}Mi" + memory: "{{ ( 200 + ( 200 * multiplier | float ) | int | abs ) }}Mi" volumeMounts: - mountPath: /sys/fs name: cgroup diff --git a/roles/intel_power_manager/templates/cstates.yaml.j2 b/roles/kubernetes_power_manager/templates/cstates.yaml.j2 similarity index 78% rename from roles/intel_power_manager/templates/cstates.yaml.j2 rename to roles/kubernetes_power_manager/templates/cstates.yaml.j2 index 17a8b7c6..0105d6ad 100644 --- a/roles/intel_power_manager/templates/cstates.yaml.j2 +++ b/roles/kubernetes_power_manager/templates/cstates.yaml.j2 @@ -2,7 +2,7 @@ apiVersion: power.intel.com/v1 kind: CStates metadata: name: {{ power_node }} - namespace: {{ intel_power_manager_namespace }} + namespace: {{ kubernetes_power_manager_namespace }} spec: sharedPoolCStates: {{ cstates.shared }} diff --git a/roles/kubernetes_power_manager/templates/global_shared_profile.yaml.j2 b/roles/kubernetes_power_manager/templates/global_shared_profile.yaml.j2 new file mode 100644 index 00000000..e4e9ddeb --- /dev/null +++ b/roles/kubernetes_power_manager/templates/global_shared_profile.yaml.j2 @@ -0,0 +1,13 @@ +--- +apiVersion: "power.intel.com/v1" +kind: PowerProfile +metadata: + name: shared-global + namespace: {{ kubernetes_power_manager_namespace }} +spec: + name: "shared-global" + max: {{ kubernetes_power_manager.global_max_frequency }} + min: {{ kubernetes_power_manager.global_min_frequency }} + epp: "power" + shared: true + governor: {{ kubernetes_power_manager.global_governor }} diff --git a/roles/intel_power_manager/templates/local_shared_profile.yaml.j2 b/roles/kubernetes_power_manager/templates/local_shared_profile.yaml.j2 similarity index 68% rename from roles/intel_power_manager/templates/local_shared_profile.yaml.j2 rename to roles/kubernetes_power_manager/templates/local_shared_profile.yaml.j2 index 26379eb3..f04916a3 100644 --- a/roles/intel_power_manager/templates/local_shared_profile.yaml.j2 +++ b/roles/kubernetes_power_manager/templates/local_shared_profile.yaml.j2 @@ -3,10 +3,11 @@ apiVersion: "power.intel.com/v1" kind: PowerProfile metadata: name: shared-{{ node_name }} - namespace: {{ intel_power_manager_namespace }} + namespace: {{ kubernetes_power_manager_namespace }} spec: name: "shared-{{ node_name }}" max: {{ local_max_frequency }} min: {{ local_min_frequency }} epp: "power" - governor: {{ local_pstate_governor }} + shared: true + governor: {{ local_governor }} diff --git a/roles/intel_power_manager/templates/power_config.yaml.j2 b/roles/kubernetes_power_manager/templates/power_config.yaml.j2 similarity index 87% rename from roles/intel_power_manager/templates/power_config.yaml.j2 rename to roles/kubernetes_power_manager/templates/power_config.yaml.j2 index e05cc073..9fa1a191 100644 --- a/roles/intel_power_manager/templates/power_config.yaml.j2 +++ b/roles/kubernetes_power_manager/templates/power_config.yaml.j2 @@ -3,7 +3,7 @@ apiVersion: "power.intel.com/v1" kind: PowerConfig metadata: name: power-config - namespace: {{ intel_power_manager_namespace }} + namespace: {{ kubernetes_power_manager_namespace }} spec: powerNodeSelector: # Add labels here for the Nodes you want the PowerNodeAgent to be applied to diff --git a/roles/intel_power_manager/templates/sample_power_pod.yaml.j2 b/roles/kubernetes_power_manager/templates/sample_power_pod.yaml.j2 similarity index 93% rename from roles/intel_power_manager/templates/sample_power_pod.yaml.j2 rename to roles/kubernetes_power_manager/templates/sample_power_pod.yaml.j2 index 99d573e2..28fd1d8a 100644 --- a/roles/intel_power_manager/templates/sample_power_pod.yaml.j2 +++ b/roles/kubernetes_power_manager/templates/sample_power_pod.yaml.j2 @@ -4,7 +4,7 @@ apiVersion: v1 kind: Pod metadata: name: {{ power_profile_name }}-power-pod-{{ power_node }} - namespace: {{ intel_power_manager_namespace }} + namespace: {{ kubernetes_power_manager_namespace }} spec: containers: - name: {{ power_profile_name }}-container diff --git a/roles/intel_power_manager/templates/shared_workload.yaml.j2 b/roles/kubernetes_power_manager/templates/shared_workload.yaml.j2 similarity index 89% rename from roles/intel_power_manager/templates/shared_workload.yaml.j2 rename to roles/kubernetes_power_manager/templates/shared_workload.yaml.j2 index 8ef54dd6..99624fd7 100644 --- a/roles/intel_power_manager/templates/shared_workload.yaml.j2 +++ b/roles/kubernetes_power_manager/templates/shared_workload.yaml.j2 @@ -3,7 +3,7 @@ apiVersion: "power.intel.com/v1" kind: PowerWorkload metadata: name: shared-{{ node_name }}-workload - namespace: {{ intel_power_manager_namespace }} + namespace: {{ kubernetes_power_manager_namespace }} spec: name: "shared-{{ node_name }}-workload" allCores: true diff --git a/roles/intel_power_manager/templates/uncore_frequency.yaml.j2 b/roles/kubernetes_power_manager/templates/uncore_frequency.yaml.j2 similarity index 75% rename from roles/intel_power_manager/templates/uncore_frequency.yaml.j2 rename to roles/kubernetes_power_manager/templates/uncore_frequency.yaml.j2 index e659813a..0bb1bff5 100644 --- a/roles/intel_power_manager/templates/uncore_frequency.yaml.j2 +++ b/roles/kubernetes_power_manager/templates/uncore_frequency.yaml.j2 @@ -2,7 +2,7 @@ apiVersion: power.intel.com/v1 kind: Uncore metadata: name: {{ power_node }} - namespace: {{ intel_power_manager_namespace }} + namespace: {{ kubernetes_power_manager_namespace }} spec: sysMax: {{ system_max_frequency }} sysMin: {{ system_min_frequency }} diff --git a/roles/intel_power_manager/vars/main.yml b/roles/kubernetes_power_manager/vars/main.yml similarity index 72% rename from roles/intel_power_manager/vars/main.yml rename to roles/kubernetes_power_manager/vars/main.yml index b67da666..74a34cbf 100644 --- a/roles/intel_power_manager/vars/main.yml +++ b/roles/kubernetes_power_manager/vars/main.yml @@ -18,12 +18,18 @@ install_dependencies: Debian: - git - make + - cpuid RedHat: - git - make + - cpuid # need union of all profiles for power config combined_profiles: [] # used in calculating Cpu or Mem quotas -multiplier: '{{ [intel_power_manager.power_nodes | length / 20, 1.0] | min }}' # 20+ nodes will double the max basic value +multiplier: '{{ [kubernetes_power_manager.power_nodes | length / 20, 1.0] | min }}' # 20+ nodes will double the max basic value + +common_governors: ["performance", "powersave"] +acpi_only_governors: ["userspace", "schedutil"] +available_governors: "{{ common_governors + acpi_only_governors }}" diff --git a/roles/kubespray_patch/files/kubespray_dnsstublistener.patch b/roles/kubespray_patch/files/kubespray_dnsstublistener.patch deleted file mode 100644 index 03e09923..00000000 --- a/roles/kubespray_patch/files/kubespray_dnsstublistener.patch +++ /dev/null @@ -1,13 +0,0 @@ -diff --git a/roles/kubernetes/preinstall/templates/resolved.conf.j2 b/roles/kubernetes/preinstall/templates/resolved.conf.j2 -index 901fd2473..50f3886e0 100644 ---- a/roles/kubernetes/preinstall/templates/resolved.conf.j2 -+++ b/roles/kubernetes/preinstall/templates/resolved.conf.j2 -@@ -14,7 +14,7 @@ Domains={{ searchdomains|default([]) | join(' ') }} - #MulticastDNS=no - DNSSEC=no - Cache=no-negative --{% if ansible_os_family in ["Flatcar", "Flatcar Container Linux by Kinvolk"] %} -+{% if ansible_os_family in ["Debian", "RedHat", "Flatcar", "Flatcar Container Linux by Kinvolk"] %} - DNSStubListener=no - {% else %} - #DNSStubListener=yes diff --git a/roles/kubespray_patch/tasks/main.yml b/roles/kubespray_patch/tasks/main.yml index 4e4328c4..a2aab8cc 100644 --- a/roles/kubespray_patch/tasks/main.yml +++ b/roles/kubespray_patch/tasks/main.yml @@ -32,13 +32,6 @@ src: "files/kubespray_delay_wait.patch" dest: "{{ kubespray_dir }}/roles/kubernetes/preinstall/handlers/main.yml" -- name: apply kubespray patch for DNSStubListener - ansible.posix.patch: - src: "files/kubespray_dnsstublistener.patch" - dest: "{{ kubespray_dir }}/roles/kubernetes/preinstall/templates/resolved.conf.j2" - when: - - dns_disable_stub_listener | default(true) | bool - - name: Load patch checksum ansible.builtin.include_tasks: load_checksum.yml diff --git a/roles/linkerd_service_mesh/defaults/main.yml b/roles/linkerd_service_mesh/defaults/main.yml index 1d03db2d..1072f402 100644 --- a/roles/linkerd_service_mesh/defaults/main.yml +++ b/roles/linkerd_service_mesh/defaults/main.yml @@ -17,7 +17,7 @@ # defaults file for linkerd-cli linkerd_cli_arch: "amd64" linkerd_release: "stable" -linkerd_version: "2.13.3" +linkerd_version: "2.13.6" linkerd_cli_version: "{{ linkerd_version }}" linkerd_cli_uri: "https://github.com/linkerd/linkerd2/releases/download/{{ linkerd_release }}-{{ linkerd_cli_version }}/\ diff --git a/roles/linkerd_service_mesh/tasks/main.yml b/roles/linkerd_service_mesh/tasks/main.yml index 02332223..8b877f23 100644 --- a/roles/linkerd_service_mesh/tasks/main.yml +++ b/roles/linkerd_service_mesh/tasks/main.yml @@ -57,5 +57,4 @@ ansible.builtin.shell: "set -o pipefail && {{ kubectl_cli_bin }} delete cronjob linkerd-heartbeat -n linkerd" args: executable: /bin/bash - warn: false when: http_proxy is defined or https_proxy is defined diff --git a/roles/linkerd_service_mesh/templates/cert-manager-objects.yml.j2 b/roles/linkerd_service_mesh/templates/cert-manager-objects.yml.j2 index 1c33ab9d..90a95f5f 100644 --- a/roles/linkerd_service_mesh/templates/cert-manager-objects.yml.j2 +++ b/roles/linkerd_service_mesh/templates/cert-manager-objects.yml.j2 @@ -1,6 +1,6 @@ --- apiVersion: cert-manager.io/v1 -kind: ClusterIssuer +kind: Issuer metadata: name: linkerd-selfsigned-issuer namespace: "{{ linkerd_namespace }}" @@ -22,7 +22,7 @@ spec: size: 256 issuerRef: name: linkerd-selfsigned-issuer - kind: ClusterIssuer + kind: Issuer group: cert-manager.io --- apiVersion: cert-manager.io/v1 diff --git a/roles/minio_install/tasks/cleanup_minio_filesystems.yml b/roles/minio_install/tasks/cleanup_minio_filesystems.yml deleted file mode 100644 index ce45b391..00000000 --- a/roles/minio_install/tasks/cleanup_minio_filesystems.yml +++ /dev/null @@ -1,93 +0,0 @@ -## -## Copyright (c) 2020-2023 Intel Corporation. -## -## Licensed under the Apache License, Version 2.0 (the "License"); -## you may not use this file except in compliance with the License. -## You may obtain a copy of the License at -## -## http://www.apache.org/licenses/LICENSE-2.0 -## -## Unless required by applicable law or agreed to in writing, software -## distributed under the License is distributed on an "AS IS" BASIS, -## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -## See the License for the specific language governing permissions and -## limitations under the License. -## ---- -- name: uninstall MinIO - block: - - name: set iteration number - set_fact: - num: 1 - req_num: "{{ hostvars[inventory_hostname]['minio_pv'] | length }}" - - - name: umount volumes - command: >- - umount --lazy "{{ item.mountPath }}" - loop: "{{ hostvars[inventory_hostname]['minio_pv'] }}" - when: - - hostvars[inventory_hostname]['minio_pv'] is defined and hostvars[inventory_hostname]['minio_pv'] | length > 0 - changed_when: false - failed_when: false - - - name: remove paritions for nvme deploy - parted: - device: "{{ item.device }}" - number: 1 - state: absent - loop: "{{ hostvars[inventory_hostname]['minio_pv'] }}" - when: - - hostvars[inventory_hostname]['minio_pv'] is defined and hostvars[inventory_hostname]['minio_pv'] | length > 0 - changed_when: false - failed_when: false - - - name: list loop devices - shell: >- - set -o pipefail && losetup -l |grep diskimage |awk -F " " '{ print $6,$1}' |sort |awk -F " " '{ print $2}' - args: - executable: /bin/bash - register: loopdevice_output - changed_when: false - failed_when: false - - - name: remove loop devices for test mode deploy (block device) - command: >- - losetup -d "{{ item }}" - loop: "{{ loopdevice_output.stdout_lines }}" - changed_when: false - failed_when: false - - - name: remove mount points - file: - path: "{{ item.mountPath }}" - state: absent - loop: "{{ hostvars[inventory_hostname]['minio_pv'] }}" - when: - - hostvars[inventory_hostname]['minio_pv'] is defined and hostvars[inventory_hostname]['minio_pv']|length > 0 - changed_when: false - failed_when: false - - - name: remove block device files (diskimages) - file: - path: "{{ ('/tmp', 'diskimage' ~ ansible_loop.index) | path_join }}" - state: absent - loop: "{{ range(num, req_num | int + 1) | list }}" - loop_control: - extended: yes - changed_when: false - failed_when: false - when: - - inventory_hostname in groups['kube_node'] - - minio_enabled | default(false) | bool - -- name: remove mount points from /etc/fstab - shell: >- - set -o pipefail && cat /etc/fstab | sed "{{ item.mountPath | replace('/', '\\/') }}"/d | tee /etc/fstab - args: - executable: /bin/bash - loop: "{{ hostvars[inventory_hostname]['minio_pv'] }}" - when: - - hostvars[inventory_hostname]['minio_pv'] is defined and hostvars[inventory_hostname]['minio_pv']|length > 0 - - inventory_hostname in groups['kube_node'] - changed_when: false - failed_when: false diff --git a/roles/minio_install/tasks/cleanup_minio_main.yml b/roles/minio_install/tasks/cleanup_minio_main.yml index 06c13fdc..c50171e4 100644 --- a/roles/minio_install/tasks/cleanup_minio_main.yml +++ b/roles/minio_install/tasks/cleanup_minio_main.yml @@ -74,7 +74,5 @@ name: kubernetes_ingress_install tasks_from: cleanup_kubernetes_ingress - - name: cleanup MinIO file systems - import_tasks: cleanup_minio_filesystems.yml tags: - minio diff --git a/roles/minio_install/tasks/create_nvme_partition.yml b/roles/minio_install/tasks/create_nvme_partition.yml deleted file mode 100644 index c3cc8eae..00000000 --- a/roles/minio_install/tasks/create_nvme_partition.yml +++ /dev/null @@ -1,46 +0,0 @@ -## -## Copyright (c) 2020-2023 Intel Corporation. -## -## Licensed under the Apache License, Version 2.0 (the "License"); -## you may not use this file except in compliance with the License. -## You may obtain a copy of the License at -## -## http://www.apache.org/licenses/LICENSE-2.0 -## -## Unless required by applicable law or agreed to in writing, software -## distributed under the License is distributed on an "AS IS" BASIS, -## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -## See the License for the specific language governing permissions and -## limitations under the License. -## ---- -- name: create a new xfs primary partition - parted: - name: "{{ hostvars[inventory_hostname]['minio_pv'][ansible_loop.index0].name }}" - device: "{{ hostvars[inventory_hostname]['minio_pv'][ansible_loop.index0].device }}" - label: gpt - number: 1 - part_type: primary - part_end: "{{ minio_tenant_volume_size }}GiB" - flags: [lvm] - state: present - -- name: format the partition - filesystem: - fstype: xfs - dev: "{{ hostvars[inventory_hostname]['minio_pv'][ansible_loop.index0].device }}p1" - force: yes - state: present - -- name: create mount point for the file block devices - file: - path: "{{ hostvars[inventory_hostname]['minio_pv'][ansible_loop.index0].mountPath }}" - state: directory - mode: 0755 - -- name: mount the parition - mount: - fstype: xfs - src: "{{ hostvars[inventory_hostname]['minio_pv'][ansible_loop.index0].device }}p1" - path: "{{ hostvars[inventory_hostname]['minio_pv'][ansible_loop.index0].mountPath }}" - state: mounted diff --git a/roles/minio_install/tasks/create_persistentvolumes.yml b/roles/minio_install/tasks/create_persistentvolumes.yml index a11b5dfb..721fa8a3 100644 --- a/roles/minio_install/tasks/create_persistentvolumes.yml +++ b/roles/minio_install/tasks/create_persistentvolumes.yml @@ -21,7 +21,7 @@ force: yes mode: preserve when: - - hostvars[item]['minio_pv'] | default([]) | length > 0 + - hostvars[item]['persistent_volumes'] | default([]) | length > 0 loop: "{{ groups['kube_node'] }}" loop_control: extended: yes @@ -31,7 +31,7 @@ state: present src: "{{ (minio_operator_helm_local_dir, 'operator', 'temp', 'minio-tenant-persistentvolumes' ~ ansible_loop.index ~ '.yml') | path_join }}" when: - - hostvars[item]['minio_pv'] | default([]) | length > 0 + - hostvars[item]['persistent_volumes'] | default([]) | length > 0 loop: "{{ groups['kube_node'] }}" loop_control: extended: yes diff --git a/roles/minio_install/tasks/file_blockdevice.yml b/roles/minio_install/tasks/file_blockdevice.yml deleted file mode 100644 index cd9a09e3..00000000 --- a/roles/minio_install/tasks/file_blockdevice.yml +++ /dev/null @@ -1,44 +0,0 @@ -## -## Copyright (c) 2020-2023 Intel Corporation. -## -## Licensed under the Apache License, Version 2.0 (the "License"); -## you may not use this file except in compliance with the License. -## You may obtain a copy of the License at -## -## http://www.apache.org/licenses/LICENSE-2.0 -## -## Unless required by applicable law or agreed to in writing, software -## distributed under the License is distributed on an "AS IS" BASIS, -## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -## See the License for the specific language governing permissions and -## limitations under the License. -## ---- -- name: set iteration number - set_fact: - num: 1 - req_num: "{{ hostvars[inventory_hostname]['minio_pv'] | length }}" - -- name: create local file block device - include_tasks: create_blockdevicefiles.yml - loop: "{{ range(num, req_num|int + 1) | list }}" - loop_control: - extended: yes - -- name: format with xfs file block devices - include_tasks: format_blockdevicefiles.yml - loop: "{{ range(num, req_num|int + 1) | list }}" - loop_control: - extended: yes - -- name: setup the loop devices - include_tasks: setup_loopdevices.yml - loop: "{{ range(num, req_num|int + 1) | list }}" - loop_control: - extended: yes - -- name: mount the loop devices - include_tasks: mount_loopdevices.yml - loop: "{{ range(num, req_num|int + 1) | list }}" - loop_control: - extended: yes diff --git a/roles/minio_install/tasks/format_blockdevicefiles.yml b/roles/minio_install/tasks/format_blockdevicefiles.yml index 276860e7..3a3bc252 100644 --- a/roles/minio_install/tasks/format_blockdevicefiles.yml +++ b/roles/minio_install/tasks/format_blockdevicefiles.yml @@ -17,6 +17,6 @@ - name: format file block device with xfs filesystem: fstype: xfs - dev: /tmp/diskimage{{ ansible_loop.index }} + dev: /opt/cek/disks/tmp/diskimage{{ ansible_loop.index }} force: yes state: present diff --git a/roles/minio_install/tasks/main.yml b/roles/minio_install/tasks/main.yml index fcbfda43..5ce2d9dd 100644 --- a/roles/minio_install/tasks/main.yml +++ b/roles/minio_install/tasks/main.yml @@ -65,22 +65,6 @@ - minio_nodes | int >= minio_tenant_servers - inventory_hostname == groups['kube_control_plane'][0] -- name: create MinIO local storage - include_tasks: file_blockdevice.yml - when: - - minio_tenant_enabled - - minio_deploy_test_mode - - minio_nodes | int >= minio_tenant_servers - - inventory_hostname in groups['kube_node'] - -- name: create MinIO nvme storage - include_tasks: nvme_blockdevice.yml - when: - - minio_tenant_enabled - - not minio_deploy_test_mode - - minio_nodes | int >= minio_tenant_servers - - inventory_hostname in groups['kube_node'] - - name: populate MinIO local persisitent volumes include_tasks: create_persistentvolumes.yml when: diff --git a/roles/minio_install/tasks/nvme_blockdevice.yml b/roles/minio_install/tasks/nvme_blockdevice.yml deleted file mode 100644 index 315ab6e6..00000000 --- a/roles/minio_install/tasks/nvme_blockdevice.yml +++ /dev/null @@ -1,26 +0,0 @@ -## -## Copyright (c) 2020-2023 Intel Corporation. -## -## Licensed under the Apache License, Version 2.0 (the "License"); -## you may not use this file except in compliance with the License. -## You may obtain a copy of the License at -## -## http://www.apache.org/licenses/LICENSE-2.0 -## -## Unless required by applicable law or agreed to in writing, software -## distributed under the License is distributed on an "AS IS" BASIS, -## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -## See the License for the specific language governing permissions and -## limitations under the License. -## ---- -- name: set iteration number - set_fact: - num: 1 - req_num: "{{ hostvars[inventory_hostname]['minio_pv'] | length }}" - -- name: configuring nvme block device partition - include_tasks: create_nvme_partition.yml - loop: "{{ range(num, req_num|int + 1) | list }}" - loop_control: - extended: yes diff --git a/roles/minio_install/tasks/preflight_minio_config.yml b/roles/minio_install/tasks/preflight_minio_config.yml index 8901a076..91e212be 100644 --- a/roles/minio_install/tasks/preflight_minio_config.yml +++ b/roles/minio_install/tasks/preflight_minio_config.yml @@ -26,10 +26,10 @@ - name: make sure the MinIO tenant volumes per server >= the MiniO PV list assert: - that: "{{ minio_pv | length | int }} >= {{ minio_tenant_volumes_per_server | int }}" + that: "{{ persistent_volumes | length | int }} >= {{ minio_tenant_volumes_per_server | int }}" msg: - "Incorrect configuration." - - "The number of MinIO Persistent Volumes (PVs) '{{ minio_pv | length | int }}' defined in the host vars must be" + - "The number of MinIO Persistent Volumes (PVs) '{{ persistent_volumes | length | int }}' defined in the host vars must be" - "equal or more than MinIO Tenant Volumes per Server '{{ minio_tenant_volumes_per_server }}' defined in group vars." when: - inventory_hostname in groups['kube_node'] diff --git a/roles/minio_install/templates/minio_tenant_localpersistentvolumes.yml.j2 b/roles/minio_install/templates/minio_tenant_localpersistentvolumes.yml.j2 index 4de0c184..9e7f14c1 100644 --- a/roles/minio_install/templates/minio_tenant_localpersistentvolumes.yml.j2 +++ b/roles/minio_install/templates/minio_tenant_localpersistentvolumes.yml.j2 @@ -1,4 +1,4 @@ -{%- for pv in hostvars[item]['minio_pv'] %} +{%- for pv in hostvars[item]['persistent_volumes'] %} --- apiVersion: v1 kind: PersistentVolume diff --git a/roles/openssl_engine_install/defaults/main.yml b/roles/openssl_engine_install/defaults/main.yml index da1b6bbe..bc062e84 100644 --- a/roles/openssl_engine_install/defaults/main.yml +++ b/roles/openssl_engine_install/defaults/main.yml @@ -16,9 +16,9 @@ --- openssl_engine_dir: "{{ (project_root_dir, 'openssl') | path_join }}" openssl_engine_url: "https://github.com/intel/QAT_Engine.git" -openssl_engine_version: "v1.2.0" +openssl_engine_version: "v1.3.1" libarchive_url: "https://github.com/libarchive/libarchive/releases/download/v3.5.1/libarchive-3.5.1.tar.xz" ipp_crypto_url: "https://github.com/intel/ipp-crypto.git" ipp_crypto_version: "ippcp_2021.7.1" intel_ipsec_url: "https://github.com/intel/intel-ipsec-mb.git" -intel_ipsec_version: "v1.3" +intel_ipsec_version: "v1.4" diff --git a/roles/openssl_engine_install/tasks/openssl_engine_config.yml b/roles/openssl_engine_install/tasks/openssl_engine_config.yml index b13ee926..6d933724 100644 --- a/roles/openssl_engine_install/tasks/openssl_engine_config.yml +++ b/roles/openssl_engine_install/tasks/openssl_engine_config.yml @@ -105,6 +105,17 @@ dest: "{{ openssl_engine_dir }}/imbc_ipsec" force: yes +- name: workaround to fix the imbc_ipsec compilation error for tdx kernel + ansible.builtin.file: + src: /usr/lib/x86_64-linux-gnu/libstdc++.so.6 + dest: /usr/lib/x86_64-linux-gnu/libstdc++.so + state: link + mode: '0644' + when: + - configure_tdx | default(false) + - not on_vms | default(false) + - ansible_distribution == "Ubuntu" and ansible_distribution_version == '22.04' + - name: build Intel MBC-IPsec Library command: 'make -j{{ nproc_out.stdout | int }} SAFE_DATA=y SAFE_PARAM=y SAFE_LOOKUP=y' args: diff --git a/roles/opentelemetry_install/defaults/main.yml b/roles/opentelemetry_install/defaults/main.yml index 7d78e5c0..3d50ab2b 100644 --- a/roles/opentelemetry_install/defaults/main.yml +++ b/roles/opentelemetry_install/defaults/main.yml @@ -17,7 +17,7 @@ opentelemetry_repo: "https://open-telemetry.github.io/opentelemetry-helm-charts" opentelemetry_operator_namespace: "monitoring" opentelemetry_operator_chart_name: "opentelemetry-operator" -opentelemetry_operator_chart_version: "0.27.0" +opentelemetry_operator_chart_version: "0.35.1" opentelemetry_collectors: gateway: true diff --git a/roles/opentelemetry_install/files/otel-agent-cadvisor.yaml b/roles/opentelemetry_install/files/otel-agent-cadvisor.yaml index dee52845..0ff1c7cc 100644 --- a/roles/opentelemetry_install/files/otel-agent-cadvisor.yaml +++ b/roles/opentelemetry_install/files/otel-agent-cadvisor.yaml @@ -37,7 +37,11 @@ spec: scrape_configs: - job_name: "otel-cadvisor-collector" scrape_interval: 5s - scheme: http + scheme: https + tls_config: + ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt + insecure_skip_verify: true + bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: endpoints relabel_configs: diff --git a/roles/platform_aware_scheduling_install/defaults/main.yml b/roles/platform_aware_scheduling_install/defaults/main.yml index d371b684..e61ee5a8 100644 --- a/roles/platform_aware_scheduling_install/defaults/main.yml +++ b/roles/platform_aware_scheduling_install/defaults/main.yml @@ -32,7 +32,7 @@ sigs_k8s_io_dir: "{{ (project_root_dir, 'sigs.k8s.io') | path_join }}" # TAS deployment tas_enabled: false tas_build_image_locally: false -tas_extender_image_tag_default: "0.5.0" +tas_extender_image_tag_default: "0.6.0" tas_git_version: "telemetry-aware-scheduling/v{{ tas_extender_image_tag_default }}" tas_version: |- {{ ('telemetry-aware-scheduling' in tas_git_version) @@ -65,7 +65,7 @@ tas_verbosity: 4 # GAS deployment gas_enabled: false gas_build_image_locally: false -gas_extender_image_tag_default: "0.5.2" +gas_extender_image_tag_default: "0.5.4" gas_git_version: "gpu-aware-scheduling/v{{ gas_extender_image_tag_default }}" gas_version: |- {{ ('gpu-aware-scheduling' in gas_git_version) diff --git a/roles/prometheus_install/container_prometheus/defaults/main.yml b/roles/prometheus_install/container_prometheus/defaults/main.yml new file mode 100644 index 00000000..21ba4b72 --- /dev/null +++ b/roles/prometheus_install/container_prometheus/defaults/main.yml @@ -0,0 +1,30 @@ +## +## Copyright (c) 2020-2023 Intel Corporation. +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## +container_prometheus_stack_path: "{{ (project_root_dir, 'container_prometheus_stack') | path_join }}" +container_prometheus_data_path: "{{ (container_prometheus_stack_path, 'prometheus_data') | path_join }}" + +container_grafana_data_path: "{{ (container_prometheus_stack_path, 'grafana_data') | path_join }}" +container_grafana_datasource_path: "{{ (container_prometheus_stack_path, 'grafana_datasources') | path_join }}" +container_grafana_dashboard_path: "{{ (container_prometheus_stack_path, 'grafana_dashboards') | path_join }}" + + +container_cert_path: "{{ (project_root_dir, 'cert') | path_join }}" +container_node_exporter_key_path: "{{ (container_cert_path, 'node_key.pem') | path_join }}" +container_node_exporter_cert_path: "{{ (container_cert_path, 'node_cert.pem') | path_join }}" +container_prometheus_key_path: "{{ (container_cert_path, 'prom_key.pem') | path_join }}" +container_prometheus_cert_path: "{{ (container_cert_path, 'prom_cert.pem') | path_join }}" +container_grafana_key_path: "{{ (container_cert_path, 'graf_key.pem') | path_join }}" +container_grafana_cert_path: "{{ (container_cert_path, 'graf_cert.pem') | path_join }}" diff --git a/roles/prometheus_install/container_prometheus/files/dashboard.yml b/roles/prometheus_install/container_prometheus/files/dashboard.yml new file mode 100644 index 00000000..d1be169a --- /dev/null +++ b/roles/prometheus_install/container_prometheus/files/dashboard.yml @@ -0,0 +1,12 @@ +apiVersion: 1 + +providers: + - name: 'default' + orgId: 1 + type: file + disableDeletion: false + updateIntervalSeconds: 60 + allowUiUpdates: false + options: + path: /etc/grafana/provisioning/dashboards + foldersFromFilesStructure: true diff --git a/roles/prometheus_install/container_prometheus/files/dashboard_node_exporter.json b/roles/prometheus_install/container_prometheus/files/dashboard_node_exporter.json new file mode 100644 index 00000000..90810ba7 --- /dev/null +++ b/roles/prometheus_install/container_prometheus/files/dashboard_node_exporter.json @@ -0,0 +1,23379 @@ +{ + "__inputs": [ + { + "name": "DS_PROMETHEUS", + "label": "Prometheus", + "description": "", + "type": "datasource", + "pluginId": "prometheus", + "pluginName": "Prometheus" + } + ], + "__elements": {}, + "__requires": [ + { + "type": "panel", + "id": "gauge", + "name": "Gauge", + "version": "" + }, + { + "type": "grafana", + "id": "grafana", + "name": "Grafana", + "version": "9.4.3" + }, + { + "type": "datasource", + "id": "prometheus", + "name": "Prometheus", + "version": "1.0.0" + }, + { + "type": "panel", + "id": "stat", + "name": "Stat", + "version": "" + }, + { + "type": "panel", + "id": "timeseries", + "name": "Time series", + "version": "" + } + ], + "annotations": { + "list": [ + { + "$$hashKey": "object:1058", + "builtIn": 1, + "datasource": { + "type": "datasource", + "uid": "grafana" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "target": { + "limit": 100, + "matchAny": false, + "tags": [], + "type": "dashboard" + }, + "type": "dashboard" + } + ] + }, + "editable": true, + "fiscalYearStartMonth": 0, + "gnetId": 1860, + "graphTooltip": 1, + "id": null, + "links": [ + { + "icon": "external link", + "tags": [], + "targetBlank": true, + "title": "GitHub", + "type": "link", + "url": "https://github.com/rfmoz/grafana-dashboards" + }, + { + "icon": "external link", + "tags": [], + "targetBlank": true, + "title": "Grafana", + "type": "link", + "url": "https://grafana.com/grafana/dashboards/1860" + } + ], + "liveNow": false, + "panels": [ + { + "collapsed": false, + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 261, + "panels": [], + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "refId": "A" + } + ], + "title": "Quick CPU / Mem / Disk", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Busy state of all CPU cores together", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 1, + "mappings": [ + { + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" + } + ], + "max": 100, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "rgba(50, 172, 45, 0.97)", + "value": null + }, + { + "color": "rgba(237, 129, 40, 0.89)", + "value": 85 + }, + { + "color": "rgba(245, 54, 54, 0.9)", + "value": 95 + } + ] + }, + "unit": "percent" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 3, + "x": 0, + "y": 1 + }, + "id": 20, + "links": [], + "options": { + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true + }, + "pluginVersion": "9.4.3", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "(sum by(instance) (irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\", mode!=\"idle\"}[$__rate_interval])) / on(instance) group_left sum by (instance)((irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])))) * 100", + "hide": false, + "instant": true, + "intervalFactor": 1, + "legendFormat": "", + "range": false, + "refId": "A", + "step": 240 + } + ], + "title": "CPU Busy", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Busy state of all CPU cores together (5 min average)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 1, + "mappings": [ + { + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" + } + ], + "max": 100, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "rgba(50, 172, 45, 0.97)", + "value": null + }, + { + "color": "rgba(237, 129, 40, 0.89)", + "value": 85 + }, + { + "color": "rgba(245, 54, 54, 0.9)", + "value": 95 + } + ] + }, + "unit": "percent" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 3, + "x": 3, + "y": 1 + }, + "id": 155, + "links": [], + "options": { + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true + }, + "pluginVersion": "9.4.3", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "avg_over_time(node_load5{instance=\"$node\",job=\"$job\"}[$__rate_interval]) * 100 / on(instance) group_left sum by (instance)(irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval]))", + "format": "time_series", + "hide": false, + "instant": true, + "intervalFactor": 1, + "range": false, + "refId": "A", + "step": 240 + } + ], + "title": "Sys Load (5m avg)", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Busy state of all CPU cores together (15 min average)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 1, + "mappings": [ + { + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" + } + ], + "max": 100, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "rgba(50, 172, 45, 0.97)", + "value": null + }, + { + "color": "rgba(237, 129, 40, 0.89)", + "value": 85 + }, + { + "color": "rgba(245, 54, 54, 0.9)", + "value": 95 + } + ] + }, + "unit": "percent" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 3, + "x": 6, + "y": 1 + }, + "id": 19, + "links": [], + "options": { + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true + }, + "pluginVersion": "9.4.3", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "avg_over_time(node_load15{instance=\"$node\",job=\"$job\"}[$__rate_interval]) * 100 / on(instance) group_left sum by (instance)(irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval]))", + "hide": false, + "instant": true, + "intervalFactor": 1, + "range": false, + "refId": "A", + "step": 240 + } + ], + "title": "Sys Load (15m avg)", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Non available RAM memory", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 1, + "mappings": [], + "max": 100, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "rgba(50, 172, 45, 0.97)", + "value": null + }, + { + "color": "rgba(237, 129, 40, 0.89)", + "value": 80 + }, + { + "color": "rgba(245, 54, 54, 0.9)", + "value": 90 + } + ] + }, + "unit": "percent" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 3, + "x": 9, + "y": 1 + }, + "hideTimeOverride": false, + "id": 16, + "links": [], + "options": { + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true + }, + "pluginVersion": "9.4.3", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "((avg_over_time(node_memory_MemTotal_bytes{instance=\"$node\",job=\"$job\"}[$__rate_interval]) - avg_over_time(node_memory_MemFree_bytes{instance=\"$node\",job=\"$job\"}[$__rate_interval])) / (avg_over_time(node_memory_MemTotal_bytes{instance=\"$node\",job=\"$job\"}[$__rate_interval]) )) * 100", + "format": "time_series", + "hide": true, + "instant": true, + "intervalFactor": 1, + "range": false, + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "100 - ((avg_over_time(node_memory_MemAvailable_bytes{instance=\"$node\",job=\"$job\"}[$__rate_interval]) * 100) / avg_over_time(node_memory_MemTotal_bytes{instance=\"$node\",job=\"$job\"}[$__rate_interval]))", + "format": "time_series", + "hide": false, + "instant": true, + "intervalFactor": 1, + "range": false, + "refId": "B", + "step": 240 + } + ], + "title": "RAM Used", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Used Swap", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 1, + "mappings": [ + { + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" + } + ], + "max": 100, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "rgba(50, 172, 45, 0.97)", + "value": null + }, + { + "color": "rgba(237, 129, 40, 0.89)", + "value": 10 + }, + { + "color": "rgba(245, 54, 54, 0.9)", + "value": 25 + } + ] + }, + "unit": "percent" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 3, + "x": 12, + "y": 1 + }, + "id": 21, + "links": [], + "options": { + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true + }, + "pluginVersion": "9.4.3", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "((avg_over_time(node_memory_SwapTotal_bytes{instance=\"$node\",job=\"$job\"}[$__rate_interval]) - avg_over_time(node_memory_SwapFree_bytes{instance=\"$node\",job=\"$job\"}[$__rate_interval])) / (avg_over_time(node_memory_SwapTotal_bytes{instance=\"$node\",job=\"$job\"}[$__rate_interval]) )) * 100", + "instant": true, + "intervalFactor": 1, + "range": false, + "refId": "A", + "step": 240 + } + ], + "title": "SWAP Used", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Used Root FS", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 1, + "mappings": [ + { + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" + } + ], + "max": 100, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "rgba(50, 172, 45, 0.97)", + "value": null + }, + { + "color": "rgba(237, 129, 40, 0.89)", + "value": 80 + }, + { + "color": "rgba(245, 54, 54, 0.9)", + "value": 90 + } + ] + }, + "unit": "percent" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 3, + "x": 15, + "y": 1 + }, + "id": 154, + "links": [], + "options": { + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true + }, + "pluginVersion": "9.4.3", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "100 - ((avg_over_time(node_filesystem_avail_bytes{instance=\"$node\",job=\"$job\",mountpoint=\"/\",fstype!=\"rootfs\"}[$__rate_interval]) * 100) / avg_over_time(node_filesystem_size_bytes{instance=\"$node\",job=\"$job\",mountpoint=\"/\",fstype!=\"rootfs\"}[$__rate_interval]))", + "format": "time_series", + "instant": true, + "intervalFactor": 1, + "range": false, + "refId": "A", + "step": 240 + } + ], + "title": "Root FS Used", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Total number of CPU cores", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [ + { + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 2, + "w": 2, + "x": 18, + "y": 1 + }, + "id": 14, + "links": [], + "maxDataPoints": 100, + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "textMode": "auto" + }, + "pluginVersion": "9.4.3", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "count(count(node_cpu_seconds_total{instance=\"$node\",job=\"$job\"}) by (cpu))", + "legendFormat": "__auto", + "range": true, + "refId": "A" + } + ], + "title": "CPU Cores", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "System uptime", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 1, + "mappings": [ + { + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 2, + "w": 4, + "x": 20, + "y": 1 + }, + "hideTimeOverride": true, + "id": 15, + "links": [], + "maxDataPoints": 100, + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "textMode": "auto" + }, + "pluginVersion": "9.4.3", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "node_time_seconds{instance=\"$node\",job=\"$job\"} - node_boot_time_seconds{instance=\"$node\",job=\"$job\"}", + "instant": true, + "intervalFactor": 1, + "range": false, + "refId": "A", + "step": 240 + } + ], + "title": "Uptime", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Total RootFS", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 0, + "mappings": [ + { + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "rgba(50, 172, 45, 0.97)", + "value": null + }, + { + "color": "rgba(237, 129, 40, 0.89)", + "value": 70 + }, + { + "color": "rgba(245, 54, 54, 0.9)", + "value": 90 + } + ] + }, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": { + "h": 2, + "w": 2, + "x": 18, + "y": 3 + }, + "id": 23, + "links": [], + "maxDataPoints": 100, + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "textMode": "auto" + }, + "pluginVersion": "9.4.3", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "node_filesystem_size_bytes{instance=\"$node\",job=\"$job\",mountpoint=\"/\",fstype!=\"rootfs\"}", + "format": "time_series", + "hide": false, + "instant": true, + "intervalFactor": 1, + "range": false, + "refId": "A", + "step": 240 + } + ], + "title": "RootFS Total", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Total RAM", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 0, + "mappings": [ + { + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": { + "h": 2, + "w": 2, + "x": 20, + "y": 3 + }, + "id": 75, + "links": [], + "maxDataPoints": 100, + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "textMode": "auto" + }, + "pluginVersion": "9.4.3", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "node_memory_MemTotal_bytes{instance=\"$node\",job=\"$job\"}", + "instant": true, + "intervalFactor": 1, + "range": false, + "refId": "A", + "step": 240 + } + ], + "title": "RAM Total", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Total SWAP", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 0, + "mappings": [ + { + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": { + "h": 2, + "w": 2, + "x": 22, + "y": 3 + }, + "id": 18, + "links": [], + "maxDataPoints": 100, + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "textMode": "auto" + }, + "pluginVersion": "9.4.3", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "node_memory_SwapTotal_bytes{instance=\"$node\",job=\"$job\"}", + "instant": true, + "intervalFactor": 1, + "range": false, + "refId": "A", + "step": 240 + } + ], + "title": "SWAP Total", + "type": "stat" + }, + { + "collapsed": false, + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 5 + }, + "id": 263, + "panels": [], + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "refId": "A" + } + ], + "title": "Basic CPU / Mem / Net / Disk", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Basic CPU info", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 40, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "percent" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Busy Iowait" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#890F02", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Idle" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Busy Iowait" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#890F02", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Idle" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#7EB26D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Busy System" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Busy User" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A437C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Busy Other" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6D1F62", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 0, + "y": 6 + }, + "id": 77, + "links": [], + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true, + "width": 250 + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "sum by(instance) (irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\", mode=\"system\"}[$__rate_interval])) / on(instance) group_left sum by (instance)((irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])))", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "Busy System", + "range": true, + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "sum by(instance) (irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\", mode=\"user\"}[$__rate_interval])) / on(instance) group_left sum by (instance)((irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])))", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "Busy User", + "range": true, + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "sum by(instance) (irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\", mode=\"iowait\"}[$__rate_interval])) / on(instance) group_left sum by (instance)((irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])))", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Busy Iowait", + "range": true, + "refId": "C", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "sum by(instance) (irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\", mode=~\".*irq\"}[$__rate_interval])) / on(instance) group_left sum by (instance)((irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])))", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Busy IRQs", + "range": true, + "refId": "D", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "sum by(instance) (irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\", mode!='idle',mode!='user',mode!='system',mode!='iowait',mode!='irq',mode!='softirq'}[$__rate_interval])) / on(instance) group_left sum by (instance)((irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])))", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Busy Other", + "range": true, + "refId": "E", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "sum by(instance) (irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\", mode=\"idle\"}[$__rate_interval])) / on(instance) group_left sum by (instance)((irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])))", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Idle", + "range": true, + "refId": "F", + "step": 240 + } + ], + "title": "CPU Basic", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Basic memory usage", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 40, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "normal" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Apps" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#629E51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Buffers" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#614D93", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6D1F62", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cached" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Committed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#508642", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A437C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Hardware Corrupted - Amount of RAM that the kernel identified as corrupted / not working" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#CFFAFF", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Inactive" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#584477", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "PageTables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Page_Tables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "RAM_Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0F9D7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "SWAP Used" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#806EB7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0752D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap Used" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#C15C17", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#2F575E", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Unused" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "RAM Total" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0F9D7", + "mode": "fixed" + } + }, + { + "id": "custom.fillOpacity", + "value": 0 + }, + { + "id": "custom.stacking", + "value": { + "group": false, + "mode": "normal" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "RAM Cache + Buffer" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "RAM Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#7EB26D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Available" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#DEDAF7", + "mode": "fixed" + } + }, + { + "id": "custom.fillOpacity", + "value": 0 + }, + { + "id": "custom.stacking", + "value": { + "group": false, + "mode": "normal" + } + } + ] + } + ] + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 12, + "y": 6 + }, + "id": 78, + "links": [], + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true, + "width": 350 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_MemTotal_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "RAM Total", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_MemTotal_bytes{instance=\"$node\",job=\"$job\"} - node_memory_MemFree_bytes{instance=\"$node\",job=\"$job\"} - (node_memory_Cached_bytes{instance=\"$node\",job=\"$job\"} + node_memory_Buffers_bytes{instance=\"$node\",job=\"$job\"} + node_memory_SReclaimable_bytes{instance=\"$node\",job=\"$job\"})", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "RAM Used", + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_Cached_bytes{instance=\"$node\",job=\"$job\"} + node_memory_Buffers_bytes{instance=\"$node\",job=\"$job\"} + node_memory_SReclaimable_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "RAM Cache + Buffer", + "refId": "C", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_MemFree_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "RAM Free", + "refId": "D", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "(node_memory_SwapTotal_bytes{instance=\"$node\",job=\"$job\"} - node_memory_SwapFree_bytes{instance=\"$node\",job=\"$job\"})", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "SWAP Used", + "refId": "E", + "step": 240 + } + ], + "title": "Memory Basic", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Basic network info per interface", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 40, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Recv_bytes_eth2" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#7EB26D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Recv_bytes_lo" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Recv_drop_eth2" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6ED0E0", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Recv_drop_lo" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0F9D7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Recv_errs_eth2" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Recv_errs_lo" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#CCA300", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Trans_bytes_eth2" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#7EB26D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Trans_bytes_lo" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Trans_drop_eth2" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6ED0E0", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Trans_drop_lo" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0F9D7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Trans_errs_eth2" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Trans_errs_lo" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#CCA300", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "recv_bytes_lo" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "recv_drop_eth0" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#99440A", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "recv_drop_lo" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#967302", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "recv_errs_eth0" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "recv_errs_lo" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#890F02", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "trans_bytes_eth0" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#7EB26D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "trans_bytes_lo" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "trans_drop_eth0" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#99440A", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "trans_drop_lo" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#967302", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "trans_errs_eth0" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "trans_errs_lo" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#890F02", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*trans.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + } + ] + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 0, + "y": 13 + }, + "id": 74, + "links": [], + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_network_receive_bytes_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])*8", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "recv {{device}}", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_network_transmit_bytes_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])*8", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "trans {{device}} ", + "refId": "B", + "step": 240 + } + ], + "title": "Network Traffic Basic", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Disk space used of all filesystems mounted", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 40, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "max": 100, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "percent" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 12, + "y": 13 + }, + "id": 152, + "links": [], + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "100 - ((node_filesystem_avail_bytes{instance=\"$node\",job=\"$job\",device!~'rootfs'} * 100) / node_filesystem_size_bytes{instance=\"$node\",job=\"$job\",device!~'rootfs'})", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "{{mountpoint}}", + "refId": "A", + "step": 240 + } + ], + "title": "Disk Space Used Basic", + "type": "timeseries" + }, + { + "collapsed": true, + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 20 + }, + "id": 265, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "percentage", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 70, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "smooth", + "lineWidth": 2, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "percent" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Idle - Waiting for something to happen" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Iowait - Waiting for I/O to complete" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Irq - Servicing interrupts" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Nice - Niced processes executing in user mode" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#C15C17", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Softirq - Servicing softirqs" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E24D42", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Steal - Time spent in other operating systems when running in a virtualized environment" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#FCE2DE", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "System - Processes executing in kernel mode" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#508642", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "User - Normal processes executing in user mode" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#5195CE", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 0, + "y": 7 + }, + "id": 3, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 250 + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "sum by(instance) (irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\", mode=\"system\"}[$__rate_interval])) / on(instance) group_left sum by (instance)((irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])))", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "System - Processes executing in kernel mode", + "range": true, + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "sum by(instance) (irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\", mode=\"user\"}[$__rate_interval])) / on(instance) group_left sum by (instance)((irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])))", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "User - Normal processes executing in user mode", + "range": true, + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "sum by(instance) (irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\", mode=\"nice\"}[$__rate_interval])) / on(instance) group_left sum by (instance)((irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])))", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Nice - Niced processes executing in user mode", + "range": true, + "refId": "C", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "sum by(instance) (irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\", mode=\"iowait\"}[$__rate_interval])) / on(instance) group_left sum by (instance)((irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])))", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Iowait - Waiting for I/O to complete", + "range": true, + "refId": "E", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "sum by(instance) (irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\", mode=\"irq\"}[$__rate_interval])) / on(instance) group_left sum by (instance)((irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])))", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Irq - Servicing interrupts", + "range": true, + "refId": "F", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "sum by(instance) (irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\", mode=\"softirq\"}[$__rate_interval])) / on(instance) group_left sum by (instance)((irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])))", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Softirq - Servicing softirqs", + "range": true, + "refId": "G", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "sum by(instance) (irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\", mode=\"steal\"}[$__rate_interval])) / on(instance) group_left sum by (instance)((irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])))", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Steal - Time spent in other operating systems when running in a virtualized environment", + "range": true, + "refId": "H", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "sum by(instance) (irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\", mode=\"idle\"}[$__rate_interval])) / on(instance) group_left sum by (instance)((irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])))", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "Idle - Waiting for something to happen", + "range": true, + "refId": "J", + "step": 240 + } + ], + "title": "CPU", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bytes", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 40, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "normal" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Apps" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#629E51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Buffers" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#614D93", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6D1F62", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cached" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Committed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#508642", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A437C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Hardware Corrupted - Amount of RAM that the kernel identified as corrupted / not working" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#CFFAFF", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Inactive" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#584477", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "PageTables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Page_Tables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "RAM_Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0F9D7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#806EB7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0752D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap - Swap memory usage" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#C15C17", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#2F575E", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Unused" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Unused - Free memory unassigned" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*Hardware Corrupted - *./" + }, + "properties": [ + { + "id": "custom.stacking", + "value": { + "group": false, + "mode": "normal" + } + } + ] + } + ] + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 12, + "y": 7 + }, + "id": 24, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 350 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_MemTotal_bytes{instance=\"$node\",job=\"$job\"} - node_memory_MemFree_bytes{instance=\"$node\",job=\"$job\"} - node_memory_Buffers_bytes{instance=\"$node\",job=\"$job\"} - node_memory_Cached_bytes{instance=\"$node\",job=\"$job\"} - node_memory_Slab_bytes{instance=\"$node\",job=\"$job\"} - node_memory_PageTables_bytes{instance=\"$node\",job=\"$job\"} - node_memory_SwapCached_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "Apps - Memory used by user-space applications", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_PageTables_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "PageTables - Memory used to map between virtual and physical memory addresses", + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_SwapCached_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "SwapCache - Memory that keeps track of pages that have been fetched from swap but not yet been modified", + "refId": "C", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_Slab_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "Slab - Memory used by the kernel to cache data structures for its own use (caches like inode, dentry, etc)", + "refId": "D", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_Cached_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "Cache - Parked file data (file content) cache", + "refId": "E", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_Buffers_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "Buffers - Block device (e.g. harddisk) cache", + "refId": "F", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_MemFree_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "Unused - Free memory unassigned", + "refId": "G", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "(node_memory_SwapTotal_bytes{instance=\"$node\",job=\"$job\"} - node_memory_SwapFree_bytes{instance=\"$node\",job=\"$job\"})", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "Swap - Swap space used", + "refId": "H", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_HardwareCorrupted_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "Hardware Corrupted - Amount of RAM that the kernel identified as corrupted / not working", + "refId": "I", + "step": 240 + } + ], + "title": "Memory Stack", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bits out (-) / in (+)", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 40, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "receive_packets_eth0" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#7EB26D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "receive_packets_lo" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E24D42", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "transmit_packets_eth0" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#7EB26D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "transmit_packets_lo" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E24D42", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*Trans.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + } + ] + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 0, + "y": 19 + }, + "id": 84, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_network_receive_bytes_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])*8", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "{{device}} - Receive", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_network_transmit_bytes_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])*8", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "{{device}} - Transmit", + "refId": "B", + "step": 240 + } + ], + "title": "Network Traffic", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bytes", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 40, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 12, + "y": 19 + }, + "id": 156, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_filesystem_size_bytes{instance=\"$node\",job=\"$job\",device!~'rootfs'} - node_filesystem_avail_bytes{instance=\"$node\",job=\"$job\",device!~'rootfs'}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "{{mountpoint}}", + "refId": "A", + "step": 240 + } + ], + "title": "Disk Space Used", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "IO read (-) / write (+)", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "iops" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Read.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#7EB26D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6ED0E0", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EF843C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E24D42", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#584477", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda2_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BA43A9", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda3_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F4D598", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0752D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#962D82", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#614D93", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#9AC48A", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#65C5DB", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F9934E", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0F9D7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#FCEACA", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F9E2D2", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 0, + "y": 31 + }, + "id": 229, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_disk_reads_completed_total{instance=\"$node\",job=\"$job\",device=~\"$diskdevices\"}[$__rate_interval])", + "intervalFactor": 4, + "legendFormat": "{{device}} - Reads completed", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_disk_writes_completed_total{instance=\"$node\",job=\"$job\",device=~\"$diskdevices\"}[$__rate_interval])", + "intervalFactor": 1, + "legendFormat": "{{device}} - Writes completed", + "refId": "B", + "step": 240 + } + ], + "title": "Disk IOps", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bytes read (-) / write (+)", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 40, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "Bps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "io time" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#890F02", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*read*./" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#7EB26D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6ED0E0", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EF843C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E24D42", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byType", + "options": "time" + }, + "properties": [ + { + "id": "custom.axisPlacement", + "value": "hidden" + } + ] + } + ] + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 12, + "y": 31 + }, + "id": 42, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_disk_read_bytes_total{instance=\"$node\",job=\"$job\",device=~\"$diskdevices\"}[$__rate_interval])", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "{{device}} - Successfully read bytes", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_disk_written_bytes_total{instance=\"$node\",job=\"$job\",device=~\"$diskdevices\"}[$__rate_interval])", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "{{device}} - Successfully written bytes", + "refId": "B", + "step": 240 + } + ], + "title": "I/O Usage Read / Write", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "%util", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 40, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "io time" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#890F02", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byType", + "options": "time" + }, + "properties": [ + { + "id": "custom.axisPlacement", + "value": "hidden" + } + ] + } + ] + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 0, + "y": 43 + }, + "id": 127, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_disk_io_time_seconds_total{instance=\"$node\",job=\"$job\",device=~\"$diskdevices\"} [$__rate_interval])", + "format": "time_series", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "{{device}}", + "refId": "A", + "step": 240 + } + ], + "title": "I/O Utilization", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "percentage", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "bars", + "fillOpacity": 70, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "smooth", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "max": 1, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/^Guest - /" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#5195ce", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/^GuestNice - /" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#c15c17", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 12, + "y": 43 + }, + "id": 319, + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "sum by(instance) (irate(node_cpu_guest_seconds_total{instance=\"$node\",job=\"$job\", mode=\"user\"}[1m])) / on(instance) group_left sum by (instance)((irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\"}[1m])))", + "hide": false, + "legendFormat": "Guest - Time spent running a virtual CPU for a guest operating system", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "sum by(instance) (irate(node_cpu_guest_seconds_total{instance=\"$node\",job=\"$job\", mode=\"nice\"}[1m])) / on(instance) group_left sum by (instance)((irate(node_cpu_seconds_total{instance=\"$node\",job=\"$job\"}[1m])))", + "hide": false, + "legendFormat": "GuestNice - Time spent running a niced guest (virtual CPU for guest operating system)", + "range": true, + "refId": "B" + } + ], + "title": "CPU spent seconds in guests (VMs)", + "type": "timeseries" + } + ], + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "refId": "A" + } + ], + "title": "CPU / Memory / Net / Disk", + "type": "row" + }, + { + "collapsed": true, + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 21 + }, + "id": 266, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bytes", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "normal" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Apps" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#629E51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Buffers" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#614D93", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6D1F62", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cached" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Committed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#508642", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A437C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Hardware Corrupted - Amount of RAM that the kernel identified as corrupted / not working" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#CFFAFF", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Inactive" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#584477", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "PageTables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Page_Tables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "RAM_Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0F9D7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#806EB7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0752D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#C15C17", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#2F575E", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Unused" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 38 + }, + "id": 136, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 350 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_Inactive_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Inactive - Memory which has been less recently used. It is more eligible to be reclaimed for other purposes", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_Active_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Active - Memory that has been used more recently and usually not reclaimed unless absolutely necessary", + "refId": "B", + "step": 240 + } + ], + "title": "Memory Active / Inactive", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bytes", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Apps" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#629E51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Buffers" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#614D93", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6D1F62", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cached" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Committed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#508642", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A437C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Hardware Corrupted - Amount of RAM that the kernel identified as corrupted / not working" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#CFFAFF", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Inactive" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#584477", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "PageTables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Page_Tables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "RAM_Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0F9D7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#806EB7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0752D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#C15C17", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#2F575E", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Unused" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*CommitLimit - *./" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + }, + { + "id": "custom.fillOpacity", + "value": 0 + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 38 + }, + "id": 135, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 350 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_Committed_AS_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Committed_AS - Amount of memory presently allocated on the system", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_CommitLimit_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "CommitLimit - Amount of memory currently available to be allocated on the system", + "refId": "B", + "step": 240 + } + ], + "title": "Memory Committed", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bytes", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "normal" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Apps" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#629E51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Buffers" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#614D93", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6D1F62", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cached" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Committed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#508642", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A437C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Hardware Corrupted - Amount of RAM that the kernel identified as corrupted / not working" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#CFFAFF", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Inactive" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#584477", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "PageTables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Page_Tables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "RAM_Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0F9D7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#806EB7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0752D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#C15C17", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#2F575E", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Unused" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 48 + }, + "id": 191, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 350 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_Inactive_file_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "Inactive_file - File-backed memory on inactive LRU list", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_Inactive_anon_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "Inactive_anon - Anonymous and swap cache on inactive LRU list, including tmpfs (shmem)", + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_Active_file_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "Active_file - File-backed memory on active LRU list", + "refId": "C", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_Active_anon_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "Active_anon - Anonymous and swap cache on active least-recently-used (LRU) list, including tmpfs", + "refId": "D", + "step": 240 + } + ], + "title": "Memory Active / Inactive Detail", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bytes", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Active" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#99440A", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Buffers" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#58140C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6D1F62", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cached" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Committed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#508642", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Dirty" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6ED0E0", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#B7DBAB", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Inactive" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Mapped" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "PageTables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Page_Tables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#C15C17", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total RAM" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total RAM + Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#614D93", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "VmallocUsed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 48 + }, + "id": 130, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_Writeback_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Writeback - Memory which is actively being written back to disk", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_WritebackTmp_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "WritebackTmp - Memory used by FUSE for temporary writeback buffers", + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_Dirty_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Dirty - Memory which is waiting to get written back to the disk", + "refId": "C", + "step": 240 + } + ], + "title": "Memory Writeback and Dirty", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bytes", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Apps" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#629E51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Buffers" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#614D93", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6D1F62", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cached" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Committed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#508642", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A437C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Hardware Corrupted - Amount of RAM that the kernel identified as corrupted / not working" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#CFFAFF", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Inactive" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#584477", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "PageTables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Page_Tables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "RAM_Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0F9D7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#806EB7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0752D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#C15C17", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#2F575E", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Unused" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "ShmemHugePages - Memory used by shared memory (shmem) and tmpfs allocated with huge pages" + }, + "properties": [ + { + "id": "custom.fillOpacity", + "value": 0 + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "ShmemHugePages - Memory used by shared memory (shmem) and tmpfs allocated with huge pages" + }, + "properties": [ + { + "id": "custom.fillOpacity", + "value": 0 + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 58 + }, + "id": 138, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 350 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_Mapped_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Mapped - Used memory in mapped pages files which have been mapped, such as libraries", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_Shmem_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Shmem - Used shared memory (shared between several processes, thus including RAM disks)", + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_ShmemHugePages_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "ShmemHugePages - Memory used by shared memory (shmem) and tmpfs allocated with huge pages", + "refId": "C", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_ShmemPmdMapped_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "ShmemPmdMapped - Amount of shared (shmem/tmpfs) memory backed by huge pages", + "refId": "D", + "step": 240 + } + ], + "title": "Memory Shared and Mapped", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bytes", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "normal" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Active" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#99440A", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Buffers" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#58140C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6D1F62", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cached" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Committed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#508642", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Dirty" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6ED0E0", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#B7DBAB", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Inactive" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Mapped" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "PageTables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Page_Tables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#C15C17", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total RAM" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total RAM + Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#614D93", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "VmallocUsed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 58 + }, + "id": 131, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_SUnreclaim_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "SUnreclaim - Part of Slab, that cannot be reclaimed on memory pressure", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_SReclaimable_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "SReclaimable - Part of Slab, that might be reclaimed, such as caches", + "refId": "B", + "step": 240 + } + ], + "title": "Memory Slab", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bytes", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Active" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#99440A", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Buffers" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#58140C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6D1F62", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cached" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Committed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#508642", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Dirty" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6ED0E0", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#B7DBAB", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Inactive" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Mapped" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "PageTables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Page_Tables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#C15C17", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total RAM" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total RAM + Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "VmallocUsed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 68 + }, + "id": 70, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_VmallocChunk_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "VmallocChunk - Largest contiguous block of vmalloc area which is free", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_VmallocTotal_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "VmallocTotal - Total size of vmalloc memory area", + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_VmallocUsed_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "VmallocUsed - Amount of vmalloc area which is used", + "refId": "C", + "step": 240 + } + ], + "title": "Memory Vmalloc", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bytes", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Apps" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#629E51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Buffers" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#614D93", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6D1F62", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cached" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Committed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#508642", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A437C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Hardware Corrupted - Amount of RAM that the kernel identified as corrupted / not working" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#CFFAFF", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Inactive" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#584477", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "PageTables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Page_Tables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "RAM_Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0F9D7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#806EB7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0752D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#C15C17", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#2F575E", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Unused" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 68 + }, + "id": 159, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 350 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_Bounce_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Bounce - Memory used for block device bounce buffers", + "refId": "A", + "step": 240 + } + ], + "title": "Memory Bounce", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bytes", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Active" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#99440A", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Buffers" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#58140C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6D1F62", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cached" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Committed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#508642", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Dirty" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6ED0E0", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#B7DBAB", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Inactive" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Mapped" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "PageTables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Page_Tables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#C15C17", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total RAM" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total RAM + Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "VmallocUsed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*Inactive *./" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 78 + }, + "id": 129, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_AnonHugePages_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "AnonHugePages - Memory in anonymous huge pages", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_AnonPages_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "AnonPages - Memory in user pages not backed by files", + "refId": "B", + "step": 240 + } + ], + "title": "Memory Anonymous", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bytes", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Apps" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#629E51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Buffers" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#614D93", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6D1F62", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cached" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Committed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#508642", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A437C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Hardware Corrupted - Amount of RAM that the kernel identified as corrupted / not working" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#CFFAFF", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Inactive" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#584477", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "PageTables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Page_Tables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "RAM_Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0F9D7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#806EB7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0752D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#C15C17", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#2F575E", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Unused" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 78 + }, + "id": 160, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 350 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_KernelStack_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "KernelStack - Kernel memory stack. This is not reclaimable", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_Percpu_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "PerCPU - Per CPU memory allocated dynamically by loadable modules", + "refId": "B", + "step": 240 + } + ], + "title": "Memory Kernel / CPU", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "pages", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Active" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#99440A", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Buffers" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#58140C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6D1F62", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cached" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Committed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#508642", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Dirty" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6ED0E0", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#B7DBAB", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Inactive" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Mapped" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "PageTables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Page_Tables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#C15C17", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total RAM" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#806EB7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total RAM + Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#806EB7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "VmallocUsed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 88 + }, + "id": 140, + "links": [], + "options": { + "legend": { + "calcs": [ + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_HugePages_Free{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "HugePages_Free - Huge pages in the pool that are not yet allocated", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_HugePages_Rsvd{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "HugePages_Rsvd - Huge pages for which a commitment to allocate from the pool has been made, but no allocation has yet been made", + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_HugePages_Surp{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "HugePages_Surp - Huge pages in the pool above the value in /proc/sys/vm/nr_hugepages", + "refId": "C", + "step": 240 + } + ], + "title": "Memory HugePages Counter", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bytes", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Active" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#99440A", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Buffers" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#58140C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6D1F62", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cached" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Committed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#508642", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Dirty" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6ED0E0", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#B7DBAB", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Inactive" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Mapped" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "PageTables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Page_Tables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#C15C17", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total RAM" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#806EB7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total RAM + Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#806EB7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "VmallocUsed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 88 + }, + "id": 71, + "links": [], + "options": { + "legend": { + "calcs": [ + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_HugePages_Total{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "HugePages - Total size of the pool of huge pages", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_Hugepagesize_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Hugepagesize - Huge Page size", + "refId": "B", + "step": 240 + } + ], + "title": "Memory HugePages Size", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bytes", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Active" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#99440A", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Buffers" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#58140C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6D1F62", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cached" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Committed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#508642", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Dirty" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6ED0E0", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#B7DBAB", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Inactive" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Mapped" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "PageTables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Page_Tables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#C15C17", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total RAM" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total RAM + Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "VmallocUsed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 98 + }, + "id": 128, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_DirectMap1G_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "DirectMap1G - Amount of pages mapped as this size", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_DirectMap2M_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "DirectMap2M - Amount of pages mapped as this size", + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_DirectMap4k_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "DirectMap4K - Amount of pages mapped as this size", + "refId": "C", + "step": 240 + } + ], + "title": "Memory DirectMap", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bytes", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Apps" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#629E51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Buffers" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#614D93", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6D1F62", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cached" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Committed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#508642", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A437C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Hardware Corrupted - Amount of RAM that the kernel identified as corrupted / not working" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#CFFAFF", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Inactive" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#584477", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "PageTables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Page_Tables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "RAM_Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0F9D7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#806EB7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0752D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#C15C17", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#2F575E", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Unused" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 98 + }, + "id": 137, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 350 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_Unevictable_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Unevictable - Amount of unevictable memory that can't be swapped out for a variety of reasons", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_Mlocked_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "MLocked - Size of pages locked to memory using the mlock() system call", + "refId": "B", + "step": 240 + } + ], + "title": "Memory Unevictable and MLocked", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bytes", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Active" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#99440A", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Buffers" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#58140C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6D1F62", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cached" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Committed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#508642", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Dirty" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6ED0E0", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#B7DBAB", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Inactive" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Mapped" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "PageTables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Page_Tables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#C15C17", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total RAM" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total RAM + Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#614D93", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "VmallocUsed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 108 + }, + "id": 132, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_memory_NFS_Unstable_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "NFS Unstable - Memory in NFS pages sent to the server, but not yet committed to the storage", + "refId": "A", + "step": 240 + } + ], + "title": "Memory NFS", + "type": "timeseries" + } + ], + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "refId": "A" + } + ], + "title": "Memory Meminfo", + "type": "row" + }, + { + "collapsed": true, + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 22 + }, + "id": 267, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "pages out (-) / in (+)", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*out/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 25 + }, + "id": 176, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_vmstat_pgpgin{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Pagesin - Page in operations", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_vmstat_pgpgout{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Pagesout - Page out operations", + "refId": "B", + "step": 240 + } + ], + "title": "Memory Pages In / Out", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "pages out (-) / in (+)", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*out/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 25 + }, + "id": 22, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_vmstat_pswpin{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Pswpin - Pages swapped in", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_vmstat_pswpout{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Pswpout - Pages swapped out", + "refId": "B", + "step": 240 + } + ], + "title": "Memory Pages Swap In / Out", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "faults", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "normal" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Apps" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#629E51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Buffers" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#614D93", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6D1F62", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cached" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Committed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#508642", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A437C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Hardware Corrupted - Amount of RAM that the kernel identified as corrupted / not working" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#CFFAFF", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Inactive" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#584477", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "PageTables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Page_Tables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "RAM_Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0F9D7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#806EB7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0752D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#C15C17", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#2F575E", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Unused" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Pgfault - Page major and minor fault operations" + }, + "properties": [ + { + "id": "custom.fillOpacity", + "value": 0 + }, + { + "id": "custom.stacking", + "value": { + "group": false, + "mode": "normal" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 35 + }, + "id": 175, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 350 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_vmstat_pgfault{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Pgfault - Page major and minor fault operations", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_vmstat_pgmajfault{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Pgmajfault - Major page fault operations", + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_vmstat_pgfault{instance=\"$node\",job=\"$job\"}[$__rate_interval]) - irate(node_vmstat_pgmajfault{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Pgminfault - Minor page fault operations", + "refId": "C", + "step": 240 + } + ], + "title": "Memory Page Faults", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Active" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#99440A", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Buffers" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#58140C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6D1F62", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Cached" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Committed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#508642", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Dirty" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6ED0E0", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Free" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#B7DBAB", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Inactive" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Mapped" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "PageTables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Page_Tables" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Slab_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Swap_Cache" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#C15C17", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#511749", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total RAM" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total RAM + Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#052B51", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Total Swap" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#614D93", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "VmallocUsed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 35 + }, + "id": 307, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_vmstat_oom_kill{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "oom killer invocations ", + "refId": "A", + "step": 240 + } + ], + "title": "OOM Killer", + "type": "timeseries" + } + ], + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "refId": "A" + } + ], + "title": "Memory Vmstat", + "type": "row" + }, + { + "collapsed": true, + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 23 + }, + "id": 293, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "seconds", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Variation*./" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#890F02", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 40 + }, + "id": 260, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_timex_estimated_error_seconds{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "Estimated error in seconds", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_timex_offset_seconds{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "Time offset in between local system and reference clock", + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_timex_maxerror_seconds{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "Maximum error in seconds", + "refId": "C", + "step": 240 + } + ], + "title": "Time Synchronized Drift", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 40 + }, + "id": 291, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_timex_loop_time_constant{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "Phase-locked loop time adjust", + "refId": "A", + "step": 240 + } + ], + "title": "Time PLL Adjust", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Variation*./" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#890F02", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 50 + }, + "id": 168, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_timex_sync_status{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "Is clock synchronized to a reliable server (1 = yes, 0 = no)", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_timex_frequency_adjustment_ratio{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "Local clock frequency adjustment", + "refId": "B", + "step": 240 + } + ], + "title": "Time Synchronized Status", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "seconds", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 50 + }, + "id": 294, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_timex_tick_seconds{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "Seconds between clock ticks", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_timex_tai_offset_seconds{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "International Atomic Time (TAI) offset", + "refId": "B", + "step": 240 + } + ], + "title": "Time Misc", + "type": "timeseries" + } + ], + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "refId": "A" + } + ], + "title": "System Timesync", + "type": "row" + }, + { + "collapsed": true, + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 24 + }, + "id": 312, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 27 + }, + "id": 62, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_procs_blocked{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Processes blocked waiting for I/O to complete", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_procs_running{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Processes in runnable state", + "refId": "B", + "step": 240 + } + ], + "title": "Processes Status", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "normal" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 27 + }, + "id": 315, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_processes_state{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "{{ state }}", + "refId": "A", + "step": 240 + } + ], + "title": "Processes State", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "forks / sec", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 37 + }, + "id": 148, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_forks_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "Processes forks second", + "refId": "A", + "step": 240 + } + ], + "title": "Processes Forks", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bytes", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "decbytes" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Max.*/" + }, + "properties": [ + { + "id": "custom.fillOpacity", + "value": 0 + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 37 + }, + "id": 149, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(process_virtual_memory_bytes{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "Processes virtual memory size in bytes", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "process_resident_memory_max_bytes{instance=\"$node\",job=\"$job\"}", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "Maximum amount of virtual memory available in bytes", + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(process_virtual_memory_bytes{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "Processes virtual memory size in bytes", + "refId": "C", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(process_virtual_memory_max_bytes{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "Maximum amount of virtual memory available in bytes", + "refId": "D", + "step": 240 + } + ], + "title": "Processes Memory", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "PIDs limit" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F2495C", + "mode": "fixed" + } + }, + { + "id": "custom.fillOpacity", + "value": 0 + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 47 + }, + "id": 313, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_processes_pids{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "Number of PIDs", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_processes_max_processes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "PIDs limit", + "refId": "B", + "step": 240 + } + ], + "title": "PIDs Number and Limit", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "seconds", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*waiting.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 47 + }, + "id": 305, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_schedstat_running_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "CPU {{ cpu }} - seconds spent running a process", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_schedstat_waiting_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "CPU {{ cpu }} - seconds spent by processing waiting for this CPU", + "refId": "B", + "step": 240 + } + ], + "title": "Process schedule stats Running / Waiting", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Threads limit" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F2495C", + "mode": "fixed" + } + }, + { + "id": "custom.fillOpacity", + "value": 0 + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 57 + }, + "id": 314, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_processes_threads{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "Allocated threads", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_processes_max_threads{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "Threads limit", + "refId": "B", + "step": 240 + } + ], + "title": "Threads Number and Limit", + "type": "timeseries" + } + ], + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "refId": "A" + } + ], + "title": "System Processes", + "type": "row" + }, + { + "collapsed": true, + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 25 + }, + "id": 269, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 42 + }, + "id": 8, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_context_switches_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Context switches", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_intr_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "Interrupts", + "refId": "B", + "step": 240 + } + ], + "title": "Context Switches / Interrupts", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 42 + }, + "id": 7, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_load1{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 4, + "legendFormat": "Load 1m", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_load5{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 4, + "legendFormat": "Load 5m", + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_load15{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 4, + "legendFormat": "Load 15m", + "refId": "C", + "step": 240 + } + ], + "title": "System Load", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Critical*./" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E24D42", + "mode": "fixed" + } + }, + { + "id": "custom.fillOpacity", + "value": 0 + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*Max*./" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EF843C", + "mode": "fixed" + } + }, + { + "id": "custom.fillOpacity", + "value": 0 + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 52 + }, + "id": 259, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_interrupts_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "{{ type }} - {{ info }}", + "refId": "A", + "step": 240 + } + ], + "title": "Interrupts Detail", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 52 + }, + "id": 306, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_schedstat_timeslices_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "CPU {{ cpu }}", + "refId": "A", + "step": 240 + } + ], + "title": "Schedule timeslices executed by each cpu", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 62 + }, + "id": 151, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_entropy_available_bits{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Entropy available to random number generators", + "refId": "A", + "step": 240 + } + ], + "title": "Entropy", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "seconds", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 62 + }, + "id": 308, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(process_cpu_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "Time spent", + "refId": "A", + "step": 240 + } + ], + "title": "CPU time spent in user and system contexts", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Max*./" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#890F02", + "mode": "fixed" + } + }, + { + "id": "custom.fillOpacity", + "value": 0 + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 72 + }, + "id": 64, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "process_max_fds{instance=\"$node\",job=\"$job\"}", + "interval": "", + "intervalFactor": 1, + "legendFormat": "Maximum open file descriptors", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "process_open_fds{instance=\"$node\",job=\"$job\"}", + "interval": "", + "intervalFactor": 1, + "legendFormat": "Open file descriptors", + "refId": "B", + "step": 240 + } + ], + "title": "File Descriptors", + "type": "timeseries" + } + ], + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "refId": "A" + } + ], + "title": "System Misc", + "type": "row" + }, + { + "collapsed": true, + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 26 + }, + "id": 304, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "temperature", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "celsius" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Critical*./" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E24D42", + "mode": "fixed" + } + }, + { + "id": "custom.fillOpacity", + "value": 0 + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*Max*./" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EF843C", + "mode": "fixed" + } + }, + { + "id": "custom.fillOpacity", + "value": 0 + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 43 + }, + "id": 158, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_hwmon_temp_celsius{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "{{ chip }} {{ sensor }} temp", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_hwmon_temp_crit_alarm_celsius{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": true, + "interval": "", + "intervalFactor": 1, + "legendFormat": "{{ chip }} {{ sensor }} Critical Alarm", + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_hwmon_temp_crit_celsius{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "{{ chip }} {{ sensor }} Critical", + "refId": "C", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_hwmon_temp_crit_hyst_celsius{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": true, + "interval": "", + "intervalFactor": 1, + "legendFormat": "{{ chip }} {{ sensor }} Critical Historical", + "refId": "D", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_hwmon_temp_max_celsius{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": true, + "interval": "", + "intervalFactor": 1, + "legendFormat": "{{ chip }} {{ sensor }} Max", + "refId": "E", + "step": 240 + } + ], + "title": "Hardware temperature monitor", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Max*./" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EF843C", + "mode": "fixed" + } + }, + { + "id": "custom.fillOpacity", + "value": 0 + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 43 + }, + "id": 300, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_cooling_device_cur_state{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "Current {{ name }} in {{ type }}", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_cooling_device_max_state{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "Max {{ name }} in {{ type }}", + "refId": "B", + "step": 240 + } + ], + "title": "Throttle cooling device", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 53 + }, + "id": 302, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_power_supply_online{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "{{ power_supply }} online", + "refId": "A", + "step": 240 + } + ], + "title": "Power supply", + "type": "timeseries" + } + ], + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "refId": "A" + } + ], + "title": "Hardware Misc", + "type": "row" + }, + { + "collapsed": true, + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 27 + }, + "id": 296, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 30 + }, + "id": 297, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_systemd_socket_accepted_connections_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "{{ name }} Connections", + "refId": "A", + "step": 240 + } + ], + "title": "Systemd Sockets", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "normal" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Failed" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F2495C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Inactive" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#FF9830", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Active" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#73BF69", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Deactivating" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#FFCB7D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Activating" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#C8F2C2", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 30 + }, + "id": 298, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_systemd_units{instance=\"$node\",job=\"$job\",state=\"activating\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "Activating", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_systemd_units{instance=\"$node\",job=\"$job\",state=\"active\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "Active", + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_systemd_units{instance=\"$node\",job=\"$job\",state=\"deactivating\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "Deactivating", + "refId": "C", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_systemd_units{instance=\"$node\",job=\"$job\",state=\"failed\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "Failed", + "refId": "D", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_systemd_units{instance=\"$node\",job=\"$job\",state=\"inactive\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "Inactive", + "refId": "E", + "step": 240 + } + ], + "title": "Systemd Units State", + "type": "timeseries" + } + ], + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "refId": "A" + } + ], + "title": "Systemd", + "type": "row" + }, + { + "collapsed": true, + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 28 + }, + "id": 270, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "The number (after merges) of I/O requests completed per second for the device", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "IO read (-) / write (+)", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "iops" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Read.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#7EB26D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6ED0E0", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EF843C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E24D42", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#584477", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda2_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BA43A9", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda3_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F4D598", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0752D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#962D82", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#614D93", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#9AC48A", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#65C5DB", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F9934E", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0F9D7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#FCEACA", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F9E2D2", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 31 + }, + "id": 9, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_disk_reads_completed_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "intervalFactor": 4, + "legendFormat": "{{device}} - Reads completed", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_disk_writes_completed_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "intervalFactor": 1, + "legendFormat": "{{device}} - Writes completed", + "refId": "B", + "step": 240 + } + ], + "title": "Disk IOps Completed", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "The number of bytes read from or written to the device per second", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bytes read (-) / write (+)", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "Bps" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Read.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#7EB26D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6ED0E0", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EF843C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E24D42", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#584477", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda2_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BA43A9", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda3_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F4D598", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0752D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#962D82", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#614D93", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#9AC48A", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#65C5DB", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F9934E", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0F9D7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#FCEACA", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F9E2D2", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 31 + }, + "id": 33, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_disk_read_bytes_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "intervalFactor": 4, + "legendFormat": "{{device}} - Read bytes", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_disk_written_bytes_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "{{device}} - Written bytes", + "refId": "B", + "step": 240 + } + ], + "title": "Disk R/W Data", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "The average time for requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "time. read (-) / write (+)", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 30, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Read.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#7EB26D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6ED0E0", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EF843C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E24D42", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#584477", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda2_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BA43A9", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda3_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F4D598", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0752D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#962D82", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#614D93", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#9AC48A", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#65C5DB", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F9934E", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0F9D7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#FCEACA", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F9E2D2", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 41 + }, + "id": 37, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_disk_read_time_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval]) / irate(node_disk_reads_completed_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "hide": false, + "interval": "", + "intervalFactor": 4, + "legendFormat": "{{device}} - Read wait time avg", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_disk_write_time_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval]) / irate(node_disk_writes_completed_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "{{device}} - Write wait time avg", + "refId": "B", + "step": 240 + } + ], + "title": "Disk Average Wait Time", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "The average queue length of the requests that were issued to the device", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "aqu-sz", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "none" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#7EB26D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6ED0E0", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EF843C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E24D42", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#584477", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda2_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BA43A9", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda3_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F4D598", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0752D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#962D82", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#614D93", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#9AC48A", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#65C5DB", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F9934E", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0F9D7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#FCEACA", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F9E2D2", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 41 + }, + "id": 35, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_disk_io_time_weighted_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "interval": "", + "intervalFactor": 4, + "legendFormat": "{{device}}", + "refId": "A", + "step": 240 + } + ], + "title": "Average Queue Size", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "The number of read and write requests merged per second that were queued to the device", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "I/Os", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "iops" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Read.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#7EB26D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6ED0E0", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EF843C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E24D42", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#584477", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda2_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BA43A9", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda3_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F4D598", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0752D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#962D82", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#614D93", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#9AC48A", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#65C5DB", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F9934E", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0F9D7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#FCEACA", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F9E2D2", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 51 + }, + "id": 133, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_disk_reads_merged_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "intervalFactor": 1, + "legendFormat": "{{device}} - Read merged", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_disk_writes_merged_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "intervalFactor": 1, + "legendFormat": "{{device}} - Write merged", + "refId": "B", + "step": 240 + } + ], + "title": "Disk R/W Merged", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Percentage of elapsed time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100% for devices serving requests serially. But for devices serving requests in parallel, such as RAID arrays and modern SSDs, this number does not reflect their performance limits.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "%util", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 30, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#7EB26D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6ED0E0", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EF843C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E24D42", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#584477", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda2_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BA43A9", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda3_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F4D598", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0752D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#962D82", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#614D93", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#9AC48A", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#65C5DB", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F9934E", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0F9D7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#FCEACA", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F9E2D2", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 51 + }, + "id": 36, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_disk_io_time_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "interval": "", + "intervalFactor": 4, + "legendFormat": "{{device}} - IO", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_disk_discard_time_seconds_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "interval": "", + "intervalFactor": 4, + "legendFormat": "{{device}} - discard", + "refId": "B", + "step": 240 + } + ], + "title": "Time Spent Doing I/Os", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "The number of outstanding requests at the instant the sample was taken. Incremented as requests are given to appropriate struct request_queue and decremented as they finish.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Outstanding req.", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "none" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#7EB26D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6ED0E0", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EF843C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E24D42", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#584477", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda2_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BA43A9", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda3_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F4D598", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0752D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#962D82", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#614D93", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#9AC48A", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#65C5DB", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F9934E", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0F9D7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#FCEACA", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F9E2D2", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 61 + }, + "id": 34, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_disk_io_now{instance=\"$node\",job=\"$job\"}", + "interval": "", + "intervalFactor": 4, + "legendFormat": "{{device}} - IO now", + "refId": "A", + "step": 240 + } + ], + "title": "Instantaneous Queue Size", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "IOs", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "iops" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#7EB26D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EAB839", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#6ED0E0", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EF843C", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E24D42", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#584477", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda2_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BA43A9", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sda3_.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F4D598", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#0A50A1", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#BF1B00", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdb3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0752D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#962D82", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#614D93", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdc3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#9AC48A", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#65C5DB", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F9934E", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#EA6460", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde1.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E0F9D7", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sdd2.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#FCEACA", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*sde3.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F9E2D2", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 61 + }, + "id": 301, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_disk_discards_completed_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "interval": "", + "intervalFactor": 4, + "legendFormat": "{{device}} - Discards completed", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_disk_discards_merged_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "interval": "", + "intervalFactor": 1, + "legendFormat": "{{device}} - Discards merged", + "refId": "B", + "step": 240 + } + ], + "title": "Disk IOps Discards completed / merged", + "type": "timeseries" + } + ], + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "refId": "A" + } + ], + "title": "Storage Disk", + "type": "row" + }, + { + "collapsed": true, + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 29 + }, + "id": 271, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bytes", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 46 + }, + "id": 43, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_filesystem_avail_bytes{instance=\"$node\",job=\"$job\",device!~'rootfs'}", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "{{mountpoint}} - Available", + "metric": "", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_filesystem_free_bytes{instance=\"$node\",job=\"$job\",device!~'rootfs'}", + "format": "time_series", + "hide": true, + "intervalFactor": 1, + "legendFormat": "{{mountpoint}} - Free", + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_filesystem_size_bytes{instance=\"$node\",job=\"$job\",device!~'rootfs'}", + "format": "time_series", + "hide": true, + "intervalFactor": 1, + "legendFormat": "{{mountpoint}} - Size", + "refId": "C", + "step": 240 + } + ], + "title": "Filesystem space available", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "file nodes", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 46 + }, + "id": 41, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_filesystem_files_free{instance=\"$node\",job=\"$job\",device!~'rootfs'}", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "{{mountpoint}} - Free file nodes", + "refId": "A", + "step": 240 + } + ], + "title": "File Nodes Free", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "files", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 56 + }, + "id": 28, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_filefd_maximum{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 4, + "legendFormat": "Max open files", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_filefd_allocated{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "Open files", + "refId": "B", + "step": 240 + } + ], + "title": "File Descriptor", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "file Nodes", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 56 + }, + "id": 219, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_filesystem_files{instance=\"$node\",job=\"$job\",device!~'rootfs'}", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "{{mountpoint}} - File nodes total", + "refId": "A", + "step": 240 + } + ], + "title": "File Nodes Size", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "normal" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "max": 1, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "/ ReadOnly" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#890F02", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 66 + }, + "id": 44, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_filesystem_readonly{instance=\"$node\",job=\"$job\",device!~'rootfs'}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "{{mountpoint}} - ReadOnly", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_filesystem_device_error{instance=\"$node\",job=\"$job\",device!~'rootfs',fstype!~'tmpfs'}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "{{mountpoint}} - Device error", + "refId": "B", + "step": 240 + } + ], + "title": "Filesystem in ReadOnly / Error", + "type": "timeseries" + } + ], + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "refId": "A" + } + ], + "title": "Storage Filesystem", + "type": "row" + }, + { + "collapsed": true, + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 30 + }, + "id": 272, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "packets out (-) / in (+)", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "pps" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "receive_packets_eth0" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#7EB26D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "receive_packets_lo" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E24D42", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "transmit_packets_eth0" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#7EB26D", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "transmit_packets_lo" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#E24D42", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*Trans.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 31 + }, + "id": 60, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 300 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_network_receive_packets_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "{{device}} - Receive", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_network_transmit_packets_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "{{device}} - Transmit", + "refId": "B", + "step": 240 + } + ], + "title": "Network Traffic by Packets", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "packets out (-) / in (+)", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "pps" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Trans.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 31 + }, + "id": 142, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 300 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_network_receive_errs_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "{{device}} - Receive errors", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_network_transmit_errs_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "{{device}} - Rransmit errors", + "refId": "B", + "step": 240 + } + ], + "title": "Network Traffic Errors", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "packets out (-) / in (+)", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "pps" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Trans.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 41 + }, + "id": 143, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 300 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_network_receive_drop_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "{{device}} - Receive drop", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_network_transmit_drop_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "{{device}} - Transmit drop", + "refId": "B", + "step": 240 + } + ], + "title": "Network Traffic Drop", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "packets out (-) / in (+)", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "pps" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Trans.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 41 + }, + "id": 141, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 300 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_network_receive_compressed_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "{{device}} - Receive compressed", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_network_transmit_compressed_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "{{device}} - Transmit compressed", + "refId": "B", + "step": 240 + } + ], + "title": "Network Traffic Compressed", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "packets out (-) / in (+)", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "pps" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Trans.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 51 + }, + "id": 146, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 300 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_network_receive_multicast_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "{{device}} - Receive multicast", + "refId": "A", + "step": 240 + } + ], + "title": "Network Traffic Multicast", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "packets out (-) / in (+)", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "pps" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Trans.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 51 + }, + "id": 144, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 300 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_network_receive_fifo_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "{{device}} - Receive fifo", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_network_transmit_fifo_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "{{device}} - Transmit fifo", + "refId": "B", + "step": 240 + } + ], + "title": "Network Traffic Fifo", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "packets out (-) / in (+)", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "pps" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Trans.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 61 + }, + "id": 145, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 300 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_network_receive_frame_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "hide": false, + "intervalFactor": 1, + "legendFormat": "{{device}} - Receive frame", + "refId": "A", + "step": 240 + } + ], + "title": "Network Traffic Frame", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 61 + }, + "id": 231, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 300 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_network_transmit_carrier_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "{{device}} - Statistic transmit_carrier", + "refId": "A", + "step": 240 + } + ], + "title": "Network Traffic Carrier", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Trans.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 71 + }, + "id": 232, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 300 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_network_transmit_colls_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "{{device}} - Transmit colls", + "refId": "A", + "step": 240 + } + ], + "title": "Network Traffic Colls", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "entries", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "NF conntrack limit" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#890F02", + "mode": "fixed" + } + }, + { + "id": "custom.fillOpacity", + "value": 0 + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 71 + }, + "id": 61, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_nf_conntrack_entries{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "NF conntrack entries", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_nf_conntrack_entries_limit{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "NF conntrack limit", + "refId": "B", + "step": 240 + } + ], + "title": "NF Contrack", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Entries", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 81 + }, + "id": 230, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_arp_entries{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "{{ device }} - ARP entries", + "refId": "A", + "step": 240 + } + ], + "title": "ARP Entries", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bytes", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 0, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 81 + }, + "id": 288, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_network_mtu_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "{{ device }} - Bytes", + "refId": "A", + "step": 240 + } + ], + "title": "MTU", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bytes", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 0, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 91 + }, + "id": 280, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_network_speed_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "{{ device }} - Speed", + "refId": "A", + "step": 240 + } + ], + "title": "Speed", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "packets", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "decimals": 0, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 91 + }, + "id": 289, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_network_transmit_queue_length{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "{{ device }} - Interface transmit queue length", + "refId": "A", + "step": 240 + } + ], + "title": "Queue Length", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "packetes drop (-) / process (+)", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Dropped.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 101 + }, + "id": 290, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 300 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_softnet_processed_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "CPU {{cpu}} - Processed", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_softnet_dropped_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "CPU {{cpu}} - Dropped", + "refId": "B", + "step": 240 + } + ], + "title": "Softnet Packets", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 101 + }, + "id": 310, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 300 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_softnet_times_squeezed_total{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "CPU {{cpu}} - Squeezed", + "refId": "A", + "step": 240 + } + ], + "title": "Softnet Out of Quota", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 111 + }, + "id": 309, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 300 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_network_up{operstate=\"up\",instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "{{interface}} - Operational state UP", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_network_carrier{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "instant": false, + "legendFormat": "{{device}} - Physical link state", + "refId": "B" + } + ], + "title": "Network Operational Status", + "type": "timeseries" + } + ], + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "refId": "A" + } + ], + "title": "Network Traffic", + "type": "row" + }, + { + "collapsed": true, + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 31 + }, + "id": 273, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 32 + }, + "id": 63, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 300 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_sockstat_TCP_alloc{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "TCP_alloc - Allocated sockets", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_sockstat_TCP_inuse{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "TCP_inuse - Tcp sockets currently in use", + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_sockstat_TCP_mem{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": true, + "interval": "", + "intervalFactor": 1, + "legendFormat": "TCP_mem - Used memory for tcp", + "refId": "C", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_sockstat_TCP_orphan{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "TCP_orphan - Orphan sockets", + "refId": "D", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_sockstat_TCP_tw{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "TCP_tw - Sockets waiting close", + "refId": "E", + "step": 240 + } + ], + "title": "Sockstat TCP", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 32 + }, + "id": 124, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 300 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_sockstat_UDPLITE_inuse{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "UDPLITE_inuse - Udplite sockets currently in use", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_sockstat_UDP_inuse{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "UDP_inuse - Udp sockets currently in use", + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_sockstat_UDP_mem{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "UDP_mem - Used memory for udp", + "refId": "C", + "step": 240 + } + ], + "title": "Sockstat UDP", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 42 + }, + "id": 125, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 300 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_sockstat_FRAG_inuse{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "FRAG_inuse - Frag sockets currently in use", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_sockstat_RAW_inuse{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "RAW_inuse - Raw sockets currently in use", + "refId": "C", + "step": 240 + } + ], + "title": "Sockstat FRAG / RAW", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "bytes", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 42 + }, + "id": 220, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 300 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_sockstat_TCP_mem_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "mem_bytes - TCP sockets in that state", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_sockstat_UDP_mem_bytes{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "mem_bytes - UDP sockets in that state", + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_sockstat_FRAG_memory{instance=\"$node\",job=\"$job\"}", + "interval": "", + "intervalFactor": 1, + "legendFormat": "FRAG_memory - Used memory for frag", + "refId": "C" + } + ], + "title": "Sockstat Memory Size", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "sockets", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 52 + }, + "id": 126, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 300 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_sockstat_sockets_used{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "Sockets_used - Sockets currently in use", + "refId": "A", + "step": 240 + } + ], + "title": "Sockstat Used", + "type": "timeseries" + } + ], + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "refId": "A" + } + ], + "title": "Network Sockstat", + "type": "row" + }, + { + "collapsed": true, + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 32 + }, + "id": 274, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "octets out (-) / in (+)", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Out.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 33 + }, + "id": 221, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 300 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_IpExt_InOctets{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "InOctets - Received octets", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_IpExt_OutOctets{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "intervalFactor": 1, + "legendFormat": "OutOctets - Sent octets", + "refId": "B", + "step": 240 + } + ], + "title": "Netstat IP In / Out Octets", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "datagrams", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 33 + }, + "id": 81, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "width": 300 + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_Ip_Forwarding{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "Forwarding - IP forwarding", + "refId": "A", + "step": 240 + } + ], + "title": "Netstat IP Forwarding", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "messages out (-) / in (+)", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Out.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 43 + }, + "id": 115, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_Icmp_InMsgs{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "InMsgs - Messages which the entity received. Note that this counter includes all those counted by icmpInErrors", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_Icmp_OutMsgs{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "OutMsgs - Messages which this entity attempted to send. Note that this counter includes all those counted by icmpOutErrors", + "refId": "B", + "step": 240 + } + ], + "title": "ICMP In / Out", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "messages out (-) / in (+)", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Out.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 43 + }, + "id": 50, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_Icmp_InErrors{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "InErrors - Messages which the entity received but determined as having ICMP-specific errors (bad ICMP checksums, bad length, etc.)", + "refId": "A", + "step": 240 + } + ], + "title": "ICMP Errors", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "datagrams out (-) / in (+)", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Out.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*Snd.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 53 + }, + "id": 55, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_Udp_InDatagrams{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "InDatagrams - Datagrams received", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_Udp_OutDatagrams{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "OutDatagrams - Datagrams sent", + "refId": "B", + "step": 240 + } + ], + "title": "UDP In / Out", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "datagrams", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 53 + }, + "id": 109, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_Udp_InErrors{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "InErrors - UDP Datagrams that could not be delivered to an application", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_Udp_NoPorts{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "NoPorts - UDP Datagrams received on a port with no listener", + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_UdpLite_InErrors{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "interval": "", + "legendFormat": "InErrors Lite - UDPLite Datagrams that could not be delivered to an application", + "refId": "C" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_Udp_RcvbufErrors{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "RcvbufErrors - UDP buffer errors received", + "refId": "D", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_Udp_SndbufErrors{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "SndbufErrors - UDP buffer errors send", + "refId": "E", + "step": 240 + } + ], + "title": "UDP Errors", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "datagrams out (-) / in (+)", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Out.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/.*Snd.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 63 + }, + "id": 299, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_Tcp_InSegs{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "instant": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "InSegs - Segments received, including those received in error. This count includes segments received on currently established connections", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_Tcp_OutSegs{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "OutSegs - Segments sent, including those on current connections but excluding those containing only retransmitted octets", + "refId": "B", + "step": 240 + } + ], + "title": "TCP In / Out", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 63 + }, + "id": 104, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_TcpExt_ListenOverflows{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "ListenOverflows - Times the listen queue of a socket overflowed", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_TcpExt_ListenDrops{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "ListenDrops - SYNs to LISTEN sockets ignored", + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_TcpExt_TCPSynRetrans{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "TCPSynRetrans - SYN-SYN/ACK retransmits to break down retransmissions in SYN, fast/timeout retransmits", + "refId": "C", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_Tcp_RetransSegs{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "interval": "", + "legendFormat": "RetransSegs - Segments retransmitted - that is, the number of TCP segments transmitted containing one or more previously transmitted octets", + "refId": "D" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_Tcp_InErrs{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "interval": "", + "legendFormat": "InErrs - Segments received in error (e.g., bad TCP checksums)", + "refId": "E" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_Tcp_OutRsts{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "interval": "", + "legendFormat": "OutRsts - Segments sent with RST flag", + "refId": "F" + } + ], + "title": "TCP Errors", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "connections", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*MaxConn *./" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#890F02", + "mode": "fixed" + } + }, + { + "id": "custom.fillOpacity", + "value": 0 + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 73 + }, + "id": 85, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_netstat_Tcp_CurrEstab{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "CurrEstab - TCP connections for which the current state is either ESTABLISHED or CLOSE- WAIT", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_netstat_Tcp_MaxConn{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "MaxConn - Limit on the total number of TCP connections the entity can support (Dynamic is \"-1\")", + "refId": "B", + "step": 240 + } + ], + "title": "TCP Connections", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter out (-) / in (+)", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*Sent.*/" + }, + "properties": [ + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 73 + }, + "id": 91, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_TcpExt_SyncookiesFailed{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "SyncookiesFailed - Invalid SYN cookies received", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_TcpExt_SyncookiesRecv{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "SyncookiesRecv - SYN cookies received", + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_TcpExt_SyncookiesSent{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "SyncookiesSent - SYN cookies sent", + "refId": "C", + "step": 240 + } + ], + "title": "TCP SynCookie", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "connections", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 83 + }, + "id": 82, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_Tcp_ActiveOpens{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "ActiveOpens - TCP connections that have made a direct transition to the SYN-SENT state from the CLOSED state", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "irate(node_netstat_Tcp_PassiveOpens{instance=\"$node\",job=\"$job\"}[$__rate_interval])", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "PassiveOpens - TCP connections that have made a direct transition to the SYN-RCVD state from the LISTEN state", + "refId": "B", + "step": 240 + } + ], + "title": "TCP Direct Transition", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Enable with --collector.tcpstat argument on node-exporter", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "connections", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 83 + }, + "id": 320, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "node_tcp_connection_states{state=\"established\",instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "interval": "", + "intervalFactor": 1, + "legendFormat": "established - TCP sockets in established state", + "range": true, + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "node_tcp_connection_states{state=\"fin_wait2\",instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "fin_wait2 - TCP sockets in fin_wait2 state", + "range": true, + "refId": "B", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "node_tcp_connection_states{state=\"listen\",instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "listen - TCP sockets in listen state", + "range": true, + "refId": "C", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "expr": "node_tcp_connection_states{state=\"time_wait\",instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "time_wait - TCP sockets in time_wait state", + "range": true, + "refId": "D", + "step": 240 + } + ], + "title": "TCP Stat", + "type": "timeseries" + } + ], + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "refId": "A" + } + ], + "title": "Network Netstat", + "type": "row" + }, + { + "collapsed": true, + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 33 + }, + "id": 279, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "seconds", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "normal" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 0, + "y": 50 + }, + "id": 40, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_scrape_collector_duration_seconds{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "{{collector}} - Scrape duration", + "refId": "A", + "step": 240 + } + ], + "title": "Node Exporter Scrape Time", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "counter", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineStyle": { + "fill": "solid" + }, + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "links": [], + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green" + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/.*error.*/" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F2495C", + "mode": "fixed" + } + }, + { + "id": "custom.transform", + "value": "negative-Y" + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 12, + "x": 12, + "y": 50 + }, + "id": 157, + "links": [], + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull", + "max", + "min" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "9.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_scrape_collector_success{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "{{collector}} - Scrape success", + "refId": "A", + "step": 240 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "node_textfile_scrape_error{instance=\"$node\",job=\"$job\"}", + "format": "time_series", + "hide": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "{{collector}} - Scrape textfile error (1 = true)", + "refId": "B", + "step": 240 + } + ], + "title": "Node Exporter Scrape", + "type": "timeseries" + } + ], + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "000000001" + }, + "refId": "A" + } + ], + "title": "Node Exporter", + "type": "row" + } + ], + "refresh": "", + "revision": 1, + "schemaVersion": 38, + "style": "dark", + "tags": [ + "linux" + ], + "templating": { + "list": [ + { + "current": { + "selected": false, + "text": "default", + "value": "default" + }, + "hide": 0, + "includeAll": false, + "label": "datasource", + "multi": false, + "name": "DS_PROMETHEUS", + "options": [], + "query": "prometheus", + "refresh": 1, + "regex": "", + "skipUrlSync": false, + "type": "datasource" + }, + { + "current": {}, + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "definition": "", + "hide": 0, + "includeAll": false, + "label": "Job", + "multi": false, + "name": "job", + "options": [], + "query": { + "query": "label_values(node_uname_info, job)", + "refId": "Prometheus-job-Variable-Query" + }, + "refresh": 1, + "regex": "", + "skipUrlSync": false, + "sort": 1, + "tagValuesQuery": "", + "tagsQuery": "", + "type": "query", + "useTags": false + }, + { + "current": {}, + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "definition": "label_values(node_uname_info{job=\"$job\"}, instance)", + "hide": 0, + "includeAll": false, + "label": "Host", + "multi": false, + "name": "node", + "options": [], + "query": { + "query": "label_values(node_uname_info{job=\"$job\"}, instance)", + "refId": "Prometheus-node-Variable-Query" + }, + "refresh": 1, + "regex": "", + "skipUrlSync": false, + "sort": 1, + "tagValuesQuery": "", + "tagsQuery": "", + "type": "query", + "useTags": false + }, + { + "current": { + "selected": false, + "text": "[a-z]+|nvme[0-9]+n[0-9]+|mmcblk[0-9]+", + "value": "[a-z]+|nvme[0-9]+n[0-9]+|mmcblk[0-9]+" + }, + "hide": 2, + "includeAll": false, + "multi": false, + "name": "diskdevices", + "options": [ + { + "selected": true, + "text": "[a-z]+|nvme[0-9]+n[0-9]+|mmcblk[0-9]+", + "value": "[a-z]+|nvme[0-9]+n[0-9]+|mmcblk[0-9]+" + } + ], + "query": "[a-z]+|nvme[0-9]+n[0-9]+|mmcblk[0-9]+", + "skipUrlSync": false, + "type": "custom" + } + ] + }, + "time": { + "from": "now-24h", + "to": "now" + }, + "timepicker": { + "refresh_intervals": [ + "5s", + "10s", + "30s", + "1m", + "5m", + "15m", + "30m", + "1h", + "2h", + "1d" + ], + "time_options": [ + "5m", + "15m", + "1h", + "6h", + "12h", + "24h", + "2d", + "7d", + "30d" + ] + }, + "timezone": "browser", + "title": "Node Exporter Full", + "uid": "rYdddlPWk", + "version": 86, + "weekStart": "" + } \ No newline at end of file diff --git a/roles/prometheus_install/container_prometheus/files/dashboard_xpumanager.json b/roles/prometheus_install/container_prometheus/files/dashboard_xpumanager.json new file mode 100644 index 00000000..6661e167 --- /dev/null +++ b/roles/prometheus_install/container_prometheus/files/dashboard_xpumanager.json @@ -0,0 +1,2762 @@ +{ + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": "-- Grafana --", + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "target": { + "limit": 100, + "matchAny": false, + "tags": [], + "type": "dashboard" + }, + "type": "dashboard" + } + ] + }, + "editable": true, + "gnetId": null, + "graphTooltip": 0, + "id": 26, + "iteration": 1647584693304, + "links": [], + "panels": [ + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 0, + "y": 0 + }, + "id": 39, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "exemplar": true, + "expr": "xpum_engine_ratio{node=\"$Node\"}", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}", + "refId": "A" + } + ], + "title": "GPU Utilization", + "type": "timeseries" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 4, + "x": 8, + "y": 0 + }, + "id": 34, + "interval": "5s", + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "text": {} + }, + "pluginVersion": "8.1.0", + "targets": [ + { + "exemplar": true, + "expr": "xpum_engine_ratio{node=\"$Node\"}", + "instant": false, + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}", + "refId": "A" + } + ], + "title": "GPU Utilization", + "type": "gauge" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "rothz" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 12, + "y": 0 + }, + "id": 2, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "exemplar": true, + "expr": "xpum_frequency_mhz{node=\"$Node\"}*1000000", + "hide": false, + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}: {{type}}", + "refId": "A" + } + ], + "title": "GPU Frequency", + "type": "timeseries" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "rothz" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 4, + "x": 20, + "y": 0 + }, + "id": 35, + "interval": "5s", + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "text": {} + }, + "pluginVersion": "8.1.0", + "targets": [ + { + "exemplar": true, + "expr": "xpum_frequency_mhz{node=\"$Node\"}*1000000", + "hide": false, + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}: {{type}}", + "refId": "A" + } + ], + "title": "GPU Frequency", + "type": "gauge" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "celsius" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 0, + "y": 7 + }, + "id": 8, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "exemplar": true, + "expr": "xpum_temperature_celsius{node=\"$Node\", location=\"gpu\"}", + "interval": "", + "legendFormat": "avg: {{pci_bdf}}/{{sub_dev}}", + "refId": "A" + }, + { + "exemplar": true, + "expr": "xpum_max_temperature_celsius{node=\"$Node\", location=\"gpu\"}", + "hide": false, + "interval": "", + "legendFormat": "max: {{pci_bdf}}/{{sub_dev}}", + "refId": "B" + } + ], + "title": "GPU Temperature", + "type": "timeseries" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "celsius" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 4, + "x": 8, + "y": 7 + }, + "id": 36, + "interval": "5s", + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "text": {} + }, + "pluginVersion": "8.1.0", + "targets": [ + { + "exemplar": true, + "expr": "xpum_temperature_celsius{node=\"$Node\", location=\"gpu\"}", + "interval": "", + "legendFormat": "avg: {{pci_bdf}}/{{sub_dev}}", + "refId": "A" + }, + { + "exemplar": true, + "expr": "xpum_max_temperature_celsius{node=\"$Node\", location=\"gpu\"}", + "hide": false, + "interval": "", + "legendFormat": "max: {{pci_bdf}}/{{sub_dev}}", + "refId": "B" + } + ], + "title": "GPU Temperature", + "type": "gauge" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "celsius" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 12, + "y": 7 + }, + "id": 42, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "exemplar": true, + "expr": "xpum_temperature_celsius{node=\"$Node\", location=\"mem\"}", + "interval": "", + "legendFormat": "avg: {{pci_bdf}}/{{sub_dev}}", + "refId": "A" + }, + { + "exemplar": true, + "expr": "xpum_max_temperature_celsius{node=\"$Node\", location=\"mem\"}", + "hide": false, + "interval": "", + "legendFormat": "max: {{pci_bdf}}/{{sub_dev}}", + "refId": "B" + } + ], + "title": "Memory Temperature", + "type": "timeseries" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "celsius" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 4, + "x": 20, + "y": 7 + }, + "id": 43, + "interval": "5s", + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "text": {} + }, + "pluginVersion": "8.1.0", + "targets": [ + { + "exemplar": true, + "expr": "xpum_temperature_celsius{node=\"$Node\", location=\"mem\"}", + "interval": "", + "legendFormat": "avg: {{pci_bdf}}/{{sub_dev}}", + "refId": "A" + }, + { + "exemplar": true, + "expr": "xpum_max_temperature_celsius{node=\"$Node\", location=\"mem\"}", + "hide": false, + "interval": "", + "legendFormat": "max: {{pci_bdf}}/{{sub_dev}}", + "refId": "B" + } + ], + "title": "Memory Temperature", + "type": "gauge" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "watt" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 0, + "y": 14 + }, + "id": 16, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "exemplar": true, + "expr": "xpum_power_watts{node=\"$Node\"}", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}", + "refId": "A" + } + ], + "title": "Power", + "type": "timeseries" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "watt" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 4, + "x": 8, + "y": 14 + }, + "id": 37, + "interval": "5s", + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "text": {} + }, + "pluginVersion": "8.1.0", + "targets": [ + { + "exemplar": true, + "expr": "xpum_power_watts{node=\"$Node\"}", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}", + "refId": "A" + } + ], + "title": "Power", + "type": "gauge" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "joule" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 12, + "y": 14 + }, + "id": 4, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "exemplar": true, + "expr": "xpum_energy_joules_total{node=\"$Node\"}", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}", + "refId": "A" + } + ], + "title": "Energy", + "type": "timeseries" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "joule" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 4, + "x": 20, + "y": 14 + }, + "id": 38, + "interval": "5s", + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "text": {}, + "textMode": "auto" + }, + "pluginVersion": "8.1.0", + "targets": [ + { + "exemplar": true, + "expr": "xpum_energy_joules_total{node=\"$Node\"}", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}", + "refId": "A" + } + ], + "title": "Energy", + "type": "stat" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 6, + "x": 0, + "y": 21 + }, + "id": 12, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "exemplar": true, + "expr": "xpum_memory_used_bytes{node=\"$Node\"}", + "instant": false, + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}", + "refId": "A" + } + ], + "title": "Memory Used", + "type": "timeseries" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 6, + "x": 6, + "y": 21 + }, + "id": 10, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "exemplar": true, + "expr": "xpum_memory_read_bytes_total{node=\"$Node\"}", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}", + "refId": "A" + } + ], + "title": "Memory Read", + "type": "timeseries" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "binBps" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 3, + "x": 12, + "y": 21 + }, + "id": 40, + "interval": "5s", + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "text": {}, + "textMode": "auto" + }, + "pluginVersion": "8.1.0", + "targets": [ + { + "exemplar": true, + "expr": "rate(xpum_memory_read_bytes_total{node=\"$Node\"}[10s])", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}", + "refId": "A" + } + ], + "title": "Mem Read Throughput", + "type": "stat" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 6, + "x": 15, + "y": 21 + }, + "id": 14, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "exemplar": true, + "expr": "xpum_memory_write_bytes_total{node=\"$Node\"}", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}", + "refId": "A" + } + ], + "title": "Memory Write", + "type": "timeseries" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "binBps" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 3, + "x": 21, + "y": 21 + }, + "id": 41, + "interval": "5s", + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "text": {}, + "textMode": "auto" + }, + "pluginVersion": "8.1.0", + "targets": [ + { + "exemplar": true, + "expr": "rate(xpum_memory_write_bytes_total{node=\"$Node\"}[10s])", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}", + "refId": "A" + } + ], + "title": "Mem Write Throughput", + "type": "stat" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 6, + "x": 0, + "y": 28 + }, + "id": 33, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "exemplar": true, + "expr": "xpum_memory_ratio{node=\"$Node\"}", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}", + "refId": "A" + } + ], + "title": "Memory Utilization", + "type": "timeseries" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 6, + "x": 6, + "y": 28 + }, + "id": 32, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "exemplar": true, + "expr": "xpum_memory_bandwidth_ratio{node=\"$Node\"}", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}", + "refId": "A" + } + ], + "title": "Memory Bandwidth", + "type": "timeseries" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 6, + "x": 12, + "y": 28 + }, + "id": 29, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "exemplar": true, + "expr": "xpum_engine_group_ratio{node=\"$Node\"}", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}: {{type}}", + "refId": "A" + } + ], + "title": "Engine Group Utilization", + "type": "timeseries" + }, + { + "datasource": null, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "max": 1, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 6, + "x": 18, + "y": 28 + }, + "id": 44, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "exemplar": true, + "expr": "xpum_eu_active_ratio{node=\"$Node\"}", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}", + "refId": "A" + } + ], + "title": "EU Active", + "type": "timeseries" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 6, + "x": 0, + "y": 35 + }, + "id": 47, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "exemplar": true, + "expr": "xpum_pcie_read_bytes_total{node=\"$Node\"}", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}", + "refId": "A" + } + ], + "title": "PCIe Read", + "type": "timeseries" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "binBps" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 3, + "x": 6, + "y": 35 + }, + "id": 49, + "interval": "5s", + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "text": {}, + "textMode": "auto" + }, + "pluginVersion": "8.1.0", + "targets": [ + { + "exemplar": true, + "expr": "rate(xpum_pcie_read_bytes_total{node=\"$Node\"}[10s])", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}", + "refId": "A" + } + ], + "title": "PCIe Read Throughput", + "type": "stat" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 6, + "x": 9, + "y": 35 + }, + "id": 48, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "exemplar": true, + "expr": "xpum_pcie_write_bytes_total{node=\"$Node\"}", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}", + "refId": "A" + } + ], + "title": "PCIe Write", + "type": "timeseries" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "binBps" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 3, + "x": 15, + "y": 35 + }, + "id": 50, + "interval": "5s", + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "text": {}, + "textMode": "auto" + }, + "pluginVersion": "8.1.0", + "targets": [ + { + "exemplar": true, + "expr": "rate(xpum_pcie_write_bytes_total{node=\"$Node\"}[10s])", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}", + "refId": "A" + } + ], + "title": "PCIe Write Throughput", + "type": "stat" + }, + { + "datasource": null, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "max": 1, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 6, + "x": 18, + "y": 35 + }, + "id": 45, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "exemplar": true, + "expr": "xpum_eu_idle_ratio{node=\"$Node\"}", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}", + "refId": "A" + } + ], + "title": "EU Idle", + "type": "timeseries" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 6, + "x": 0, + "y": 42 + }, + "id": 23, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "exemplar": true, + "expr": "xpum_programming_errors_total{node=\"$Node\"}", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}", + "refId": "A" + } + ], + "title": "Programming Errors", + "type": "timeseries" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 6, + "x": 6, + "y": 42 + }, + "id": 18, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "pluginVersion": "8.1.0", + "targets": [ + { + "exemplar": true, + "expr": "xpum_cache_errors_total{node=\"$Node\"}", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}: {{type}}", + "refId": "A" + } + ], + "title": "Cache Errors", + "type": "timeseries" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 6, + "x": 12, + "y": 42 + }, + "id": 51, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "pluginVersion": "8.1.0", + "targets": [ + { + "exemplar": true, + "expr": "xpum_driver_errors_total{node=\"$Node\"}", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}: {{type}}", + "refId": "A" + } + ], + "title": "Driver Errors", + "type": "timeseries" + }, + { + "datasource": null, + "description": "", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "max": 1, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 6, + "x": 18, + "y": 42 + }, + "id": 46, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "exemplar": true, + "expr": "xpum_eu_stall_ratio{node=\"$Node\"}", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}", + "refId": "A" + } + ], + "title": "EU Stall", + "type": "timeseries" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 6, + "x": 0, + "y": 49 + }, + "id": 24, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "exemplar": true, + "expr": "xpum_resets_total{node=\"$Node\"}", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}", + "refId": "A" + } + ], + "title": "Reset Count", + "type": "timeseries" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 6, + "x": 6, + "y": 49 + }, + "id": 19, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "exemplar": true, + "expr": "xpum_non_compute_errors_total{node=\"$Node\"}", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}: {{type}}", + "refId": "A" + } + ], + "title": "Non Compute Errors", + "type": "timeseries" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 12, + "y": 49 + }, + "id": 52, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "exemplar": true, + "expr": "xpum_per_engine_ratio{node=\"$Node\"}", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}}: {{type}}-{{engine_id}}", + "refId": "A" + } + ], + "title": "Per Engine Utilization", + "type": "timeseries" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 0, + "y": 56 + }, + "id": 53, + "interval": "5s", + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "exemplar": true, + "expr": "xpum_fabric_tx_bytes_total{node=\"$Node\"}", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}} -> {{dst_pci_bdf}}/{{dst_sub_dev}}", + "refId": "A" + } + ], + "title": "Fabric Transmit Bytes", + "type": "timeseries" + }, + { + "datasource": null, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "binBps" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 12, + "y": 56 + }, + "id": 54, + "interval": "5s", + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "text": {}, + "textMode": "auto" + }, + "pluginVersion": "8.1.0", + "targets": [ + { + "exemplar": true, + "expr": "rate(xpum_fabric_tx_bytes_total{node=\"$Node\"}[10s])", + "interval": "", + "legendFormat": "{{pci_bdf}}/{{sub_dev}} -> {{dst_pci_bdf}}/{{dst_sub_dev}}", + "refId": "A" + } + ], + "title": "Fabric Transmit Throughput", + "type": "stat" + } + ], + "refresh": "5s", + "schemaVersion": 30, + "style": "dark", + "tags": [ + "xpum" + ], + "templating": { + "list": [ + { + "allValue": null, + "current": { + "selected": false, + "text": "dut1132-pvc-dua", + "value": "dut1132-pvc-dua" + }, + "datasource": null, + "definition": "label_values(xpum_power_watts, node)", + "description": null, + "error": null, + "hide": 0, + "includeAll": false, + "label": "Node", + "multi": false, + "name": "Node", + "options": [], + "query": { + "query": "label_values(xpum_power_watts, node)", + "refId": "StandardVariableQuery" + }, + "refresh": 1, + "regex": "", + "skipUrlSync": false, + "sort": 0, + "type": "query" + } + ] + }, + "time": { + "from": "now-5m", + "to": "now" + }, + "timepicker": {}, + "timezone": "", + "title": "Intel XPU Manager Exporter", + "uid": "9VxbZDPnk", + "version": 3 + } \ No newline at end of file diff --git a/roles/prometheus_install/container_prometheus/files/grafana.ini b/roles/prometheus_install/container_prometheus/files/grafana.ini new file mode 100644 index 00000000..94eb2247 --- /dev/null +++ b/roles/prometheus_install/container_prometheus/files/grafana.ini @@ -0,0 +1,1544 @@ +##################### Grafana Configuration Example ##################### +# +# Everything has defaults so you only need to uncomment things you want to +# change + +# possible values : production, development +;app_mode = production + +# instance name, defaults to HOSTNAME environment variable value or hostname if HOSTNAME var is empty +;instance_name = ${HOSTNAME} + +# force migration will run migrations that might cause dataloss +;force_migration = false + +#################################### Paths #################################### +[paths] +# Path to where grafana can store temp files, sessions, and the sqlite3 db (if that is used) +;data = /var/lib/grafana + +# Temporary files in `data` directory older than given duration will be removed +;temp_data_lifetime = 24h + +# Directory where grafana can store logs +;logs = /var/log/grafana + +# Directory where grafana will automatically scan and look for plugins +;plugins = /var/lib/grafana/plugins + +# folder that contains provisioning config files that grafana will apply on startup and while running. +provisioning = /etc/grafana/provisioning + +#################################### Server #################################### +[server] +# Protocol (http, https, h2, socket) +protocol = https + +# This is the minimum TLS version allowed. By default, this value is empty. Accepted values are: TLS1.2, TLS1.3. If nothing is set TLS1.2 would be taken +;min_tls_version = "" + +# The ip address to bind to, empty will bind to all interfaces +;http_addr = + +# The http port to use +;http_port = 3000 + +# The public facing domain name used to access grafana from a browser +;domain = localhost + +# Redirect to correct domain if host header does not match domain +# Prevents DNS rebinding attacks +;enforce_domain = false + +# The full public facing url you use in browser, used for redirects and emails +# If you use reverse proxy and sub path specify full url (with sub path) +;root_url = %(protocol)s://%(domain)s:%(http_port)s/ + +# Serve Grafana from subpath specified in `root_url` setting. By default it is set to `false` for compatibility reasons. +;serve_from_sub_path = false + +# Log web requests +;router_logging = false + +# the path relative working path +;static_root_path = public + +# enable gzip +;enable_gzip = false + +# https certs & key file +cert_file = /etc/grafana/cert/graf_cert.pem +cert_key = /etc/grafana/cert/graf_key.pem + +# Unix socket gid +# Changing the gid of a file without privileges requires that the target group is in the group of the process and that the process is the file owner +# It is recommended to set the gid as http server user gid +# Not set when the value is -1 +;socket_gid = + +# Unix socket mode +;socket_mode = + +# Unix socket path +;socket = + +# CDN Url +;cdn_url = + +# Sets the maximum time using a duration format (5s/5m/5ms) before timing out read of an incoming request and closing idle connections. +# `0` means there is no timeout for reading the request. +;read_timeout = 0 + +# This setting enables you to specify additional headers that the server adds to HTTP(S) responses. +[server.custom_response_headers] +#exampleHeader1 = exampleValue1 +#exampleHeader2 = exampleValue2 + +#################################### GRPC Server ######################### +;[grpc_server] +;network = "tcp" +;address = "127.0.0.1:10000" +;use_tls = false +;cert_file = +;key_file = + +#################################### Database #################################### +[database] +# You can configure the database connection by specifying type, host, name, user and password +# as separate properties or as on string using the url properties. + +# Either "mysql", "postgres" or "sqlite3", it's your choice +;type = sqlite3 +;host = 127.0.0.1:3306 +;name = grafana +;user = root +# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;""" +;password = + +# Use either URL or the previous fields to configure the database +# Example: mysql://user:secret@host:port/database +;url = + +# For "postgres", use either "disable", "require" or "verify-full" +# For "mysql", use either "true", "false", or "skip-verify". +;ssl_mode = disable + +# Database drivers may support different transaction isolation levels. +# Currently, only "mysql" driver supports isolation levels. +# If the value is empty - driver's default isolation level is applied. +# For "mysql" use "READ-UNCOMMITTED", "READ-COMMITTED", "REPEATABLE-READ" or "SERIALIZABLE". +;isolation_level = + +;ca_cert_path = +;client_key_path = +;client_cert_path = +;server_cert_name = + +# For "sqlite3" only, path relative to data_path setting +;path = grafana.db + +# Max idle conn setting default is 2 +;max_idle_conn = 2 + +# Max conn setting default is 0 (mean not set) +;max_open_conn = + +# Connection Max Lifetime default is 14400 (means 14400 seconds or 4 hours) +;conn_max_lifetime = 14400 + +# Set to true to log the sql calls and execution times. +;log_queries = + +# For "sqlite3" only. cache mode setting used for connecting to the database. (private, shared) +;cache_mode = private + +# For "sqlite3" only. Enable/disable Write-Ahead Logging, https://sqlite.org/wal.html. Default is false. +;wal = false + +# For "mysql" only if migrationLocking feature toggle is set. How many seconds to wait before failing to lock the database for the migrations, default is 0. +;locking_attempt_timeout_sec = 0 + +# For "sqlite" only. How many times to retry query in case of database is locked failures. Default is 0 (disabled). +;query_retries = 0 + +# For "sqlite" only. How many times to retry transaction in case of database is locked failures. Default is 5. +;transaction_retries = 5 + +# Set to true to add metrics and tracing for database queries. +;instrument_queries = false + +################################### Data sources ######################### +[datasources] +# Upper limit of data sources that Grafana will return. This limit is a temporary configuration and it will be deprecated when pagination will be introduced on the list data sources API. +;datasource_limit = 5000 + +#################################### Cache server ############################# +[remote_cache] +# Either "redis", "memcached" or "database" default is "database" +;type = database + +# cache connectionstring options +# database: will use Grafana primary database. +# redis: config like redis server e.g. `addr=127.0.0.1:6379,pool_size=100,db=0,ssl=false`. Only addr is required. ssl may be 'true', 'false', or 'insecure'. +# memcache: 127.0.0.1:11211 +;connstr = + +# prefix prepended to all the keys in the remote cache +; prefix = + +# This enables encryption of values stored in the remote cache +;encryption = + +#################################### Data proxy ########################### +[dataproxy] + +# This enables data proxy logging, default is false +;logging = false + +# How long the data proxy waits to read the headers of the response before timing out, default is 30 seconds. +# This setting also applies to core backend HTTP data sources where query requests use an HTTP client with timeout set. +;timeout = 30 + +# How long the data proxy waits to establish a TCP connection before timing out, default is 10 seconds. +;dialTimeout = 10 + +# How many seconds the data proxy waits before sending a keepalive probe request. +;keep_alive_seconds = 30 + +# How many seconds the data proxy waits for a successful TLS Handshake before timing out. +;tls_handshake_timeout_seconds = 10 + +# How many seconds the data proxy will wait for a server's first response headers after +# fully writing the request headers if the request has an "Expect: 100-continue" +# header. A value of 0 will result in the body being sent immediately, without +# waiting for the server to approve. +;expect_continue_timeout_seconds = 1 + +# Optionally limits the total number of connections per host, including connections in the dialing, +# active, and idle states. On limit violation, dials will block. +# A value of zero (0) means no limit. +;max_conns_per_host = 0 + +# The maximum number of idle connections that Grafana will keep alive. +;max_idle_connections = 100 + +# How many seconds the data proxy keeps an idle connection open before timing out. +;idle_conn_timeout_seconds = 90 + +# If enabled and user is not anonymous, data proxy will add X-Grafana-User header with username into the request, default is false. +;send_user_header = false + +# Limit the amount of bytes that will be read/accepted from responses of outgoing HTTP requests. +;response_limit = 0 + +# Limits the number of rows that Grafana will process from SQL data sources. +;row_limit = 1000000 + +# Sets a custom value for the `User-Agent` header for outgoing data proxy requests. If empty, the default value is `Grafana/` (for example `Grafana/9.0.0`). +;user_agent = + +#################################### Analytics #################################### +[analytics] +# Server reporting, sends usage counters to stats.grafana.org every 24 hours. +# No ip addresses are being tracked, only simple counters to track +# running instances, dashboard and error counts. It is very helpful to us. +# Change this option to false to disable reporting. +;reporting_enabled = true + +# The name of the distributor of the Grafana instance. Ex hosted-grafana, grafana-labs +;reporting_distributor = grafana-labs + +# Set to false to disable all checks to https://grafana.com +# for new versions of grafana. The check is used +# in some UI views to notify that a grafana update exists. +# This option does not cause any auto updates, nor send any information +# only a GET request to https://raw.githubusercontent.com/grafana/grafana/main/latest.json to get the latest version. +;check_for_updates = true + +# Set to false to disable all checks to https://grafana.com +# for new versions of plugins. The check is used +# in some UI views to notify that a plugin update exists. +# This option does not cause any auto updates, nor send any information +# only a GET request to https://grafana.com to get the latest versions. +;check_for_plugin_updates = true + +# Google Analytics universal tracking code, only enabled if you specify an id here +;google_analytics_ua_id = + +# Google Analytics 4 tracking code, only enabled if you specify an id here +;google_analytics_4_id = + +# When Google Analytics 4 Enhanced event measurement is enabled, we will try to avoid sending duplicate events and let Google Analytics 4 detect navigation changes, etc. +;google_analytics_4_send_manual_page_views = false + +# Google Tag Manager ID, only enabled if you specify an id here +;google_tag_manager_id = + +# Rudderstack write key, enabled only if rudderstack_data_plane_url is also set +;rudderstack_write_key = + +# Rudderstack data plane url, enabled only if rudderstack_write_key is also set +;rudderstack_data_plane_url = + +# Rudderstack SDK url, optional, only valid if rudderstack_write_key and rudderstack_data_plane_url is also set +;rudderstack_sdk_url = + +# Rudderstack Config url, optional, used by Rudderstack SDK to fetch source config +;rudderstack_config_url = + +# Intercom secret, optional, used to hash user_id before passing to Intercom via Rudderstack +;intercom_secret = + +# Controls if the UI contains any links to user feedback forms +;feedback_links_enabled = true + +#################################### Security #################################### +[security] +# disable creation of admin user on first start of grafana +;disable_initial_admin_creation = false + +# default admin user, created on startup +;admin_user = admin + +# default admin password, can be changed before first start of grafana, or in profile settings +;admin_password = admin + +# default admin email, created on startup +;admin_email = admin@localhost + +# used for signing +;secret_key = SW2YcwTIb9zpOOhoPsMm + +# current key provider used for envelope encryption, default to static value specified by secret_key +;encryption_provider = secretKey.v1 + +# list of configured key providers, space separated (Enterprise only): e.g., awskms.v1 azurekv.v1 +;available_encryption_providers = + +# disable gravatar profile images +;disable_gravatar = false + +# data source proxy whitelist (ip_or_domain:port separated by spaces) +;data_source_proxy_whitelist = + +# disable protection against brute force login attempts +;disable_brute_force_login_protection = false + +# set to true if you host Grafana behind HTTPS. default is false. +;cookie_secure = false + +# set cookie SameSite attribute. defaults to `lax`. can be set to "lax", "strict", "none" and "disabled" +;cookie_samesite = lax + +# set to true if you want to allow browsers to render Grafana in a ,

#u5-StH;3FqjA~>E3L0grs zQFM1GW^fdC*wDqFx)mxK!*zom0#by0I)gGDL;DX`Y!-IBZM-X?7{g~*CM4n8V~T^K zbCBm%&RQe}NIyA|^^TucboWiFr+~*zcm8*mw3NANifn3F@A$84C7tp00&s=h(P5=h zhA8wC2;2E2>(_igF)O|v&d0u1I5HpP(>%C^CzbMR*(kg}2Mld;g6G%XD(9_!h_Nw> z9sU-^(^YkgAK8pCX72fvvg_WU+>sFUs0}6a_w$=Y2>;&5KSU*)%(w^0z4}Kn{)not zW#`_-$Z>X?W{z7S+5ThbT~R&$NQR5anC8|~;&*!zW{%ywkH;;!1aS#DZwElh|*l57$ogW2_&TEFVqc(D+c~^A!GTqZ*QEfp*uPP z(!Di|xDFMoF1x(q6oY?wn?&pS1}Du6eaqICY50iU+|Egu3)x%43%5+!7_>uBoeD=@ zz5t%8lYD6{mO;RM%Hz(Xefw~1?&|oy7NO`d8R|=0EuKfndVMCrtHr05azP<_sdy*w zIvvXSSCoBpekX2rr>y&jh>R(V_G?xSPm?hCQ*s93yXgmt0L&j2&oCL@4q{=7GyS-= z!rQ-Z%V9kQ{=FQ+gJSG^vRkxKr(*c~^7u zI2kG+&(jSr6d2!x22X^>lbzmd==Sg1xUM#SZP#jK1NB1v3rcL4ol>C|X0X;{iR^>% zA61h;Qc#QA13HA_yVSJR5ZhgvD4uam3tTsZI=7nSZzLFWkr0MrGn;$ctFKoALN(Gf zFf(yqzPp0mL1;RL?`U9GTB!b-5uFz=B7-0$@NeQeee%;MX&j6#{*}z(PAU{5sv>!0Pw#~+&gfgs2J+#B`#&D z>OnFZq34JbH=yQ|+nOsl=d*8KCH)LP6SK-{Bs^fjY^dE=Ya+@Jhx+D}QlywAJkk?} zU2L=nYo8^D23K!UiciL50z!+kqxwficQalgAq zH?}n9oqT}-q>}9A5EV-xzfC4kFbU|NKG?U{+SHQuaHzB|kX%IR+&2IJbpdiG3R05M zeb54(n(1V_3p;!v4e<|<@5-L|8e8r@e=?X9h6wrSD$#}O+2HBd`CS>IlE$odL0fbYtM>*ZTqMj~chcv0&x=u`73$g9pfRDzbZM7`=`CrXNf$;8T zr^!myF!^$Y9`nzEX?C2|K_>OsQxhwf0026mE$NKm#4-Z(Sjm zDr#&%$LE*I#3vg4X%V`Gy3}|c^os+Zjps`uO}PX6S0suzaO~Jh#C+8fwXPao1^&ZU z!piJqhHHaR-y4cWXA?%|^L%~y#wZm}7IXN#nkX2kkr4gnwvWjK`s?m+jV?lI4^|-c^Y5 zQn_BWY8M19k(O%s2@*%Bs_RZBT;5!yI@_?6cg`QK^=|WLElTn4n&t6q9l^~)Ie!wm`uS`y55B-lL!`2$(smfk)`4vpdTSc4>(q1SvoI1`LJ2cW z!E~6`l&P{?@usABDb%mT_L4)+*&!MqP^&%&Zr7I3Owwjwj$26)!H#3i7bpJk$Bu8K z2d&iWucSQlJUaOu;gd^s4{R#e|23?BB7dDe5r-#cmsLo#X^TU0Wsi!6wynK!fqx6i zyM*d_+XD(v#Y7kLK1VCshKXC~n`E&t+|7*;QB<3LBFLEuY1JR|gYv{fC1U?l4R+*w^zM@mt;fE%m*0<9onbxucY=f*8EB zS!RH;pqZt6YP5$$#fYg?dYJYUrse}G-adS#B_WE<;V$>PUw`-_6XOq`SB*EtrL2vq zjt;TT;I*SJMHhC}uUmX$%i@>S9lmE1^_%DO6M)|fJvJk>H#(I5n`HgJHSPZkpZxQB z89x{e14TeQYa0QK_6thoy#=`Llf3^NFqV{=A#CIpIvbpo6k^q27#~SF8Gy}(2ljH} zNhopHgi1ME5jpI`ZYwV>WOyea%Y*9CLH-qhtIs01b&o!wV&GsuZzY)z->Jn4$9@2tg$?vUw+fQQ;X~X zeG1>_hz^U^dPPSJR*?{hr{IqEm5ui|5qvoQ#P*VjAwASAD zH)Dt0qO}{8hSPOI3kEcK^9DkH!l@F_%CY|x+tQLdaM!%)jn6iQBpOw7GlSTi@HpcJ`zKb9ASS0A4kHI55YkdlgN9Zbt~gE0WYXaJTl zpO=6&2C+lYzHLHI$SeW%i1oFZ!l2wy?zVFfZ-ybch}6Z|!TSvL*bOIiERv($L z4EJ5wlYw!r3>j#U5_W;gS8sYUzy4Osu7zdpaCq2S%gy3-PooA}L&;q8&aWyRqVFK` zXl)|j?hRk%R15Fe+^HC~Kr5x9hY+_npA3yUWYX{;lH<5}?Y6dWfex)sOz(pF((}So zO3J&0^|=|(V%5UU)cntsfHpT=J`)Mz3n#a_1mvQxK@#$tr`6xZPrC40G?s?($!wqn z4_{HZi7a8W+C5r|o6wwCJ#Ml&w7q^deUu^iU?^m6rB@gl?GJbFyTBZ^A%l`oAD@Ie zCrqox5(hv5Q8GyYLZPxgqf2u{5o@9sjlIcAp>Us##fQVY859v9K1G}9Zp;s+O&Mo^2%`40=;LhT03>9o=6ifVVD6l<&$Qb^U@9d|M=Bn!LMCIhRZdk8 z2WF(NFQ~oAi0I-xq&*yb>@fGT-};>K`dDy->f$rmU22Rg3d8Y~v#`!;GbcBz_AN7; zlZ;Ugqx@(4m3&{W->KTqDxBH8KR9*MbVv22OtGR*s!YRY#T(}!68`5d-Echoy2xKi zp>}Ddu~jJe!m^m`GKk)d<}X4_8o940p4@eTpYUZRci2>rTi#Q-Ca!J!gM_AuinCmf z?5VChiZ>SruUWgizA>_O-2pNKm#Iyu6JM)vhc-nN8B)MJQs#v{b8essZw1Ixw= zNe(ny^|bC$<*f4$hn?C4)q|;|b9iJH^=9e|)u`KwzMjMTr(#7w7Ie5L^5!(Xz780j zcY*KZZ)4R)&s1E6P?MSY@~U0fL|dLQpIGTb`@%lW+J z!c9-J9_6eSq`a= z*mWgT>@(PP!QRn%VBF#78C921qt(5$7=tvYvf_z*o3$m~H}sR%&<B*%b}aiR zispd8iFk4Yq4S-Q;*daGO=h-5i(rMFNUaSVDH1#P4bnAw1pL7A-F3xC>H~8!Y8d^hf5j=4-@hm(xggGidu& zW>SR&F#2^TTf@K3!Q5%JIQI7Q8;K5NN3Q>zv(;DY|zd zvAP3Jsf`5}lK5(IZ#Um;%Ck(k3M3a<(9;#LBj2Fbo(pOYhGulT)zim0KO1ho`IKN9 zuhVw2b{s|T+GZNcVK_dOS5g;p%|$3^?mBD!Nd1v@i-`^0Um~}dXUedm1P_#85Pbt3thW4cV z$$E5!WO$~O*NudoY9CH~u?~c}A=#u6AAYwS@{IhM+ zO{C-~b`fb*veI&JasySYX7PR!outvfCSox@3!9)y!v){uL%Ea+xSy~-&EGa$x_m0- zc{Rfa^%!JEFIUN0%8CQoWhqY|cGYurn|#~FLq=X08$nu{ z$MT|}7Sub#dBb0TTFYA^it6;Q!Q{TbS{j=#Wxk@cwq9@iOX4+rbbv{@qiT`ZDCEe7 zvn9kskE#!(UY9V)C}JtGNx6{fXs2n?UNENQTclpnY3;2Giy+olL#FjBHB^nc=(J?2 zVtA5@YPYj(Y+lM9X*U!sMu{h^!H?pQ0GR>CbV&I9YbLJ!nKfdOhmzOX{}^0@kj=?h zuEx*LuZkj?$;%W|Hv{UIjcff1I=0lY(*^5ON$aeF7-bmw*1nmP!8fyh!Lu-6CYton z@s+SNo3HkvnfoHJHESDgVs7w1lPj9WDm`Iq#Pm*m-{agm-{?8~KATVh1JHGAKQ2T- z)_2FXwE6qkUUMK`l>aTD^oTApZ8lN?Wyn4jIMiMc)wG)081|cd6NeE z2btYxc;>2MBtFO@Yt{G?;pERrCEz{E(m>E^{t6>*g%6%h>u}vKabH*Sq#^{oeRG>y;yK%ZRy^ z*#i;{lW?`QmRBk<(B9n!d{wDlEzEL!&L~HN;NVTey$9)7F{P#T&YdMyjHnmVkh6Zf z*hg5E-v7GFQ-5J=DzEYTnAp6~eAT+0Ez;qc`oygM8Jris_WH@us7yS%s&xjdW-+24 zDw-uBm3!mynjhwHgkaYw@5DSZk!G7r_2Sj3C zmWXEf+0}>=JiOLwSydJ{s49D2PJj4W>!xv-;f(;WU*y7yKDjVs&645#{4$V6M{f6R z#U5hLgpH!4f$PQbP#xZFz;mP*_RqhI#f6b;@^Wa*rQ zp)P-3V?Laf-@!ar$+#08R@*XxX4vKY0WA5M$rkxGblb_ne1|ac(t+0VZfJRY)m}NP z>2dg%;G_yjHLzq>(4}>0AaK!5$`mg8br)iO`sa>4PhzCW+SFE2$P-+a`6C<;w<@JhGe>@s`BdPuE; z&($A@4Z~;xx6yD}8bK}@Gkw77h)H1$ng4j3OX$3TAc$1{M|DxK;~P76wL@PD#*-Gjt&?&03Uz3T zy2_T-DaFymgLmOezevH80QMVfB5HUZ;+gA#Ll%Ol%eSb-;6$yKEM$>pDK}5Jx6sP~ zOD=LOQaR3GOQ&HXzOKj+mMR%5k4?$$PNB-0ls+a9-)_`t&}{u~tZ7Zt1%2VfqMCQA z8>4E$9-$>wuPV(~$$!*+V7KPoQ<+bs!P)LndSEF)T8G3vmuib~1ABeeIo%~<7^6qF z0f56^NX7n%>bRn1|FyGoU?pANn#X*yTcOoAW$4Q_Tm}p5#^LU~8y=y^tWk}%@eS&$ zD1rgn*Bql|R#rLUgo)i-@)BCFp1#Efl7}7qh~*k9Z}Q#v5V>Tg9=@uQ;?}?F(~n^R zn)g)6Rk@8D_5|O2$$sD@U6*A@#F{{pIU@zUXxa5Tp1rf*XaDx<>7;8WMZ*$J!$BaH z*nV!b#Nz(2eR*ajMzE^*2@Zvg$M>_t%5)=w&c#so2A)RaPX}p@tA~Di%##^>1eZd% zN>K->?}sB?HhX<%5+7%dxv~xy4Kc+q_{Jr_; z&(I4Us6KP+i4IJCcW8-?+cly@h!WMaoU>i#O~vf0m>%7(NFKFuGdEf$=Z6fv)*Y*Z zFRWY(oy~d@0yDc2U%r#DTJ+z9=8n)<(a3~PNera7xoda%ihr|g2ZoWR6iA*@a-rag z#;A6%nOTa`Nj5$*KfF;A#)i&a1X~M8;n8`koyXi2e@2Kh>my86r3z*8j&=nqtV+e) z-(1mFDAk0^r*uiGTCn#n;lb@Ia0E3Mn$mi$%hg?&g6q?NA9b zMokY>3X8d^Z5Ro=)68%nNd_EU+*`A70SdPu@F}M9|Cm7 zhPyaEzfZ9Cvo@Ola^uXcLq#zFpP|E1%3{=5IuId+OU`+g z1k9@QO^8WVJ=-Z822Vn%ORVbF_i7PL7aa754^_?bS`D6nZqVVTpa|GCh!(5g#;K09 zr1xZR3`3sA=4ffk1T7Wvzzjx=wooEb$~`rb^7vAGyGs#}NqOkMp( zSlX{reE@EFB1C&8dbwfXDE&{HlZy>P669C&2e0_devF}A+J=`T)aDHPgl`^*HPyeL z1G0jP^!w@wpP%z(#)M_)NH^d44Wt~c5Sp`N%CN(0cCmnLN&Fz=vd-NMLXdXf+D8Yz zz>XeP6r;v&gPzVjkoUNROCG=#YlX7hbH#SS>=tg!6mg~YCsd3c?UJKDP40t!|68nJ zWPa=;v1IjQCYPm2j&z3Xeo?TUnyp&hbGNKnS209SZNz)i>pWS=31P(5q1_6J)L*%q zc>YhV+27~Z(kLR~E0?4%Dr}&OxTI_kVcVy*Z=aA7Nk8>`P=JKA&NONO56HtHM^6~# zi%6Nfap-C~o49e)YFnR4trk+j3`-EZJ&5i&kXSX;(v82GMkTRPaa^Fcq!v9T$xGmT zCD+UC59POdKh@d*}c}I_!Ac7;gAMa5D7oUk^n2dsMnG zEA4Lxb*?3&<%F8*U%VyEfFRnhTNle;VTW@T4j;Vp-BTj)4mJOc(}9tYuL<)l0h7mG z^@(38n;tyBYOZ=>uG?h+L@GrGs-9oxh(1Pm-SmtWlpvJ7YV{2S3YO6Ny+5Ol6}4DQri9V&+=bYKACmF#$A&O zo?39iUhSG}uH0Po#=|5WvI~e7g}c=z7vX|pH+oXMq{I#Ul))TyO;lgHjf$$E#dXS>+26iyFl&nu)3HvykYN zoxx;PEWn%4XO zS|$r{Cbc~66lojb4Go@qd06G~+L3)6g^uO|nw_bVw3w|AL~V{X(FS-_5iT*vrg{Q* zY;tFeyx4AUXzcSW&R@9bmFWt^gY5*`f#7yZa;p3TFh9w>SVjpdA0@IK|AbQ?sWnOS zpoteDJajNSP7UB@ERqp?x%HoOqnYRS@K^$}JnXD%&mnoN^h7G(yxIWQma zb4xCD7AY&og?vv4v0g-IKVz+q-v}H3=fUm)5BBo8#74*e@OZdo|A@PtfZZp?+h|3Z zfu%;IvXNvt<=*Cg6ACY4^Z)h(#slRNi6%l+q+u2s$pn^XkvI!zqCr3nJW3f5blBN< z>lICMF-|vnOa8c7$JXt}N^>em-6}C{?&m+^@MfQXzm5_5Le5i)6p_kPp*AyBz&z;j zWZf$E7CEUxXnL$9K_*SrF#8CX+wxJZL^juqcMZxmLl+{q@;ll3u}!~7vFwwj5i*)J zVT$#>)uXNv_=drbfu?g^FWf1ySk1N23V4q?XXl&8s_Q>Ab(ZpK*6CY%l4vifu3hqX zc31K8_^py)`O_ZI?%Dzp*x+lQg{k~my9%-!_v?*vD+}pW1zRP?%^)uLPA$bxOS0pr zkj7K{6J%P1mfgIshzCOQnmGwJL{_^U+T3WyyPk8R$uKqRRUTBkvoU1M^b|;B#>DX* zu5lLNhBCWsk;ycnI4$yDI>`a$aBzOBW z>u(rQ3HHs!!|6klG`7PKw}Ef9xml_1pq^4)*%x{L(%-c3KBT)w_=Ar1$9Ke(2Ld0XNPY#NSHvw)*MHk2cj`A@tSv;o7n7$$9h=N@4E_$%*8bmsLE*Ddj#ok&nhjVt?p zl2=#)Mt?l8{4hAZH?S5Mova4Ku5Vca=*$Hn75WTrfCRGq~1S zEJd6w)F;?fwqn|rnK!t0MCYs>eBj&rnC=Fh>viPR-mX*qnR5;BM9S!*U&U8Kn*u%U z>V7b<>4mN;OdXx1W)nH3M9S`H@B-KMn{4{Un$6^tfy0Hkn}n%)9{zOESBTOEi>~>N z#Y?JSc_-p;N8h5%CCcv(ZlJ_xD$<6OiY*{aLf;BLZYtYmX@);~pmKP{=h_WymzWXV z1HT9w!5Rg#l{_6)d4}Ik-+l)&_)4@xbLQ{K0_jWHdZxw^f;GY`Dcz&*>c8OUM2*J} znvb?O=Y)~VC1x_ntUpI)c`z`NYwmt->|crCJbQB2pKbspxyPZQBY4% zcad6@vS0pOiRGBwa$>o0Jx!ss+{R{Xgd+)Rr>z8T@5BH>;X1F}q+qyDm||*_!wZ62 z?y&vSl6kO)gt)m(B%|}ZtTuitZWXelFLsX()eJgUs|jLR!bey51uq>_S!u{a7SwT+ zT|{Yo+Ywvo%%-Nbs5=Rl9T%=vv#4rnE~|wAa5?+(1$JBsEdmv-RZ$0_D~Bud(^F(6&$d$JT)HPFCaWgZ#Gp66sba0W3&46)kV= zSc47v+-d!GpSSt4#7#{O=9uLB({J;|JT>$i%#1SY44*IuJR^|ZSVvWoH1sC2rhkv2 z|JthR2X6NL $qFvgeN|p`x*x)%oT*E@Gt2rIm>k0k;sgNA=$>USsCtU_?###I? zODZ|{LYTMQ&Kn$hEpVo@eDHmFw(f|Zi>M*6x)VZXgFn9nVTa<#8M}EfI54VBUCDH& z564ZJ0P}lKO*8`8y${p;IifnG8Q^Q{^Rl*V^euW4e%p7eqhNJk6FaNJPqleV>u1-S z|HWp(&#}q)@dK~iQ>fiIlM>eGn8g>d^{-!ISy>%!-UYl^#G0xXacQ)8A+W8u<{i$; zg-0v*92<$x@C5HahredOo&6NW(CjRlFa^a`yNJqpt-;`xNc>;tqi-86ssJ-^JBBif8Kew_(6n>S=q|H2fp_3A3FTOpJeA_ns3T}No#PhGHlQ%*FrJW%wPkf)y;LO z_N;2o{YPdTuq+L?gkEe_%yvLdqJ$W_wJSECC|lK_11Pd$ZmOWsPS~jYVyfrY8XpYc z&hDuEK>awP&M;6vm#l^s|1t5Jv9$gm_o(;rUsS&OpGGvoxOzcz-WH2QYX1wx75kTg z50+I#n}Th#1wG0{VrE!3uNprc!d0BYr9d~f#)fx$hkvDG!DV53B4rqL(~yhm>Yp%K zihw`}*vfAoO?f^JmeDtP)auafMhD{Aq&ZlASvq;T+ zy0GHQ6yy0ms_!CE>hA4Ocp{AUGBR}a1^CXwZy<99&;{$AvRTbeC4Ii*T}q_njt(qc zpUoKkuR$V>X$3cIf0gS)p6{{InAk_$!AsG@`fT1z>}7=psb?XB+Ds`~=%+eo^eyQw zirSiIjyv30ISk1wO-{fu43h$a&!4U@mdI%fO3tmh)e0rruYbweQqWaKfbjYC45@t6 zM0;^}7usIw8GJ`p)=)>hfQPEOEJQEbW(je7L`msfW959u2zGG>Y5L*7iPN1%GuOAA z1KLQcImZP>%y2x%a|kfi@bBEi5I- zbBj_X`A7aLa;J0_m8@Ak!(Xr;UKR%?c|aq>$(|Se89FN~Q#Ws4*!m~E4L+%|9zYNK zdfe~hFN{>AcTkfgAm%9fEQgo1EUt1^-qvK$aEhbkcRoY!*c|qq>hzM1eEOa~0dJ4% zMNNc=c&MawV&y74Zn5i$KN-O2juY*~+~_Z!GHNlAURju_ggJeztPhi^jfhhw1Gu*L zy8GkLYw9D8BgyG|AR}=1HYnX3|WMpi71zmq$cm|LpeyqCWzXt8Cf5AD{&FT52#fPhK<`Yegy?>JP2@;*l`I0mM)_wc^FcV%ArOSX)MQPAC1BU{K2G zfSmbWc;7jku|=j3Np6d4m4#&YMl$P+@drv@Kl&xEW%fX#+Wn5>q{ip3LS_qp*R_Aw z>O}73REmLcUgMiro2!s&dVKJ@nFS{=pD!)8LVV_P)v}}UqEssXzlMR?w=NKm5j+Z& z`z_%00={KB+qIhBd2#P+282DaMR3IaThQ~`|C*BGfEn+fWBy+_|2rODUaKg|-u(IF z;JQlCoa5hbQ@ID`#ecmRZlZtPr+u$)-1=WcYX0?sfSd3?=?DL>cl`fx3WuNUwEO}6 WtuxQIdzx|a+8Q^mVlLmh|9=42D1ajX literal 0 HcmV?d00001 diff --git a/docs/intent_driven_orchestration.md b/docs/intent_driven_orchestration.md new file mode 100644 index 00000000..88c8d60b --- /dev/null +++ b/docs/intent_driven_orchestration.md @@ -0,0 +1,172 @@ +# Intent Driven Orchestration (IDO) +Intent Driven Orchestration enables management of applications through their Service Level Objectives, while minimizing developer and administrator overhead. + +This is achieved by doing orchestration based on intents in the form of objectives, which is done by the planner utilizing a set of actuators. + +More information about IDO can be found here: [Intent Driven Orchestration (GitHub)](https://github.com/intel/intent-driven-orchestration) + +## Ansible Configuration +IDO is available in the following profiles: +- on_prem +- regional_dc +- remote_fp +- build_your_own + +Configure the Ansible host and generate the playbooks for any of these profiles using the steps described in the main readme file. + +The installation of IDO has two dependencies: +- LinkerD +- Local Docker Registry + +Update `group_vars/all.yml` to enable IDO and dependencies: +```yaml +# Istio must be disabled, as we will be enabling LinkerD +istio_service_mesh: + enabled: false + intel_preview: + enabled: false + tcpip_bypass_ebpf: + enabled: false + tls_splicing: + enabled: false + sgx_signer: + enabled: false +(...) +linkerd_service_mesh: + enabled: true +(...) +registry_enable: true +(...) +ido: + enabled: true +# The demo workload is optional, but useful for testing. +# For this document, it is assumed that it has been enabled + demo_workload: true +``` + +Prepare `inventory.ini` and `host_vars/.yml` according to your environment, and proceed with the deployment as described in the main readme. + +## Post Deployment +After successfully deploying, verify that IDO resources are available and running in the cluster: +``` +# kubectl get all -n ido +NAME READY STATUS RESTARTS AGE +pod/cpu-scale-actuator 1/1 Running 0 18h +pod/planner 1/1 Running 0 18h +pod/planner-mongodb 1/1 Running 0 18h +pod/rdt-actuator 1/1 Running 0 18h +pod/rmpod-actuator 1/1 Running 0 18h +pod/scaleout-actuator 1/1 Running 0 18h + +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +service/cpu-scale-actuator-service ClusterIP None 33334/TCP 18h +service/planner-mongodb-service ClusterIP 10.233.2.41 27017/TCP 18h +service/plugin-manager-service ClusterIP 10.233.6.112 33333/TCP 18h +service/rdt-actuator-service ClusterIP None 33334/TCP 18h +service/rmpod-actuator-service ClusterIP None 33334/TCP 18h +service/scaleout-actuator-service ClusterIP None 33334/TCP 18h +``` + +_The content below assumes that `demo_workload: true` was configured in `host_vars`._ + +The demo steps provided in [Intent Driven Orchestration - Getting Started (GitHub)](https://github.com/intel/intent-driven-orchestration/blob/main/docs/getting_started.md#demo) will not work with this example workload, as the network service has been removed. The demo provided below will instead utilize deployment scaling to show the functionality of IDO. + +The example workload and IDO resources related to the workload (KPIProfiles and Intent) are created in the default namespace. + +Check the status of workload resources: +``` +# kubectl get pods,deployments,intents,kpiprofiles +NAME READY STATUS RESTARTS AGE +pod/ido-example-deployment-5f476b97d4-55gnn 2/2 Running 0 18h + +NAME READY UP-TO-DATE AVAILABLE AGE +deployment.apps/ido-example-deployment 1/1 1 1 18h + +NAME INTENTS PRIO AGE +intent.ido.intel.com/ido-example-intent 0.01 18h + +NAME RESOLVED AGE +kpiprofile.ido.intel.com/availability true 18h +kpiprofile.ido.intel.com/p50latency true 18h +kpiprofile.ido.intel.com/p95latency true 18h +kpiprofile.ido.intel.com/p99latency true 18h +kpiprofile.ido.intel.com/throughput true 18h +``` + +In the above, `pod/ido-example-deployment-5f476b97d4-55gnn` is the sample workload (sample-function & linkerd-proxy), and `deployment.apps/ido-example-deployment` is the deployment that handles the workload/pod. + +All of the KPIProfiles (`kpiprofile.ido.intel.com/*`) are prometheus queries towards the LinkerD prometheus instance running on the cluster. + +Lastly the Intent (`intent.ido.intel.com/ido-example-intent`) is a set of objectives utilizing the data from the KPIProfiles. The intent is referencing the workload deployment mentioned above, which can also be seen below: +``` +# kubectl get intent.ido.intel.com/ido-example-intent -o=yaml +apiVersion: ido.intel.com/v1alpha1 +kind: Intent +metadata: + creationTimestamp: "2023-09-27T17:49:48Z" + generation: 1 + name: ido-example-intent + namespace: default + resourceVersion: "8591" + uid: 5eaed99d-cee3-4230-a8ee-81c69171927a +spec: + objectives: + - measuredBy: default/p95latency + name: ido-example-p95compliance + value: 4 + - measuredBy: default/availability + name: ido-example-availability + value: 0.99 + - measuredBy: default/throughput + name: ido-example-rps + value: 0 + priority: 0.01 + targetRef: + kind: Deployment + name: default/ido-example-deployment +``` + +To test out IDO, modify the workload deployment to create additional replicas (5): +``` +# kubectl scale deployment ido-example-deployment --replicas=5 +deployment.apps/ido-example-deployment scaled +``` + +Now check if additional pods have been created: +``` +# kubectl get pods,deployments +NAME READY STATUS RESTARTS AGE +pod/ido-example-deployment-5f476b97d4-55gnn 2/2 Running 0 18h +pod/ido-example-deployment-5f476b97d4-ddh8k 2/2 Running 0 38s +pod/ido-example-deployment-5f476b97d4-n9q9h 2/2 Running 0 38s +pod/ido-example-deployment-5f476b97d4-rxr6m 2/2 Running 0 38s +pod/ido-example-deployment-5f476b97d4-xk27g 2/2 Running 0 38s + +NAME READY UP-TO-DATE AVAILABLE AGE +deployment.apps/ido-example-deployment 5/5 5 5 18h +``` + +Wait a few minutes, and try to run the above command again. You should see that pods are being removed and that the deployment is being scaled back down. After 6-8 minutes the number of pods should be back down to 1 again: + +``` +# kubectl get pods,deployments +NAME READY STATUS RESTARTS AGE +pod/ido-example-deployment-5f476b97d4-xk27g 2/2 Running 0 8m43s + +NAME READY UP-TO-DATE AVAILABLE AGE +deployment.apps/ido-example-deployment 1/1 1 1 18h +``` + +You can check the logs for IDO to see that the deployment is being scaled down using `rmPod` utilizing `pod/rmpod-actuator` which is one of the IDO actuators running in the clusters: + +``` +# kubectl logs pod/planner -n ido | grep "Planner output for default/ido-example-intent was" +I0928 12:25:55.371232 1 intent_controller.go:165] Planner output for default/ido-example-intent was: [] +I0928 12:26:40.372409 1 intent_controller.go:165] Planner output for default/ido-example-intent was: [] +I0928 12:27:28.435086 1 intent_controller.go:165] Planner output for default/ido-example-intent was: [{rmPod map[name:ido-example-deployment-5f476b97d4-55gnn]}] +I0928 12:28:56.756055 1 intent_controller.go:165] Planner output for default/ido-example-intent was: [{rmPod map[name:ido-example-deployment-5f476b97d4-n9q9h]}] +I0928 12:30:25.926758 1 intent_controller.go:165] Planner output for default/ido-example-intent was: [{rmPod map[name:ido-example-deployment-5f476b97d4-rxr6m]}] +I0928 12:31:55.554513 1 intent_controller.go:165] Planner output for default/ido-example-intent was: [{rmPod map[name:ido-example-deployment-5f476b97d4-p9slf]}] +I0928 12:33:25.384619 1 intent_controller.go:165] Planner output for default/ido-example-intent was: [] +I0928 12:34:10.368687 1 intent_controller.go:165] Planner output for default/ido-example-intent was: [] +``` diff --git a/docs/ipu_setup.md b/docs/ipu_setup.md index eb06a79f..1032d501 100644 --- a/docs/ipu_setup.md +++ b/docs/ipu_setup.md @@ -15,11 +15,25 @@ IPU board has five different connection points: - IPU board is connected via serial Mini USB connection to IPU link partner host - IPU board is connected via high speed connection to CVL NIC on IPU link partner host +IPU host needs to have Baseboard Management Controller (BMC) interface configured and connected to network to be usable via +Intelligent Platform Management Interface (IPMI) tool + + +## IPU host and IPU linkp OS + +Currently supported OSes are Rocky 9.1, Rocky 9.2 and Fedora + ## IPU pre-built images -Please ask Intel Support to get tarballs with 'imc' and 'mev-rl' images. +Please ask Intel Support to get tarballs with IPU pre-built images. Insert those tarballs into directory /tmp/ipu on your ansible host +E.g.: EthProgrammer-2.0.1.zip + hw-flash.6330.tgz + hw-ssd.6330.tgz + hw-p4-programs.6330.tgz + Intel_IPU_SDK-6330.tgz + hw-workbench.6330.tgz ## IPU inventory preparation @@ -31,6 +45,41 @@ ipu_link_partner ansible_host= ip= ansible_user=root ans NOTE: If you need to connect 1GbE connection from IPU board to IPU host machine instead of IPU link partner (from any reason) then you need to change variable ipu_1gbe_connected_to_linkp inside inventory from true to false + In that case you need to provide ipmi credentials in inventory record for ipu_host_machine + and run deployment under root user or any other user with paswordless sudo on ansible_host +``` +ipu_host_machine ansible_host= ip= ansible_user=root ansible_password= ipmi_ip= ipmi_user=bmcuser ipma_password='' +``` + +NOTE: If you do not specify ipmi credentials at all then power cycle will be replaced with reboot. + Reboot is sufficient in some cases but does not reinitialize IPU card reliably. If IMC and ACC does not come up properly then + manual power cycle (power off, wait at least 10 secs, power on) needs to be done. + +Here is error message when IMC and ACC are not initialized properly +``` +TASK [ipu/flash_ipu_nvm : wait for ssh connection to IPU-IMC] **************************************************************************************** +task path: /root/final_ipu/cek/roles/ipu/flash_ipu_nvm/tasks/main.yml:116 +fatal: [ae07-06-wp-ipu6-linkp]: FAILED! => { + "changed": false, + "elapsed": 301 +} + +MSG: + +Timeout when waiting for search string OpenSSH in 100.0.0.100:22 +``` + +Once power cycle is done, wait for ssh connection to ipu_host_machine. +Once the ssh connection is ready you can continue with ansible script via: +``` +ansible-playbook -i ipu_inventory.ini playbooks/infra/prepare_ipu.yml --start-at-task "wait for ssh connection to IPU-IMC" --flush-cache -vv 2>&1 | tee ipu_deployment_cont.log +``` + +NOTE: If ansible script failed during ipmi related operations from any reason E.g.: wrong/missing ipmi credentials then fix configuration and + run deployment from point before power cycle/reboot handling again via: +``` +ansible-playbook -i ipu_inventory.ini playbooks/infra/prepare_ipu.yml --start-at-task "IPU images are flashed" --flush-cache -vv 2>&1 | tee ipu_deployment_power.log +``` ## IPU group_vars @@ -38,6 +87,14 @@ NOTE: If you need to connect 1GbE connection from IPU board to IPU host machine IPU deployment uses group_vars/all.yml from standard RA deployment. So, if deployment is executed behind proxy then corresponding proxy configuration have to be done. +To prepare needed deployment environment please follow listed steps from [README](README.md): +step 1. (basic) +step 2. section b) +step 3. +step 4. (tested with defaults icx/cvl) +step 7. (k8s) +step 8. proxy and mirror settings if needed + ## IPU deployment @@ -50,13 +107,16 @@ ansible-playbook -i ipu_inventory.ini playbooks/infra/prepare_ipu.yml --flush-c Here is summary of Successful deployment: PLAY RECAP *********************************************************************************************************** -ipu_link_partner : ok=49 changed=8 unreachable=1 failed=0 skipped=5 rescued=0 ignored=0 -ipu_host_machine : ok=13 changed=0 unreachable=0 failed=0 skipped=11 rescued=0 ignored=0 -mev-acc-rl : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 -mev-imc : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 +ipu_link_partner : ok=72 changed=34 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0 +ipu_host_machine : ok=14 changed=6 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0 +ipu-acc : ok=28 changed=21 unreachable=0 failed=0 skipped=4 rescued=0 ignored=0 +ipu-imc : ok=11 changed=5 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 ********************************************************************************************************************* -NOTE: Unreachable=1 for ipu_link_partner is expected state. It is caused by ansible error and has no impact to deployment +NOTE: Unreachable=1 for ipu_link_partner can appear as expected state. It is caused by ansible error and has no impact to deployment +PLAY RECAP *********************************************************************************************************** +ipu_link_partner : ok=71 changed=34 unreachable=1 failed=0 skipped=5 rescued=0 ignored=0 +********************************************************************************************************************* ### IPU deployment progress monitoring @@ -80,15 +140,28 @@ NOTE: To escape from minicom use 'Ctrl-a q' and then enter Finished deployment on ACC: -MEV ACC mev-hw-b0-ci-ts.release.4988 mev-acc-rl - -mev-acc-rl login: +Rocky Linux 9.2 (Blue Onyx) +Kernel 5.15.120_ipu_acc on an aarch64 + +ipu-acc login: Finished deployment on IMC: -INFO: ######## System booted successfully! ######## -MEV IMC MEV-HW-B0-CI-ts.release.4988 mev-imc /dev/ttyS0 -mev-imc login: ++-------------------+ +| Intel IPU SDK | +| Version: 1.0.0 | +| Complex: IMC | ++-------------------+ + Built on : Wed Sep 6 03:38:07 UTC 2023 + Build host : Linux x86_64 + Source rev : 49f2e2588be2401c2025bce28e1a43d32119ee39 + Build : ci-ts.release.6330 + --- + IMC uBoot : 2023.07 + IMC Kernel : 5.15.120 + +ipu-imc login: ### IPU post deployment accessibility @@ -103,11 +176,98 @@ It allows to access IMC and ACC via ssh. For IMC: ``` -ssh mev-imc +ssh ipu-imc ``` For ACC: ``` -ssh mev-acc-rl +ssh ipu-acc +``` + + +### IPU/IPDK Networking recipe verification + +IPU setup scripts prepare IPU/IPDK Networking recipe including one custom P4 program for l2-fwd_lem. +Here are steps how to validate that setup works correctly: + +1. login to ipu-imc to check ports + +``` +ssh ipu-imc cli_client -q -c +``` + +Output: +Warning: Permanently added 'ae07-06-wp-ipu6-linkp,10.166.30.171' (ECDSA) to the list of known hosts. +Warning: Permanently added 'ipu-imc' (ECDSA) to the list of known hosts. +No IP address specified, defaulting to localhost +fn_id: 0x4 host_id: 0x4 is_vf: no vsi_id: 0x2 vport_id 0x0 is_created: yes is_enabled: yes mac addr: 00:00:00:00:03:18 +promiscuous mode: 00 +fn_id: 0x4 host_id: 0x4 is_vf: no vsi_id: 0x8 vport_id 0x1 is_created: yes is_enabled: yes mac addr: 00:08:00:01:03:18 +promiscuous mode: 00 +fn_id: 0x4 host_id: 0x4 is_vf: no vsi_id: 0x9 vport_id 0x2 is_created: yes is_enabled: yes mac addr: 00:09:00:02:03:18 +promiscuous mode: 00 +fn_id: 0x4 host_id: 0x4 is_vf: no vsi_id: 0xa vport_id 0x3 is_created: yes is_enabled: yes mac addr: 00:0a:00:03:03:18 +promiscuous mode: 00 +fn_id: 0x5 host_id: 0x5 is_vf: no vsi_id: 0x3 vport_id 0x0 is_created: yes is_enabled: yes mac addr: 00:00:00:00:03:19 +promiscuous mode: 00 +fn_id: 0x5 host_id: 0x5 is_vf: no vsi_id: 0x5 vport_id 0x1 is_created: yes is_enabled: yes mac addr: 00:05:00:01:03:19 +promiscuous mode: 00 +fn_id: 0x5 host_id: 0x5 is_vf: no vsi_id: 0x6 vport_id 0x2 is_created: yes is_enabled: yes mac addr: 00:06:00:02:03:19 +promiscuous mode: 00 +fn_id: 0x5 host_id: 0x5 is_vf: no vsi_id: 0x7 vport_id 0x3 is_created: yes is_enabled: yes mac addr: 00:07:00:03:03:19 +promiscuous mode: 00 +fn_id: 0xc host_id: 0x4 is_vf: no vsi_id: 0x4 vport_id 0x0 is_created: yes is_enabled: no mac addr: 00:00:00:00:03:20 +promiscuous mode: 00 +fn_id: 0xc host_id: 0x4 is_vf: no vsi_id: 0xb vport_id 0x1 is_created: yes is_enabled: yes mac addr: 00:0b:00:01:03:20 +promiscuous mode: 00 +fn_id: 0xc host_id: 0x4 is_vf: no vsi_id: 0xc vport_id 0x2 is_created: yes is_enabled: yes mac addr: 00:0c:00:02:03:20 +promiscuous mode: 00 + +server finished responding ======================= + +The first four lines correspond to four NICs inside ACC. The fourth one is used for sending traffic to ACC. +Its mac address is 00:0a:00:03:03:18 and vsi_id: 0xa -> it is 10 in decimal. Once we want to use it inside br0 rule we need to add constant 16 to it. +So, final port value inside rule is 26. + + + +2. login to ipu-acc and check NIC name which holds mac address found in step 1 + +[root@ipu-acc ~]# ip a show enp0s1f0d3 +5: enp0s1f0d3: mtu 1500 qdisc mq state UP group default qlen 1000 + link/ether 00:0a:00:03:03:18 brd ff:ff:ff:ff:ff:ff + + +3. login to ipu-acc and start tcpdump with NIC name found in step 2. In our case enp0s1f0d3. + +``` +ssh ipu-acc tcpdump -xni enp0s1f0d3 -vv -s0 -e +``` + + +4. login to ipu_linkp run scapy there and send 100 packets to ACC + +``` +ssh ae07-06-wp-ipu6-linkp scapy +enter +sendp(Ether(dst="00:00:00:00:03:43", src="9e:ba:ce:98:d9:d3")/Raw(load="0"*50), iface='ens785f0', count=100) +enter ``` +.................................................................................................... +Sent 100 packets. +>>> + + +5. on ipu-acc you will see 100 packets received + +01:19:46.679164 9e:ba:ce:98:d9:d3 > 00:00:00:00:03:43, ethertype Loopback (0x9000), length 64: Loopback, skipCount 12336 (invalid) + 0x0000: 3030 3030 3030 3030 3030 3030 3030 3030 + 0x0010: 3030 3030 3030 3030 3030 3030 3030 3030 + 0x0020: 3030 3030 3030 3030 3030 3030 3030 3030 + 0x0030: 3030 +01:19:46.679264 9e:ba:ce:98:d9:d3 > 00:00:00:00:03:43, ethertype Loopback (0x9000), length 64: Loopback, skipCount 12336 (invalid) + 0x0000: 3030 3030 3030 3030 3030 3030 3030 3030 + 0x0010: 3030 3030 3030 3030 3030 3030 3030 3030 + 0x0020: 3030 3030 3030 3030 3030 3030 3030 3030 + 0x0030: 3030 diff --git a/docs/monitoring.md b/docs/monitoring.md new file mode 100644 index 00000000..3f77a0b4 --- /dev/null +++ b/docs/monitoring.md @@ -0,0 +1,60 @@ +# System Utilization monitoring +Telemetry stack in RA project is consist of following components: + + 1. **Elasticsearch and Kibana:** Elasticsearch along with Kibana, offers real-time indexing, searching, and visualization of logs from whole stack. + 2. **Prometheus:** Prometheus is an open-source monitoring system collecting and storing time-series data. It enables efficient querying and alerting for system metrics and application performance. + 3. **Grafana:** Grafana is a verstatile open-source dashboard and visualization platform. It seamlessly integrates with Prometheus and other data sources (influxdb, etc.). It provides customizable dashboards and realtime alerting. + 4. **Telegraf:** Telegraf serves as the data collector in our stack, efficiently gathering metrics and telemetry data from various sources. Its extensibility and plugin support make data collection flexible and powerful. + 5. **OpenTelemetry:** OpenTelemetry is an observability framework that automatically instruments your applications to capture distributed traces and metrics. It enhances end-to-end visibility, aiding in understanding system performance. + +## How to work with monitoring stack: +## Telemetry +Various dashboards are available for monitoring telemetry in the NEP stack with the possibility of creating new dashboards or editing existing ones. +Dashboards can be viewed by accessing Grafana. This can be done in several ways: + +1. By directly accessing Grafana through an open port: + + https://:30000 +2. By creating an SSH tunnel with port forwarding (sometimes direct port access can be blocked by network management): + + ssh -L 30000:localhost:30000 @ + And in the browser open address: + + https://localhost:30000 + + In the basic settings, the user "admin" and the password "admin" are set for grafana. We strongly recommend changing your password to a secure option. + +Most cluster parameters can be monitored through Grafana. Grafana provides several dashboards with options to monitor CPU resource usage, memory, disk usage, network usage, etc. + +## Logging and Tracing +Logs can be accessed in several ways. + + 1. By using Kibana. +Same as Grafana you can access Kibana by two different ways: + - By directly accessing Kibana through an open port: + + https://:30001 + - By creating an SSH tunnel with port forwarding: + + ssh -L 30001:localhost:30001 @ + And in the browser open address: + + https://localhost:30001 + - Username and login can be optain by these commands: + Username: + + kubectl get secret -n monitoring elasticsearch-master-credentials -ojsonpath='{.data.username}' | base64 -d + Password: + + kubectl get secret -n monitoring elasticsearch-master-credentials -ojsonpath='{.data.password}' | base64 -d + +2. By using K8s log tool: + It is still possible to detect the problem through kubernetes tools. + For example using logs tool: + + kubectl logs -n + +## Kubernetes dashboard + +To enable deployment of the Kubernetes dashboard, it is necessary to check whether the variable "kube_dashboard" is in the "on" state. +For accessing kubernetes-dashboard follow [kubespray documentation](https://github.com/kubernetes-sigs/kubespray/blob/master/docs/getting-started.md#accessing-kubernetes-dashboard). \ No newline at end of file diff --git a/docs/profile_overview/BMRA.pdf b/docs/profile_overview/BMRA.pdf new file mode 100644 index 0000000000000000000000000000000000000000..40b4f8fdd134d51ab955ae54cdd9fcf53e383283 GIT binary patch literal 54198 zcmbq*1yodB_r4$~f`TY1(u&gEse~vcrP3hX(hVv~hoE!}illT(qm(d&IFdsM2qVJK zFu)N1bI14nVtwE1`&<8St;0U|&YZjB+0WkREVzrScNLVl`FVthuOgNfXNdWk_?Vm@ z+7U}h@yfeenz=i>GF_LqadmdI^>Ab-=Dq9cY~f*U2|m=ew>9Gh&oViR@qsUJi*22q z6wKT$nXW5H2=ECA^7HYF^NR=w3JG!Y@m&K0tW$Hg`27+Y8DdK(3&;h*f4X9B#>7v| zdsB^xSKZmw(ahnWi$wmpNXgd09eBp8MY#evsUv)d#p#VZ6J* zJK@S|BgdBuH{g6^`Wro%`joW+1K$HIW;Ly4xbH!W#v!uha2t8-|E}<;Dd<4(tMkz3 zro1lMZ3Ee;Hoqh-jL3HM;k~9qv~2U?(sa`yCU~&%zDH=1-u$+6$DLUILQyQn+*Ag^}Hc=)R`rHQs$+uHEkl1ekG zN2+~iYe+^%UYsk_%+BsJz2A(pPOAMep0eHLznGtV z=1L7Ukpu7f7u2KMgilWBW<933&O}HI6uf{Gtnd>HsS?WYQSLr&Y^`S}BFG2&Z< z9v8HZUrVWn`Nwx>LH<)dgF=xbqBx6B7%XSX@|lm^2H->)fA$6?kWo)5Ao-YLbngl) z4WXw@H%%^n!_D}`rJM20`S?I;zJMh83w(Q+1wl4YPTQtqYNRYMsiQ1FpqTaa{I59h zZ)TZ7ao}Hk5)v*K+}PUu!h)y}k`!+tN$|FH>Kg5b>t&eRxl2az`~WAC+aX!RKtk>V zDMwrs5Kj(*fLWZ?e&WI0U8T+3eVzhB2xo+t=*iO?zzC5l!6lH44SX!lMIeETKmr&2 za4rHnz(o^f0Y~5_fiC1Mu@)rpwIIk$DIQ?2m?uLvb5n?c55Iwl8V3_K4kjQ0Ojrn} z9UM%+rtQzagK7K!8>UV^%2Dcg)kzAR4-_~bfCN75LOzJ#d`P<)?{m;Xt$g6kp2;w7GP_uw9=C%4S|pXC^dz-xKsfNQUxSPRVOZ0 z>$f2vDB@L9|3{c^tAR}FJOT2$G9-zVpG0d2Jk>Ovdl_ZMQMP_Nw?aFEgD=?c38~N@ zgE&>LLpAFm+lx_Srm`kEIb+jR9OqYYoX0? zFQJlv5CUQ5;+hZ84H&%v)eL|ym*H<}h6-2KRJgJR5|s5mRMs`PvPRyFFMztA&J&PR zP?f-A(7ZMP2vF|7;$V{ds`cc*G~n6QKYRS&TG0AS3t$rf=HI^lJ((PF&jSgV4kW1R zVjKobxT@y9lJe@eAo<6qoMN;HL(j8Y{iG0diAWLopqeU|tIBloluao-tBFfKy>K{> z`r(_x?J-6Ee7gWw^0S5dNd^Q@3x{*4f)#X^1mQY+pKhw^=*pB`Kc$-x%lY(%?)=9H zxx2b``43;@7y93&Jq^6#bRB#1PRM}fP2u=oo3Ojp zM-?KbTRcG34w72={NY{TGc%1&HID%ifo?IJ>Qs2)uyth3#u$jQ&>xDD0|rIegNxD? zic*&{vt59S0sLgP3G6$(rpf#WCT1F5>aAE)TBDj1SIA6hCpYga!MIb0L9h_1ZbGE0 zIS-MF8zPk@2}G(&MeRL@8xW}sAW{ubL8Pj94w1^i43LUgbUW&~pKs?!Zt3g+s)+S7 zRL`+KNS}OG%_veD_Ml%X`Pyn~a*_=mIIc{S&nT%IvHD0?PoxOun&vI{Y zSf7gp$M=TqX(VIp%-(m=2L3rTIrZ`GQf-iGcjnOW+pdARf%?O|TG^u|q;n8g;~gK} zJA&(ecQT|elms3`#WL#n95~;Md+q0(SMYB6^`oEzEHe0WP0Ht0rSgn5CyEJn(b$K& z4W!Qs14?ttotCOC?$5PEt*RkqW%Njw9w6pWs5um-YruR?ANh&HSzcgy)6vFfv9pm} zuwlyau3%%HRsNSMQptslosUQBn;9*&@>5Zbde=p)8vA$65&Cmyb&zi{HO|W%o4V+o zrXP8yJy`3odv$M{o4L-U*{s^tHf~Rv%XfaopzG;nWu{~Bs6+aZPqgN&DmEnJhpG;- zLLz-aE7Q;7)IXBgH}@HB@0CfV20AL`NxjoJYpDA`_kQ@;TO*{phcKG%&A>QpsA#?Y z?YAA-g2~zD@*EL+k5!`~%wn8OP4)Qd*oU-`kBxmiLAlw^o(Py)ENuN7Qt}?Mpla$# zaG*&ofkCH7e(_Xj*SN9sB}-qepi80I&Y^lIoe9%wY^NEQY(#tf5+xh$Equ7S{N)AV zKSiB48dC$Ny@MM3zb8_HC&&o@ZE=FZLro-$&Wn zq-+YkKbATdBdZY{r-8UM!6NZ8w{Jt%$+sD~Y#544G~ndH)=zK#6qTtpPIsh=s&r`< z?hU!h`?)Wb`iIpZiF@Tw!g{?O{K+D^JW+3BIL}Qd{L$4VyYZv4YV0Y$((H5M^D(E3 z=^uU^W@h7}fB2=JpQGq<_Qy_s4lXb?gFCu6BHpI6s_EzlMEKW}uMqUYu?H^KZoQnZ z>{{i~PbhA5xRF7ZV<4O0?=*k@;yqG+v-$Hc@5N}E&C^i+>vX>QgLbbaVF>;4EA(h7 z=c+5s>#3s7HivCgqS39w1&(u^y4(9>Zw~v8jwU)Wx`Ou8_s=EyPTo7m{m5M0K9UZu z_io}r|2V4Cby!O@u~MX=G`-qbkOyY$R+i!=FN^SXxBFy<@;jm>t3hft`RWv9`WGsy zWIW7rROkPEDUYV=Se%$EFb1Cvmfc#2x?VO1ZoMKN)?6w?M>JOtve}FV$&cpS){5sp zc&ulPH8)!CL)_oQ)LX{ICb@_dU9xJcF%#jwoY&UxB2h%2*S6py!Nr)@c4W4nO}V_R z!WNmJnPqHnLPaz}wsu{=nxp+q;b$=seM*`>hhvXyy)^$xo8fyVqL;?kNJ1-X4nNKm z^|d?_GoWe)LJ1O{%}!zlxv4YV0@VcVRZi`d+hi(3T6;xNpTZuLG`lNs>6u8dBo_11 z$El2rkcQ7M*@w1Y*r}F(NQ)h3E`Jq(e(|8c*cXcV8+x8@9v!aXaxw?? zjZT}dE$W_S^(@&R`TH-o^**t5RsQfI{Hn7VOj`UrBQ^YPa^Kaf>RlV>^RkEL+^ZbZ zW2mAKgX$nU;rgodrHtY<ibu>25{Kn)zGs-L>lo2wrHZZWTX3lv zlj;xY1LRu(@lXD&uHFUX=ES2T?}H6IK+NT1GVjAz~Dx?f-7_~D&1Vm`Y# zjXS1FIXRl@OywYpnTWv^^;?nci#Bm$^xPNnvK43D!Ix8`)PW-=<17QB`Fj111J(-* zFXwcpxDS>yfYEe1)ul4g+3PyIUbb|zGKI`owcM$1jPsVL1paVr zw0%u1zu6ws&@#i@)S+OVN}m^YRAgFxQ~ZT`4JiZM+|<$oc|KMuq=vx_dncBiPjl7l zM7JM`(TSX^G#lbe^NdEAjfv6MfN4^Ueg{l5e3K3n11O3K@u#*%(Jnk@#^ETxE-R$L zcqZBW0E+Rueh#`vHqX1NtC6}avHYI+qqFXlvv(t(joHNU(Q#j}s>DGFcoB{mWO0$u zhj4zj7)wQ)ge^0obBOWcdGh`7^N|Sa;7?Wt_&d?j_QTQqW=N(XF1oj=Mj!;c7j`1& zwEEug7miih%9NpeyEd#8VS<<)x&99a2kXj#L62H{YITFFo!^~mn7$Xy{m4YTsv}_j zGY2L?9y@WLl-mP*Fy=U4w7q{+Ih-cq7j6Wnx2X>FyZ>l=G`+BABqnH}W$C2C9d6G$ zV*Mpo%BrvP91b~4mY;8qz_UC`l5W5ad4uoO8K2^It&6JeteQv|%b6!|8Jhfj$=-6Z zk^T56KQVBV6*)~--KTO(PNm76pbDvS%apU(lc;LZK2(mR1zcdj#S*ySBH^W|eb1!tNZ^Tm3nLg5mQw3TdU4 zg->?nTlEI|Ws<&>Ky~yLEMFUt=A>J>J8#!)y3>B{Obw%-k;mwv<TWKvtjogHY_uMkKM)zij*=vs*kM}Y?;cYnq2>1DAzw~G; z!%-AQjh`d+x{_jWdp=)K6tQ_%KA&TlokqDzpQ~McUlDp+f0MbjZ_&-rK>${k!Yh@c zQ9HmhH38>u*lMlT*Zql93o>jl|L&vV5K1S~)MyxU;Lqq&PQcLMRZNii0SLxVUgf7U z`+;B>g07Iz)hu-N!>jm|#KZ9n(+mvhW?r)6wZ=!Qmm=Uq=v-;bsDH<9*S3)mk0fEhCX^?`x3 z``Xsx-dwE2`yMQ+FJN<1ICl~eV-y6#96rDcLD?owHact|L|c#i&uU8h+>vP5->Fz$ zLBcEG&b(<2he!3x``D!l>}n}o_uEmu;kgxXF|$)&Bk^^6%fTa=w{+4_U$VuiiDzBz zeyRM8*2Z!+i%+sw$#=#V<-A%-r0Nem<_FQGoC766W&!TUBunO$-ev(sr+ABEb!VP$ zmrFF3$lMD*s;pdqA5GnRW~+v6IQ_~$^IuxmPgzC9h27YI^*f%CqC3z z#qPEL+~C$d%`=H4ZAf>Uy!=#bt$jE0>gZn5ow5ergirMT#its!^v`+DR$p4Ku@@fI z9I-32=NT&NxB0n_AnuZQ+lTV?w2pkFfR;-tN_$-TY(Gx)b-Hy1a=VCrO>>y`uCuJj+0F{F;hX=%}smc=H`yY9`FLFyuV z9}JMIhASJ_E?!nl9=O_Hd+_R~e&4m}(2u2ICA>x-F63=4MYU}mPlG60w8P0^z zb&6QZYhTDSza#QSJ*b|@wOo+cwVW0FTqJfiYhj`uj&w0=aN2M6#2R$m;cO?zdN5|N zlxXo348T=xIr#YLjc@N%jP+nfa{2&CsouBnAm@(i9WnrGi=9d%6Vrai>X%Fxt?Yed zU&dRoKI6PXwv@QNt4zluKYPyx36mAC=n#=qGpEbQofi=wFsDmL4KAR3s@8oN9DOry z(Pb{A^zHj%6X5WSMX%yz`#ppEJ6QO~cuZ2xw((u+VgDOO57r1h%m%SbA3sj+eD3LWgb%FrS~O*gKm|DI)B9{jd#8 zsnv3K0#Oy*eUU&dUA^Ji;+TMojN?TVLc&D`eQ_b}c}TrI1JdQd7MIgi>~s|h$?7WZuUY4-$Cbb)nqaLBAOoZPQ{(LacHt$em~FBw*H9I-c6vz!~9bS>&~9m1iqAnhX8 z^-4n}cZ+78r0z}6CQ{U@D}C>rSiq1}zt`@7`DKZZah_g$JRFV&FvG*%I+O3kvAU?$ zp}6eJF20#dVlcPM3&;s|k3-Ip&uHn{&HFbVG{KhAd62rrEw$5Ry-OTuRgsVBo%!hl zOw?|)a}OCB=80w|c;y3Ia<`j?>z0l8D3U#1-o!SG(GOl%w-y?2e4R%&Ku{;fMF0JZ zFF#K)_3TIAxs|LJ{>`1RX~a?7dDsJxJ?lpb;O6g4xxqB=n+uv6OMyaHQPM0_=jgD45 z3L|NVX}i%-Z%t1BKHFZqxj#I4`D3_%=YGXK_r;T+$B}wA6_$HZ%^l@&b`(E&tr#9<5n{-mn(yh&RHZMpxPuG>Xf~jkoXIxxNWcl*k z$9|*D=bnL=xE3NAH*M?WIk!<4676Fb+>ZCw5}T~B(+6zyws9r`Ldd10vl+FbQ}v;`;}mfnQ!7u#UI3@ReT<`&eq#9#=91>`HyA#cx~`S@HX!EM2y zI+(-gsM%&~pS)yIUv5UD0*mr<`iDCD%%7g0dusTAsk39tYiMloc-b!H=cs<$BASJa zV|FVfKYi-Fu!eIKd@p-!I@x#+z#7nvNZpq+X4ie8hT!U}wFa}BIVRo0q zb3#pGZ7-)aNMeGok91$;K<))+7}7Re4lv<~85=eAlG*z7wRA(hlz^FooBWNHAK>Ag zC>7+h_g1e z&bq5xCTHde_M={P?xhcdO{Au@{+f1!`?0@h=R!=b3Zj{I-vQN`pS(hopiTe%VQR3U zs!LSlqGJ~c0@){dn$86~cKAughrV9IHGCX+b&0G+N@H?;#H~K2B~#mdcRM+;yk3Bt z(ePMcT4ZIXpu<*5;5poiAwsvyq@P{tU_am#l-)2F{_|kb zlt5oJN20=;AxJ!TaThMGcFj1}R$|>(6|p{Io;p6$E!1@IWxbCreR7~9KfMOz-VEeE z%0;{yKOadrRItMY}8O z!u7~?fA~n*iahUCB;Z`e5e+4RG23+?#2fS?axvD0|IqQ+VL!=W*^MHVdcz?1eUrPR zVS0Z7DPq$d5E+~o0FljqhR(|=!SZ%K*IG{Sqv4;+!3HgkHj^lgZ*I+Vob?}P>+hoM zN(+~5`O@CisoUbxtmj$ylO(ZeZt#7Y+WXDH_q^9h-c6e%a<4tDw~40yst5-iQh$-q zW6+8>KXM0@;dlkQWchaU25U!s!E0%a^nTB?2sdQMobeoJ|)V%pv2eKhcNhAo)ESZ<)}tnA`X+l}MnXQRN2b z&+(q)i27l+(tP{k4d8c@r_7I-rq_$e4LCPlf3v&W*9w?{`@| zPM$rfJE^8vN7I}J>E&8KAK?+!fv2i+LKxir;{9< z(|k7BJXT%DZt2084CQ3}v zFa338Xv(p1Rh5Xq^EcIM52{4&M$CB{;)j&u6KUcpUdak0pon{X3yf%J=xMh?L*OxU z8#|LCo(}nA4*a`{CnAXC22WhSbmH|H(_sQW6)L`JO1UuN>ny(%NJS7x*+HW>6~7;h zc*3*Vgy-MjM<~mYp8<8!#XzNs2*ey9$t3L*%V3r;2-RiT-Xd)so zy76-#;62NK%yQwb0#MVM4TcglirBw_!KK&Q82uODn_fKo?cn}NvHoN06YRuU5hs4d zpl~AM6o`RH?L;NpW4(&U?7AnIIerDta$)WIg|&QgF!=0=#lTRZDl1P=C8q>3iu}yz z=p7|Re39IE&FII`1r}NCrskX}PBaczwH5vvps_o-VZ_M}SIrgh*$le#m0K-1XorZB zRj-~@!+Tk5QUChy43w;LR9^q<43ri3zM-JRp#wnnra0sMl?aT{aXle>;?a9ngL{ZU zd@Tw&O1ylQ6S@RhVJ9fgJ${3CKJ|nZfWQK;p@-{^CL{}4Aj!dkCn*^jq14*n{{|4C zL?F`B(UK}B|1prdagd~pdt98MsK{z+Z#`JVZfcOCh|oz<{0+hxO$=Ob@I7~YKZg-4 zFK=f>5G?m@XW2oLNr-TnVoy0gGl zT3Hhc0GCJ>{JTO>B5m-Kh4Fgh370uXf+;A=N)5>hm0O!#r2*Oe{G_j6f@HEV?(6bm z9RaWeNRA7TWVujQoe%|VI9`1zOz6Owpem`$K^uYp3#Nz@6yyLVVCCk;)}k|}iCm^8 z1s2yiAgOZ&k_uN$ojGX-iAIrk*Z|V^IZ|r=)`H;wXd(A+EhKX!oc#@1N^uwf2`~T> zU{DKTu#CfC8DJ2Wa3|yqe&?O5*(`YGIq~Fij|cAwydmcshk~R5%W}K#2`Eb8U*bRc zYZNBq>B1iuTFTkolC!xbr%VMjf{OW1LZ;n0m5RSp6MtK~3R(a(c7_o|)x>K86FLHg zMM%M*n?e&(9i`lA!%0hdhDPmb_9eV*+yW5YrKiOf5&4fl;9M+Ah>9RM zi_`EyN8mFBpAJV#Gn9wFj_?7*wEyf$zj6SQ#0Vvs@N5F)TgaWOtSorJ#TE*zrez7L z!#MpV{|Ei1MGy=4?s3qPp8<%m0{@FGDsmr}YjwN7*6O~i(|Q+2^?ZmXe4zCS;h!?- zwot>9&xyYskQ4v<`mZekB#9gkYzc}dnS}6qL0jpD=p(NQ(Ff!fs0+iPG)M%|N0=}w z;SSq#+ybBoEC9B^0#m4zKr00cKwNC35DkCjAJn5F0mwgSTb~K|?D3PYlF)EM@dHI5 zeqam44_pWF1L#5g_c&5$Z~%e*47YDEDB?2+L*pO*+lE38{AD)*ntxLSAOUoLmjEk( z+TSJkHwh5>y9As8*!?B}xj1~aarhSC@U?*O1-u9NssenU{WbwL@!<$l&J-2~yy)aV zhHfuZ<3j>W)vQcatV~x(9_ZoeD-aeHoe*Zh{wS9l2zO?=me7<_jruZQ?E z3x}yCVLtg8a1VAD@YN8J&HcwWwOhe=V)yPnruvT*v<{Y>c+U3N+>-FeffugIfbM8z z@W5TlGvL`mCAkKCid1~?aA$c1epMJ`%T58hFH{C!=?LSGQ)YdpD67C1B2A#oQvAop z_stYMeX8|C$%!1c$H97d#NE(CRGN_Ue_`PD-x%2WHv@m}tYx*Fa^QGm0x-b-4F;q4 z|Hgw7$b*0ENXyFr!4sonlv!U1$^<#(nDrnae+TlY?%x;?()sO{R|r^4$pu+Y-R?TJ8wCJQ9k6U}jZ;wWxU1f8z#X7%IW6avrz_Eca7 zbRuAe{0yKd=tSUP1UeC5ZhJ1&iGH0P6|a2L1cMa}=->s;9>Ah&)JTMy2BSolhGLLfYuLpavyHoq=0 z^JM(PLY-UnS37jrQ`}V|x`#C@n5!-9Imcr3E4Ha>ko8aWD}*0RIgJQ^jW2+2ST(FX zeZ`TJx?@^hS$L?0e)@B)Bx7`3yeOk|ysjW4lz6ii2Hvb@C7Viuxv}o5y>eUH2zl+d zw};_9zI>A!1zRyO4m`q~-5OUlPa8iNikCg!k3Y7LJQir~JmC2$RCLU&wmIOuIS{*N zogOsqGiP<=m3mBHacT|&a~5KGVrSp+(3y-oqRsvj#w-{8-h3=*stgv?yfLt6WLlM7 zMIuQTIYkxVWYClO*#HiP+ zh~XE`El!_an0O;EJq?I%IO0IPU3+`vamhTrbI$ft^v|9UZVmU5gZ=lrTJIY&BoboK zK>jaskfrbMr{T>~8-Lh^KS=gbyu zH{OS_Z$F|Jg+1NezE>r@p6Y0e)Ak^Zn!^{N$a0eWP}i`I5*S7r(?MYF(5ki8Vc@Aj^sV%e1%>pDt;EEF zYR+(;N^Q~f=x*P9E3u1_h%~`dd7`|Nfq{OWn|tW7VpOfQXy%lRmw{xGmO-rtJby^f zbV99dvSir@Mf9U&l36Ny-1(N5ljnE)lz#PSo8#Bu7ikASe65k>Cy&d?BgN9gJ8lJv zEvJ5uh(q?%)S<8536#zLju8aY-1i+;b$K!Jyik8Jhfs;_nMZ+co!_<=a*zYg15Pq6 z(hb`OJIkF2B+4n0xp~lWcT@C;ezLk{m}}(Cq-%8J@Wi!xXGsge)q0=27)5I1$}GB4 zz0m|}=^y=(4DO4HWcAyrr;X^a^VwGuoGveiHE!3n(0>#*!=jgG&#i5snh|gwFVP0~ zmetXyC-6wIUh9KHF)53uQ=}etPi0!f8xDal?|?5W(!iHZ;LAtI7nRr22*(_XrRL|` zxr((di=NgWr&2%n*5pbK=8SOt7`CVKL<8HHH>ZvK2ZNoO7jE3|TIqdu7)QA`d}V}S zwN~(glV5n~^g#aZVN~?=08!mxn6_*)`L&5l^t9{;RrA>vKJ8@=)bKcuQ|vD0-QH|U z_S)l&R|}BZ5ne3cD+FYC?89RQ;G?^ZQBec%_wO6UIrgXvCZl?`vhp3&>Mm2lC^gsh zXeslkHC=qkp=sTRJmLM6`@TL4e@K_H?lP&1%Xr)7?UoF7S7gfDp}>OoU6nG&uT?G_ z21?a6%vh%$2C|K>S{F_RUTI)dzk+3^Rz|*7H)6_|PA8XY@(J(TL({(8L{=JTIKvvu zWRk4el4FaWdZ}&nUJsP4M>tz!rq;3>oOcHg17&B=&!uSY6{RI0XEdgZ8m-KzO)6eF zH7%IkH{LA6^7wqV3<%1FC)U!ZPJY)bgnxCK4EtztT;HFwvgACOwS^ojcJf<%*aDkG zV=9*>1KUkg`&GYdFAi@HRj)1B-&is#Q9Q2qr-Frl9KL*?V!~7DgCnoz8kUx@1*Z83 z>qZeWnwdgA>$EYIq=)VHKOSqsO38LEC|s^#n=xLDuMyN-(<46#OCW#bJ9wzhTR(+T zti>#y>uYvDtnE4b2&L~L`OIDiP^KSH=Hm6dq{+q2S(d(7!`W|=B0Qh{jTLM!X(%i* zQ(Hp_r2~ZG!~sIx0HNqjq#!9MAPj6e(aEMzt$UZXM@!T{Qhk^sj@y_vADvPULghzI_>Nk>?(%I2 zaPq$x(`S3LSls%}-c}pmM-5rk@cqr$Hswj>RrN9FQdqokhe3GfSSM4qf&Q z%Aa`Q-M}U-=$>a#TDFhAZ0F=hdoxfrG-P_<6ZSBwVtOE~{!rXk`|P!dqB`t7?4L+Q|3Ix|#ByC>GU&$zjp=e) z?c;J#keP|QpdU0cI_dAx~N`|Sm;<#mB9R%oQ>tAKYhvoAfSRlBY&os-NXt!+B% z*8-CtUPV3H#JXvEcMZI*+2;+qDcKaZ^EE)HDz9ux(xfHC%7;Gt4D$YW|8rzC#E6V* zKBCK)i2=C80Nkh31Fyef+^+8)UWRUik+4t&M9k{h@Y;y6dFhiB8I1gMtEoT3>!$ky zrw)>GxwguWM}K5Dk742(e^xfP*Q(y%sF>Q{*Qk5&N!tK1fsQ=t&u*ggR25G<+<>1n z=FKI}OCn$M;cq|8BUw-Ra&fk@h1T!jW3Ax&SZY!35S<+ki00~a8VHo67+6NtvX%r&n)|lJdQYvf zCW;<}WlzV_?105Ht&Vq398$%sBDsPz=Dl-FKOWVWl~6V@!jJkNM$&n9esSb2H#{7X zP0ldx97I`CO!kJ&QotxX7!7)d>DpyIkLC^TG78W+(UgqVp`$X8#t&=v>B~2LXeD7u z{mP-5z=^d;Hl8(4kVOKP2Gp%;fQQ1i2*&i6e_&{0nMT}r7*r~wu<|QUv z$qoa*IZX2HV71iDfd2hNMbpJW^xcxMeRNp2~Q`gUr9^YobUg)0p zG^@MzD$yX+e9xJ_x~h>l;971&o8QBqwXC@yrS*HIU0)X!Re^VZPQXcoQxoP$x~L_J z$i?uDmt@3baVTe?WcH7mr0XStww0f?s{5bWC|SH8loNu8 z$louDDnLYrNi>D!AtG(Slsgdlz;lt+y5F)jpp}|;Wl&16c)EWV1pEg*gwPy(Pnw}YY)>f3!sQ-z$~jCzFW{&MsvGk+Hy z7BStQKe5Kj0lIC^IoUU}7*-A=H(`!ygE_cY@nxrSSh%{EbwK|)$?UnBq#xk;7&AS9 zn~K}YM*veYWH$5%-M^L_Kse+Yt$MwPH=47iPpU4RJlS*y_a7b zl+h_uV?MQvGBNSAHpRMD2-f)>1k!%$TCKgZeMZtM3RY%rbpS4Q!9`3|*Qyv<%IH=L zjR1VD_55C^*~{8GX~9*}rIeII&$9$A89vhoBi2@V)CEn0PR|fN4&MiY9NM|mS|aoY zsJJ63hu_Wkzf$Zfx?I?jy&iZOk?~wubQ#yzlP9JF7q$grjywfpn#~1cR?Jyr=6nU; zlP&QU|Ja|j?Au7}J<2JWcg>wvbkXxdEMK>n+^J7A(<3Wy52zb-dw6xZq`Nt+9vzjB z><={&J&1ZgJ&+6zQ2RfE>|EwXqw0~c?95cJq`7C7xm@e&W{2nMmZJ89<3vp+Trtd} zB|e!Cu;{RDtCpkly+JnaD!v%6SIbT{$#4o3J4#%omf_qkI-S^Q*JtW5P?b>hAemU? zc)Q?RXE{IR?%}72Q8B^VMl@OR#4Y1yE>v$%+~I4Z6p-A7gFvhPRcmN+at}Hl$uJjm za|-G=9C<65qS7F~IvJaNQG7zU>!o~4hIr%5EmF|K+CUFm3t`j)w^WCL7X0V>wQQOy z<2@-4HmJ1j?^L$j{Nai8nqf9+c<UG8KMrs&ZVu!29UN-J17&Xk4&rX# zAJT}M@3zNPPBm&+VkdrX6b6krz)rx6UvD$~*nfrX{c%ZJgIlb^@Lmi@QUYs_C3ki= zYF>H6M?AvcIED$9l--4z|K#GcnZKB}E%`OE;Wk|I?V*!%%YlPf+yt7hBz)5O)>*F? zN#;McSVGi!AO6cE!}~CKpM+m%n*C97(LlS4^yQWr^!RuIsMJA6-ncC+tx5}Q5KI~^ zF!F6It#&P=rcq1Y#@DF5VU?F~^K3JuhSw}{+&ca6X_D&6AkOU9LV0@uM;Nc8XS+98 z9f;tRdySYqRByhkR>tV|a4L39D%?6>Y!r6$xls_QZc1?G(~f|k3)HAi)FZQLSbK*! zR@`tDbIoPF^7|hbf-8T1f`KkjkLFWN!mk#_$!jh}dKSDVnSIROQ z90Mg~btXLnee2Nk+8-}OrHgAc!#pE6uge`edXL>ueZho&xa+}{lZ3=hntUOq*v2{Bp6v8H5 zdd+LX7!m*zD>XpWl!~&;y3LQBohKb!m249CiCq(KunLm0p8oOfs7Z9aWQS5_IHc^T zKB1&BxrdiT0&&a!CC`Uk!GNt-MoBWhJGOK`?EOd-Em7H7DX|`7{OI8{vEz@K38D95T-n{643#_rcQLTG?p5om z1L!(Lw#$uS%6e9Bug_^|Fj4fGvrHsXm~c8i$%OD{Vx9F-3fYLifef$be#+vOc!Sab ztkRw|`Wl+9ZQ(Yy^9AsSeGoB(l)VZ=K!L^5g%nZtoaE1ABKY^S^vY;oB-cQqgbV!>8|O*z@zghnvVvWNzAh& z8>=k`dz{B>wJrX8b1=k6%fV{W?CR{7K=nf%rCkl>*H!b3i8U?^U8VBs<(G}hldN2b zZ7*)}kWM;uOknOU=h}48D6#f;#|Z9tCNnSS!m&3N%Iu8JrVSI`vGt82X%*6-dhy@> zN>gTo|KJLV*w(ej=l}B$fe(ZT4Z*lVl3aiBzr8!am5R>;Mm*kfqJ1dd+yC}gCa%;I zRNeTbZD*Ji2>;`cJXC6U)L?Yuhdes*_I;F7!Jt57SfVe-+pCJU-J7i3>~~Z$g=}wY z>TjstQHkhElxpe<`5J@v?ukX`(<5Cj-D&6&y32`nx4W627c%}uJs5NTlnK-H2frQuOh1F(chfLqSULQw2<54`~u?v6^QIIMrp{h9s8*eGct^$h zzgS5AI}2@pWx<)RArqkVNUJ(u8FF^~g?DhVz5?r%={u^yFc6z1pC4FYEK#7ZbH%tq zH&W!^xlo}CH}#Zarfm$A0W1D1nVfednZZHit(|5Zq`QiF{ggpM^+GDjaNzZubCCSyy8~J+e2?y)*|w~$IpAS;Qy+}!N;f7)*{%v6)U_Kaarod@c_s- z|bQOU4Ac-!90W)%NCaRFD-?}gJ4vn`gws9lIlghw zm;SmO=5gg5myh6P(4)O2vdWr z2bd0bfE2E&R6$L}7)YD;NL<+s-s1*4fVWol_jN!=a&JW)DMunH@+V+#!ZqJuK-G(l zEP9RpxRwE08(=$-pk@4N2C#katz9?te4IDX2ih8tfX?Gk)hOa(oV`61-wDO|4g681 zig`_e;8$aRgGUA$2B-)N2nN7*;5K0UpUwj};W)Mf1h01BFar8OQ~6gE|6$=v^zZB8 zU(*1f2VLpUKmf>a05UZWMQR+1Kmrus;83i>p$IbI0=4G<+IWBU3;;CX9}?FyfE1ej z(+t!zK+}HscUgD_HHH5n3uk_t1rSY$4^*!t3<@6v&yi$C5NcFE&C`dra!twE8!ux| zQFlW=1GbOh*ba&eupQK%T{|4dc2JD)*kb)C^5$X2G_uhGt@FhkcF0u2wcev;YuFZ0WAYp@^5e@U!@D@Q~!NM zH{m>-!iC_AYe(R^ki1k>mqe&z$#F=4>-o+ZPzEFl!BDnY)`4c=fRHYUa7W}q+RF%w zwM0tW-yPUWWx*3yjGb}ykgeR9t@Ov3!WGofTRU37c}O#k(0}siSKWd34al4WgEq0h zKC&(0*a+|dJ5XsYRI{n4n|*Bj?a#!KNIkv-2y!xpZ&O2)zpaITAvz;m#<_;MHFcpoQ{y7#iCZ z;Zm?p`odldn~vhSmtpNmv~Qi^Iq&<8&HIZF!@fLfxhQC9j_=vLXWyELIZv&RyIAb zyMGu}GCeQ~rXpnQdv(M@BIn9cg=n$nQMppB$z1A=*5Yj`4D$t(y*o34Z#NE;v7E#2 zcN!92B=NMBuGXf=ClR8cR4Go3LW^b;!({s(L!c?KA4<H!_6>K} zvlu4p6McgjRhmA;=8S*q(%Q!Iv#U_As`lKw^86>|0@Bl@`rxqE<1Jbp^~FLuWK%71 zdZ2p)E3OwPdkv1}eOaw~1D!e2eS5|wZBX}qvRRXz!~D$v%k>fw@n^`3qu;s+M0WX~ z1j@2yX$at1=A^6PA2l1*sH{$pT$D8w@5b^?(lA&~?W6WZD!3HJdc0TLTWU1aJMFRG z397gCj9(YZl*dukY&2k-cgl8X5^vvWET#WwkeO$B?lR`mKFVwODs{kA$gZU0hpP@; zGBLBqZB&M`HG)kRBr$Yw?305E?)L=~XR6hIc5uJnpszjMd4IVz;dKAO35{CJf;8yU~WOycs!rPxU5fLT<9k-{8&st)Rv>kneXS>ZZToKeH|sq zvd?PPANepxVI2L<_iN6XH9E`FtRiWYLdneBchCPZbz)_n@iXoO)oCpRcH~7;z6PzTq>yWE?U!*ru?~Hu5=QuPMOlU4oxdTfoaQ9K+=qg(cAM(oB z(_~ono7T{3nCzIngv}$#p1a)=u)7gsh_Mia4e(A?FhygylxL(#UV0&d+b6e}EJdZ2 zN7Tr)JO+Y=<{g8{{IZb6Ydc3;9wafZjFg8wxf|UHl)nZNt#@0-+tyE{-v&SI^%MST zpSO2Z2f$B1_~`>bz4JTD@MIZI?tn4l#`wg;^E=n@_uqB9=%uhEC4{@`jktsOQU~B) zp)d#FLiLSTQ?zM*N`|2MV_{1HxAYPF5fa6hdbIQQ(8{xIfeq@LvAsefyK$dR$!d*# zyCpz(pqw`y`z&`)>t^W{@Y{azva;{V@T(@bVep>sbw#R~@cP_-l;y0yac&>#|55jr z(UJ5_x}e!@Zrj~vW@ct)W@c_Pv)jzf+-7EGyUonZWoBl&^?vis?%bW7J^S~bs>rA# zrIZmAlffv^47bHXRgeFh!nD$<12MP zJwL#~nUNR2Olc;0GQB#?+t%IVvF?LlWWVy_m7aJmc{A3K)1-B6>cCQ*8yyI#t)AUw z!MZkJ_aQ}sqbp2!ZCkFLqUz#{Zm73Ex<1vpe z*H4|VPTo6BeRlrm{rzO_Y^!5iU8j#d7EkyFUelGP@s8sA#mx%!!+^1dI`3`^^MWn6@I^E8itQVgZBH)f(;KE^ zPp%-4xwhc@<0D={@UE7uxuKNDN*W@+qwQzkQ|I9y7B2wXy=uK%E^~X@(&=ry(M7N) zb?2VXvZn+l*H2yrH^H3i!&RnadrEa-vkon`FBbYBF6 ztF?t~Sqsc~!(!~oo#$!Ywy?nHka2*03%q#9EIrGF3so}ovhh4<-ODQ=%#*&pj->Rl?7tWfY7F8O3$p)H|2diY-N_HD;dRcZFiRz zr-$mvN4l9r`DjO}jP{*I7MD6XS#A8R{9?W_7TN|+b5a}mHP!9hq-FV(3;g-+YZ?Jo zYmfda?~|Eia(Ns!nPngBl9Ap1#*HUKDy4SDpKtxK>o(`F*VBBskKh+qGJ||HjtwgS zCg*1l2iBL1EUXrvY@(iA<%37#v5-71zjgFo)}BM2c9*V0prr=|n2Q3;1w0x{9e$h% zw9qrJc8{2*P}%x8U)qDN7hA7?U*n6p*+LZg5Or*aQ}jFO-Bs+OMqXiz8^3?{XfDMB zZOKNqU-uB6*%{_F55#}ULhuy>+M0tlKT&r17WFTTrHZ%$9suPyC1zIJ9Q<);2xi0&NfV`p0~8`m~s zDRg(s_FacffMYCgf9jlktKHre0{b8)_V12Q7xfeZ7q7BR!ajTgLn`5B*{iAVRaj+wrB z85ZSr%*y1;PrS$NDmKLH8a|PBTEeA|7A)S1)y53=$TI_LE+$}sbo!g3}-aeem{&9-;b>!SNwS06x`SPCB&1imCL=NFp z)Wp~D3RF$er}efh^XXKc%5k7&oM$7nORSfjEfuKK07jR)N%Svq6*`f;(=AlU9$ZED zl`HYS6VV^rSV?M+U{ z$!hXNw#MFMGWAuly0E+6@w@B5oSpeQ)_a&{AijguH8C3ylE*lJB;@v(fg8)W%uh*!F@IeRhEve0IV4u)OY6PTw-TF!j<^ zrdx7R(2>0?bL9K&fLn7+Snv%Aad+$oyGx&?-KkNukT!mlIFMw3q^OK9-B>&meI-(1 zid1>}c!`%buLam`Btp}(3lCh@HrjbeIRW;_MSu6hOCNpKd5fL*3RQ~6$@iR9+xhK~W*p?bC_+?rB z=Eb*Nd@br_SGzZD{Yd(5)rIcjoW9miQUSVZ#mlCqj=D>%O?>pIXLn3heDsN@w<$@7 zbRS(xS{HI0$DK-2_*%b`u<@)dWDVnpNN!$^PB4h42y3} zoy4sPBOkTXY|zc(600A{N!^v)WKX7>jFa?dwzAJ>ug0hP%lPVl&tCUV-8KC*80SCJ z-o&YmrQK)!D%6C`G7n#$95AY1Tk>0Pyg}zL>X}CA&N8%eQ(+!F(#3L9N1Z&R$6l+w z6vz1X#skXxJG#uT8i+jjs1uGFm^|d@6OQhQ{>AQyJbb>&_gSB3i!0Od4L8|46UIJf z>dIPN+U%tDB+Zr4Wh{a8OBwx_)%vS}Cy*k5l#%%=P+szpDR0dTcM!+BDZdzei=45AR;} zbSm)E`B25-1yWlVcJV9Z2f-%&F*K#0AispS7r%sWuAk|xe%IRrhqrZgYvT3> z!oO)U6Z_~v(Um;H*>d{lYMBY?)OVkTz>!O1rHyOrNGhHjyMYeo; z-1)Urrf_=v2XM9QVcgOq&Ze?F_R*zpHQ8j%8dZ`4TqKe7;9B0UrCAEVTC;U{KFQWz>A-*<=y$O)A;(WJ8jr>hScN}N6mB^VC~7%<08Pdcp`8u-qtBG z$D$)U2YtnQky{gglosE8>OrS-rT3V$w)sO|HW&CbN;hlC4hO#6ssN{v=Iv?ce`b?) zeI>rZm(z-yJigAja`9zLxZ~uG_&2t~<8FkLHalZ09KOQAU12F8+v`RuJ3f1Ie^jx) zswlbMHHUq&6JJ-lFR)sE_2?hYYpYiEJL)4hJ4$eHx@lh?ck`1Nu25u9s3`s^vLm!0TV=r z%B<&S6~xeOyp{GY4WwNPVeE5LF6{Gz;*!1?k;dn7S%t_RZIJ zlc~4V@>%&yR(D%IUx{6H^Ylq3^^Mc(A=O7xcU?e<3OdvJ(q;3eV%7fAMT{rK@>$fn z9*1$~EvuTx=9sNnRmdF>5N$Adhp<%TyYkg-JAGvb4*!sx_VuMP7&P4cVHkNn<}sFI z-r^DC*uT6<)8aYZeR+7oS-w@?E2v&$@q1P8q&&8F_#)NK7Kxw|Y3ngFX0O$8t2m{q zAIgii?N^S|DUm+ zjf3I8x(TRKTesR^`0n$he%$t5v|#oatSF9Vm8XT~9(;{VK_c(uOIW<1j@AqJDrG^e z^67cvcw$zfW63yvoTJ;$8t4k_v03Eq!t|LJL|gIE#i;yjY}C-ayX} zUY|crAIBe!2d2s_r^hAT)>{tc=#0eR^y6T*F?9K8_YryWE;nz;%fMG~%q4`}U+171 z`cR0*AY!@?!~;BfNg${Z1ZJF5xtEn0Vp8)KrD6c_&V#t%_gU2MlPt=e2u-GJWJv8B zCIF^#kWL80{PPC<*R3fM){uG-Q>1q zea;SKn=#zal32RF1rPfV$BQ5Cro+EuKF_PlfZ@y8g~yef{WNaXrvT`lGWb*9b!y(J z8|^zR=Ljp|mP;p?4t4=5X&2HK#zf792XK_tZSBNpaMl)OLO(Jnx_ZcFHGGOjy^Tm$ z2QkbDpWcMD)a8Xp)GwE$d1pO^8iO5)wM0m>Q;C2C+Ure9)(*p&CJPKE0tj5L%K3^8rW>&*uUfn}IWl?^tc^>Yf z&*t4Gfb~Ce|H+p%HL)}lvU4ZU{s(Wv$i~4!$G}0r#LU4&$H@k)SQuE|)YjRF z;GdW%z|zXjjxNT|3WkpVN@M^AZ~50x0>=Nm;=dh#umexG`;U+>|Hp{N)F=l_8s{tH$0-(+O^A7y0xFQI$>kLgFofAs6W3Ca9F3;ADy0{!1YI{h>FU4Yud z0!uG#Y2rkn{jX{X{u$Xy1pjot62ZTkM@bi`NdF%8jQ=Rv|2FQ04V?|G?acqVxH(X> z|DSro_>V6ByTE^h6=Eb{VrAn5cFF%&=pjZHR*wG-_IBY7?fzr<<%wIn`aF5_j68cR z8!_Bbh@2$IS>sPp7+EhtZB>D0KtY(TC*vTf_4ihTK%3gV2$(ut?<50%YzG`8>%G>Y zu8>v(d;4(NTIVM}b6a_oo&8*%=_l{$Ctmr(#-8l#^i7ZT&bz7eliwh}aRouX!cN!v zcAz>5f;FtVy=7Vp4+n8uc3mL#mi0O9MK%-KRcrPhvOxgD%7-iy(9G~)GxYthe<`KW zUi|vxy^(6%KqWlp59M>2op;vfL9M6v9)eB~;En>30zfpgYrB|y9&at#d(`g`1zZ^g zW($~&oKkoh9=NRa@A!US&SENK8*TSxWv_hVG+(iT!)2)%^9fvtbzZU`y%Rh>N=pzR zxIvj>6in}Os2^;ZwMpPe7LR(}e`BwR#W-{Y%^EK0&C8gf0gq&#g)fn?1VeCH$Qual z^%qHYWt~$P1>Qi0toIvA{=f_bnFw=QVwC|Xb^gzN6S$E-1XKpXjsCnZs4IQp#|Fm` zNcnJ`qj#mhBJf58?uE#VpiD@=K-1%*j{|WAXhuXQp!{xe_60x~(Qm(h%uv1%Ltq>V z@)>=H{mw0DKHzc~|{dV-n42p$8r!VVO|HToF9d=j<=?2aSdF;xiFgnITh zkHJ5X-~I{=P?d_QAb&x97<@PK13Eea2_ZlX${Vf8!wMlp48R)bW599v7A2A6pJ&er zV8960053oeCC&*!R*2+486wg{R40fDphgz&R~@bSgLe9TSm1{PO_X#_0JQ-RAejO5 zhAJ&cW0(O}oxCT=VdghS&Mrh+!aEq55WEA5jA$oTF)4qjXIjh(NOV{trksCz7-L%U zJACn9oB>M)bO){p;VjU*M89CMRwxHD8i6cGIArsFz`kS}xb*iUrYvY_0YV{>J6Z#SwNx*ZH2#ZW(uTAErU7{a&UY{e$o}M7yc?~R->tCK^1kpDyHEqH zwa5;%)>6I@%Yvu?dILBBS1Ymu??%%5cQyiGV*g}X52&r+#}Vcfff104qJfX(ZQ{q<| zx%4L{o>43IBk?CJBEhAgH>xi}=y&{(VPt%NvjK7B?cdwIh67#p5I2nOL@!(eyOsk! z@K62BVbD7i0KsS7Th?bw=)!{1|j@51DJ9R|pEgZHVge{_aD zv-F$qKmZg6`1`JQHo)DYtZvVZ^jW=F(_ zh`dGwNMGCDl5&7v5qD2q%U^>6sILugMcse$iFqUN33}u5iF$*7CU}Ex1to_0h4c`0 zL*^28Bjl3D_H$oD0y^wrZ+M?!Z+r%hcELFUp9#IApOO1xcdc(@uVK6cpDDdld7%6Z z-BG&5->KhmZ?-9F}7P5TM<`}*Rk^?=+0q$rwPuB>$ z$UUadxX9mo$c=6_pSfp=KLy~!_)&-foJ6F4&|d^(19t;a&!}|-&EGjaerSaUq#2}< zu!(@gOcA+rAj5FO9S2)-RtYMt__G2m?-!iSu8Zr8Q3zBM-ig}Ea8Zd-b!Ug`Uq~EC z78AJRd5E`=&}=1csre$Dp#Jcm#wC~sDP>W+=9$_K#kaB4vFjSoAQHN)7XGAPm}Zglfu;;ZeCe1eDVoPUhE__dGCs0N`(^i0Ge7*?h9s9z z0^=sR`S1*igU}=H0WaLcqvZ{ph{sQ7dKL`ve{nZ{pbC#2Fx`h!&#>3@b|{^2kr5lX zIIX;>(RBPg@W$T|)Em|`LH4}ly+E3Wbf}yhXznkI9iDvxd2GTm%j2`rA+Npp?X>7} zyk7Id-uB++%shqJ(my+p7;abhgzA4Zi(_&)2tF%*>5ti! z5WbX?wxX8T=TpfC`+xcHr!DRBv?Hi9U?+Gvn69WE4Yc}whnHFif8Zdt>z#&RyZeM) zCUL>o-3p|h6ype^?uj|E&8p$=3oC>&?u-XH{fTX~H!M25q{rttaGhZ&914%e^2d+A zsoXAiY}`|vYRoJ7frj=(5r^(n594&HN%|<@U9PY5H$OW$_jqdh?nA4tT~gagr4pTOcSHKbs(NzH-r2ei2C z24}e~e(BHIRyt>g#CQij{TH_1?6KD`6?;N_pOB%f0{~sCGcdt}>F(xNiY?S>90l}m z&j201PlQ3LckHE|*ui8)4ZQ0jsgNN<6 zEf_LHb`2K)fah0z5`a?i_&mk@B}MEW>$v2<7K}v7b)4{%a+n`AZ-Xh5hThz{MZmYdO~fvN4Sh8L8U|s4=q#8Nd0% zvZzJcDAq=GzlRC``#uu3yl%vA+#43*8SD7h6dg3B#Ak z$4C9VxSwa-g+neh7tp~Smt#md0r677h7xNcs_NtL#I7pC$B2ef@*FO9@cBeAQk>Zq z$t|6Z#8kouJ-&B>wtd*weY|RT#WU3PnI?t0S4J*cbU7BqO67>euWO(6b<+tP7#MDx6y^w&WCZKX*3H5lyvEAei1DvUok})qHV{ zLpV~rqhzmAk~6w7eU|v;J+RcvQmketlKQ7}8WE}b!Y)VJU+Wamm{`VVdwg6Zx4a#8gM!)goug8erab@^X0#lLjaXEK^j zGMd-V0!yjwCeywKi@DdF5@{`-#Q(~CPHYYV#D|GukEj44u(F#mpe~)U^Nv}uC7hH& zmlaHxFm>2vGBeU_l!9N>eXS@Fk>IB>uT%xTgCUpE2APC}TvjuY-6%YArB@IoDCEfw zfZ~JS-AQfETgv{)HbZ+gw&lx}6~x+W5Z^Oz?m7*zNyE-xlrC%@2@bAF59e6qv-!hf zNc9O8jWk$ni#|DQc_%?m9uP|J3ZD9AxiU=O7e6soRvr941P-C{yJH3?M`O>Rm0Chc z=E?XbE0LA+yJnU?ec<#X(R?>kj^P_GTh|acuTu?MtQ(`M<_lGJ=f#jOeo>en{C#Dk zEMC{4+RUg!*Ku*D%l;0gp*zd=IJ`6n9XuB+rHh34!lj4UR_bZdj-`O&yRJZ`lI2I` zjh>X}yQqTv8n4>@)0uBq;>$;{-r=4v*&2tr8qvJGa#2vrM;wc#|(B$BI4O=&K`$9ky3%LQ4>&8GgoJR)l~;!DST2O;(cH!~va_HkFI zrM4ye^@ORaUp}a1omWh;;V$a6tR7XtZ2;97P3XFiOZMYQ>&G(%j`Yg#bE{Bt*skhE zuE)l5p@JK3Y)Gl}8}&nAv<2DQ?dXsqu1Cgqx21)cr$Na^ZmHkU;1W9^K1?u7V!8f) zH|ar&=fv0K&k@e#>Iok@=(coEO!}krKWY9-?WV`zJCwRhsn>AlO%F{6zTzw6_cQcs z1KrAYCxzxFD~%80CnE5>Wn>>FAL7$c)8K2WYw{;{jA9=&NQK+Sb>!^qjY@ zIV@=IZyb-g@$S(%ZJ&LHzTxKpy%ANo%Qp}|&X!dijP(Rg_x?Q{{?vQH0QzLHQ&^I3 z&_W!Av4qvb()7yUSV17DL{I%Ab01fhaUX>K8(p*jWEb^O;u#{RT<)HNt_kQ50aV+7 z{mXEi@KWK7!C~~J*ygW<_Z)hy*a}%tFNE+Sh}})kRHrD+J%1$rrnP`eg0x<#sikG~ z5;j6?vjJZnTD3g%bksINeB9u4h*9|~S=gmD8FNhn7Nq?00rp%6eQYG8F<#ajMX2L+ z`{N=x2g#5T9c^^Lp5n(C4>GL(tuUFR1ex5#Q6Qp{$2MS)997OSqZ_{~IP2)qiMlS7 zMs;^2?f8I$=R;$cJSd1fgj~qoC@eu6ZRfGYcjO@cZh7|YkX$}q@tSCiM7qhJjt>uK zXxQ#~b=j1Ple01^nDZp7h4M z&B5#yzYr@;sc#fy7=j3CaG`lrQM{pKC@3V}PfLp2tzCdAKbmObJl;3Ly+;vm8NLWcIb)QSCZaw9RA?>)K9N{17+sLIzC|-_#O$*e*8s?LxWl^ z&3yh$9N~u#A?^^BaaCG0o)$zw;oai{1D?bHEKE(m~RNgPgRw?J?jFCl2K{K;c0*|wKFuw|W z?Mfig8q8L#z)NosEMwOhJg1s0;B|ybjV?4tMxSVKd}Z3NBMEilWVAcb1y%o3X|7-0 zNk0vz3%#uVL%!9@x%M?a*`;N=>eTzCepY!odgMUQVs*px@8c>lwrXzNvY#WrpIQ7G zC!c}+$-Xw3-fW|h?-k*E{_Iu@LQbve!|6D6vFtCAcJ+KVMB+_dL;tnuF3S1YaQ6KJ zOoK$KaYA7+vUuk3@ac(rGcPwW?p0jGx2e*8Len!Z%x@S!0=IjyJpu($NM-G!SaDcT z%WyD5FGL~QL~LzFs>uH)S&7CcZ^B%gM3vx-@F`NGj_x zeJ=H-HKv0XfRX97IDI?o?It4yd1L-eZ>u4GZJaLY&7qL=Ta%TcnsV`QVuD9EAv$SR zCbdUbH?CsMl-ytx*0TpP^NVLcp_rajed<_~IcNg~^xF5_daBRoI0Sh*@9y0tTw?Xb z;9Q=#s&XGDPkXn<&$J8tgr=RHa^HQ0y?*Lf^~-D(m5x%+@h%+t>y6xI(jh1BgU+`y znj^vwm_}^=`yfFp@(zehFn_%~c$^VUNbZ7kP0hjj&C8^0YXx*SjD;vq>;!Dd=qnVi zUYmH9MbR}7`kPja&1xmFdOd6mI&(zvHUKJk>V<9E$H-i_v1@0S;d-kF^_kmLmrRpmQ%P9ZQenv#jBLJ8H_JKt zWeA;Om<@;7hgLc8Vinn}b@Rt&>Y6pS4#&&q?nG#3tk&E3lX<-EW>3H649$;nvzTu# zxO@$M+e65~MK2%K=d*iMO$7Q~<%Q?vvw&WsrvkFokq70#3ei{^V-tt#7e_&3PiqlYLNtfI__b=?& zH(9(+5D>V{)V`t0$*pkq*gcHt51V~i2_AIMTn}}p%2o&xPMo5rYoP)H{Uoy7o%BZO zp`+GrZkGB#QZ6%?n7hqpRC`|auo~*}XuJmjr@P}rKSDy}@;d8YH0}4^&->a2ssf+g zvyGaEhcj`-wjMO9aZ)(jI6X3;5bk9Hp*XYp0z!T<*w3DWEN&P=9r>BDA^kG%1}kw< z9klE+^;?ISu4CVI#ewvU64qCQbMR$34pU_8|2AciBr{-AH`Zxz2x&wnvI|p%+(`z9 zDVLF`k5h=rOvkA)6okntYA_qkP~rOYy+Mw31c&}kyWrSkmK@upA^fOP z3{EahTC^TMENoi2ii{Z^nqIgz+)$hHmfL|yj^sO{`mPFi#1rR1o6JF=!Z9b!g(40t zGJHX4Zs+c?;@ND$7{WKeYyn293_QdO2tG*>V)q{>g_0xDw|H*ttBx0#YJ;n;+%%a0 zlb5iKj6aC)X0}ptUwnN_`g4r0BvS;2N#RlLqrp$A4YNYdWKq(JAYeuH-;4KjDYWOSN_M0KDh z+Tb-#9U-kTRqQbvDFV?u>PbHLnS8i&3SBLOBb6nWC67~6LX24@dR`Iz zhkF5Rhmngc;+vS+4=Af2V5%hols`+kf>l!v_n%av(I#1FGPG{Ir5=n>BL~e2E3qgm z#H_@J7G<6&u_Y|n;y46k3977$-&C3>c@=E^%U9(!MVpjz6sxpb@lp0rpSo|%6C^!b z@KD^E)yNUa4#PI&TM~%?0ImYGc;-(6N3(zzD|y`8g1s=?U!@x44kAL`=UHK=LYELa z=bFwaq#F4U(n$uEth25CcOFs<+{Oqe7F9(>&V;jOd1)}jtDrh3ZHs%`)khrNVAsT25{>guYoLxhbE4vD8fBM7Ed8 zUI=lCw(7B>_-i85D+w!Vh?_Sdrk9wQqjLK>2pAp3B4Mn)l%6I#hKA*TqLhb=vF7xs zMkle?6--(N21Uo0ClwL)y;rMU;mQ4U%aONGIgPWp?n7Q|B1)ujjRO;>hi>VW-LSrG zC?w}+`;|PO#hn-?7CB|xuWTl=&Ae{i2-x*Ul5{GogfmUp?i%9n*vSjA<-b6O`YGRb z`$S8Tgm(3JBw_1)G^qtOCf}s~4>M2RV|5BljaK*sa>MZ`BZlM5HM~+YrPr^80n}0B zM#XL$mG5j+VR9AL#{iPzY!b zVZR`Zm1}o%__{PkL%L8=!o8u@#{o&b!>N`KS1MSKaryse|(VCC2(tkA9o zk9cRSHT@Glr<$%4u}w4QJ>ctmmkMTQal6&nNG%Z)MK1gVNuW+)2|BJSyVa)q<=;v? zF%yMwop4O-DpoYETL`;Y8Sez7646}shkW}kuX??tgvLvDBx1&#jD=!QnM}%H^ZO}|qhn4JeuBm7oc^3i(Xx?k zrOC4@dMW)45_w@~Is+wOC-(`61s3ve5yl={i_y#-+f4<|8)JQ}J*D(Lj^qB}b}W@I z#xEPxUM{#zwPqsoQGCX>);THYnT&N$byWG)iWd{3&MFnJb!1dp&N_L6) zh3P@T>(eAX2KU99BC}oQ%;Nr-=qOJVb6=)KGMYx-;|@v3fJ18*KnBdbuzGtU^1bL2 z2^^L=C7r>Vx5l&fFUgrDF|SQwN%VaMmElCBObf5}14fWa+;AA*wgu){s(-STwBoD`}iNt`K;ppqV^ z0U5E(!mOr}c~;QKw>%7K$ev&@Ju#WWQ~|V;@NPI>WA?0xRRH)PPD`Lz&#DVRCa}RB^U2Fbbz&wrb6-l5w&3R=0VHJ16TI= zNwIN=ev5S4!&uCAoR-q->+XfyyFbh2t3zMD=Iqa>t>yRL2NHH1BETu{L%6>6xU!gh zP1|d{i**lYdoyb&?OkU{5bc*&cdxw9?2Rw@v!?zr_+6|zx!r(%=m z@93|ZJ4XkaNU(V^DK^H3xVsm>>m z+ASCa++5G;AhSVg80A{ZolOdBf2G#GkMnywe`ceb^EaBk)m*jh9(JuahnHxd=}Tq3 z@Wjh>jv#b-Om-*I^WC1~&+47d!jcvI1lqy5x?{-2IE4@GUw+HQSK%~^bF#x-Hj5*` zK09=Rq?N+T;w13li*wU5>+%Zw(4g|T14t4=5zbpr=G78ITHQH zLaS4{D$b>Jkj88#Oa5LFj*PBiJcx^xl4*qo9s0*vcv}62Q^rA#Y6s)^Y?nRqy_<4|C4wWLa5#1n#Rib6;oIn15xT-E?-jK^KyR;-Y<<8_onORN1Xs_nY{6_jno5>HG~sz|@z4_8hE6O>w2 zx+FJ|g23U&s9G1+)goDp%v~~^mq7|iD`+x*5wE)%bAF*1O4Q1-SeDE7Fkk%X0=rl%grJ)jQXRzIg6#=CeL*QpKTUV7r7~Hz3EJvu)Qw`0 z8RVO3_8F0#;kPx}608^Kl`Xi)9r7bT!87^fKlWA%Zd z%dTBhXFrMelbbXo-Zo^LS%O+y6VS$9!fs2tM#;EFamJp-aCN>0@b3*&%&kMU1}IQ8 zuuaCk*}S*u>eKf-(9w9LM(v5AJX|j=kC)ltBryoAPLw zMmjLrG2SI64x&#>p3!32phdGq4i*=}rwrw-jjxs)qhC02#pT=u`@EsK)|A6uu93nf zi;DWZ{`mRGyDjup6e9g7x6i;eVUgOfkb@H%(8N(3)skkCJ=KfdEdE0CWRg@9AaX*7 ze2Wz>fn}H=mPTS~R`@Mbf|5Ihh?bWXXWmuoHv5E~2?Nx1Bbu~d&)Z7j%kYDLi&d(C zdm-UD(@)%uDVkq;?fA;#r&$(y(oeJc1rhtvMk%5S7)Iqcc0l>>ndMr6n%iS6|3nc0 zif%$B_^1hyGx9-URw)K`KzjQi4b`)B%JZ2hVbg{jo{@5HGNzc7Yl&9`>a2cY%JT^K zI$L2pUhN<$Oz{MmG2iEgdAJl;c4C(u@Sm(FLCQRvN}^jbE6VS2h9gKwU8-3Nt+`Dn zL1yDJOF>~m9mpU}!yvjHh{CG5a=eP{Xg+du-u4!y>F_@#5bK{gq(l(E^TC-zHpST* zbcAbu*R{B^@2;W_hdREn5n zdk}8nAwgu2vJ9ixSl(<4KW+)VuJLA>4RJM-sU;LqY;#37dYNsj`D}buTj9}3&!&^xtFNRm1^e?lTfD~3=I~T< z)~=&b4krMrv zWwtaNx=vYN5OZk7%HZx|C{2)K8~esy&Z=c!-Jn?)i^VfP zE{>NJ{Vu1-hkoeDbHWS=y4&{0r8Jmb4_i1Cy8m#*-mKzig?o)Kao_ z8Oxn_fZJNo6Fn{qye~JDEzZ$Dpm4iyPus^sCMqZ?)_5mW6%McFDztN$8#+C8ws-GB zUGh<2N#z}u!wsYMYgFm?(v!KeU8m&K3la)jPGwpF+jxrwD75#o*^rCoOd~4c_0^Mo zmFSICvV!X`Xvf;$l&yv-aobaa`tzKRrO3CMNhziLyC>@FO1rYUSPXwDUbb(klXpiB zyCKuXW3aT_xsA%hwn=JFrIPAx&b?0DxAQzAFfZJcV6d$!40nZ`e*3&ZE7nLSH^X(f zF`Jbc&LYA#akQ)?K~2Bx*IMy9MZ55Efy1`1KW}|5n(^#fXdWD5RCdwf*;&)4DIC&T zPx`iI#nLn`<^u<%nR|&+{K4sP7I>7h*>vVlyib7si#N&LaeETy!;Aon@r&rzEFihu zloPHn1*Kkuz}k3H7BKF$GG6OhKupm^MIj zox)hVna3*RA}%>-g*}4E+K#46ddKKD5&!0Eyf|cfde@O(vv&%a@Bf zAXt1m4g#D$xBCM@lEwa(T18<4dUK3RR0}*!9)dWSl&7(AKh=$H57KHhz4Stx@#uG~ z>aY~FrUmKvq)EJp+N}v=b?n^LM3oBO;eE15&GY@w#AH$IV6zMBd?u%v(F9xk@34O? znBkHsH-9VGoJS(TiiQ)iBI<-2PaO_&=!o~*qno*BPZFll)WdK@hVR+q2``?r&$m2N zh;r4>Cs?DJgeD~v#Q#b)mWm7NVXX^Y;B(rDrwrSa;}Inzd*pYTxIp>FT5|`3iJ;%sxz%A~aQAAU#9>#KStZtPt%v1^$Bd!exwyljr z=4i$!P4mM8i5^9eV7(v!4&F1+*K2{`&Xafc2FyhS>;n(8eq7J9;i9Pg5nVZHzA@(n z4pOM+`BAzjJnq9HeRgi)W;M&CV^GVKG2Q+m`ZwK=!Tb2>R|Z!zMDeM~=KZ9t&vQbW zk7nv`V-n<Cj1DkG%b6$2mKK8nRFaarvUAQ@-6Vtiy zdz2;EX|gS^Dvje$Pe6f(y3@kpEAb?3fk%ecZ)s=x1+gx~*+>@dbK`tOy@lxT2;T4cblDkDw$!Y*?UO+m~9 z;o8+KJh?XRn_E@k7Q3FULGO)(bdtRdB`+Pm};|Cre&YEG8umC-l zi2RYg{zd4L^&4mwyn9(COBAPSW z7?kd>XwO3)l$xvjpEf`-~W{qVwfgjw!{7l>TVYpZANrF*Fx7^O5}PmxeN=LBibA z@w^WPWFLXXiBBm(L0=Sc+M}VcU35vaiI~*=g^xv7CJ8E5ReD6Y*hM8$Ls=J6fk|9c zKFKVf2@GnvI8$2L%=Io}=1Tw{&59I#6KNGv!)O#!lL8~AO_`ixqK#{S5=#cd0#mq! zbOa14)rd(U&WS<-U1&0FBdw-rpalc4h+73P+Kk#*z`jA@PIA$tT(3#W1Xv{EqX+RA?# ziO9#06Y(w=*@Upbe<%%U3;G%HuOUt2$iFv%Pw!?uag zvs*N$V9S(cl8#t3t<#F4ko0Sg1WMHwkJ3;Dyk7gcTvFM(@L)>z>5q*mHg3>CLT>AX zep?2L5SotuT?S2A*Hz+i4knUs`#LM3&~LRX>q_lNq4UPzNEgDRhUaU2dse`kEPA!C z<-;lC!--H=Gg%Y&?S&&~dWtuULr^o0F@f;@Dto=|6YH zjCL*9K5*@zjTkh2T0Lh&wW`qer+?7)*BWD6i;5@#-r#FzI>o&8SU|(?cz~E}@l~Ba zb$|SBvxz(DD5=Y>nTg4WYsGT0#b|ytr^+82O7(i_$EfB)hqT1>gQwrQ^*fhR zaP^&~qxZas*Uvh9HH)d1Odc}2Og?dL&=A4=LzW3hWKev0WUQh3Ss%ls5s=azW>+`KjdDqsDMmeAL zU71yQ$YBH!UR70H?Z~H}T%XXt|8}=Mkr4b9Lywqa`YBBS5j;@LyhomuJ5L@?r#ymgM}Li(Cv;r^6k4c9x2O17B8rcEocfKGx{8kB-p}aUDx%reV9NzvuAVDY-a0Uh^Qc9G zj7z;AM#2-`1fHii*+#GsIf}i;{SVPwPOrXh@|^ez&c72MJ!$q}H-++&uj|BJjC(Gf zdjHl;?W7GRnEBe(9>I71^5`T^itRU{&Xq4i$8ZRy@-`@2d{PnBi{kc(;TnWT@_2IP zGg0Zpql`%#M?ELQ9p1`$w*vdCuF#@eopk!;3Y!;et7sb5q$`vkTO=O5%sQX5Yg zEZ9-l(m2aKkqQ^&^m%sW>643hmYOP`M2;L9F z@9?64ue5`pxmHdg{}L6lA!{$2ghj@ch}!xpzl*8g@bQ*x)Zs9^QK%OOv|#rT1N*3& zV!MmU=G9woo=)bl+0!FCx|!i1H4{N2HB#3QI;j|d&dzHJ+hRwP+96XM-!6le`iJ7! zM@yBxQ`lKEM$-FKYqK^#SfApEen4?!uCTzG8GRk*L+j!5Scm?Z*jQi;H2)lh)|3;z_lskyM#qa%s(XW;qn= ze^#G_D%Z!zE9pi3@Ef~+^8Mn{%DkB=fW18~FftH5uKSkoES1*LA;>Dv7W-YeKR-K} zUzak^?-P`L>eqRb$39#JK?Vi`F#Wk1U&_H|a=X&jr1!N-y5sJNeGW4#q1xp2zuGOk zlYcK9{^mb_XH?EEH?h*v23})PyQTyC3;+T_MatGb`5@L^WmqPsqBYherJE3QF33+v zdg{Fn-vFSqB6Q6Qr`^;$d2NLv%hWq^?FMJX@=~ySmNLualBk=Ug3kInoTWXUq$128alXzUnR8aL>XW*Nd`KCz^+It>k7of=+E@d zEN1v{0r>sKd0*PY2I*>pY*FF ze>UyPX1r>6Y9B>%y(=Ja#yfNS3*SKAe>Q;-`QLCyKSV9cw<3H_Ye#{aR8b zq!GK(UIQtN2mSQhA*B&+XWd(r7h#>dG%yN~@h8K~cc2+Q8$aEUBlX?vQA&)>iKCaR zY&jAcDn6*@c&ky|`1IJKS{G`vVs5ZgIn9aL!nZijoVB;adcx93CusVHj*0pls_=cJ zgclBftD+)RXxmD%5bbVI9~PojN+wl2TQK(nO4<0XB`ZX5-3677`bekV=&^s}FZ%t) zzMr|LKI`0GFMS&gEctxZQVjNhDGkx022Ez<^q!J$i;W*5pI$lB!Xj}B#4jYmvU|_i zQnK?j-#pe);C~j&_%TAW!_s?d^_YncCHbr?-UvX4Pe(FAeK`5!TSDU~1Q|_bi=OBHkZ0PLCwfGD{7GhAdl4@>(}v zFxs2Sk6`Nwqitmv6pL$zS+2`wp;>y+y*37*uZ=#*DHVZ^CT#qE*g$Eq32MXR)aPE|;`D1IRE zb|!k4o*#D_eEMs_TcMEc1r4O_1wAA5NhNs1(SZ#h$mig&ePlroc;ROV8u&0x?H8^d zPf%X8;%p8k8I0}U!RIa=j_p&j*-rk>G1a3a+wb4`E2U*p_wxgd>n3TjnsOycj~0Co z>6VJ70ES|NDP{NRm|>YuFB~Lr@Eq8}raVNqgHdbczzNGWH6NP@qquFjJ-C zO$T9BIghU^$u}pCOhvVhbp=iz1-~N;Ln__e3m1MtT%>vF za{Br~uf2~&jLy2S-73UyuHO68pwZqv$vXD9Gsxxdk!Ph7J?7|Sj+P^&3-wMtCOBF> zX4u=RwBRAMkMJkIVe=F`ub)esE1J8h^fGtqH;T(dw37HTdWxL;eb@QkI9FYnQ(0h! zVSZy4*IU;k*q?6p&TQX&-^|{u*u2;*!92%I+H7X7ojc;32ET1bNO%A5kZucTjNjGz(sF^_k z0EP1j00ba{Sp4@z?`=Q`fDaB8yfIG~?)s8}L(;;@%#ss}U(MCR($)5bH3P@LiXn{}9?s4VmX7yhh}^jW zi(mU*XRa>@hVVfEP-MsNMdv^$5|08xpfE!Qq-2LPyu8^5CDV^27!a_rQm`vK|TNof*eCA0NK1C0Ddp-y?=wD06r)H zgw*oF;7~p|7!Cv)BAaK>haiI|00!ST0}|jvx^iC&ln)>v00lA#0tNU0a1az?$RLPx z2`m6XRu2c~>E z?7rLo{#6iyECGc>8IT?16M#XHL-DT*Fw#-Lz0dzP!H^By&pxEHe-?zmkX0aE{EHv| zt{()3@*$@@vRhyvm=7c%2o_)ff&TRE&q~0^1q48LO@IN4^!J}7ATXrAVC2ApVIV#j zvO7QqFa+u4eFPam$N)f*GvIzg{CRDNTmdrLf6SYI&HDSW{BwQ(Z|Z0e0QA>wLJZXx z?YaRt_uGV2`f@bZQ-?^*mc0o#)TzUlcgO?r#JVhn-`>#_k*9VFev_9Ik{@pqo09cK zh%IqYpN-k)(5P;wEg)FX^77gle745)2+>?XYINZXXE_{7uXaE(?1xr5Kc?o^ zUL%$>r3!{IwN`G}R<|8~@Z;Ht^nCmzVM@={T6bPbOTZLgYEQbpFuMPt(b{jAjFB~? zf*Q`1;lMp!yB+c|y1VFuNSFpDc+XhG{Q2zA`x$jlvZ@VOOflLPXZE_h4KeWoUE#!& z>Q=1=s9$!?W=B$%!n^Hgy4@jSHm_L{rg~*MOv@#n72-5fIMFg)FY#<>Gb@CEwYFugTNnq zG5jUp{bjR-KV|X)001&=|8Elk4ibR;bte2U!1U9co)*7oh2AJz4Jfk>SiDSNG))Uh zQ}(1V`P@g8Ob|T$;H`2zCZ+{5HiEi_B~raapCqNpe`aPz&G^HUu{yIT`yp+wyxP!D zr0NoyGQ}3#!WY9VX>SN>YwrA44!=>a1dfPaw789kep}iKm0@^55YrLlc5w3X#MaW5 z?ZGWZ!Ijor07r_IYKW9Gzmd|XBbwUv93+7*VV4De^O>-HFs zM3hv~x%Wo(l~L_xRvM^v?;a*jQZLNb@f3Qy@xJ=W=TL66zwdy7J@??d@rRZdj;GsU z{^fq2^kP0_5}=&Qwyk*Ksn<(E_@;83|3PRW{v}ypij&K+d7No8)}p?? zKkqNYW^-W09ozNA3txR}ZD&dc6rEdcu9f9AC*$8PD0_O*0;|BKwlXVY_V)3W@%gD4 zA?`erCQzu8js1_0kyd#;7v+wp`Qo(V~uKsWqqMA&CJQgefLf% zz3Hivwp_h<=%ItdV&eGp)WlR%otu{8>+(k8H0C(#8ZZP3HC4;!78e)iht@N&vyVa) zKQ7DFJ{R;nRay`{8F5%n6K6(l__h`m$^9|~G#MdghQ2^83=(S6zY5E@66ZRDKnuQT zop?9WGPhQLr43N_6!u`Bin`<7@sQQlmo>~OH+PiN_Y&q8dqWs0+ntV1qb)b7S1B${ zlA2UiQ#`)%wo9&xtk&_wi{#4rRjgB)kIH1b#>>-owX62=!@wE)dS<4^1KGgIg}$=4 zJR7-mFGAA&H0)b{$Ln(H!gV?Qs>^3qqH9-zLp3(-jinEjt&LP0*mH(fCYE`U*E!xQ zFoF`ET@Dj6*}L?qrir&!=au<$otwPOVauS8J%zs!Ekjj;e#A!8C2oEtMGUKSPb5ZO z!{3U}L`dNrQhj@bhd}vOy)q@l^>`#AnmpQxMlR*8?hyn}nE=I|5%{QZ|5kMquleah zOZ{Ll3xxL1PW?82l%EjpNDCqXNZ-#0AUm2lJfkN>EJR< zqSQ3uvhnXOT<6r3w2Dl2sWw#>v6zmY+x03e5cazyB_scU2Ux5%q3vi8xSF|0} zk!G4d*^4Eowg|#QFhcADdV5n=c@VgdeERxpap}Fyh*1BK_lPs2-(oa09285%#!>C! zXcd=m;$%^6`q}DS=G03h=Xc^0s`YYXK7)-^YOmf)S@iaS$hAT0Q43F_ZE;&Sn>a~a zVv1q<={znvsaAG+rQY~ep!=7#JE90mcH6K zrWxanV_7IbXZS@zt!J22iC^a16Ljlnmf@=k4!#xQ0V~#-8scqE^UCMV-abA~tvyrH&N%7L;ynCX! zmOUujZJYb_p9d)eI({Gk2)Q>e}^pft9!6;rEWe6b~Al`i>5nL08IzW5Ij&35W zbeT=AcOymj2+k4w@aZcebIn4@xLIhtI|Lpd zPkceJA#ejQ1(?zq@dU`$2{#xRFYyuKGBVgKjBFA>GbW!~G?pMz0m^3tFH=cwX-b&# zCAdD~P7ttnTJ&h95{aPN0!Uqe%F-Dq=d0EL(h0(XPInn$Sa1O&Z3cq!O$3bA0D(uC z`G)qE1jvXnj@F3KeG{IFd)2ANfPqEeW{Pe`*Fbpsgy)rRXPhbe7ugIq9j05UfHe13aT1MIFl92U;CmJy9l*^@~-r z9xlSx3N~*XtzdkYBt&sG-=&zQwCH;8s+<8TZfDL@RC_}Iu@D_@GujgAlJK9M7ajP# z2;8yA2C@YL>TJ;kh$t*q6C*+sBrFICTXE5Sgw;X%z4L@TtdZ>*S*OBB5Stpx|Z6Gbf zqzgNgreDWj73y)UK&`|m!~aFtJdB#4zFP$`+cDhPEuSjDFVqN)Ux%CEuaB+@^t|sj>Io+_4*ebGkBIBoK*V(scWO(#T4BIL zG+<}REaBb|20m0nGY&KELhAN94hiUA7Z6@BwLQ97rR?cngZHqnzU(pX#O@?tn?3UF zSwUQ~(8w&sUCMc5{X$&odlMdeG10wB3I2_$*3FF=?I=d>abb@b?cDEteI(Knbcxau zap?;Sxx~2CIKp=(SeCk_C;a_5P|}a^7uBt3%Og1gqOjNAByAFY^vAX^bYrG~2gZy69P2I=ICQUwnX*C{*5yJNvZQWl z{gicj)MRcGoe>2QESLJ1(Vq!M3E=DBWP&cy)Y0Du2NIC3r^v)yl8!TEu`a9zP_PK~ zyh2ERlzU{d!xvmn_!hB3Q`VKJ<~v4u+($RyGT5KVsM<3(euIgT7*{1HP0@mR?Ek7m z)4;sk^WLwjV~LaLpuqL;8QAFeYAl`Q{gm@x?AoxFaMy#MdFxT|9~=!*9e zeUjSm(F#t1VvP3zf9QlnQZD+6q0ezO)e4bPM68Vow?dy`EzZhffJ@E)ALUAc-9A5q^&=TwS2g_llgR83LVyhuTB-Cu)g_hf63a)g6A+G|0E&zs7Ts% zS00tGS9>R9>8iKKq92*0rmCo1(bM0XZm!&C^V+B&W}@)9VyFRIk~5DqLuLE z`ymYvp{*TQ{f7!XKbVP3M2Brdf`zNMxwQCNN~97%E2ZNH`?TJ$HMO{Mg~s z0Wr5&%UoB=V*{>V*7`fe7m?SAF-77_lY|a*Ys+Z{pW`s5;m+`V~ z_=}o#=;K<-31X4PD?8=cmQr>Oy>kT|pPCk$R+uP`nq|fo(QqSe`TS?Q&Xalsk89>< zy&(KcQ3tYTdYop2Js6%H@g9CFwE-@Lvk}tWK3hsNq?7N&5YzbFN4rVS zS!3$r317pUS>~qP-q-4d!i4=R%F9EVc8cW=rTNVJ5x?D4b-q8k<0KMES|`_xgiA!x z8COETU}2Z!#*JsCo0ZQ}4GKqMRZSJMPCEM0Pje`dMtf%dd~p6_RGz=Mkc+bpkwc#P z@slpQ887=AK5lxe1f9fPw~UGqGtIgstvVh}v>5598kFfBU!1qq!94tSq7?RUgnLnG zO@4X#RA0*Z^x4%pdi~6TcV?-RMCpM&f6-cRn%bM+m$hWsXF}y4ifvgyfPtE2Mef0n z9m`@HAH^$KW1GrU^XhH=5}(62%2!uS-KRL*bo*H&)nZ>=&QxN!<3suLhK7s`NO^+! z-m7YySv(;fc`a0F^`zsfk|s%iN7=Nsh*f2%`(c+I4mpnDMRI6kVUJy&+s;BOU14un zg@^9*!eQxBOLA`-&o73!Kjy`!%=ILF>+pW!h#Q%A8}ugg{!UMiE%_$aI^01H>$!{? za%`<=C+fd^(#9}~Um)C67+DK2jjXIle74|n<%(_9e8~`lSr+dvME(1t%Gf-bg8Fno z)Z)p%Tx2w(? z?uRy)+sf?dk+AOSkjPf&kB+!8Z9crSjc=rIhyFO)%%}F1oA$4E>b22 z(@%+bWZVUvu)y|HscgXU=6(CUwUZgr>pm}?QiNbNm_cNZ#g^4N8LjD8cC`m)))N-% zBM+q{Jc8S3W$-3FU*p))@2W;D=t+bFn*w2x(m?YZeJ%JW((IHMkF~y!&}ZEwJ(nu- zmw#R;I{G5QVMM&ZxAQ5~gvI*~g-UZN@Ps$VJJnbX5xW1GG+AjkpzBT7L+bf9hYsmF zeq-HGVS_a>+6V2eRd(dksUOp&4m()BmoM~hn96Sg6P40xIw#Q+38xyO-bS4y`_sKD zHRERo_IQ3ZLSa%9>tTLcpMrVp&M0mBkr5v-*B$C3B0WKKvdOrt^_JZTRigh;T~Wj} zt}Dk19BujP=?vedi0T89N1nwIVLIqXMObfJQygfYdh;+wlya@;MoDYcK!`&u2-@aRzL3$*2O={og@qd;#c>l@veYZc-$4FNMF??imgLZdMn5 z$4efA-24JN{?UNz0B(H31kr9>&-*6DgXMR1yaO$=`@e8_T#qU4cB~KNb}!yJ<|!DL ziu<+-ekhUF4dqbdN2hzV>cc`UfK9+Z7mc5=^xa}@DC|{+)|0@ zNct$QD!c&(kOVI6a3&3zSw>4yv0G{$TXm&JBtK94_EIuWLx3uROCUXUPDWwf(MUnL zT&p-@=KGoKe$6L}ovnxjK!{H5vGxEhJJ#6f9}FFFNWX46_MoK3y2 z%TOM`OQhw9fV>#wZH$&}fO;qT%DB?Fzj@4>#aR-OT9Y48E+}iKu5;arMT@cL=vM+t zj)+mQ!kM7U@~Fazl6GzEdE1#$G0FJFJ%2$t70X}iMe>lzgg_mM;HIgnXFBf_#jI@< z48*(rfYY?ucE8=(eW8}&cY3$>WpgzOu6{=oUxPIIUlVu-9fPOuR@v*FuGW5@tptBF zxAH`!4P3CKDUU0=x2Hs?7lB<>+O^9xj#qQVhoJj)3N-A68oJVzc!Aj(6vCg!gQhKK zG@vpmCv&v>#xOeXC=!BQ{*-CMrn{`nR%&SO2%CZvG3J0Xx+;Imx27Lne7P=C^bK{+ z8DrQpOr1OOW(dx~+%E4TcfB#|bQ*%zYN22_6c=TOJ*N3#yg#|+~ER19VNlq(^QIE6zVvyh2(P zdp_@U;!J0Wydf|_=rO&m=utj6PtV!hZvG~`_1)3vrE;Q60aOtxr*r$%xL(*}yyoQxXkhOeoCrwY-XJK#G$mddFBu9ep*!nAz zABBA&5gO-5c6nO%#L@0#b?wB9{Ot}%63FYk{Z-e=@TI`8CHe#VBv+G=C=t9d!X!=G z_+nP7m}{+aS?s<5e%Zy2GhbX8HaCzB1GisRkf5%D9;+Q~yfm~?3>YYI8!_V}W# z6i*X4hb+3OYPKx4EX--sb@xkdGwJ;Zysy{7S45v~thGhoxeQfS_v@Ti^2W8jAUUO$ z-xf4s*)Rply6Tl{a#?TH`^-O5`6QS$5)Lbc&nGS&LUviUb+EBTxu!K01_e>K)7Tb8 zzE2chNmw^`asHGa%7Z+^`_=#gkro3itBf*;)?CLTw%)j@!W|xe#}v8gANE@DDo4h} zbyXCFUsPRXXuWY{S|!DOYWl?iUJJ3eC!2a%z)qhfKJms)2y+>=ATFKpcJp@7uhP5v zwT1=2PGB+(8tDO}hKK-_Mxqv$M{({46Vj5V@L3onS{A3In{ZGx?Cq+M^ahzZP%*~^ zm@st`7ItD85SRw@IyBY*I&BTUJk<{L5Rh|Qh&mz}!6h>ju`}x`nj_Xjl^&ALcjJQy>(gNmTG4nA(Ig|w%8J!G@Ihtux)rm@J4ZtW3FTRME1fuE65Ib|3H zDe=?&L%t%B7DnTsUfceQ&zHn#cweqC$pzgDHq=6%NxW9Wh!53>HL0KeR8VU-5$C6` zqT5&l*kR%lrSZNFVyz8Az2H5K*chU*5z<_~!_)2hBJTJ6ZuKygHN374Pi&~OmDK*d zAMhzX9Y~vdq(wO{ltWpuG=1{pg&yr00a0o4#SvCPu=-eN{7k>I3u35)LwpRaOz`Mz zlxrqp2HK*)l@^?!eT{zn)cH4YP}17V0CGKzwU$cXF)q~v3eG)nO-Z3_teSI$8m?`s>W(3WC z3rA-SPc8Th%PiNo)ho>$>_coH)&o6YLG)Yo_fk6p-)IvR3<(bb}EHOqH*&IUS)hFHnB@yDo&UjsmxVc}p@559X@HyEbxta2n#It5@ zK_HMCJ@JB40VtuM8a`i5nUQCuK^4m!DL|P(;YNfG!ZO!u*c^#hH1<(=vQV-IpQ++L zxG_5R=smE_e%ODu8=K3iP^sspy1NG_H*5n>TZlMT z7b$CLbS27E3$Znytv438weuv&`q~|FxGBbmvrp=&wI!=qxtz;(moD$vv__9lX6hCt zK2dlMOgqym$5AVz!Z?6a6)BeeSc5<7u64utR)|-=H65EBs-ZUP++-fr94iX;a9P>@ zxh|CTsldL8(93aKb2ql541=BpwBqX&#VwbqsP%R>_Ri9B^~yYG!I!uw?kmig3-)+a zmAE*$aQ8)Nf_?4!nPx0WNP|XqM#k>d6X z;q6t^OnXO`_PM%`>y`8=NtX0b(VCA84o7Wsp)R5bqu%A3X~%~ou>M$EY)UnX)aOj< zTOY($SF5)zD8Haho3elE&Z@9YaC}B7ygoiwUnAnoaTK*!6RnxwC|&_|P9-^bs+da@ zBLO@}9<(HS&i^y1$aPdd9)9`Y+Tf6b%L6EkvA>};oZ7kc#6Km9n`5=8tSiCr1DmZ~ zywYb)DV?N&+;PQepDtVFj+v9_1k-#V+NI|}YK|RbZ~^?hEK=R{#7JBcKkghIkUm{ftkH{kLbwBB2Cwvp4W0{5bneN>R|QKB!VQRC(oYC zmC)>ff0`U1xm+I~u~0i(v}JS_FVM5*i#sLwu+6YbQHml>mDfhA@htB{`7KpN&n~}?INk71s#dL{0;P{4ss*#J6=V1lZ7_*=$H_~KafPH4B@2Lv zYhLu%52vV-B&}z5U#WqS_2KTs=;atW_9(w%5?HW#q>?D4NAEQD8nWB7H!EJ(7lj5qpPj>IT&SB&eU}GG4^D6DZa7{>v3{EN|BOe4W?$hKI z_KXvXlB+`%s}LpVq55P(d_pG$ai&FDyh-*A?DHW#jtPlcVC}CVaU0Zb@mA`lPpp$W zqsMK{bCF}v`bSs|ps}I0ho_B`-r8M$TKJTS`FF~k{`UBW@)v_#*(tuLhhFOEIR1;0 zhIoE$rA@mtJ_Fb&$%F+ zSim`@?C-X}0LWaX{K(DO`gwLbU97yjU98mxNgQ1-#gaK(FA8y#^yilg`nR>$G5f1) zSYOn?ee#tq`BzrXr5AOy`iWF(AEfcT@Nf$Wt*3ZnRw;e z(tY`q-QM)vDd)NbnVF@%>vFf@VH5%<6VOsAA-0)z#&o34KIq)|4@kE%&uFq`NC?^#bV(LO1Dm4Q1+llw+Iq9%5aaWvF+N}7&f*M^GN)ho3x z^-ZzNg8Ut0N{H=QxMoBR=9TO-ntZrCvjP(>^Wc6Lc(9xEZEViTkyfhg%Q@zWWCV>a zlWU^<-9tv@$(1#{Uf#69?w_VqTiMdN?Bg_&(y4tkYu`VffvM<^LTsb)L(GyG_Y@va zYT}o2F~uR;MVNk_AY2BR`|@JnXv=*rWaG}%rqxoL{VBViCy>SKdq{0Jh8 zRoXVZ@WFD&mqyf*0!_T8+hjaJu zjpE+G>f4CP@rwAo+u`EL80c{tF1c_Z$ek{rjqx*E66_=q`DFkKo;OUEqR(-#S5 z>icwN>R;?44)+ApiDt^AZEU2Dpj5oS`j&BzT`~ny$a;058$VKux0g}FL$Xd|o!{pa z5u7X)eWmgbvgW_&SEV@)AWWat-Iuc!_9Hk(VJ zuLmSQV09Cv+Kw8fys<7|J)eB{`nA-p={@ z=AJu>0-IS!N#5B5w^DKD*&LGJ?XYmMM^$e7&Dry#JWj38pX%{qXH~`>|u2YEv3t-z4 ze|lR6d(rU)+w(&AX2D)E$Uwe|V(fZCZ5%Ou5$lgEh+U~_xfaKByCUT>D|iIecrepv#IV@F+m_Z8nXC9d2Qg&z@8qIW;KOCyXrpF~g-wLWEQNX8`7 z&%RqK)(9DKYe3bQPzA&O0TT%efl*Ttz{>0FhINe$^z&@Y^&%YGGI zDapU{ovM=Dv<37?8ppvia@pOJ5v#I`CVD`pKp2ArEVe}&Y8l*ua(MDwss)#N)RdW= z+D@ObjQlrt`a<`o|E8ya_rT!)Hzt)=%E`e32{!$ofT>8h?jEy)LXgbn|2dcn0`Lh6 z-lK>A&tU2w2ci5M)B^t3fieG}#qK}<0;c{kD*qFh`p>@q377r|1N#+2^@Zp zyCS(zK9C?1_+)?}xl14z$+IFU)qnQqzgbHpWBTX3{ofEfprGJi2|GjeA-hlj!Pjrx zRxg_yWLg)V5v-Xtu71R~yLpJzJ$_(g>Ns} zicW$x#Io{fO1eRbw_7KqNV zw3B*fEk;%7Bt{jicw0=a0Exbhn(yGl9v!I~tVXqCZVHTE$n?0Hrcq0@%uSl@8A&)# z&fw#m@3VNGXK~`s;kmL&vVd(bV)d=KHsE{1KFldqlS`zn+lE+k?e@XatMK{%P6U5B z=;t4Y4E>mF9TYX^U0RCrvU=RTIpZ+07>c8rPAQ^dNrubWZ zNW%QD@o;xFMJfecv5;)OhOLj~eR%nwIXNNq4EOvlGS&V;?%!wU`>EjOZtCj($3%dD O1O)}KSXrN_$^KtLV^0zQ literal 0 HcmV?d00001 diff --git a/docs/profile_overview/CloudRA.pdf b/docs/profile_overview/CloudRA.pdf new file mode 100644 index 0000000000000000000000000000000000000000..4532be85b0963dae4a8c2e12f95f0f738f3643f4 GIT binary patch literal 34370 zcmdRVWmH_v(k>7n1b26L26uON5AFo_;O_2{fdmNd?hxD^f&_PWw>!yu&UfB>&-uP} z@6WrY_U^rVS3h0#boHJ!t7<5e#3UG)8QEYdW{wUvV3~;l#P-J4uzY+>qE2Q;F7{5u z)S{M7_O@28wluIzN>28st|n%nPSsCVMogfw#I{@j(1Sn9R`zycMlNQ=)M7j=02Wqe z05dl;2MY%a7aagV1rm^+yuIoFtRf%)Yi4Kq+XU8snPOr@%nZvUB2UbuVDDsWWb-d2 zj(;ghSlPIMEMtcALw| zJEwBr;#~!qcw{2jmVfr*qE`d6zhIZUKvx&R`qkSz4T(hbjN;Qom68T`_bUD=j3H_> zYSmoo1Lk0;x3=b+H9ucAKZ0&PzSQemBEP5GwY4_IxVl$X1bbgxBD)t057`+$eu=&Q zMhd9Mq0W~rrnT3XoYmJ3mhK)TG69K%#?BK7oq==v>$A?6gBZX20fWw$4aJR<%D1bS z{U8HD!qCs(z+chg^Cm8ljx65arS6*;d0ai?2sYhw^cwkIIhK9J!-8Ve_j8bWbhetz ziAs9?2=nM%Dj;M;P>P0~stkut3<(Pl0RJ}TM1@n=F!a>WOltRfLnHm6tj{Q$W zvH(b{0GeZ%W8uF8fyP1y(B(rxHHln2Brk+l{9kka7ohMu ztX05|v;RB?WQ$p05)AU++#^;pB#$R%0Gzm>g8!Qza=gIa5TF975GJ+@6$&;Sd`l8c zG!*olG)y7{Kf#?;04jL^D%sye7Ybq;BI%SI)iglb6*Z!|E1xH#SIaOF2pRtyB7qnV zz9dQN4+MEXM9?w0f1OYvZv>?FS5=^xG5ut*0x2`*z9|)WKIFKxJL(6KuU#Mm|4Cvr zBr1rd6>0Lp|DyUYw}}D!3Y!-H4UoeBD-hLx15{8qBMk}j&$#%@E>tOyV{SYVVA_V0 zKruslkRhZ1Ol&nO5-B(rUZ9vi{p7*#p8Y@kxs)U!=R?NMC{UsNrzl%EAcrsw5Ozlm zsP6j88_}g@s00iqW|;hsYXd?jSNJ`{KiT;!X#a$m5onR)e=`LMBKedY!X!Z06*aI< zs1fAAzi|hGPyDm||AFk#Ka2mjp#KxAAb7~Xpn!kC6@FVRRLJu|SZ4jdL+bx4j9h<* zQS_fMk^qI#G{nkpF5q1L^zi&K7r2>3uQ1sDB?RKVdMn*T!k zO)6B&AN+sN{1+MsUj<_2zgsl-KZ`c|XVLybQ~%eZ{d151wGZJTKz@$o<3)5yW&2A3 z{iVM6OJ(^>W%_HF1ydU-b0P-_YpI=x3L#^vUm+t0i!buPD0jEn;X?!gDwjh2iAPHd zNQhysYRr&HL&lQqup!8dAV~ivIw2^z!Qv~_kkCf_kbe`g?%Y%4+Mfs_hRD0>ULfSp zbNG;uM*0oVBKcKT(=oz^9*&*xyA;YZhpD*@6NC-`!^Y4k8dx znduKQYXnm$96gnC1q@Q68&d4=Qp!fD_n-|QuShbWlCHB0 zpejI8*d#bFKt-%Kepig84?4+3lEO*`tMZhRe0%cu_;g!UhCfr|K`%5!!*K z208Rch;RRa4<`LPN`+Lkl2eG{(J{?--jj!jYd_#5{*F@rd;k-jATiD8$S{%=>{ZPr zD|zsDZIHSMw(uVbZwCYh{}vE{T1pNv4Jb|&K``n&R8p`}C716&FeT3*4)xW21M@+_ zD=4f2iW#DKWRCHYOSC_ZfqV;CfFqNF0m(b^5Kv5@{oDE7Qn1h4%?VQHjP%HI<;`Tu z8KJI8IaUA6a%HE_6V_hn_Q&UyW=E~J(W8kZXQPi7_XCXvpaFZ8xIH7-tIep}J`{1f z@yt!^!;fo$#54EdBP^aRL&raJ)-*(-GvZs6_pjUWpg7%sBj&uzJ`aJtSV4!CX>AiC8raP;CHidwp?=H zT)jU)vSoUEsB+YAF}iBRGx-SJd9BN0#;@354tv6hGvbEbB9B)P2AAYMhMd3ZL+F{Q zQ_nqnkL@F4w-T|ebOD@;l_#vC2y0`{@s2XvzZ5=Bej9CL$Zj5PowZ%2D7?OthY7Cs z%JY-+%?KS+-nt}zUEi827e?$&P4_q&7-+eRsa-G?)xhahXs&IQ`{rFfYqM2@)W&K4 zh4U`;1x5GbDM{GUuPD#Z%l`Bs9*Q@|{JSB}r%ZYih9EQg+v(4^^iG+W4nGx07@GYN z7G=yVaI=g#`4x9DQ8d+!~CBG(ml{vp^nmLObMKE%1oScZPmyZ#lAx-in`Ax-R)AGEjA|)oZ;U)K`lt| zeu6+lqXaM;{nac&a?r{d=1wDw+R^tW{sLyl`YT*tO)Cyx*a-}Go$aYuXXBZu0!){` zsq$B&sC+02a`{nv!RC;o%SiAUz~5!O;ydg>8fjokli>kdd4a978U_xP&_z;3RJ8iETzjlP$j$hXv&h8?L2l z$~)76lJcLnq}iv{b-$g5U zSNCttXO60mkzT!Qn9jCmo*tWb9g6i|FPdSux3)~ev?ttmCCWGAhsFzlqvHhx{o|BZ z%Zq?ZD4vXI1*)JBo-70S_%Zg>;!Gnrok|6$M!oyTg9bFZk6^rp$qr`yYvZBHqc+%m zqrV%DNFZWf`hjV39X|8$L^*ECu*){KSCkz~a0Y^C`Xb=3#s-F$AQUk3j$f*Als1T~ z*;c>uxw9_w7Q#!rSh$~>fld>xuBwIFm=%+0)*)?TAFmHOX>rk%g*Qvi+#+e{*e3lL z0w`x*m^9RV|DDo{k$NuUj73(vVBCi>W(}C>!sp3RCqA&DpP(K~fP^3Vqc|-*O; z4;gaovS|;HlP#tL;u2&+tCfeyzhx0H!R3ciHlv^OrwKp4OuQSaP-gI%9G93Lvmsv{ zmAD+U;qV6{qJg*wRx^K^73;21hnC|?E>W5%#oSWDquqNlOBDHNZB?pdWe?CQmJ+y| ze^=;{Wt=oLYLh-X8v0jbp0s!X$jIo(w5d2MPD#j|XD+m$_$G3y+@5MYw?m=GMvlDn02=Y}11?D(j|SKa0B7Wh-VkYL?)YUsH# zN}=CR&n5jeWMk;O5>C-v(FCte?Y`!E!7M0kal`%R3794$1sfUOxE<&1ViF!68YMfk zkq}zam|0#o7bR_madB|Bu$5j<&) z2-Cb%kY3KuPb>ZHwWH90D0EM(%c`6Tlv3hU2r=TMPakEE)v zwRaTzwIe91o%DQJPi7m7+e~m1=6>4uHjh91EUr4~1>6F6W@PD4eo22;P+anTYkMSG zcSX z;tIq#jH&?)U6xPS%?cg|jRq;fC-xq*DSG!hOjR=uY^Yg*W2ObQ38U9Gq`Wx@rn^0S zBOYE9PRm>Fm!q!QJl;jNoQd#1xp5yC;RqyXvj)Y?-enXm@^LWY4fImX1RDt`IXbYQ zIbkyCKUs&>rFScYF(136_b7x3AG>f2K?y_~@Ac3Z>(iz?`wY{8$=ocIa@>Fe&1f8~ zAVNdCom|OI<%DXg4l2`MEuZhUa z@w3^wtc{WXNG{Nfgu2EQzO*bGi%smq_Tfwqk;Z<;Niy2eON(Rd#As2eG0d!Xg^&a9 z;n6OZI-StGHO>K?RXkrPI|$UmMo{`qx}X7_-|7?jX7IS z0lhW`C(kHl2~oidO;Y__jdImrG2Z;#Ujn}?YOJ3yh4R;IltiEIRz}3x9U|7SJ@C~0 zx_+(VDryp>N8~w0iV8}6>M`$GB}yCf_WFqJbK#E5&#q2`zc6QbxpTj?M79gsbXlNH zXR{01g%S2}X=7dnd>kjX7w1cP-xMUJnF)cmNCA| zGE>`p>jm+~KdJAz5ugTR%hISRUKq z?|7h1@Jc^e#B>u_{xKScu)Bld^)_2XeY7+(D!ajnbkkxN@Gz|Ef%kZjYoE6} zyWI4}?TWLYj!6`F@!T#geg(O6+C8p$dnS=dk>bWPZB^(j!*rgY!AoRCaMY{sHKP91 zw0QQgmsnFeaJn2{U~(sMv8eu@gVthEtnZ>FCUeYatA<(bW4f{3p3NuCY&{soVw%ER|zO4?54MX|9;)dt50z2V&xc-Ck$r()SAzR(sVRjFB^uayvC@oZ zCA0D(Ii)p6M9c^!v%LjN1knrI)Fyu$}7hVRJ+b(AUdADk{z$h&avW?{x zfYsbEN5+VpMG)wn$-)YZF)y&FU3{?NVkHXA8ek0_DP?VaO70vry-LZLf^*@>$QY+& zomn(B_vFX`PEfL54VnJz9N(SDy{EBU zMIV|mk^E9M#LWFe&Vnla;24sXpZCW&=<;DZc>m~F)2L&wEVU6iC566^UVoQR|NW^K%kLF@Eq+e(W}=?zw>nX>1MC7ZRC85poOHr{e&abvn5g;*6{xR ze1xypsEGMWv~pK6oJL-jRjcvqB)`}9a8%|pD|@xxn$I$WEUZI~qu9-JZJzHI-N8D+!Cav9tH&&ue|9{cvB;imQstddX>?jdTg8(c&kA{xbAV z5;>=+`}VA)EladoyR-xz}4u!#G8?J@27-RO}0Cp+;RA z0yDD3QpXh>nq^&Zz1_L=QMsXl^)Yr|t4ccQ~vM8}D&w zRcwFB^}^p^@-^?J-Aucy1(;*YVl)YbVSw<0GR9K@Gf9K--(f#4xb!NFsCwZ``yn`Y zKr9qVwMGona4K`JP=au7A7Bla<>;Ip4?j3rFRcN62ic}qgBy--u=bN*V|*W}zhoZXwTN{-JkO&Wo-0&NWo zY7s`SXDvbunvxHg-!9(M^@PYv9kX%HQ*%zw)kE&MW!Chs2i;7%GL@u>#cCBI@+af9 zE2qaU^h<=8$*DC&MGR{&E34*wV^wPiwPkJQWKk?;cUEQIOuhBBTbqNEz&6|F4r`Gz z^42dLL>;S24Rs3X(Ak0j9yC1`RGREzRCMo$SmlzU|!D^e8zW3|lj zmP~PY2~wG77Fo=Bbo=ELNk37JEN~K3&&+D1Pjg)|53Y)aI)n|mx5$QEQV z0eLwtJ1gE^nb1d$9)4&4{*k8pM;?&t|1!nL{=a4T$~Bg)e*utv?lkwCk;TcZTmn?2 zy6v)cF(1Kg(Wvt$uiw70l6|Ha@v^rKZXa;wc4o^I(@on^2|eXqd8uR61_nLea=c`N z1tPkLmlEf4xM*V-N8-i^rvzN8(7@b+$;kFTSUH8 zURI0XuKG(5#J3!hPsRaVZdxHKziN?<{ppM$V|rn5Bfu-dtG9%rzb0S{lQD-v8^<3@ zMSlyx7Jc{oX0wUBe^Ll^pDskk48zYH;H~dSQQBWGWXe8GRVvX5YBj_(%5xzS=CsHy z)(=WgCjCr(j zhH|lyMCQqvnHd0OY}N`2Dx0z=X7+&Ap-4aOrF8fGS7#^2!dNXwp|a36!t|blw(%Yo z`*NiA2~4NbxFeVpB`!49Es-g0*jA-9$~s3BxwN2%teI*bm#75_OzU7|_!Bpx5kd5h zg_DqQtJ1F)hh6blOI2P)bQs*{a^Y1rjHjrJ^R_0WqfUz&lPFGg<>i2!SVi@Z^zR$L zNufl&>hqFw1erI~xJ327u77hZUfcZ0n_n`b%Rp;So7$q+b(-HhlQBHBP?#8`!KpI+ zedg525O)M8L62)or?;KEmOR!$jMVq;rfd9tDcd21Q(YB0^dNzhcGn|ct-Fz-n`k)c zm|%z@>iqpD9!p)soCRY}3wh2Q4y?hS(-0GRlAe!nk9)Z{dC$d;Z$`IuRh(nlpnh_!CclCw zrhu5j?0?@oEdSa&YM&j$u%EvQF*xR}M?vr36X8YX- z>R~5l|Lf`R=kjKzRz@QB9>lr;P$x477aJphif+>T;-X~a^rsU5sy+B~NzDBh3xlw14vuRN(NRJ^bzPzj^5V zn=n@p<1DaDvR0BbUlZ7)tTozVZ+O9b(UQ#PCZKSj-(ja2`mN@R!?ks8d|K;yQ~)cU zNp4oPK)}1cRRepv((3fIetB+Q2Nx7Fx4^Gd05-b}1NXA;p^4#g(w<)G2FoZO0?;73 zh?)G2mgbDckwGBo!ptVR{dl2Am>dnh$AjN@=Ohh9JM_xZ#;I0Ztuxp2QOoQpn1TE)li zhr;)|3`V~pnF?X%zOO-HC3(*b*LG1Gk08*CzaP9GGJfIbA7@0hAGja(v3GnDqZ-a7 z>J7emlAxOITId)$;9hZ(;1bzONWd5@@Pdd02^QH)gsIn?S`40r-hcIi=qnaF8KwwP zFX?{N0-ns~nkaKH@{^EYZ--K(RyP3hWou1X5Zm7aCz&L*yXzijA89B2oguO@jvA?T z_*%CRcC;3vC^DTP^(HYk<{LDv5!a?+kqABn8FEVjL$Ge<{4h-q;es+JQwZS> zpE0z1KAVL3NvaU`Ih@r{U{g*DTku1zA(|X@%_#n}Q;mB_z{0pqHPtc!oA+iajfy6FwdF!l4-r@b` zSQWtq=0bo6-KJL+&Z)>R2zRno=sRM&5U;JXZjL6T3%4es3q%jF3x#{VXMzPpUKIDP zpm6sejJ>TJcNdKtja7ITg!k_@+pFGR2=q3t``(jnzN^Bza6gCHRIrSFLm)J5g1aSo zCA=j&mGDM;1$&P|6ex*uCj__I6K>x{w1K>-&lP zivLFU{2^O{4`zZ`0Cs|;fw%)YY@K@3$b;~LnJe_3s}0!?BikPfCCBt09bb|!@E-OV zM_NWv|Ua{|s_;_mr-h{FBv$xyZw zdlY-a?^&5iUW3w6CQxZ!E$)eoh4~~+dkw-dF8Gu;CBLQ=g)@gb8cRa1?=uE+LnTy| z`s=4R%n*`?m>r`iek&~C4dRU;K(k9Lqj(kf7Uk`i=xfP~GkcOgt`bP(B@|>>Mbfad zeEuX5?FIc+@GyR}Fog6Mom;M%U2jS=R|DswiHxamB?m?39#sV^z#xjQWn;6wLi7>O z_w@?B&$X6Yc{L_kIP6kyesLKZbUE###Mdh?gGie=}zMlVPC&YR8-C zm$3eTo+-HJ>B>nG&U4dz!PWxG?7-Hc7g$a`s`*!elaCfvSN>je2>Z*GFPzQK^)7Iy zIE_6C3tQD^7Y8WUdp{io^L)UxLY5sX3Bq{SUGlwytC{EFH0QTY8es>1*fqs1$-z(N z5cK*4>+7CBF8C=(;TtFESkyB{zEO7QJgYpbd8lypNd~+8N~n+=TMZFNPUXm}b9+0h zec}AuM=)JuOP~cq5%6it>%w$lB|gyc%@7;E6y?Z4Qa>uEbaDL2i!W)`FxU$zn+AZF=M}MCS39 z=2TB@p_Dj%jg@IK&#SSbS&!$5_o0n@Q9C0$+QnU`gs$(Psx)$pR+qeX$PyVUn=D@P zm$JLK{LZzdr?{TlEz)J_Y7yVz=%67wK4H$*-=1TiN9ajPu!uz8;>?IQeC0%Mdm;m9<8iq$5~Io$sY=L-%PNQfH57tfqE=WaJv zLRZNYekF%xx&e)%lQd{}(8bRz zAB6(9_*2>ej$W>SeNtm_OTT`wxP*)P^#e2e7-n&|0kd{Dv?0PIy>?Zf2%bY^^OzW+ zkXAqv-4naBVNLEme)2Dw%tFcf7+0-xB#~yLxZ3tY3SU#z0miQ7M2bTF>4oKFDL& z5_P^!5NCu2s@!7+i?2ea_;B87QD-*~xd-5$CaS6w6x9NOvq$76lDsc2@*WV72gTvm zTMJPA%mnLb^^d|EVyebn9-eq7#dzq5PcH3SlSS1dF~%Kh%VHL6ZV?5LYHPI%?)et_ z>aOxT#7J8L!|Jowf+x3M`km(X8_YNRndN`X5b8;&o4lx-xX`7i2DdmqLhLTF#fhSA zEtU?$!meRwq2&K_MRDCj^WcrO!oe|{6>)u zXUw%Ig{l|cM?DPJ*bJdL)&<^jeRoVVUxSDN3nZ$6=vRxC2-i@|I5)+g+DV?UR!LK2 zpWA|py{v>Q*CR?OTrx;0G-ua&(tB-E!7JjbxthZ&&;@>duW_^y5weW=5cJL0UM?4d z>f&J^cWEJ4_?<>>Y5(lLD-)Z?Z2%_x*V9AGi3HQQBiN-{+Qk_+FCdZYP6h5$=Yor* zmtD;h!KzaHP)_r*)b$|O_Jxh~tXiT{V{LBeokIlb-iDPl?AIsSm;$JU7&Xd$iY&Fx zJ9R%b1hKxzOtS3hD$V*lFETE3OaBVo>)gu6IFv(P67iVCX=>^3C;lDD%`O7vki5&$ z1G4wc`qvRqLvfA6%;&_7T!$5KH_@*P#t7-Px;A+PM?!Ww1YcDn3$UG~#Jov_ab zq3b=Ti4FCF6_h+&@n2*rkv@3vXoTs{^3yLOP=BcO!N#CH6{N_9TT<%GTn+X$4Y9yk zQRGrtJ*GMuVEbhHvT(uP)RRJ_0^W8sFn7EKqp7S(Xm=X@!XDO!4PUJh+b}Jmb1@C8 zhVsRc$ijGwKa9;^y-d>(p-iAS1nd(l#WIYX^~N=ax}OrGS3ETSEsSSeDn5f@mbATY z-;$O!^l&F-M5Z+BCl-qxzOi7Alb+Nb$SygW6Nsc7lw+{|2Wz=sV6_bO#av?nc0gwuS{;jWu*9T8nIT6@>rvaMlpQe*OO5ZxJ zG%fxb@uun2s)8M1dJbn}_$df&+(de@XiIPR)K=yvmu7GovA*}|O6_W78gS6`ee$vd zKT&h=Oq;HZ)i%+#;{?r&)Yq+)#5HWi)V^wfU>O+cOatb%OuIUBIhQKVTcIrD2N4$$ zG8^}-_6XD7!EfQP%fvcPq-H!#UY5D&k;%0HMO=o0^^Gq&{+o7mFqS`-crDnEtyraV zF8$%yJ6h_Rr(->Dw>8Gso~!rGLFGMt;(M75uz3yzOzWOTrO04#@H5_R@_@zYOfkqj zWe=7z?1&W3z!(_gW@?-vG>Dd@ml~sdWO#3USOO&E8xx23l^I%iRlLOL!w#i|b2II* z;S;BgPmw-u-W65s9L}~jn9a5fi^JTOMg+|a9B?_m*xgc-yQTBnRYqVh$RUGg?tLIU zEK20NU<86+t#Kk0q~=3jS&z$x~t z4ARif=?-z*5jEoH#1|Lil<< z^K)e6^Rv?zDqohS7ws0I)>!z`i$7YHN*nkO0wZO=lr^o_ItOHyss7! zM*P+{!}oj7pB;&c)yjfVpdfZS_UZ6j{;UppP4# zyAO};M0)W)UejD&mP~va7|kJS@_uW}@Y7P#X*gDd8)9uWA?l6YGTt9=_a(laQ%$Pi zmKyaUq*}G0tY2!DI@+BdP=cnKYFkA{3PPx1E1iRp?==;32B2~#r)<27k~+Vh%GHPR zNUyr+-vswwBR!1)jvJv-VqI&uIOv>$Z7^6=1`=K}elIGUmi*2G=l=%Td0+Dqa*uu^c?5@}4`+uU@z}&aq))SkCkoShON8={w3Gc>+;HJ>3kE9Cj4TX@b>(Na zQLc_=$t6-#T=#4Bh^xx(4074nlMYfieUh}B5s@qA*lzgbXsx^fR-twC5Ea#SyV{{J z$Zw*Lp|{BAXSuM{;~dAd@~kyCT^d;3992nyPB=5dt1@vID$`g8`a54XjuLZqnO3=D=e6G2=r;n|(#{IxV~=L;utcw8I*EewwH)%Dr<>(SpXPXy)eHxJZ&O|DT zsHDDav>>{UO>b9Ej+=Vp3;xi_tqoEu0wS&XY*%Ge`p{1Tc)+F0kUs6t)#lOMFP5Th zcIwa7-#!uLym`N*p5$rFLK3aze7Aa;@p5#pdnFs;WpiE)z<4+GqdzP)CG8ZLZqWYK zX{H^Y>HHU+t_d1fhs*ejTB>vWEA~M)d97$5f`CDe5D(+_=r4ke=+RM#z6~j5 zm4#f!{i-6I8#i*!{K_+tmada;|BCbab%O>WJ)^OMld(x>d*Pvfn2UJ7GiuCb?z5Yk!d@E&AMT;}30{O%o} zO!9y7MzySAv|Kh4RjgGqcpFO3B|7q7G1<4-+#|U91Z`oetgx-?yb`*C{%#c3B{gq_ zD9@ZRX%t}HSi0Qf21=z%Iz6aNg|QSZ*zyhFy!hZFek|U1dpPr{pD|a!&y^rh|0x$2 zEpLC)w??av>A4?6WU^->xKb6V1@o}`a1u~{-8yt}!IWWeEox8{_(7-2G^n_3h@=Id z@>!@q^!Ygf$Z5Hb8ufU`w1jfH*4fHl_;l-_(Be86TQ!%CUTKG7Gh%&}bE_1dh%@aD zpt>P%Zl3+NFVJr|x~uo$094r8#6{w;Cw_AQq@zrk8$T6vK1$KF1#0tnb^C`#jt8Zl zW$^5LJEFaFt;}SuvUwa7x6dCwL*N_WU|KCItfG40f9sfmGUpgte*3pbOf*zN9h zC4b~6`!$1=7N_;}nP+P$*_9~TSvIR-*OyicjZyrm_JQz>o$oPuSMSIy95!-Gq1}cE zJk*WOa1komXCEnZZ-CZ~0z@`|@5R|g%T^TZZl)7SZi9HKAsT{H$$cFqXbIL&qxUwojICI_}Eii>oqj$ z)WJCKM3YJ(+TNNME;#qlUmJ|CZ_%=l5I(j!Phaof1XVu1#7!^U-1PmR!_JtYnajh4WZaVKflAVrYbwq5!Q>Mnbi_+QV9$!}kGQwF^t7#K^?h9ZGL5 zTM1+hWRkOZD7|Eo0K%f1htEW3RN@K(jiqab+o0M@Ufr|S9cD)Nvvwj~`asi{aGMC_ zfzNQ9G_7r&oi~jx;tJ7zBsL;Gt%j`yYBHwS@f&=~)8V68G*K+W{_ZBvB-&l#c(hf= zPTbskyMR4NxY$~)GRmXfDrqoTa11FNnd!s=TqVL1@GU0SkRoGkHY05pV`9o_;^Pq^ zFot{%rP&`RlrjONUMME1PO#YDaGv$6O4n!ju^2EhZZSAkym)S%jy0T+aL%V&2*Tb1 zen^j}*{`Yc#Im8p%*L;zopiFe0o-Wa7~RSkhv@20Toy_*gh?ZX)Dj#&<2W5CEaSVi z%ExcZS8r=^6$4xA*wU=4{r$`Qg+%4wA+T%aDs_BQrfjzcRt@c3ejMaA%laVH9`Ejk zfX))M5k7I_~AbkfLG&skB#v5k0>5+|k9HXWUxCbh;xYf&A> z@AqfhEG&#B-FR9l$7ywU&N72pU$%(yL&?QsyLmxd#DW5<=QKti>PmXN zr5oRddqa!6b7s@lM|$D!;1SSB`12n}SO!I?hFotQovu`*X`PAZ^WPcKbuux0H%Feb zVdpwVe4pJ>bO+WV($$*}6vksI8eK`|5cwbhJMj`Row_5ISyCbP*B`H8H z=<$*P+ZTK77rh+@n~)|%j9tWF935hbpBl;#pyP%pf@FNQf~EI9n|?ARzop1p%34P` zrz3w5aCXP><0SgK#=!K);=qFdBx%_MV;IwT!M7AY&()wCe&0nZc*b;>9o9W%>b;dK z(Wu7@s;NB{Uf#gOq2P-1^!Z8^k~VWPSV>c6hGo-Xi5^yI?bNa{BRIu;wp!q&ZDC-Q zs<4Z`k->0+S+IjNxQb@kPK^qs1=eovb;favMOHESs2Zv{?T0HF8j=rf*?EbEn#T!6 z62(%mb9n(vbs~qZK}4*2qE-8WF&&!V6@n^W2(Fb6Z8=Z{Lp|>#Cati`DItV62Cpo< z-$83YWw|AVQLVOCwe2m0^zdL|LDuSIDW;BLr!<~W)_mW-?5>;l>LvKmqI+1P!KTIo zRqr(vPb}DQV{;l`y!4D0kM~~Yn6-f==V$&7W{nWBELY&uSm~1)%e096Qi;LTN_>I{ zcmu3H9aLE;eSRG1PmvMTcqs`en!%~DEC7`R++ryPiP$}EI1F3`r)=0zeRkfHBZ)qm zt(a?fi=qS?mwF|4Dj1jg^MzaLle1y8Y)|`S+?7UaxC4j}#)gUrv_{I0HNqpgVRQL3 zA9Cum1!g^OnVphUQRNOy@hZO2Y{nYc9q`uGsigO;TKBGQmTu(%oKz7}Pk{jN!1rs0 zn}cJTIcp=D1FJgG^XvkzP@OssqOjXtqB`}$6o!-~hK5-seXd>$BK)a+i3h}t0;;3G zFfmmzK5mO*^c_~&!SSk9s(f|WKV))q{{&O`1i3o|<9cf4a)6SP*s(-zk`)0w7qgCz?J;{1jXj?ifi?n%}IW2hK}o4_S|tf z;;H>suPU~J#g7#oFn6xGuGVl_M_V|$Vu`rLS_T^psCb%6mL2tiw&Bv14_NyJa1!P} z73<-Tom!pYzXTz=Pi`Rc{HhMc4KN>&$nKHIHY6h&omwjg_!%rP!efr9e%KGE6$5v< zXf0YU+(QbJ?Yoat1SlY>L9@s#G^=Vo?l=5uM&MZW39h$MTwSsq?+nfuaTN7*wkl9CKA>??)84^U1ex7W&)nD zyyXRBxft_TOx6rxXTm8_hV&*IM!3V-Tw23M{WjM!l3crH3$#=(iK8F%oCESWE~?Ng z$-;NmXnF*ccePx2Px_3IDiHBl#@jL&n0@b>dzv19I z_B|JHNmO@>p3D2%^w5vA#x0v{GO)TPZ}+ig9h~a(r-_Zvytq<2M~(Rw$Ip9xoNKk8 z2yT4(EFtc2xUMbb!-T}auk)%K;dJF|EbE7tyEXcLeA^=(1h9`0PYRaDi)Urjm=)M$ z(CF^6qs9bYpPWVY-ri+UPQ56~xI(E)m$wJS!cBcr_9L|MOEa_(w!~5AWZ7~$I5T_h z6F)vN!*w?hehZuU@~}b{hrgC4qGODV@EdS;fFC zG&j2fH^Q&pS*@`8JhQT4o(D>2)+yFn-$N@@L^Xr-mJ!;6_7v!3pf|BR7)s1v*dTt+ zx+(h{QyyU?_$@j-|E?lS*Pz*CaTV#cX79ds7S_#Z)(g|epTO>CizDq1B6VWMT3Jl= zfXoiaG&`*mM;oV|oVl9CVfSH5o+UBKtbNNFT-HK3rc`A8=nfLj_dtTS`%12h;5TSo zx{_pVukJXnljlX?8vPGfMtaY*=sgLPlS$C;?I|-wKOQ#(ou2Z+dGrf{Xlr!r?+-r! zRuBv5rOg42!``B8>zn*mG3>%wAepn)bPf8RptDIAT$5LXYY}#Lppd`!an&FEGyll& zGp~E3h?TGJ+v~&UcjLgq&g-$hr@k5-0+yv6_;SvHZ-(XL(r9IveTIOG0$U>i)_U-r zQc~qm*v3y7)l|X>>u++~FrQeeY=A>oq5R!!&cD|FZ;Vn(6x_wQD_q&MrvC}i6 zc#IGH%IhF-Tkh_q$Pywby0m|0iNSITcgT`K-+&Ast0`qlLBdk+$4Y4O;iU%5vK(!G z^b0v^+I}MXJ1I=KZyQdqJ7PUrD5i&)ti|h0+@s|vP%nlUf`bhLc&zX~?np}dQwIBC z+}pqoW77n}Dig?yBl7@eBEQ>OEt-|_q18rXy}@Eovh>g!=QHc>!698P^aDAbwl_LO?mbLSL~CHG zq`_T2sy4D-G2xHH{;$L%T(5ImidGrd4;(`zDTNCMLc9c)h3`&2;d|6fdQQO% zNsy6G^hqYLCyMpaSYa1zC7G@sCC#JnKUorgt=Y=p{h>~RJDC9)bZV%qtc~NNn61&< z4cC^(@s()K3GJxTmZ0XYx!B5u8Mpt$rmfUCZwCb z7~F94R5BjAq^`x$GkW;l3D{BNc<)mV&xtB_I`g4O$4JD|$3x?>`L&pDNCcYbVyhrD z#o^9I)z>Yhz+ioRH0}oWsIPBU-IBn4`&i);VpkIxtOwgJu01vb3@OB8&KciazlIYn zTH>%PQv7-(*1Hhj(Fv{zdp`66afev(IJ4kRzBRh`A;Rb72R}pDS%>=sA?FacJl8bM z?P7C|@!5Ins^{jqfr0PkU@6;DoGVsQxdHNtSK?I1S8UXaS3k@XUM`+5c|F`bU!IWi zUf|p&OzYMPs{K$-h+fWr@%m*fW1ckHp{?o^08kQu|8!3M9Aus+Y2^8ZeL9V=pM&t2#M7KtUkjUF7MjmCv_B9y z@XiF+UPR=jF|=Oc*<)>rzg!UR{BiU)*~hLv>Ld$sN?Gh+@kC!v;9b@!4cX|K+%A<5#p^R5x`pL=Ue;dF(7?13 zyzh>vclG&&!Ta**YqAX^!-H#Io$Foa%J<_){h$Jk#}SJ)wfcsOrRB>oEDN)gXhvK% zXR*7;W?hN3848|*`O|NFoij6yHM=$6oJ*Xp+!5S&FuhuD4yI@*^6Yo+!O3YKYct4kY@y+@{Tf4Jio{AoUzZ{eYtzw7#QG*1e$B%~ zX!3|`#^eaCb?7YEW7a{{g#0ih#u@xy#l2-zTwC_9ouCQs?oP0(!U}hH2?PnQ!QCNv z;lZ5{oCLQ(a46iJ0Kr{?1}D75>F#q*cmLme@3>zsH3nnX+-@} zrk-1waKsvRWOLQz>+TNPu;2AiIt;gRWn43L6!xneC_k-f9UO03wBK@VJNsMwwV|CIDFWT7MZb0Y+d>v}iC7hP6ph~mi)YQX z3O^TaI*gs)N7Ye83oZMi$^>M>$D{>dd)0P0k9$Y;v|c$O8V&03clMo6IgtCG7cD9h6>WFO9zERg21tpw1~hKNzkD;4oo zlCXtg-7s3IS;YOva~d;ZL}q+K>@4Kxx<6(fWH45D4OAW@0g+rjtChO@IG8$F;sgU` zO_Bsy>Lv_+=IftgN4t~*fS1c0YCF63N4?@G$z{XJ3YxozUQ=H z5=F`%W{0_2&g$f(EYyMH9bGC4){0~(5~H~Vd_qP^tSDgzIw?j3@37K~D+lI%<`52c zdEg_0*pc7jMI1ZrYV#}7jc&e_{RF8W#d@Dltb`7j4sV{Yx>ARHZMZMpfG4% zcvo$SZ+gGmd6;sLWpnNRhTX53;2f@g6K(pEfR%48WLy6y-MO7vsUm@_8J`Nly0zHO z3}8}vi<;2}r~Gt)Br%!08f6PQBW}K3poc7^h>McTU?8rVKDp6oVQH9Aibo$d4L;G2 zq;Dq8SBFQJfBR+?-M!eT`W3-y89}Uz&hC}U-FPINW~8e*BmH-Vehj(P{B)bte0JF` zhlnGF85MFDix(zw8!8_b=G%&6i@qU z;N$!1ap9(q2mciQS9@u_X+^E9^h0Gb?f?|E#mGt_LW|bFD(*LbHq#!uw}l)+uo@Q$ zLVkR)VG?nSG?rv1Q-+jU5?fK!Hlf#P!j00cAx$PWT8d%z$~aFuJzsmLEdZkRefn}| zh%0^XzPG`(^WK7|y^MF457<6o1|UY%F!A5i;MzSW#rp!gckx9uxRi}vFHo6HZ8Ann zTAHTTOmsBbBGHQHWsq7`#s`cEs*LC)+!^<^djkIUCbs$!l|uVZ=b5(N!8y1Ka>EV; z2#H4dUq6WtGZ2^e$1cMW+bV$6jatn7QO8ToUy+b-@4M(N?C>fu46v@Jqwi?b$>pKM zD}qoutW!DCRw8Mm)%E0slOJsGS@2`NV?;D|@8RAcLBd<8Wr%4*Ew zdDYQihO3k}Fqx82ei66aDJ_?p6&)>!^p==FCxJ~`hd@fswZFf3!ZwM1FhbzFa10S$Vz;SmdwO4_VX(%7MO{HA^*kE?uH-7q1Y# zNoq*4s;}+YEhy0YPWkX9w{k4R-S%_Yg5qZrF5MXG$j?FFO|;6GYhI5II=WYxx%SBh zO4-3ijqw2oU0pxv3{7_UNOqYv*yQU1e(hUs61!g48GB^86wS<`ceUQN`sA#{Z-MGm zi?Fz6LKmN17OIt+?O`I!B76gB7O&SSD2EReWPr%~i+e1bn5-jB&dm+M0VJaiZgqvI z+ba3zU%r1xTTE{0Lvr`?XUtqF9Xr}6aVA{NcP9T-RH-j&rXQepb1yCqUaxA>sL5s?z5n?gmIalJvux}IthYwO6eT(6 zh3Wd^zC2wB&ZsfRKk-RUPc z{p_XQ+yIx_q1afJ_qumydp5N21ksD^+nJOji^0y>_tV5K7*)vGI@;AjR*QbFJnRDC zHKK^~T5r$#DljGTLc{>9tRGnS^XPg7$I-@zfPxo^!`AeQqPa>5yx6w z&+hEERwUP1*xm~f&8R9VZeo1$4hR>Hl&HilcqEjllhfYFk@QLYwl7^A zJ)9-Ptu~rjnc6pT2DwG7Jr5hs6r`;**=tGng5MeG+TY|mXBzH!=O?i|K7f9HCe1Z! zc>OWDr^#ht=mUhZ5zpTL1j((8rEFqQYe06I}QfC(+jlg?lnrHir^osalj1?~K zYH+E!j8*+-6XBch9Sa}L%o*_QIao5^+)!vq$MrO+X?g1u%WxcA?3!m&w6eMqJ29@=tNVATF`5qEOIESG&+zm>h*7Lj{XBalz zK(PiPM8>q`jXUFlczPsNmLS&N*&(Od0q`7nGccv)u)Tw|=Z3I`aSS@CDIHIEPIlY= z?PPhgw#BZFP!v1GdGVc}T4EB$;F>rPKU! zzi0Z5%3gZc=31z8881Li*<|e80Gcm6mW5DkaG4Z3FA#B>nRI4S#Jf=QVb2Ma>GOON zm~L%OG$~m))&DbdWYdSZIRHcoB<5b|C?Gf9RZ>kNJ1{>wPq!zHPbnZ+yvcn)oqt?S zkJvc7A@}8cM0mv8mbf>*NFcYfHRzI0B;Y~**qZdlDQs{4_ZH$TFM(dMi62NmP+xxl zoVjv&VUpxQIjwug#6eoDH=&u-t9aRhYC{fH`(6mox;?c#+x=Ses z4Q%9|xVB87p22#dOn{OYYE<*#BkLwTs&fq$9jb4QXr193=21ZJiC{M+k$>L_TDJpn zj>R>8_Z4y5(1~4GC8yx%34Is`K(P13D@=x1c<97BOom9fQ?L`>HW^Daxi z@qhc-^2Tki@&SJ+ihy=FBMRVx(G~r@G>5ph@?&bXk#Qu8%f4#}bIgmx;zCab6+It2 z>D&8hbNs!w!ZZ$4v0TdS@rF-)oqeHWR7@t%XwUAlPqT1UP#!xhKN{`gj~;$qkkE}f znGsfy-&MVboE-V96svw0WkIS#u*dcnWtCn18okYNsW+|KiMxqXR-Px@d$cmkSA58< z9?OP8{Nl=Zdz}%(cTVWN2CiMeBg(@<6KQM8(HQFfT%Dr#QDxyalK4lzQRKR@^D~Ks z9SU2D)I{98_N9c|ycNpH<-KKEu7(U$&44b%7PTmDSBRsPECu_xw*kui8HDJX!Y9Wo z7S-yws4#rF8^sUzK~9W=C*mQo?C`_o*Sfm#IThzXMh_znomz+HkW|u(p;r&itwpQ9 z*oPOL#$)YhlwH;*9Msh%^mxwe$_7OEH6)4~UIvczbNVC-4P;YffFhRKVy2L74<^|( zCkf&F=AbU)3z!wFa62oBaN+&W)W2cAgwoe@6uZ1g9c#k}GY+h@hherv+U%%E`jAEx zBV>L&nx-!hib2rei8b@w{WU@Rnb!55UcN70ChpT41J9h~`e{z4_)0D3z#t?Y9!)Nh zE<_5QQiWBwp3WlAJe#lM8)oFT9aKW1vp;PT=Cji?YicD(GeShCNSLdz_Zru{ao$v` zzbMnsuNA0z-emJe0`35~0`X$op0Nq(NVWNGi9XjD=K5Ab(C_U}VRr56p^3#pyrxa@ z)TY!S)z$T2_I?laVft=4{G2=KMC^+(!2!p*H2+OdT@nrs9O<6KtqH^mR-ci0`_(Xq z2|-mMX2V6`%QP=lIgysJs<^lvvNna6X=~K7VfCMSPPm9{&pWAAm)Z} z80j$Hk>Mqe$##U;K?v@ad%fGiXS}Ow{9kI`2)16&3lmpnzgL>=%(w8N2ugXcnfiR< zJd_e*U>?_sZxT|zSj|l;$*v!D5m=eiQPpg=O<}F+v`X+2OA2&!uAt`Z>u7etYm0t_ z>>P*2z1Ns@o(!^XNk~YJ zmD1+v*ZWMbCBWc_jXcGDrz^{UWN^S2k`Q9``o}_MSD>gCm-u6P*XJ_Fx(@{Q zBPbHD>l_uQH#piieFAUL0Q3jsgMr)jP014!$-gFA2EU1wxrm_xHF+AaN^7bqCPYd? z(=NQpQ|^OtH0nP1*VPx)U}q}qGHD%m@B7`~>XVmHofERQgw!eQO}CSw3_ohab2M7} z8~qNV^QeRB=Kr)JmcP}O=FKdfmeEetmQtEYlTm?KTvwk7YTX)1U8dHG+&FAA5$zhT zsgw}!(!f)R%LA`^^&xx^K1vq~3*y@M>3xcIL$s%|tj@w`5}VEa>HS~hr{XJVj%Ai( zTFAYGoCVL+H$(fY`_JR^<1J}bWMU3O8HxzoNZV*GDIUpLNiCU*xU%Rn2$!U^ zOqjH7e1JC25%c}W6`U9Ml*ykFB>r1Y2K+BQ*^}x^(%i+=+1e3SEA}5MFlA#qn9@wc z+|t4PwH(Z3Yi#L4#f8El=3)xda!~;RynO7i+OXU_C>&2spKKs*06QNq&yz{R)!a^t zikp)i1OokGkT7JQ!1 zlaB45qEq1Cs?YvISN8PmKU(~iFU;GM-U?Ql*4V}T>4Cqct>I?k`iJCBLQeAO){_^Q ziVF5k(?Lc~Qpwm6rm8c8C2#HO^@mx*%LSIYoV|s^lZ;N?+|nARgY}|f5`$I4HHVeX zRdF^mceb{-q+Na=~u?p}YeDIH^ED9(GT5MVYiJD7(T7BM#nrYwAV0~IF_b_E7MrSR8X9&T6* zUOrwb5EqCY2w*gw8{jF-zf54*3!Y{YCl&Xf4Y|QE7qAfj76*WelZ%%fHf>?O z#0BDF=LGU_0jW4S|BUI+#$2#P0Dz?gq~e9e`Og;IU|5`7oM0+0Fef`0mIjE5iyIc* zAEN-9uaeq-jPk$c=u@x$Q``5y66bLOU{m9-+Q)C>$LxavXuZc?kz~yhh#}(KFR9&J zlfj2^y|>Rkq_I!FP?N=Mc?eR{vyM;LC0x7_{;u|#(k5RhPWT5C=}F6+G+$cpm&6W% zMyqa|&g2w-dcm!6U4qbciyT3;`aKyfyet_*YJM-?^AWKaa>44|Zf3^?{p)Rg?!e41 z_3eQ7r-Byd1Nawf#b0OJzQUip7;30TaXQ+uEAcY(9V2KIRql@=!&hubUe$GZt_3wt zZUKkB1_AaB1&cpdq>7YxA99)XRYQhCp%~VwSX+|J>KWALLA&WpG7B$hCiMfx>~%41 zD$`Ol3vR?;g_C}o&!JF}wN{M7Fa6LQ&AMX-(z7E!U5J`_Zq1JV2FdNXqJ^K3Bx(v8 z%icPT^b`JT{@c$f56Hdh86E$<0sgJW|Gq2yxz?YIuqpZ(GxosP>=3{-;C}0Gr*g-RXZ( z0-FyA=K1?HIP!hzt2y)QvFW(=`_+IV!+@DrBDHaPh#Z3enpuA@P6}4g$g_Afvl>yJ zn)i)j3VB4G-=v&M7snPBZl?&Xun%b(8XNBSE9x3l>uFH1@h3iZ4IeM@CYp(hPfuQ+ zKL!Z5t@*7q?c{ijHtl5j9Q!09P(8y6?GWqDGVj5k8q^qh21U}nTPNj|NLWOQm2u&@yG_Wm_z=6>}L zwA?I%mF00z``HN6v&A3t1~aP%w#Y2=&&dpPTiQN(x%hP-_*tH63C;+R9O$w)n*wfM zh<3U^vo_6N(Y|7~-s}1u%=A%#n4z48BKHKZfswnx^=-of`vXyjX&2Cy`)m0xr8xIJ zHG|lPcKg$}v!niHH!K}w)5yM2?L%KnHc$JiA2e(o?0(f>dmKk7BQs?L{cc@cp8bxm zkCOR}ptRK2C%oP7q3^WBMvVlC^~ce{yGyq-(g*TvCA;$41k@r+y7OOluXVgpgiI%M?@1wJCMnw=IESoV<=+5xl)4=N)Eb|H!+%3XQ#)1tQlsfXD18Ix;DB5 z<$B!m-s*J&W$+0b(s*ar9SPc!dxjp!?Ie&~&Nytxll|IyoD%N1w*duq<-aSHN?CS% zHTEi;ET-m2Nr_2cHCD9_WBYWoz$;VEfr-iXK2jB=qAQWiA-o(OCZDsq1moYuL}Son zb1QP`fcips1kM*35;AZ%7H8(yu2dXsj_lJj7(n}r-vp%VGkIe735Ns{CKjei2wv>G zY)$X?n|pKB{!237{n75H3YR5k?~v!HBem~hG;cU(EC~Zst&^QaT;iF7S*?G+#iiuA zhhY9=k`&2$r6IfiMjEL{&$TpC`#|K{AnBNyyTP`IwTo4p7&awt8Xe=jUy%62p$hqoSpN`AM9%jy=#T_Vc(jQ#Z!8h3Ag)ZT7H zW0T2V=x$5@YIJ-7A0aPCHn?_@H`M<4q@D)?pO%~@LCCf*gl-On+ip8NaOfW-ZiBI6 z00?`ikdwezKt9GxnYwP8+C6PTQBk?}iH^jrOJFC~nBj$;IN+Ti)*$p3>XJ_OScJws%| zPNw-R8d`$nc0y()8dZY$_Jnm;G_C|0KJR&SXJy9=1}GD@l93pfgyM+;gkS=$8Un-$ zCqg~#d<+H4iq?tKmYcs`vc*i`e}h_$o5cQY*Zty6VtTkwb+*8)On=a2r8YdOyXJS^pH^1$X&7f7u%x5 zCfLJLtYKaddVmP{MPb#mYP3P@oi1^xR~3^d&UQBr6j3$s4Tb_XAx+MEDOH$Q`7>5( zhcgtV5y6Ssm*bjRQ=EGpbV8%m8M%sFMW#h#0QUu}9XXXoD{41%rgNhT&KJTWrH^P6 zd@O-JO>KbU51mn^6hx{Yf4>{d0Oh8s2z7(FRS92HTz^8WrqIPE%|)-qUI>1K>6qU) zV(??lNidplx+1TH63j#9Uzw2Fira>dKuEDx0x=rlsx_eOjX2e$6)0`!q-*b>N>%u$ zAZUBH#r5;`EgFO8LJ)0nj`i1$N+(SE=yNpofkzO%wQsA~*TNUnzn{5N*#~Dsgd8PK z9LA$O5h?;HY5drhAdTeq(k&SFFMs3Tu4+LKK2bU$FA?~nx_2Excss};A|V3U4wCn% zO9&p+%V;nyHQW`|GRhUwBie%_<+`1t$okRAvvv0C%2ne=cnZlDT>ofKLctJE-g~-h zhWi)(k)D`>(ViUlWH2!`FHB7B07YI^gFXiJVNr$zt{$A2!qn40AR>W65D_U~bc(Q6 zh)8fBb^yj57NwXk{%_JtT459wDq*B{lr}iHdjcm>=#LY+bqetNYgEry*PweG|L{ga z|F~tdeF*CHoeEkW04Es@u(En(MWEfj(XW8I_>j*d)7!251UK!^zPZ#QG8YROLh) zrm}YE6zNcK#5*;=7FcIF;elzal}`{o(U+xs$$tl4BejHIGmxPQV}`H!K}EWeApwDy z*el|`7{5^%u^*9wAu^JJD8)fzT`Oy*(64J#tDGmXkmD|EXh4T0gCklZQ(2-bFHw2n#5S26=||L+XT!8s~Iy?Umw99r9aj zmBaX~T;C;P6RBw8){l;&H%vaPA)T4nE66Vtd$_fhRNz#iWlaSl(+LF!wATHfHTwJI zOHc^&EnL$}U(v_63M}~%jus#TinaAsJDDdH<=fNC`-3&(O%mh|X@tTXBTUoSsRTz4 zsQFMQaF`KRafeUT%1TrK$m(UW990b>L^9P_#`%e?RPJZ=S)rV!n+6@2ScDU%zwG1s_1-jEVs;ZjOm|y4D^Z1uHJRBzG^&h~~ z7v6m?&LYBTWZ<+GH`(F_O8r2#%*Wh)6BJeo2ka`}EBMS5)LNpJ- z#2|z{jz`QvAVWkMP5xl^yTU;RAkVFy7AS-tSrT)+arf#Ax{hq5aw_T=TRDX)bJS6f zKRrm)tmD0UC>g$aZsg_jyc{LTAhpy{A|ug?&J*I&=E4hoM$d{$sV;*cr|z7R_#D4Z zcmbN`XAY!!dE%P=FU53O`^A+{l!E#gJAU1ZE(Erb-(6G{&3>w=ox9O`jar6@i1c2G z*{%#{eTYSi3RP)MXR&|EH5DS7*q6$4<~LtdQxCbs$+%Fv|0W&4#)hkI!_g@$<@Bq* z0bZ*UGN<0Tip9lL)O7!rWl@`C3OZ$BxKiKZNcoL)!X_Ugg%w~Y1b_Gfq4DSB2e?OLAW(n*tGl5$p1w7wAYd8;N6^B>fFsPKw<9G zZ|<(=(@(7MuwR3z6)c|^Td}`TrfmFR`K&;CR7i{Dv?^Q0yt86c`?4Ct+tkpI*=<5Q zd^uddHx~PGFRP0G#jOUb=VvIYW(1#T6op{~?=T8#S?d1V;wDo-*M$M`P=T>LvmWp1S%9VcL*Zh7$4P-saZi?#RQwX~sCHB;TVQO zF#I?-l^varzY%VWdhrJaQzy2}DJxL4#6GGNGke#xknD8oR$EAQ+*G3_>+LBTw=x$h z4Rt;5dWS}c_Vy|ztf>TQk?*p5>qqe>7~S|_a$hwwoO|}isuMrJqrPsZLp9Ee@8~=K zG^ejmIk^kIM%$syJN}xSJ2ZD%!&STu4)(8ZU4gI2HVAKm1VyrbmaM;<9SmXa9aNX! z>)0A%dqcJ!!^O}jf4O&bALTA~j78TogIs3MQOUb{Car3b6Ff?yFJ)LKqO!a2lA%>} z2`-|@x1(NDe^N(5*d2oq({MqXw4$QlIF$GOx8BpfFp#$Z#u!NgFVT@}NB~}H-HCj* zy7W%M3_-%1Rs1?vA%Z$aNzs7vo*BUkVX$6mB!b}}@1z;3uU$kVnrC_ZplIGdTo)ZcvAqC*H7z?ArB-1yu zYJ$U+jL=6YsNa4f$887gNhiddVxS+9}?z7`$%Hq*ipYi}x_Xxzs-1 zIW?Bu1SKh?*L6-ICE-ptX2wUI%|1}PFE`;}1VJws{17B_d?7MMN5Txysc^Ky$S}bU zBTT~mQ#3i3{={%kJ*gZ|v}fe)GR&v_LMU+=6M)(0;yTUo=LpYLvF7s`NKD8(nqNs> z3fH)au2qN8HROdbZmwN!(H5mvmMrHZwJ1q)X-ZQ~*y>5nK~$!)tp#In?jEHr&}@_=ORiaHc%W-PXm7=}CqP6zOU-*><6LF8nh zXY84=9CK4VnWb~u-RXJc_05O3(95d!jl2`iR39-M4jqOjH4qbVQcA?iH~F96uNJ}$ z*>%fZ0+6(tu8AJzEr0gy{u*hhfmY6bY!xF^t&q|QWm3UJqJUfVrXvLsfFS23W+7Ye zlV$P9z?j6SS>Y$@o=lvyLq?nh^Jop>^X=I`h7N_kiq@PCwdpnWXlksYH-H+6xOXo> zn^mdL#T{JCEU@%qW6A~ZnDmG}O>q}qy`V6~bg4E9qiY<{U`XxhCko}#8Z2374XMW> z|57vjD@eZ5gbU}jI3BR}WkSeeYr?KUwlZ$v_pO-81I>x?p+vMUER(68x3nHPtfC?| z$`>fmk|}`FW@E@xa_02DY$-dv-`&OjdK6z4D-7(1IWt=Jd$X{FC67g&%~9a z8Jg;l-q|*gHBaYcW#p^K4Ky$gIXk=5dIrROVv;M5oxGNX~jDL?>Em ziDJWGk=?ZNB`QT1f7ZWJd@RtLlrj;!8w#ZZN?T)&SyZ7#NjNt_fV2;T&R_zf!+)ojTY* z)nURR5v-zKToYzO-j=IDB+x$*JYo)1<&{i5o6kNl1Y>(f;bHA@q*k>0-nBM+^V-X% zE8(m#;3VW=gIiaZ$rq1xMK?l(9qrb@p9srd2TjCF;?~ZVb$OOv@B^^si)I*lY!jml zj7GyvX$?S}U2k%=WM>xo@-vc;=YSUiArR%I$kMS^fADD`C|`0OYeD1l zY1XDF=RP`n_bXJLCK;!jy9L*7=x_J{N4(IP&jrOlh>S7ez=VZ-l&~immv=!@s)0&lu6M5SYysVq{??Js9s=u-H1jT@CJ-iW5Bh z*{O?pWj-9I7yD6YR5#Ya>(DJRZABF4>vLLSOGbe~mj zyTHFvhoc($MrT*8v`9YTD=~14>wAiEs`Lml{~|uIdR(2jF3XlwNXb z4YP7!96$oBQnqBeB9aa|K#S3<`%V|rHK3^eYX5*G<@RA~eaX+rZEp#mWqVZb*GwVV zh&4x-2AKB^r%f*Xce(d`kIN5PVz$<1oqG9Q` zf^)|)bA9R4zm*o;><4X_2Sl<1ITrD?*77W6X|rnHhiqLCE)Gd;IzTh%4X8kV`Y=ZON4%M?fUH>8$O6c6upbBiWr9mrTQiJzw-fXQht!MvY{ z_!o0sKlx{;h|nC8bmVW!c&xU&I7g|E%lKn0iLQ5;a0D3scX=d7cx%F%q4AEL<~dwC z8Pw@fqlfe%L!0hiocTT*>o|G1?q2!7oSX)%TdAXJY!jAD9diW1bsf@zh$Rc_4N#yH z1#5a}Ead3prS&et*#{JLg@p(_0VlQwJTtRx;yekTJy(k}R~|+mp5WSP6XhN!E8<$_ z_-Z;6Kt0h^UXMqz*3G0mi-zS4-ojwzw?`j(0r{mleEKb=sB|vViqGWMUNOuN&llqMjOL>s z=w_Va&haJBgsvaPz<5eEH6&uwT!C73<|+Btp!5r#O0~h+@A}>r8U`sKQM(PE!wV@3AAJR zL(AwY$sZ3QJI?#yZGI}`3cGLkmTpoB3XGKL<`p&98&ph1Olk0`W34Nrhw60C5HA!= ze2&o_8UfBX4t7i>{0t6k9vGHMnM+LfcuWxNEfe^9cT(hS@myCzLLWZ}MKFgsQAfSw zgP?fPGvnb`TB(EEHN%=r&!l@Xgg#q=XTADAY?XBu&y%pq5SI6l2;xDdZPg}H%9dJ~ zDqdBwRk4h#yz*d_2kWdGV2tLKyUdkyd|ZU;OZ7!mnogqB*@mL1XxWpc+cGpUBz&g1 zkxwY5YRco#2;9_zXj&S-j-IrQQQzw_pexCmMke#sGESLLSH*SDV>o=7$^V@_xtf*& z{cRzBV0WP5QiW*&XaqjcFOastsu=dy!3rV4KTeZ9vBbNyQnB9W=lZYlnmLq%1G`B#D+BU3TFO4a^*E98g=9 znsOaGw}_hpb)EMTrRj?yUXj6-6zH}Pfy~LKQ1q;3>)r(7s5JBoyO7^%=H>Ft&AXI} zvZAjN`wOz7f9s-rNg6QgF!nD*`4FT_|CqX)IyfFuVkq@%1NW0+k>mr(uXgcvMdd_R zLR3Sx)S>4t95N4}aqDElazhm7P(|uKPt^zAP;oKjf)8I#L*G<3dp>|vnYQFZH6v)B zM<|zSsz1nwA{h#yd(Xfe$yccNYv%d4s+YnI2daOPdQe09C3TP*tX{6lV*XC8U97yf zqu=~pAx4r>?4&`RKVk`&e2jcWr{a4q`TPkZ@lBHC=Y)-ERu57K+CkFAv(|Y0(RlrN zZjB5quTgwMkh@PiKiKc5twlD6dA^?X4JyK`a!l4gq4P<(uvDv z(;*j(872PNy=Kwyv;}m1zO6x(d?#$9CUt3b1j7g=3HfXU<9qcJMUwja#f7>Yx*~0d zz3J<$x8PgU_5JC2Fj}-4rA_sk!pVho1Yx-diV1GshH1=c+RJgps5q65_rx2f8qX*n zip~kkSI}k)(&%hpdT6-1Dcr<_4m|Q?tL&gf@I$Zs?~bNt(x(^N2krOy?0|c_`EfPc zbm?}&yPrlZ6594aJV(piR6qD_r{dGHAg{9LlgvtY>7ZKzGX4)glV6K*Tp{Hg=N%H? zUcUd9od=VO zMoId&?b$Fu_AGJU7pB(nxtR=}uifp!<(@7wE(=+)1h%(}{7hwiMGi^}f{z)ob)ibx zKUSm(+o4GV$-XHSR-vF{4mZ41rJmJMx@4ZAn~^7Q|bkzdk!Pc ztjk}L%U(>8og@&`u!*&)Ctw%xK~f>@=0kGv9LZMQT*yEoOZln(fu>GCjC+%|ly0mq z_mzIyHdXvxy6gsA4-~)~zBm3x+|`Penx88L3-*Ff)gim_)l7SD(sBSXrpEh{x}mX;4K%Ywr4`~M%5ADFsS1c@yXIh(2%nY9&~dJb~DGv7M`7GrAd3uvrNuH$aH9( zlSCh$Q!7rbL}2u=MGSVM8m^$|%)8(sp*#t(j>ZfzNoL=Vz?{;= zEPq842Wc0i`F#d)8ldehjQyZ3)BlC`ERSMv2y`;l-YvRm%r`8f4=Lxjf-^+mmPRRW z8(H{dzRMq*;~X7CxnL~hOuKhV=Qpkh`otYX!ty%IRhy{q!dY&~)}BFoc+2@FD?GC& z*#&Xqw0ENU&8F)rxcP&~H*T$OaC&v^nZNB&C(En6^q1!ab|$JjrOS}{gC@$WCsQH! z3hd(ogylE6xFU5Qz4=o(4G`k;M}9dX9Z|^KoG%hKaJrIL>Sp9(-|^6-_LljOjY5G} z*>m1RcWUHE58kluwV^*F8INLb ziE6kv(dQp$6w%^Jo*oZ>C%8jx8S^GP^FuR2{jHidzD3pB3lD1D`RLnbsi6^*qEZ>b z9J>pLc*%B&hHK*$PAb&W=`}5if(ZZdtgrdDB_z61hFm2?sfFy;;?MGu?GN9w#Ck9dW}Yl##81VjyO~^g zZt_O9NTv0BcD$U6^tSS~jJ&#&dMQ%z(gtcZ7ttRsM3S&2c$qvdmtBA6x7us+cQLXW z-(82*Nn^h#XRVQg+dfU~tjfAnzw3QR`OS3KyA2xfZrQ3Wlk;(9IQ?Yh8i{b~@!aO{ zln&gKc>SR>zsvxvXMuxl~y=-NKfAC`m1GSzYy@yMz&+!NF z&$eKfB{$@a#&e9)jadJ}?kn2S(|kRmzVfe*v6&7DVEb>OZcCF6%9E8_b0C*vW3uL3 zH@UA@wXuu0#I8R}WEMYy3Re^S&wNF0E}|7LuA+N%sSN7B1w9l*aCsM1$70@r`;h(0 zHoRL5<`~^Beld})tca88Rc@2%owV$~OW)Y1(3<9RQBt(+%%E7GkKL*FG?=1wy10}j zU-UM^u*udA*?bt!YP$_`mxSpYSlH~?>~^X^y8-;-M!F! zxpe7=W)R)OqNR0ATV2;Y_l=ShL)^>mSh=!0o@#nxWqUDW6<a9c;~D_Wv7%6Na!o!EC(TFuL*o z9^vE!u=DUdfqnmbg!7+=4g864^8Qch{v*l1l7^i`@Pt?W56sp-bN`d=`hUrB{tpx+ zj41?wxnX1_4+uuZ0-uP-C#H}O4B+DZ8zy#-@n0k-6&DN;<>LZA5sxr-6vzXB_3dA+ zpCaMo1Hs~iQJ3sbX9_%>gTVc-Jl_5{-}rxc@(YOf-}y$hp?BQ?td;|o%QT80+dlgj zL=uiKug8(y?w;4wxg*rmncb|mS5kC_vdaRRIrD$u+cl>u%OPKMzgx}^R(oaaH1TEY z1-!C)e9JTHCZAl-^K@eI*2WcraF=3frG1k3{LNy}bmxekzZ96CFJnSGx9W5_+Q9H@8RLJL$(bV0G@k5NeU~^*YEpF@cO;?4s z;w0Nd(2>5q?j`lL-+a=UxhOk?!D5mB!-*h&g9rcG z1D-hIf6~hTe+1({SHu4)#`v`M{sUwDXoLn$C``>JwoV>iS)&6fb01RS=VbcF@ z19O7^(;tABlM}W={^L2Gr}H)b)5iPX{+?Fnzdr{80eJrNIT-QI_n-N~O#kH%#Uf8<-_jho@Soi=K02Xdm&(@UvH|9?CfXPp25 literal 0 HcmV?d00001 diff --git a/docs/profile_overview/VMRA.pdf b/docs/profile_overview/VMRA.pdf new file mode 100644 index 0000000000000000000000000000000000000000..b6d67d361b44c9fdfcb596568d8df5b2a0da7eef GIT binary patch literal 52146 zcmbTd1z1#R*gn2%ppuG8hl+r7cda7bEiF<~Lk)wpN=kQ!(%qd(4c#e>bPq!eFf{)& zgZum9zx%ts?{{72ywADLd!GBf@B4Y)GwhySTIJW0EF7%d*tDZt8w=PRPuQQ>zO%#@ z7G@K7Ff{_(Iy`wH{@%gX#=^;l0h>+P!Pdmd*c7;_W@TZ-2IM}m;b#XXFb`YUf?gYe zO`p7YEy&5v$;H9WA;7`Q$-^zc#Li9+9Kdr5wkH4gM?^)jO+hAT3S9p|#n|Wx2R7R) zg(qx^whlH%*8lMk?|(cbX<-cpn6XJ(0~}tP8rzzfVzbGcg3Q71pYU;VaEOXNaRdXJ zY_LBh_v_DuF7rd&cS>^2Z?Z-`a=M9obG!m}W}We;*GD!Sk~cREuEgd|C{)uIOBu6h z7-kyY%PYxu8fB80NI%MQCx8WCHPu0{t}dHSCx;XkJg9zMZkNnmt|g(`&UTOINPeA9 zwKPDlT4&f5m)v&ru6PObPm{dPX1v_bCR^OjcJ5@jp}s+(sw;bjZ%6g!)xEB)!5%}ZG(3cjH1$?%QrzIN1!7Kc5_M1i zyAd{7a9iPzAp$aRncDoAfG?h>`Wvu{(NnbshDgMHvif_EV7CH6HzZI*k6^m;U|Wvi zrjA$HD;2T;qx^JG!nLPbe0*Uj$Nllp*ed`n_qr^Btkhm(-+ zgfbSt&Hx~PeNJ}^`hsAgtFL*}rCub|{_lAFZvTd-_V0M>jp|?5@deo)zo_@_k?VZz ztnAVaER^LI?_|J;p@#J>=`Rd5Z%`{*tp^Po>7(Cc(|}y-Yv&SVWNCPwj`RCvTJLlr z?)}J@!<1kHLb`fyVER=yBkW?fOAdg+h`r|nYW<29i?&Au^YHI+z45s`m_U#Xg@9=wWrT7Kn7IqXeZ}WJr0@j%5DVt>k`Jg~72;b&9B$z65tHIJ zKTs9}q?--TnG|fL&jc(7vFC0QC~TM@)*ywU!AO z58A8W;|5y3^R;~NSsfq-EfL`M@GeH;kOEqZ8^+|{BcLySoeH3bri{Q){ykzS3t=d0 z)WiPw_B)3(nka*Rj{v8KuoO%!%RyEniOx9CDZg-MB#UfIb48!hQt9 zbQ9|9ek9QAI?g#r)lcaSqb_-W4a4#pO#q>cY^3NEfFtm*;#$@Xi5fMR?)Y`c^#oW6 zXr9~uV;zBg3+Y`DBcbPVV0V~EOI@eQ4f6y;Qv$q!LCuj28@7M zU|~QeunEvze$5C&cpJ0G^#miV4PDKD#QL{lIyGQ=Ss&BOis)XR0cwrzWmp(s^FMB~ z>xZvLwDYe42Qizh#tOa3W-x1QW2$^z7)F@A^}qD;zct2q{WX3udfx!H0lkbyME5eH z^WQcZ=3&J3h|vO+1GC9~Q^GVM1k*9-Y4j#z=LNd_pWbs&j4#F*UsN!Ad=ul##P&4` zrU%hY`~}Sp*h;cB)Z{MRnB59&j!&4~3haM$?XOYFFt31I%(is8-WCuH%5@1aPS#u# z&YB7NH}kJirqH|Ac?`Y9LVv%M{-uzNTd7i%aly3X!L-A{wEaG`-M45vZqfd@b$fxK zP9^8LaX7h?Bqvs#3hnck;p7^kgpYi3mR%`BgI^X;4tJ3(yFL!S^|Dy2Oj`LqRd0%9 z%9mi;y6eM-w(i#Tfq{~GiiWz6hU$2NhPrzVmGTh{bpi#TVm#4M(^6}+Lkp2aYXYg~N#0+U1T>+GVu0qng^_Q7mcMl^pIX5R&Yl9`(G6zt zRL3a-%SZZ;9Q+#jqQ>&{^o!qwH%ZYNyH1!tV*aXi>oY)sVs|9w|#E29b{n>WU(D+VH0Sv{l&uiiv>Ku!a891%!sTv#>2R( z+?HAMD5R>VxlDa}E2hVQP5sCr6YH7YWuG$qwHaaGdh$e51>i*HXGW}kNRLz~Io$@5SMVSrGNt(-|x*%GOZMXaG?kMD=cW>m5{=71E-C5k_AElnGrkZ2C z+&e}h_tu`W(nW|`+1)$S5kDv+QkgQM$t9oD=X2#N=~t|CKb_w_Mn3flGmecg{WcZ7 z7ry6T8PA{LJ~Pk+{U+DYHDRku9c1LGTu?H5wRfmR*80XHYf7$!lcMe3BdJ1O*4ZR}%JS(;bB&W!BB zND8bv!e*0V>g<^1!o<#~?31^Qqwqe8sqqd~5N>RYM@a*yb?Wi!Huz4 zr9U*E+!3MZKIVV)Zl1MSY&F_5-8dHZ%b9zeo93K^Ep&CSKe%8mdkv};n}g8 zsoRW3(C2WGnUb6U!-k%Ap8|P#`hzu?D1hVg#=HkpzN?(hm4qk6*_se%BlAMh>j>6SQpiyzP-l4mDGDZ%8 z$nLD0j%*e`M8)gtv*L_UoHbF&f|}l3PAwc|t*khl=BWRSYK8F5TAKyAZG&sa75#;P zFs$f*)_1U6J**3>PN=Ne(6`){S=ab6>{HFL!23`w*({^g6c7vqeO|_7)AZ~l1J>$d zHZPO7<&FU+{mSYOV-mb(4iO>uh)MIhB_u5!A|ev?Ofu<+r*pd{^tFxI9ETiE_WX6@ z0-(Yh9?N2*&5jLK^e1ac!NA z3d+uD+rMtf{Bp;ucI78W${KUpMRV$j@h^co!7|2*Wb)~0yNCpq(DKm*Rv}8oiqQo@ zAxeDi(NR&Vn!>cGB;_ZCtnCb$o`$C-um>VY?>J+002mkN2ztzrQzNq`n}FCE`do{P|4sP);{G z;^HSAN4??9bID_?r1lqHHc=!SY&M)BN8e_Pb9Z=|C-03PMVPWlHfNQ&7Yi2X9a$a= zA1+&kd6;JinT2Z1ZJ(EdANO;^?(dM0nmPc!%wqLm&D8~ci9Y*b$}_vxebS&7 z^wW(^d~%b2g{8db1R1I}Y;uUyh+G-e`gF+lBFgXkZPO9u+OP6tgv8M@;!Exc%Uwo= z&`D~IPhpmCqQBX{yOAA>ulOBk)%RvWI6#jMD*A81!L|L0{=Iz%2gN=s0&(1Xt&y}n zy|?rxb}acckcJ}nYCsQ5i1!95Yx4!_>&80e2P!xg1Zj8#2BZ~0T&hu{!DGvuQwsP@Q zEs8ZKgVr9CgeW>K5!;Bt@t)SV+WncjlZO;<2L4gR1*qmQ!NNS|6(^O)0kWN&c+)|&ns*1Ek`hwg90v2w*trFstg zPAicO{+CS7gX8YC_J{aN0oLb3JynWyfK&h~;}nP4yG}mYRFDSHI}k zIT1EDGFE|$bdb%Boz?Ag;;1VJL&tEr(s(7u*s3#}RFz^{lHvx%p}iyT7|-L`>7EX^ z`@wpRxIel&$R7X~L_a!x`~6()DJj(zHS={vl?(!qUpyc&N2WQ$pAP~%Jw`CJ2p|{aSL{a9;BQ# z$gg!%GdYfDZ9wdT?)NHi(qtmo&7b;D1Oidj9I>3w724f9hE5G?Nof0bdc}EG3a=cZf;G;ZSoN_&SdY*iu0^( zNs|6VWkGKZr7cQW`<3579-+&J?LDw~AV|b`IN2 z?v&z|LQ{ioH9p#Li@l%Lpj&8>9mLXqU@aqL2!Fp8^myC7>{|*O5jS#Ajn$~pT@t({ zSA|Ro*>CSJei$tqx#E|Y!YbxR2mvzV?qEUM32jdIouv{~Z{3r_hOWJ6qU#9L|KQOsNX zX1w!p($Cs=Di4WU=!pHarYj0tz7)roV?hN2yDRfaC;Z3AeFS3~lU`W`gvBcH^)^^OeLPN3_zPKG z{Ob@s`E<$RH(i~N&*E#l1;y9J$_60xHde{or9(QYY|)Lqv?q^mfc&e9*>HOG4K?8( z#P#1C`qh{2;W-)Yk#DwMPEK|Vt>+B-bt?B~o_GqF&FpV}wX_Ab^mza+H1eEkNP6RN z`KO^@n^EG_1v|^k3BMBDHJKhFJn*oAa_vhBKJsmxi%b2}!N^{bIXSbDlM0DrN3M+( z(+Vk|WvS|$i$AQK)-PuDZY^q=h!^fRAN|aG*j!b(;WMZ>D0Z*mL=!sj6DV4Y7ErX! z@pcL7qa|~XH~B+5Z`jTT3qqQ=(}cUGqTJj@)-Lj#jD6$dYD*W|9yk3gCaS`yMuy|8-)m!LdIe!(n8E5Tg) zWBy!-D88(iJ4D%lo1^C?T&d~N2CgA~dP0~tKMPv{gL&G@z3kHq?GyRROPRSDsuNw> z1T!)reY3>P01D9tb_e^zcgulyto8cjc|rRF6ubp8(X7X7{xhP6d%os<-IYC4jVWf) zYHQ`p!C4EYC8zk*`i4ZrqmyxuXMQOY$2Nasg?ozY?p=EWT6>M(E32P=T_qR*3hDo*SgBDZ6X@@iM zy#+@0uaV@7kd&vz3MZ5WEq6py&up&NvPzD;3{K5~3nhq`=%M&7GS%}%Z`jHtqKDN= zE9I4fqKID0T3w~=4Jz?pWn&-0pIv~%y_*HQUV2e3e;o|9AF}nB;-YTem$i;DNfg^Y zm~8J^;xpMiJP!q00gTRFXEHr^Bi22jG?^_o-WgRaE?)LD~1vokCuS`VQBCo8GiWTP6M{*Qp7l9uLZf z;5_vv*;Yv276Ao9FrK%VLi!=`9eW+tPJj0&EnD6nHkYk!TEPxxIla>whj?KK;WCFT zC2PIp2h+8AVt3|t)(7Q?=qIxwEQ*M$+Jl)-gBd0G?o=0!^T_q6i!>>*c8kGU*Oh0J zjNRP$K%vrrLX`l8vH@NUNhiB)%VJ1z@Rmh{u83VX-ays#gvhvqRzcTZF|BzRAIO%@ zZT%9`ZiT9$No{PQ+(T?{KMxBNr8@YqWz=$1*NITWxh?0{&+Va zC9FAYDHR#&&Uv{q_(>~l^g9zj{q*P+@Y6>A0h$znuj(L1y(B$HwIuBYoYYi{U+3s* z)`YWyyPu*Iz%x?Y@)uRyO)V|>Dh^(hz(fD%T+F;obVtc}v8>%QCB>R4o~<{FIbBf_ zS{tcKXc8;83xDR}2sEld*ULGI<*&9tqw?A3o~}+$EUorB#+iSeEx*y;`#z_*L+eX+ zaOJ3dM!Vu5qzu_12oVKEC(j4r0Y9;s2?TDTNF2CJw`uRBs(tgRuli1^49xg&j`5{m zuWd}Y|Lo5AleQz2W%#!%H=eG8Jz#irpv#kKVSUu=x@$hu-rFWJH>r92Eo~;MI?_T$ zM~G6*KZUjJ@)ouD(Jkt>Z6L1PqUNnZ8Vygi^rTnl68WC(D=XKv{B(W09-TE=Vp?9A zz#7XtT9E}1s!U;vtwwf?K$_qD?DNlp=2dgUVhoBof?_PL+E!1wchYt{5N1N5ISBKq zjcKKflP?g=t03@B-`vTdj{^P%+!l zr8%)iKCeUM>KtliTkP^^!%Iw{rjaeoOY>@V;#gGVa?iH6zG%ljadD=(Z$k;Oz2kLq zumk*CFfjt?Il}#QMdk96ZGYrtseeb?(Hnm^xvR^Ww7FydmaD^2>MP-vi^D5X+6L#a z)ia{C%FkDyxwERz+cJt~j*f{IDX10J_@HK>=|-Zqa;sseM37p1Ol^PE2vH3qi~LIq zQQb9`F{kjeGDGf~``s+B9dIIMo1_s2G|nXv4U_KGj0m61+N`@EhR?cz^F+^TeX&5_ zKIy&})#X7o;@0{~gPum}1}q~a98hF+ZXvTxs}&lKO307EJ)&oIz5vj33{V%$N>ukT zs!=f!fQfyEMtu5N_x(H(S*lY21x>Q>FrcNT6t zoPP6yRea3S{x6%NpVfJz!JLOMDLg3d+&Id|kwWXPd9NMrXs~5TBED{BjiI>#d}t%le7-EFsy9WG)zJgf zfYJ801;uf)%aqXktt`SHEU)cw@Md4#tikB{3fw}bL;~~Aos|z&Etm204P?=*Q^8H@ z&+6y_8qx@vak;z*H&tL7Ewp|USQ@QT^;XUOA83^T9N;Hf9Do9d`+`9UnSBZ5p&2Wp ziOp=F84IBqgK{v884;%Az{CI48rN>MVafpL{T2b}MLUx?g9o_Wy^VFK8n_I>R1*LP zYNj!3PdO%sX;`47zIQPVJ5L1kOg|;&;p-~g(P#4o47)BSh!5?54LOj9X2-zoAOuXK z4fA&b%KT6L?@WSzSO^zQf)d>HhzwIvw0IbXD+t{lpd559w}EMN%U;x44x_EY^ac*l z8-VjbZva|;!(7wDXhBOw8+HwIO>Y!v7K~J&DvL)mz%m%;0l5ECNOrWA-}KO4IiRJY zcUc3b@viBiy^6p{O)CnVu|YTHKNV8nkpG3>%Lg$m41O<7F=HKO#EWHNz|xPG5Vw3u zY&9IJGjv<8GJ{B6MX5hAWecg3hS*ssrbKis;FE7B4TDH*_TMA1k`_M`*!C{9*s`8_ zn;q(a2|$5r-p6mMtm{}*EVjH~S=!sny6K@*kYQ)tH_1nqC|FB#JU+6#en+ojgkt#u zvua21yhD+dak!2cD>ZbcPDmKbJ(#*rTmAl?^|wED-+w0}E2YOx1y9M0Ooeer84#EIfU>rWG-^5N%DH0T%2B{I_VK)c2dKL ziNi?k(NwuE_*5H&Oo59P^eDq}-OGCBtIhyNxG*?zdjqc|RaOlZD)MqI>-vp3M zqAtTgD1@7z`CbP8wIFjS0Kg4?38YzA^W{dWKCdQMml=+nQq+?9_F!g%ku)VgQbu*k z2)N5|gFL;Rh~*QY76%Uy4w+Jq!JYXk>&K4HLjwbOFVT5uy`O=+ktvWeZI(0|IGwgE zr8x2x4M3HXUxPL(vvkb{n7eBoK&V@J>)JyARpueOh3wH;aDGfy209Berom}JD1@KC z3#@MMeu zlo)9MCDU34o|~UJ7C@;H<3m>l*lC2Smt~BYGSHHlj+`>U^kD(xy=%Z~0Pyu&@#v{^ z+EQPv>%Fa*_+2rwGZd4gf=Ne~y* z+=KqDv*_%}9jI$f@jc$xDZQPj=-qCG2s`TLCJD6==7d6-+%Jyuk8yqQ3 zaPDu$nKl1IN7Nv@ZKbHrJC%8sO|Hg{mAP4}hB->N-q&}qPAbXA$T`-)tC}`Gb{3wR zeB<0&Kw>bq4Bo8Unc8r)_2Wi$ivHFwt(ubNF9Vk*x7|*e54uGn_rscgPz)}3c+Q4c zzpGgpg4XFelBD9T7FkcRx%d}C71{iCx`u;Sh71$My#yg<$D9fSE2hlBII%qD`IW)@ zC}qJXSD@aOG?|@!BRwHe-kjvaC6*^9%nhih zdD${7k(|D(Hj=TLsHXCR!!8--y3?P>y%twUCI?4rKL$QCw*iGiatC81Cm%1Hqwz1Z zKHbz;_Fgup`;#TA#E54Yz=7-3N8yCB<0lE`#Cg(+gd0OzsjyBFEiH2 z4w&8!c+>K2a^Q_5Te|7NGfhdhXFyOq(hrgDm$ScOXy~OZr{IexXE7Nu1~17ZhmOZD znpATsj)MZC`g{4O8-xE2E?P8-y}$5xqPwEcidKWi4QxByuRN~`HMDGl5~c^@DapM= z%m4{yNY1~*s(wgVtx|KXTxL(H5*689)4T6`TY|>yEWG&4`ZrlPDrO#3w0Vk{NwU2& zJ9zeY0Jz2XJAF8sq$p55mj#2QEJAD20Zy_2r^x=^7XYWf1HegEj`D_xy)y}N)W@|* z1FPO^BglfX^hSR|tp+aYO`{uUYvw-5HaPTk>!IEQt+ms%Xoi$GQL(AY; zXtZcU!x`v-0K1~pa^md=b!JAdSzYf(1W~QAL zi;UarXtq5eKGR{8bz@Jv4IPbwp#u!a{mt992}2x8PR+V}u@^r=n=jtByyT(WF%t@N z2KK=YK;|8aiw+_drd2`BF@vHesjxkzI8>f=q=P($#pSV*sPI1&h>Ef2(smrPHt6hI zHEjn%?{Y`Mck?Tn!MAhqbhwLoC?8=$->#T~JYh~+Gy1TMN$bRwVDlQ9&k!Nw4{c#D ze(lnP@|_1{!ouFL?n!l)Gnk^o-DsdjW>lSF3AbM4PJl@|Hno!a{XkvOUKS3du6t`CaZ~_9C%Yzn=i7V@fg$ zJ$o#vDQ)aZu$HKTjwrP|s($w`c9_b{cArwk9aga$JgyNq8N{TeF2i%Zv#G1ud5XDq zb%Pv<`z!J-EH&AEGuw4@a8HUgzH_)(Lz>6?qYP@)N`~hFbZ*q%#Vxr-J0-8A|7dm+ zB6ju@VQx5c0k^84HeBCA6?lk{EihA4>`2O{p*$K~#pmWqGj?{4)#m2*a++I~lsvrJ z!kP`weVa>cRl4HxZ4^r4(zeH99ma?BEPsh;Rd!`J#%~L-LBxTX9dwzCw51wQzHeJH1qzr`au&L!X0RYtB6Hrc<)>|s+=Q9^LF@~3_Qv4{0R zy4Uth@TLCj_$A&bhku_E6z92Oumev)(#aTOQ@X_u)cpzKlVuURM(bsHXp3i^z=JuX zAHluOZ`dmiD2e*MHhKP#o~II(ek(&xi1KBSw?U*?gpMZ;a072|jYIMQZeF%gBP8b^ zt2T-k@QYIwF}CN76{=g5vO03Z_OS`|HNzd}*~K$1?{}G9N@o)M+q^^wB-!@g9y}vB zvXP{U6_l)6w%m8gPtz{aN_S`#uWQvyWR4~sD4xdN`%*4wQzo60 zn}Y4qKp%V|mtRygBtT0RtFX=bQ56UwlvURwvUi(!L*hr|np z`X5lQx8SXL{rFAY^xgu``)m7>lbrsIxIf44h|n*3STBy>Ymj7mXH&sBBFVOBeekRY zc5F;>HsRDwkH1gt!)h~!Bf4Z>F1 zD{7eTwA$?%IwW$1o{cVDwd|WG8ru8V2@uRIt-YT0qSB)<5F=nESW62a&Iv;g1?T~a z9+uMliF2;rP)oWI%z-P|MX4ZJF?^ty!e?V8TP?%_YC2CBFwL^L%&Qx2$%y zZ_U%E)JPUbu4bW~ujoJIF7)O&i6=oxo>Mh_S+)-FiKH|5Kuw>GRpYQ2HQS8B#=$uA zd4p$nvK)Po+0OEPz0*j6mJ<+ZL-8$Cvk7b8!T}Y7nwG&Kt>c3I+c-Qcv)5xZ=Z;Z5 zM*%%K&+s3JYU6|+)h7yao|sgI-x)I{5A*NkmAoux$vLSH`_YInCMh|KFY9fN4j*?B ze---Y(!-7W1=2yDoK;vGwIFf9;+gQ!E#B<5d2vBp(M)tGJ_`>W5-INrd$H??bWE0) zELD7M{;Cf`M;o8Y|Ad9kN-?;ddsZb+JLTiS)u!<-bL~&H~;&gIJKv54rKN>&1Xgg z5K8nDsgaOE(I&oDv4n|Guztl8l}TplVIG`O$6@bwy@MwV#3IsNJOq&sHsf28Lbaf= z_Ghb9xL*V!!W}?@5+A|pw@}NK9VW@30d?SfoeWAHwQClM0!}sHd}voM;_9W}qO}9% z=5)+PF}?E0krz(76%~`muAOi{iYKRI)zrb-AiHbTG+q8zbTMq)VyQoA%sxaFQE%YH zGFsTiG_B#%;5(O|z!MvBY76t3zL3_=t!OvRuil{D`DNbv*@;u|p;)_{5Wc-SB&Oo#at^^W?a$SCUv%PE~@yT8ISA2BZH6rf{LGcI3k2aE;&6v+g_ zm+mv^t|h&6x{EcaGL(Kf+qiyRcty!nR7s)XeVfagpGa0tnIR>LLnnf&fG-QWsjyMz zs^%JOkV!wb*U3tGvf1F`Cf|hYtldvI7?W5^XHmd*Jb8v_*FVM+8a1xwdTHjS7y%o6 z4zN?(37e+g4-;}RsQ>W^T;wEN?sC<)OX<_8gyOW=O4-{Whe(?85IoXtg1<|OgIVrp zvpT6!em|d}wQ*rRH$7b~@ZDVgWCl`6c{6n0>=vSNvfTE?sD)^h@LmHYJ#_=+m+~2x z{kjDyM%&|vK61~@QH0bhalUyyQK^U7OARFZYmSyWcjmZ%Drzmvk0gwZYZ#swJf<5k zG5sm8N!M@E&rso#z@8MhQSOq!97UX`v-Q@A)YW=)T=Bc>>Aj)~i~NGhl(s%d_U6=l zLEId#m10bPuljVOJUS#aGH7_}u;ekS!Jjdir7)2zX+n*wvQ4J9rDI{C;OTAoDLgS% zNLyGY@anaJh!GOkf7+vOyM35ls@EJBc)azTsww%4LM_jP8+4G;{KyzuI=yF<1FmJiiykxE?E6;Fo<(HK>)AriKYgO8jtf;8t0 z)+^1MB%))tt6-fM%|GV?K2bW)8_wCUI^|FjXM^tAq=iZadjC12E41+@#)h6JH_kcI z12!dtJPJuMq5QFmFF@45=bdKCez*axFH=LXcnhsfaF%*&I@3y+Ci<|U9;scyq-t*n zSKQpP91XBP9jQ@M8etYil~GoB7UbFPj$3Z*6-@ z4#AZk=w*XsM=A)gV|0fy(~j?IH+<+lEP;i-r#RZGc`eCSKejI6{IhYr$E9v6X2>ox zZ6j#jA;h^G5m*E)4rmC~mn1$o_O*}W}-)v8tlv=UP zOl8H{)CrZGO~p)lz&m?I!3(O?U2$8}D^gCz`iBKXy&%t%cck8NY`oCr!(P*1gBHC` z_)4z=$1EKRF25ggyg+r>F7^4R#v5#JSm^Q8c5pYI*dp*xprD$w;U>r9C)^o#B(~|H zU#n=mRtEUx4{6z}DyW)k#Y{*I zRJ|5Gb0J>`8^na^E9n!?ekb{z?(^mat>#c$Y7sr^19 zz6)JO|DTiG^SPbNX$M!1D(`qs&mqN$oL8}il79nzH@!4!o77^NY;*dG`t{~2s4MqG zT8CNg*J;)y!_W-ay=FVAh~JTT^s8P|5QmQa>zP9SL>o=2Zb`P#cNxS|rNcoPs05TD z_0^@A+Rz)s|Nny~2mAl{i>APT`J$<>AGXZz;GSkUJcv~sMo24rBlsp{LI`DT{O8pl zUKDTVac57iZfZ&@6UKBS(p5D?j#ypIc@rZ73BBbRCGviq)y}8YlwO|sqZYl+_)$A@ zu4gqZo@X;P#d13q7v`ID^%m{~TVfZwiK`nPxJK@}s(M#PN2sggqmrxLqdBXy<6nm~ zWHPf$SAqOq>#?jp&Mzm$!M6K5E$1g8UMFXU%@=!vd)sVRNQGmMnrg#Uj-CgX50%5K z&wGDXlb3&J8-GcV;gE8J=IJN%>^ z4D;0Mp@Ifl-&U>uD0x&zYnsNBneL}(mah>WAem|N;POEd z^dklwEN;HRa8H9mYrI#yf3fIR`B?kDtZsdxYZT5BAITCPBa;1p1AP-w^j8njP?u;; zCY($1H*QIMY||@ANX;!KUGH!_ZZd;ccOPOJNb$*s^t22b_qO?PBI zt!Wsq9Aj{~R1)p&(Q>G4EUpKwoF(vQs%)$}3qwPVwyIGyY=)U3J{A}gVmr)aZU$O^ zmaPUcl=IO2K0c+G-YiEW;7Lzs^heeMm~?WM0A>{JCkyl)iw&Dt?kA=wUb)TKXjl?+ z9!C#CA@)`_`cAe>nfA!Xe-=-Vf$TPufkl=_bw`qAz)i}Zv$F_IGbgP60+^^mY%4N> zA~j*=ewoO;)dB99BoYC6cc=fTvK}Wae~k58{9D=TzceA`z+%xU*0EzO3_=NLB8qf! zH%7~&`Xje)=l4fqWx@eqU^%xaS{_L>@b^mKPIskZkY+X3Q}VB|MLMspiw;zYNWPn> zt4A@0IqxNA)KNVm&G=nifbR|M>3=m-t9-duy*%@O00~eAY5-7vB;N}xCD17wJNCpB zs1#5ip`Xlo>^-5pa@g7EI~1|#Ig^`X-@k0qyv&18O5qVL$}0j=`S{hq?8t?C_(9;-6pCKZ{jFMz@T zjRM@^{LduMf>nOoGHfFA3D`paPmtvVYkjZSHC1y!GVGfB3_3*A;cp9{Yg0zOtxVA3 zc}`@*tGOK@k@QVN=-sfHjfGaSPA=o#ACwTVs+_1$i}PxO6SP0zD*dSQpFJd$4LH~1hKT{IM$=_KVmFl z%Ia3~zmJ1)1RA?Ui0RU{Y)F0a8^};vCWA^XRahh4i|lfQ3rcoPdR?@}_7DnP>nDgV7cm zp)JHgqpFdhEo=p-V`Hcjqp8=UQO6_z3uPuH{FhN^3e&J z5@;HXXbUwS02WH1EqsBtP~rjF!oWW`^QH73Js2nRvHY`Yb9mx4H`aJY(UYp&qb?2C9!O2#^l?Efc34t0kN ztULfEmrV7LG-H!Xa@;&P-@7~H2I;s*KCs#=r!;GjI^H)Oqu)Gnvej`+G@$6!(DXFK z^Fw+07=zJ-KU<8KMUtN>cZZN+=vhjS|5%muRY4PGFA`LDFy_+$mP7Zv4v&we@5`#z zuK+Gfyedn$1|?ISinltY4X{|A;Lq7yb~G+_HhLxaD~u)H7)$P8ECFxQq_Y9mn?m4a%OS^Q1sf6Qd2lB2#s$tWz;Uqd$@_2#uM2q4ZIE{97dXZSH&k6+7d% znenqP!Zk%$qWVLy+Wj2<1UAzBv+l=!7t7dfo({f7Vs&pV#p?dj1i*$rs#7X}#sI2` zv$-7kp>%vVM$s0G5?X_}pJ?2PB6kwc$EV}7#F+WL5?Ek~TZxyC4D$xj5dU5A$5`W^ z8OCpDj{_{QGB8CD&$)S=5-OAP=;N??PE5#^~$0)eQ z!zcijQAnJ+RPSt;cunAWWVoqc0u!?{7#W_+3;CZI)9Ttzmt(bsL2UfK479@#H{*>Di&FvxWq`5iR( zsqm7dg|epE_TjmCHkX~9*Q!wuLi;^<@!Lk6JLO0mdG9 zyL2{{^{5_<&3hgxRLM>BkfErm=*;O6UV7!flnhm zHYa*ZS_OK&Z%Th^J`4`Hu;+ch#NPU4P@YqJ^K)*g6}eZA%T3S0RQe@f`X3x5PEpXZeF=#uEiQCirWT-& zVVz695|QesF*Wq`i0rK|CXo@Rnw!CPwj?F>uUrY|=3QgrOTl*@-FU|*A~ZgjLw?~d zIM7e5)zR#hRUjHzXZm?(;Z22yU=3gG?(qv4nPt;Jw-_G&fy#m?_2U7g%7!34aj2ui z0xvbOBDsIsN~O*1UGAK1>MUJxF!FIuT`o8=NEQgw7NV>vt4kvmtgOkrN@71O%A4C| zoekAi&%M;OsA?_a^c$W~A{R)SJ1mrNq);L(6qF~GcGFpdK~ig?Y*euG$`6aXk$ zgUO8os=IU8YxvfVZZPLAyP~r|rHGPlesaFrVJTmvyLx?Jw{J$WUM&>rtJZn^JO{4r z$n-Q6s#i0aq;}+Nik-5ecIevyR?j zyo+as;>9dg{8n+Ht%F1kZ2DFFaLv1d+rWv?41hPfajbs_&URgn_6XyeIhGe%+a3$~ z*&3G*%jceU#9W+%`FoZ83(moUn^@GK`XByL=kB%d;c4sX99JGCn=#H-hAJEU%S+FM zV~7`tizRmazdf|||3+i$-#ySs?T_C??XTEGc_%alS^|bwFwyzD>-9BIx(VU_KII|s z3w|)>+~!}L8tWhI8+)iBLJ{53yJu5!Zx~;=x?m zZq*+d+=cA)ldLbgaSZjh?FWLWMAuix*mJ%nIcr~nnje3Lxih^+th#Yb_1C=nnzVNL z7$!W?K<%o$B%KFKZ(~hm^=Db zQ@9=)#=hK2f}@A0xvOIICv}Ye&T@LMhaUK_)b?Wx>!qiWgGg?C4z!c=z=@2gf6@XT z(aX)6NGUALvpQdfDnU*m<6PDcU6`o4B?MTfdU5t_#DDR|d)YX(QxU!M6zCM*oNsg4 z^Sjz{Oo|=Z+iD#(V#86QZ)SFfz_%7hr%QqcAMDm2Q!uavQLhxdY4}E=asrt*;D-g~ zS{A^M8ZOF~{d$M@mFIfXpfp`FF~bX)fyfl~+Mm6Tp{{j{l5ASZtIMqtZ3k}+{H&hq z80hW&fZP50+60?Q`23SS5HX9E z9t`PL*&T}~p>$Msglq1?wq6qR2@*X;xy{!qTH03y$)f6W!FfT6u=S-8Cs9_|%F>7g zYjJJX)*?;KvfR>evwJ3jh+Gw+k>dNn4a|m(?`ptbyXmqpU%BR+ah6I#5bFL;xc1=7 z-u&!Zhj)_y!S{aC4U@TpCXHb~KaQLJJN_0u@a_JMq19FJTSp|k^ymnz?B*tz*(5?a zG)pBS%o4U*aQ~>^?+kCx56?$p{i04SuiJw_;c*f<_^3Wfk+^na`mv{|{^Hsw4-l4D zg9=WIy_Ojrvxet$Y2YzG)4Z|n=;!9LLYJ7}D1Xpzg~i$*)} z_-N}e@J@__p&CDcmywIZ*Y8A@)LH=u>?WU>>0=KHEF{Uqm7y_Hz-(@2@-s()S!BzSpTpXVIcUuGW6lQ1 zVaeC7q5RNQ-+YQ35ZNl66GkUj>sD!dlx&X6wT~SH-_qy9oY+Azw|_4=WFOM{)lVNA zw{~1g3VJJ2Q*iX_3(WX&C`jdnK`I{zQh7l)x7O@`R9?%~D|B0_gHHmhFy|5BTQO(< z)N64-9JDfMx(JSvXyRzGiHw8608JP1Q7}#gZ#icb=*<8JWf$Lz1+_wscw^e#qFE^` zF1-ChW92`8fQeZ$P9Tz;hLA*CGHggow<5HiDKV62Z5ItPZjic1hNWum?#?iRjQbyT zuWUBBc&shRbZZQf&16JC3%VW?^5jb2E9#Z<Q!ZbvPsH92r78b=~ct6 z4{G6o1kGcSVG@hY#moPzs=o%b(?b;Y=b-w$0%G+%eJ2tA*TPL(BIHi!RHe(HRFRwG z^ZrS8RO&kD0RW%}_)9>ei}CSj5+Uxz(4xtx?Rev&vCU&nH$d#@4Psz*<1|q%A`AZU z3H0TCmNW+FQN%%y61Nkzz2sd3hby z22>+!39ULaB1qCa+pSgC)BqmWVtDYW%~-WTh9>D+Rq1SWE!nN=$-AXEU%$*^3|LIM zHK7K6;ve3Bn`$qpO2Qu;xDkD*p2$Ji*ElS|7E4LL@ogqxfjwQSM_RnxKM z?Ofv?zTflA(J%J%pt~_$_9Ni@wGTaASH|2K6+u$vjqUMYVrRQuO6IE3VA+S8!dN3` zzsj@_jThGYoW5T;t7^ry4(j$GP=BXn$rOP|S{i+%I;#c+#u-LHzk1qM&!%syz@kpt z&Qv9elV23d;o)vyL{^vJor;8oWS}ygor~b0dcioid!{Tq0XdMzF;)l;={T}jkP%fxU$&*9#TDyXmbcS7@?J?S|jx0fWiJAO;`=kQqF z%<6vQ%UB1wZFD!c(^9*~IGH0@3?(cIoj4#|z(IRHSN4j7IQG^i+YJsj=`}m5BHHcd z4i(Cc4Q}x;H+B-4nH@Umb&kgL{3Qf}bW&?p9P)gqHq?Nrit6VpP99E5cKxlrwJ;wP z7HMUTdP|q3OwLVowp9!b9o8kg&yT$l_bT{M9=Q$VZvb1DGtp9E<#dDuGv15sNNIJ+@t(T@}W&$_~4pN5%n$?0f(k z%d5NUom*kXR!uj8~CCIGV*3Z_1G46bfr z4Z|VxC+EE96w1vdEF)^9(m~EX>952fn?Kr-SR7y1N|ECxdF&uk&v|i+>2CYHsa1m> zwP}sMbU_MS?O3*p62E`$$gzeQuCxG|NZk41u7A{o7RcYDSx6tZ)dt{NK(a5W2!1C(xsI@>(@6 z3TOANxHK;cjgwyf09&$T{vm1fblR;Qg^xN_+Lm9kPqf-|EZxp@yn4Sm!@<)SIO84_ z_4;^nfyk^6Xtdjc{&c#4gdKm&rJQshD^Xt0! z$Q<11{@$2fbQfN+P4?YM-H)60?d~FzmO#3EjgCM^c|&3D5~kD)#c%_9Wzm`>YLcbV zgz3}09>ZI$@E1~oa2L0`PObOpBmHJ$zCH$gh0#t^N(}oeP1b#6FV4U*(>Miv-ap@h z3e3A+tAVdc1Z+`>ReW?}ql*MdBroCc= z$}`P2P+fOOHJ`G8lTIGwAK0*G&^gZ5DZIJ2PEQu?uDLgxU1%SdM(J72-ap#7#|##s z*o476M%ir?uI}(-BwX0b9rIwYwoMEc?7C~Tj`yP#-LF5@rT@~<;O#dZ6x)8;rsEhF z3z&6R1d)Q-bPbO2;ec6u+YgtwM=!9z#Of8Rqdl^c>-K_2%Qf^BdM8CFeHi zSy?ecxI)rXA&^;TgHug~$60cvvGb(mr7Si4QbV#M^+Ytk`nJyTYno-#vC4`?D2SC5 zMv9o5efQ-GMRUM{I*|u=DDeT^fX6D0R}i`aA*Wy;Qye!`qDm;o98sJiI`@<0B%kqw zbbfAty*LHxgyPqnAz`s3++4xezr1rC4)JDYrr2{ozc`>$pD-c*tGvXA%PgHs0<8hA zw(x*hjyy)8jz8B3`(Jg6>(7aomPnb%G9%5&l^2il%O+v}tIlHE6N>q{8Mgn>7L^$a zc3n)EaE7q-_~(%S5*So1d4wE!Ng-Rr@{ovZnk!o^|KFSmZmuvhLVS0wkP#=sVoa3I zS8bs#2I8OGG&}L$;wN@dV#IQ>kUpE;e~`J65vK*pOnp)m{hz$hXJw&|(KLO9%_t9g zpTAq)KwszPN4X4Jp-CuDIf8lOy}>?O4N@(B6AfY-$tlvsD6YnNvS4@gou%U)4TE!(j3RHQV&U{s!Bu&hTX*66hAFM5TQAM^J+3zWHQJ73 zzYdmH>XQO#1TtSoCxk%R3grS&39&zoE2jf>)N?KRw1^)=o8YSLI)%hut|r{WpTeanAW>YIit!V~qz0GNlmTW59VfFN0YSgkoHBw#0KosC#<_pMI<4OW*0ZgTfFbpX^`|Z;{=EYd(X*hmJ0S7f z#gBE!2b4neS%+e{$|E(%len7?olToEtHxX#dHVRY8QvV@D~}hm;Ef%MdX?6Xm*w96 z*eH1Mrq5-_(1pS^Pl&L6MAns3SoetV9(wSo--h3^xnoHVC!%{Oy5_PuvB}s|p&t7REJdM_G z=(3Y{YqoPsPQ4Ut49UoFKe&d%ZP^C6A5JGhA37BDSb}8N1<9_YO_x3gLf5<2%ecI- z$z%$;O=#P~zOYHoG;e+fHs^pq)sCbmD~~3Fn4fRABQIX}5XtYvgW4S~Q#KSR2hwdu zJe$SvyQ3wa<%fwpO7Nc@&n#2( zUFP0Kt^*_ohf%^dFo^j|h9cD;hMU%raCZspb~@)W?SFZ3;Vx&uD?1h$8$TTOW`N$F zi4$+j9IbJYnr#Je08QPr1N&?Y?S9wPrn}el=)KppYZXsz44mR-1NopekrPq$8By&cazyaVfP3A0v~vt%si9kVO$ zv}~`#5IELFhg-ilIC9R;fE?v;W=lLOp9T1Yp>w=4A9RHIY__&5z>39&NgcPXbPk)f z%9|xVl@A+2!jL6?8HmpRu!I-%jAC1LOglbIk-3eWUq7fpo&zP5TOByFTNOC7I=0Qa zaq|i6x_7766Kl}Y-l;}X%X2Wb zFK!9CQSe_>=wDP^6PjMc87QUz{EIgaVijX1{EHVa4?>2g7qzwm{w)(_A!PYm1}b4C zWc{zJf3Hg!8kp(|T00Y}|BZZNWaD6=W8ffUV&-6?<75L>ECk{=v;x`_{sor@aVr4r z9Q1**x_18*GJsI<|M^SE_`h2G7cvAp==OhM=l_op@h_Uie-jaewf`SP|i z|4l}w|0pBlzny{bKc*iU|Jv7o6O#Eq3;A!SEBtRE?f)vj1IT(K}{)^$i&SzjG zWMXAv2X)E+=?n)(7G`#ifA`hT+rK$0C_VGNXSR{Dzevr{ShkW%sVBx4*2lA{92Lkm zeijt4^wxvXrEEJ3BZSYEw*f1GKWc_!qJS^l(h*Pu;)*KbK6(fQ*$T+6VOhj&x=CBD ze4cRcy5;#e{CM@8WM=Qka%*&(Y|D6TKU`zs|MWYM(A!B-XEwkwC;n=TFyUAW(Zp{x_$1)rbo{7tGf?myTF~+p( zIe5_%!G0DOoMX_WH*P&tF(c^V|A|ATA;~U^?T*8sBX9D7iftC68Gx4A8R>$aHgm&rG?nzl5omcDlUb9GWN&k6GC4OtU@9!xY^w-A;1{Fg|}!QC2Ua-O&`=)q}SJ39l9N0au5gj0m+0v z#I+YciH08s3_9^dph&R2j`%C-SU^GWE#^d!VK9si#13n)S5QDTv@KF^zYXmUvI^b@ zYGMy|pJ6q+E$0q-^=D7N=Gbj2l_+Og3w{ka_CA^4*v&}ST+QEXX)VINU>Ai@cC2-n zcet9dYyoS?A5d(Bp0Mn{E0Q~rtc2RXH4}Z0cDLZt4XDuWoHTxlO@75^aDzl5BiKAbdynB=Ev$?3M1ryCU4t ztcJO^dJ=lYd%`;tZHL4cY{z&Y+z5JNTSM>?cK`H1{Em+=vi6Pkx@k)wuuX?xhd_sK z$EuHgn~1~viGe4w8Bz_2K)@aR9SRcZv4^NnY&+!X6fvL(Q62NbuL$X}7pZS>E9J@j zirg9f`qL}cwe1s(9`#O{4PqoQzV9JYmbZRz(00HR^%ZG*@Dplp(Kg2pQeRg$^A&ra z@itl?AG15WRzw%9m(LYly+y&+zc+X3f2Y|XU(-H8?yO#6?QmbYykfS)XcKlq z79nm1RUz{6yW{ivGY1oNF9s7Z^MD!edIiXAsrSijzdXr3areeuf$#8biS3B^XyPQL0nS@7Uclmanp65So@H=h`wCY>||}L_mFrFUbztyzVr7z)m@QH z6Tb_S_GN{@U*k>>X8Cv_gYUTL3BLv;Bd!LMd45?53CGmOB>yG+DRP3?xsVXq5p4=) z)=teYzZ%87Z}zC9n_qytd zVFJthy?7Lfj|6pBqXvc69wljD>fM2Ru!pc&b8b(=*X zhs&-$cW0Fk+nRqVn3c$L;Xh7gc)H|(Q^=+M@v{5H^TID~kFrNo-U;0ElUQ$-96reu zeH7Xz#^zwH+8uqv(%=f6<;Os@T??+qj>O$K^>t;REg%l5x)p@+jW*htn||rXJ$%X$ zMk6}e(RQng%@^6Ay5vZDf6eiX|GRVHx5Zph!YCLu0*A6@ROegj2U${=<65Zw0G?Q#&0r+VA3t8}Kkz>Q z?{2S;PnXx5yB|IJcoTW6HV}10ofyi202WHjM+6_D8nqgH<2|^ils=;kzOB@JbdohU zn}TJisO^!eQ(N|l#Vj3R`3S}hlTZ7ha4{h$J-g;~1OPLG`{3=~KzJ-gPT3O7sjk;r z&z#zsPBM?sFjtJPFs}QMt`A2WwDjzQS?9M#hDP~rGxFhrMZHF6~NkL>M2hB-q5$7px7uU9X8ndA#9|!n0rJpWQPllES=t zYoI?wS)57aTwoJlctT>gcW zS5E+j5uT8y8>{TO!G20|(v;o2<9Yq>I!KNI z#&19Al^%7zD!Ocxrfhs4n{B)ge(zR1#)t=b3}r<|f+uf&HPY zth&szBXRhhB_r!c1$<_E#XVe_$L0-WxfkQjA#4m=DT-i~WQmAp&hsyqr|w&>ute0frql#J zoK&Q;CTHG>)_XR8zRZ+WXPwoaUnXLT{KosZMBDzGj3cxX2+1Ksfxx8mR!`zfg<~ zd+(+{FYg{aDK0{ROV)HzFuSwO) z`_UzQd(tY8&YnZRHM~T4H!=1_QXTv3XoYN*?gl4@{LnYaDP z!So~Bvl?kUpDbyOrI1=CE|+qi6$K!Nd=Y@WCGZ)rB!Cm;2azYUS>^e?$MLmYq`prk zFt%h*;D%{2t}W^%oS=tqz>CBJu@gg*{)vZ#H;;1uET6zq<{bO}wv?r1_qOhW=z03F zeNVwBD=$!@S7-ot*D7wfelT6Hl9iy^KrK-1ohT|Xt3I&&rQEVH%~E@nd>J7>T#|D) z`Gn)*EHLZdVo<5QXkPpJW~+>LpU+oJPxn5LFOg58>T--A_I>oM5j-*3lgh}U901^0 zf*>r%NXq?LA6cibIPk@{x|F@wvQ(x6B&jLv9u9g8v1|_V`bs zWsiyb%)t`Tg@i36uR}I6q9Hhz2t~ZDaIc>-PXSTUC)iknx$sy6@E*!cttXwWIQ~y< zPkR+I*}1t+L?tj^f(go*C`GphtRp)3pRL01BJhY6IB1JVZuL&}SCV}?n}B-TMdg!7 zkOC-tZ3Nt3fQzAEC%rX!`rkDky_&efO@$0?DdkygskxW%x!xSE`R?^nO% z(YagUwh>h&JV`%D7Y7;}Ki1x5Rvs~IGODGZyukvgc{gZ&7**tw`~ITlL2EP#30d?_ zfzm4|UW_G&0Oq{J*n_!22Sh>!zkX}>Q1#bbcVYpqObBm2^lb)1H3`1R$e(_g$^7!mr19=n)+oL93QwYDS zwp(OUb?5pitR?%$iLVbo;kh&6ryCj5LCQggmiA5QkxTX3Jp-D6^uqRa(FyyPW~C*e zR$_YtjVC*xm{$1}vY@X+Hy6|b1VsxOua)+TmvjbE=qT$ z8%r3%4K4wT?&4y0=rymhqn3D9XTL}@@2tB9gN=?rb?uC*sw{(vtbhO+FS#d^)vjBP zmdg*NpQBJ<%b)U&y!U+tilOu-eYkvgV=m`;jkueX*RCh`Q`4qGtHQk|nnGUqzp~%S z;#N%Tj>R3!o|earGJEvmdI-Z?}foJ{f-#Wyhsq1;p%!a4;gGVsk`%bJLgD^VY)f< z{$*r~6uyYPPe>;aw%=ez5GAJG1YT`N6s8wGO#EtWdS@e`*_-KhB^J!xl<{#u`sFHQ z7W$=9g^AhP`>L_SNmpiv4x_0wC;nE1%lhU zYljKClmEDy!31>wg$_hr8EI7G6qd&KI9`Z!zSS{P$>_n{pA#{@rp%XDobF^EUu_D& zowbe_>pRPBW|Z{bl-z#JZ!lKnuI`TxN~^O|g}LQeDm;ph+l#VO;S;sljqqEm`OMIA znV4M1ubxzzsI_EoRv&NMbgc$Sue2MD?;Fk*u@kSC$>Ko9T;NyBVjZm|8%=H(oLPKh zqe7SDmyjY!WYOE2oms=Wz}v=6Ten(8YFxd?TIoav6HjbMuG%FNr$15ZU|g`7nv>&X zg69r{vI|*Li&0k&rkoF7bELLK$AH_CFv#l4Uot=69TmO6x~)AL+WX_z%Z{sN=UDIwpHIyS*q)@!g4K&28L!dOEV@09vuqK39uI9papbH>D3je$S-BnoiP<--OptOA@8;Y!`qls zh&%Xj{SMQB7j4Q*jwx4y3?w0o%hR~7=^k}r9b2xm_}zSdX|cqn`(FBtcT0y-n3?W= zc;?x58#qyZ#%FJkeK;g)`1?+GeOMbO&pb%l`6%dICVJzN?%Y=SXk^m^IO}OR3_elMhnpuG{(XbY_=g+y*6W05ua`LkGK8nzQQBHoIqVL%(OcCoLo5 z&He2v`0CNi(zk2+6zw}9O2uAbGL=TTQj6LU_%wflAFC>|3{=K>pW$noy*_sEw%^~} zl51Tp5R3~_123S`>cy|W)A&lM;=RveL(4GT;wwtY{-?*Sa{=|xBcAn3~nkDFLG}DecSq7&zih#buuGAzeteMm{kEKs_&kybI-hlTO$!{#Z(s0;%I&N$1%<%(rMcGzz zQ6e1iE}dCMk?U;|7oJS9S*EDruxYv#(OES7e8S8iix23SIXiB2UjAp4kj_xaS((QxX$E|7d!^k(d(1LlLf;BUq4UknqCUR$}xuL zdfp6>h>eLw4zUvli$&^a$RFO`@@~k4p1W(YcZrUMH41fEc3m!zH@9~@dg?Jixy}w6 zn1^#+skfA*B)q+ycjaa3+P%W)GOuKSH&Q230kGf=zR1wf>Fqhhi;#?XJL%)*vwQoY z!}iqo>K>k{_oU;&E=}k#h!@i8ihmL!Li+(BF|!rRN`av2N0p!)##AVR?HA@aSMkHI zn_l%v6NBPn=Y3u=dr569xght!_1v zsU&6+fnwO_0>$WPdEJH*)_3p*$@a{3YrVy+xmU{m2&tdn%OYz_^U6yZ+- zSw(T`sNtfO1}G1~SeJFz;?;|L5!WPD_bdA2jW=~`Q(h?97#%NhtYcVas@}Yt^E#*= zR4gio-s~VL3E?Z#N<~fp^14;I;D8#aA&l8g!{WaI221COge9zDM5Ditx$h6`I1T*3 zzDU&Y4;N_s##Ve>jCXZas;wE?zS1aGDwQo%`y*KKor7A>ey9J0wo%S_qYW_`@6gP> zD_5-*6G^%Yg&Tt#f7M%ZQ=sT#{pG%?Ch5mPRGV^tWUNe4IceH3qQ>btmWv6##IDAf z#n}>J?R^sk%v>%1As2YLY@j#?6j=m?^m=V4 z&Y6{o;(-~JiJB_9><%@oBT?-Gq3*JtzkbN&JME#neXO0VesUJ+b_F z>ZZBY^$(OHMZxCmz({FGB{(M-C z+hd~}&!1&Qvt`ApzrV6`$5@%s8&g$Y+EML_ycbsVpRNwVMbu>SdIr^$ltc_lRssjv z`-jP(;{lr7aTO?V%D6}rsPn{3#lLH7%q4wISciTRPELS*VjQq@7|5fpt+Gm=Kdn6P zNBIR;@h;_YoRIdo*fDLht_@mDSZ*3j7MagWHF_X$vOm2$8yai%6k~?*XcV!3*5m~0 zNN*esj}H7?W@^cP8dg)zN*$FLRD>TI$KiWhVXU*tn(N&f5*%KdqTX34JdJ=ifVuwxltB7&65~wcmcHP$`%UK16BR$g z^(3?y1$yU(1@#EC!6)D1I8oBB89J*Wi*w2I;+tKaCjF0+e5|)QpzR=iQzATd74m^X z6huwapVTK44KEdV#EW$XFl1Ok*>#Q`^Oc7i)|#fIZKzD|hh}LTA!vi{UWo|N@7W}^ z*}6n2-L_&&{KgP|MC+os?yboixab+^!2Dhs^tkak+{R%oNut2Y!1A!7_0P4KDhP|M`Fs znIqJ4y4oBiY~%L=}l3$QZS#T8(6$6Y>9msOcT#yV`JSjB?pSH&KJ%1 zaB-`z=H#E1Hr1C2Bxgtq0sU639`li@7z7~_lbIK=N-)D&(AkWpuBUVdhY7oa1A~(; z46FHLCc1gam8$cEu-U|#c&MOhKXl#biCTA~)U95R*6sW3&XyqCh9gEX_;kpvFFEUN z22pp+J_7GT8$7h^MgeVB*_~Biq_%kBg77Pj35O`WJ%RfNzvYK1QhyE{R1fRzPT3eGzyK-TW3x?CrA?`AZ$mqj%%pw6}xy&)}Fme`i-+7i= zNQ?}9JRy|WnRY?DIT&wb{TA~VI~UYvA4AJ^U0!$7LTRP9{oqvjx*5O`fJQ zj+UjC-{x|xSyluct> z2{jgmUFs?7`%H%tuqcrGF6+%mZA*!IO0-|?aGG|A7mVl61eO6gb*ouQ6}0`8a63$7 zh9mFNCkYq`yOc!&ybh)1DCjAUHoFMDe*a;6zQopO*QSY^))j6`uejUhKrJ8Ub;upo znKmKSGK0T8o}i@A#tNCyAl^H@6RtuoRQ@)5IxCOjKQA{D`Td8aG@RJHhtFF4-bE7{ zzz`#JI2a=fS041KUEc&F)w)mP68pw^#}M$H<~X~nObOFgD_R+QmE z(gm+?A*k(A~Z>;2X~^R^q_p-tR>ps zcrtA_0+!F~)s#%Pc*KiRT4YJ%n1NI<1rQs|?e-*(x*MtZWHyA{Y#?uP*cSl-mOXa+ z^RO^Ffmn!&yT8n!s=#njT7a}3s&LRON%9&dszI}1n~03XN9D_`1lRdydDAhqLV0~o z0$cq|;KDtdMh9hwslP!XQtaW*{)THuW~f4yPI4pw%cV-DtCN8->g#7F_0Mfem7!DK z6Q!`Ne*OJuXm^%TWyt{;&SIK`HMkQBgW7`}2G(ThbX=IMyH0FctIjVI^oJIr%aa}r z*B9XdS+Uqwd%ax%J0Uk}Wf|!A`hpH6Z%BCBmS;HJZ1Gb)W`=vt1?jB3Cen_Hs&7X{ zUfcHa?~A{RcH{l6nVgJ9s(vD3v^Z~E2L4rO~G#Pk@p$F?beHpwr+0kZI7r2*2l=z#Tu!X}l4 zYoO8gvG76PH#LNH zkIPJeRjfmx4Jr4-@*e{I%bT_h&zs-o^n4C`XxQh2u26Fe^uZ5+VH53YlR4)PzOV;N zT+4~+15oxBH2n@yXUw#6IknUk@(|?-d|&UPI|_bSP3vzWcu6qH8bAZ6XbAkV`Xzy! z!+3pVW)n`$&C0>1I-c|u?n4`LuyRMz8D92dk&Fq1nDD!Q{c+M=d@Ez|`)u}Bt7X)g zn$v2g3c|&}AtOg*+y*8*uC-R(<6BoF?u(?Jfn5*;jvX#9lqhg{onlli$N+50-YA93 zq57CL9vp6;Ux;(6e6auHF*Jql{rIz%WD|S?o?2bt5!Jo3oLn_A){X-#E&2u?e7I#h zmeOt)ZMnfMio4ij{RipqQ~O`ZB~8g8*R2o@vY#HQOICP#xW82~BWFaae0me*FC<+l zc9%4f&>qn1xJdj-RFFi?O4h)cPOLAqL)Pv??oQvyxi)%i=P;>ZT*Lq90CZBcomrBi zLFJUA`BDn^L%C_I!Xhr{;tFVSi7RDsspETbS{)og*Kk@{h{H|s_+hJVd;s^@6{l`1 za@oyX`kHWSRk<l`(JCgS7+mf*NW6eOX`!s)J$ngp}3JR6?WoS=z~w_ z)ts8T0W7PE&hJkfT+K@B`9~QQ8nk(~xSEbp^eglwzBsUkrlYGHo~SbnJp+pv-YS+B zHnY?{7bhU$V+Naly!b3_C#rKkS1_~ay^HLPe=0+rg+~8DOJY#Mzci5QbCK~)D^$b$ z6*8dJQBPPUR8smObS!(H6VjS-N)1~Ui(k6^E6X=ghrDo0@0((*MxkG*I-46%H61m3 znfi+njPLzRV=wxPyrTZF71>TZguZNu+3T9KUUQ!!McME8GV^}aL249v%MGs6L*r|2nFJo7+@rl)iGRpoM7F^yq+x~jmhU8&m1_3cN3vo(3s=lXr zU#e2&jHnRwnTm=yc~d~xyd`;4#aeKRMvh#nGw0xfUSn|8S z6F3UXt-^a*Szw)Xd>2*DeY*`9KN&OG;7;flr+S!(fKiEX)yBo1AsQr+5Q*U}QV#kP ze(PdwOq=8J7|Z+~42a?rKhb;1W4>bNSZD|F_I4=Ok+Ch zOV~~2lR`$!3R6poI!s}5L&EJg?xK+f_WFeGRVz0o#+{bV%uM9m8I=#F#z|z8eyo78 z)<8hoK#-aX!(78$fzXKS3Rf*xGxzl^i)hg@Rj|u_(6JYzp6~a3BVa;95x#(Mc#xV5(dZFCJM%H@c=Qx zs!6E+N2CI-M^;eh4*30O79aCw_r{L^LHg0juVFUV1%b6^08?MySp9~F<6pkg$`cMc zh%vbi97q{-bco!ZkL&qQ8{8L|YB=1a(ZsPIzMeJjhOt7=(B=Wg`j=h5DDGSV57)0{ zoM!e3KkCLzGAsACk=Nl^b(=JSB;$^&&D9wgth+tAi^W17%Pi6`UJ_#GF`fCCc_e6D zxwqdF(Jfz=XM9tk3f@d&UInj|#l`FlYx3!5Hj3&=tNd?>R4ORxq&vAEZj^e{(k@9v zoZ{-9G4!naw9pr9?r(!H@3jQC%}F0s_Xr;Z&-{{(s|Z!E!!hqx`l?i>{0;?nAhf1{ zxMmUP^~m1BU3#=9sy>0}r9n{G6~*&VYP2%i4(8X0!UEJM!g71p+oFDR%aGWSePI*{ zVK323w`B1s(gR&9L4|n;LbV7H@DalNMLJ9pczo6ZFH_UJ^>|{&1W)dGr@cea^&tcU zvM*B-*y7Yb%m`vhLsEqauvdcm^DobYP<=X=7JuLctpwrQKz{}p3Yh0YaD_z5-%k-j zsAZ6zkP1vee-|&B4Z)V1k&hJ4=V#Mi3i5f#IpyGb&YonS;w5FKo&tL@q*hWlBj6H< z{=(dWfR-N@&f#lH@at~D`S_?eINpT&Mb?e|6BieQJmYjRU^*zL4k1Hp=2W;N^Feod z*XLwfQk06ty?Nd2>`W-H8&e3m{y9?UycbFoH6|of+}JSV1f^@Dz)aG53X5TmDcD4^ z77Rf#T0o3*CYuz`kr~ZIjKD|^nM`Of=fUbLO00Du&Cz-NZ~vdku^va!k{#38tk@!Q z^ef@cCJt(OnRdc2Hr)Ht)N^J61HZTwKuo$|VF{iUF?L%98L8fHSplWAdCNL%;zrDx zCmdtWKU_5mRpFKxdDnIu9UpxBygTcdRJh<1gs4?w+_??A`#c#$*~kZ~OKIyY;>Tkx zK>Ms91ieJV3;|=t0?gx~XhM?l4bx+ePx8^oervYH9!uSczOzMscF_Y|BHt~BdBcYa zhRvo`4J$waED06A$~KCV#^K|Z^M%A+nQ=1`D;NDxC>hnbK`lb&X9uU#Q8mIYlq5_n zo#=jEwyIITcT6K4mTw)mYQh=&n$~Yvvo$p3M}OPHtFaU`ykHA_@R@X)=H2_L7Oj^T z5M2*to{PYJ+M9XQOV>tQ8}JZwYVQ>J9oL{FR)nbth+B%fz9ooMAkV1kQU5#SXLGEF z%I}MMu5-~Mp`yCucKD)M(^x6^v_Z4Fogv>$mrQBNL9pnYx)THC!5q6I9~M-(uZBcq zQRw-(M>8Pa?ie!}hg5mceyvc}uC$a?!_wGA{7<*gb^lnJqnKsWVU5YhEwV$Yw_xlrTd(_V zaI`S4Qm*5AFq@HOTrKY@J@b8>pWiEPHc zb!&S^Mv#t}?AEos>S1!cWaOZssLmzvsIkx;)F@9=k&7q=LJHjLCQdmNlb`A_iY( zmIMcVa+ILeX65WvfEs;bn4sB)&ZuZy`-d)@5l-PALK>T)?)%Vyh~#@*4O4Lq zR8mir?CckV&aBjf=qh(ssgHt@;=%wrOJN&(zGBmK$N#6ew~mUsUH|=+R63-)dzhrV z8wpWbx*H?~>7fOrLj)w0E&*wV?nXMLrCanjJbUkFKhNIZb$;iZf6lNL>oeEfab0&@ z%v!wO_gz1-&HLuD$dcX0l9^4s>qv9kIHk_7C(4jwXJ1`C(;eRzls&>*a=I+zk9~-C z&_pBLgR*7xw#Qi|)fpER<65vqF$!Wnay#s6Iu_WZc8!B?mtz&V9eQeT_3BLj>2*j0N z(sGAkkjHFkiZG?Vk88Mo!A~X@D46=5)f}ri)YhbElnykhG2dqUyiMRD#M{dW)t_{& z)-KA1Or*3rY#P=fZG9m7^ilH%9Id~*Q_082TTDK9oe3N-Ah^jUJDXRNHdiIby{M|*-^yc3 z+qQU3L`T{uapdC3My}8J8IG6CjhFD|us@J%dDAp#PBMQCcCCLf?^9{aNp}-n4AgfJ zMsb5wzMBR}3Xypn!Dt7lD0tm|v4~Eji8O7)DC6@4OKJiY*QIEQN7Bk&cP*N^+vkd- zm-FAHfSSh=SXAOBW~<4VXeLs1!kARGg<|$wyj5CKDsKxbNhhA>X{FXM;WTr~_SG5b zw7_@j2yGN?D--ecT2t?)=c(m9*gwSNyUAe>L&f9lU`GCJ$s2|+QBy1 zal*@6(cZXIneJFzW{@V3pc-csrc?@ z{C^u)E^qjVilzU`KKY2Lc4FRUCd5y%>cFzuC-dGsmp$zfD=~VXj6c{!g*C)HNSzga zg00*|(M?_l`kB4@1!n~OGi;S2tdQKl?MO825a^e+>Jr8a@(WxAg^fT&`i}&|M!+H+ zM~Y#O`}N4h`kf)2syG?sj^;1AE8ea8gzb?V^*G~pPT>>+sXLv?r6FA_S{)QP4*_J` z7>$g6AEDbSAL7&n40oKGA}V(E3azGUi_2TWc0LXed!WRG572+}%TGrK?oTpsox4n@xpXiFftH0~Px|$vH)#ZOxW9F_ zR-grpE*0SaJSif-!n6DJlMIm4w55dG97l0Ox;oLZw9hcD4gpK|>jR!|e=e+V(-N*o zE?NF!7?-j~oKhPx;1P*l?z!4PLv>(HdJJNt{Vmi*G1nMcXw!8SyhtIm$ctF?rpC5DAT%dk#keI){X_ExMd3Pv{;-YXc(EoV#=-U^{ z_lj5->sh2}HXkSqa={>ck6Wdh1mf%|$A~U6AD8^;Kw|vT6M0e`4 zy_9HvwBX4{bWZn}aJ(XqqHbKmQ~eN(FO<^;0>(wtPu1g5OUsQ1l(I36n%=|@jB55P zJQ~dB<>(?9ljvUoi+n!zBUFE=d}FF;oC;^d2k(Sm+FTqTmjsICvSBnCGYT~sOV@b) zwr5c(sld-;cM@p{=-$i=CTce09j=W0|POL{L3E`HTjx=D|^)$_p^ynd)$ zx172D#oRKjS(nr+{jA&3HxII}D$2^CJ)M9qs(!YSli4e%G#r$9>xvd-58FM%$dluu z@Ro$4tVVmMpK;?J4p$wK{Ol14;Pu1F>`L0iYV=Z8(oq72UcSXi)X%VV?FA&NDuB>M znOmEzyf06hPBKsSU7frw8wwW;f3SZ^((m`!Om}Ue?(caL$EKkV(~c{Tp3}&mTIbB` z3f$FM(kJG@(nj9Vz&c$=;HpfnoVTD_iqf5{`eI&5)8-swMDiW8%;!D+ z1!_Dra4z9Zo~Rw9V~jDqo`(r%YAm->Bd+x%!BZ%Td?kL+QxQ|IaqF3^uMSM+^z$xe zb$?f>jMmhw(%G{ZTqGY}Dx= zFg`5a`>OXgFboI~J3;|BxyVGuzOJr5KF0)QZ30T4Y%P*4y61Vf3CsPX!EM9e2a|>4+M=N^ve^o;owLF}i?JXSc#}Ih0(SCj}&erFH z!T>NJKQi!pr8bx!0uThl_yr8_MW{%{D+CBa4$=L`NDZp~KXlw+AcP(a;sfvt@(Bpg zLy*HI_{Vtt#U>yK;DZW4p~z+g>A}cv%n;<5@_}LWU_qqd^h8#nFAhJ6?py0hC_WlRT4+QW7AxHtNfFM6W z5Gn`;8zQ@-*M}h|4+MoGACap7fb9B@cEBJ2P!PfoGoXjU%pO)l5${l> zB4snM+2IXw61+&3m0aMd4zGlDB0fc)ZbQti4oQDiJewe-)jj+1M8HAq=jzuT6i+L= z028$JhL;{gBsd=??3|Ww2)#n8R6k1@-C7VsFC)vlP}JX=xiYb#S0Av{FJ-=Y#+aG@ zJlxq9L+);1D_;ojq|oz4R+udVkw*DDa|PY#cVbMN=$gp{G;gs2wSTzue>Ce<#-el7jyu+x(eB?vn>H&HWE) zM%K~H$^1Sm{FzL66#kW0%}tQ$4Vik7dh|c?nI>}LwVj?h+90b~(1ZRs!{9F|?=OcG z{3$O7B6k8Z>)F`5Te#BSr*d}-8H@YsSiJuzLobLt3hJ+m;=u2*pZ3)G?}o$XZ4 ztbOL*i3}!bA&RWRIOe_GM9Bm}Ll5F{%&R2$s@~K~D(8~7H_Eyc&yUQ_-As^K6YewC z)z|&pEvu<}TFZz2}gnHF6n6a8944-mLH^IwZ zfBBVE!qwd&s+?x>DaOm=mjQEgF@`^%K{9qL7jbejjh2^2!L27l7!T$bXN{&-_Utja zW*<@;=QOo^@OFFMx%b-YSVv?^m~u}K@YM`>^GKro)dL%|yyfSY95y>0zk=D{D^swR zGScQ8J*i`Z)wvti%>nMn+sryZ?yxVV=c;k9b~KD)?^+#?4X20wORl-vs3$S~qFM*O z6m1;$RNiUXJ2{-!UU?oysA00F2mNZEUz+|#Vt|$LfV8;S&o>-->1oe#k(~x52KVB@ zp6!Ll3Dq4B^ zPD@Miw>YfVLqmPlNg}k4TM9@-E}SlLQEl<@xoH5lPu9{@a!cLV%-HDI^x|+7e|~tb zMr$K&&(zo?{Y7CEN_9etJM2a{;aXw3qfYf#tSKGd==IREa>pA_PN!d&9KZQ*?nZnL zUdb`fEN(m5TJIzOudKGaDp%CUUpsxxn~{{6l$bQge>;7ZQF)Gb62;;M00T2_+iw0pf