22.08

Reference Architecture 22.08

New Components/Features: 
- Inclusion of the Cloud RA in the distribution 
- Inclusion of OpenTelemetry 
- Inclusion of Jaeger 
- Inclusion of Linkerd Service Mesh (version 2.12.0) 
- Inclusion of standalone cAdvisor 
- Inclusion of Scalable I/O Virtualization (SIOV) for 4th Gen Intel® Xeon® Scalable processor 
- Inclusion of Intel® Data Streaming Accelerator (Intel® DSA) for 4th Gen Intel® Xeon® Scalable processor 
- Inclusion of Intel® Dynamic Load Balancer (Intel® DLB) for 4th Gen Intel® Xeon® Scalable processor 
- Inclusion of 5G Core support in the Regional Data Center Configuration Profile 
- Inclusion of post-deployment hook for additional automation 
- Scale up/down cluster nodes after initial deployment 
- Support for Load Balancing on additional interfaces when using Multus CNI 
- Support for upgrade/downgrade of network adaptor drivers post deployment 
- Support binding of QAT to new Virtual Function (VF) 
- Support for 3rd Gen Intel® Xeon® Scalable processor platforms for FlexRAN 
- Support for xRAN Test Mode for FlexRAN 
- Support for RHEL 8.6 Realtime as base operating system for FlexRAN 
- Support for Rocky Linux 9.0 
- Support for RHEL 9.0 
- Tech Preview: Support for Application Device Queues (ADQ) 

New Platforms/CPUs: 
- Intel Coyote Pass with 8360Y 3rd Gen Intel® Xeon® Scalable processor CPUs 
- Intel Fox Creek Pass with XCC E3-QS 4th Gen Intel® Xeon® Scalable processor CPUs 
- Intel Ruby Pass platform 
- 4th Gen Intel® Xeon® Scalable processor CPU SKUs: 8470N, 8471N, 8490H 

Updates/Changes: 
- Updated Ansible to 5.7.1 and ansible-core to 2.12.5 
- Updated Kubernetes to 1.24.3 (min supported 1.22) 
- Updated Key Management Reference Application (KMRA) support to 2.2.1 
- Updated FlexRAN support to 22.07 
- Updated TADK to 22.3 Docker image 
- Updated Intel device plugins (DPs) to release-0.24 
- Updated NGINX image to 1.23.1 
- Updated Vector Packet Processing (VPP) to version 22.10 
- Updated Trusted Attestation Controller (TAC) to version 0.2.0 
- Updated Trusted Certificate Issuer (TCS) to version 0.2.0 
- Updated Data Plane Development Kit (DPDK) version to 22.07 
- Updated Platform Aware Scheduling (PAS) version to 0.8 
- Updated Grafana to 8.5.11 
- Updated Prometheus to 2.37.1 
- Updated Intel® Ethernet firmware and drivers 
- Updated Intel® QuickAssist Technology (Intel® QAT) drivers 
- Replaced Barometer Collectd with Containerized Collectd 
- Enhanced automatic CPU pinning and isolation for Virtual Machine Reference System Architecture (VMRA) 
- VM Cluster expansion with new nodes and/or hosts in VMRA 

Removed Support: 
- RHEL and Rocky Linux 8 series as base operating systems 

Known Limitations/Restrictions: 
- Intel® Software Guard Extensions (Intel® SGX) and KMRA are NOT supported on 4th Gen Intel Xeon Scalable processors. These features are automatically disabled on all operating systems 
- 4th Gen Intel Xeon Scalable processor Intel DSA and Intel DLB features are NOT supported on Ubuntu 20.04 
- Enabling support of Intel QAT on 4th Gen Intel Xeon Scalable processor requires an NDA (see Access to NDA Software Components) 
- CRI-O runtime is not supported on RHEL and Rocky Linux 9.0 

22.06

Reference Architecture 22.06

New Components:
- FlexRAN software (v.22.03) Timer Mode on bare-metal with Ubuntu 22.04 real-time kernel
- Istio mTLS key protection using SGX

New Platform:
- QCT (Quanta Computer) Servers with 4th Gen Intel Xeon Scalable processor MCC (40 cores) CPU (dual and single)

Updates/Changes:
- Intel Ethernet Operator enabled by default in all supported profiles

Removed Support:
- Ubuntu 21.10 as base operating system

22.05

Reference Architecture 22.05

New Components:
- Support for CPUSets, C-state, and uncore frequency scaling configuration
- Support for Trusted Certificate Attestation Controller and Trusted Certificate Issuer
- Support for Power Management Unit (PMU) telemetry for power C-state for 4th Generation Intel® Xeon® Scalable processor
- Support for Cloud Native Data Plane (CNDP) for enabling a cloud-based framework for networking and accelerated packet processing, provisioning, orchestrating, and managing microservices to scale efficiently with Kubernetes deployment.
- Added the Access Edge Configuration Profile to support vRAN software applications
- Added the Build-Your-Own Configuration Profile
- Support for Ethernet operator for firmware, driver, and flow control management
- Support for FEC Operator
- Support for Rocky Linux
- Support for Ubuntu 22.04 LTS

New Platforms:
- QCT’s beta (2S-SPR XCC)
- DSG’s Fox Creek Pass (2S-SPR XCC, MCC)

Updates/Changes:
- Support for Ansible 4.10
- Support for Kubernetes 1.23
- MinIO tenant load balancer option implementation
- Updated Key Management Reference Application (KMRA) version to 2.1
- Updated Data Plane Development Kit (DPDK) version to 22.03
- Updated Vector Packet Processing version to 21.10
- Updated Platform Aware Scheduling version to 0.8
- Updated Telegraf version to 1.1
- Updated Grafana and Prometheus versions
- Updated Intel Ethernet firmware and drivers
- Updated QAT drivers
- Update Power Manager version to 1.0.2

Removed Support:
- Ubuntu 21.04 as base operating system

Full Changelog: v22.01...v22.05

22.01

Bare Metal Reference Architecture 22.01

New Components:
- Intel Service Mesh
- MinIO Object Storage
- Intel Power Manager (Power Operator)
- Platform Aware Scheduling (Telemetry Aware Scheduling + GPU Aware Scheduling)
- Intel DLB (Dynamic Load Balancer)

New Platforms:
- Taylors Falls Reference Design (Intel Xeon-D)

Updates/Changes:
- Playbooks and profile config files are generated automatically
- Profiles list expanded with 'Storage'
- RHEL bumped to version 8.5 as base operating system
- Version upgrades for key components:
	DPDK = 21.11
	Kubernetes = 1.22
	Kubespray = 2.17
	KMRA = 1.4

Removed support:
- Intel CPU Manager for Kubernetes (CMK)
- CentOS (all distro versions) as base operating system

21.09

Bare Metal Reference Architecture 21.09

The following features/fixes have been updated or added in this release:

* Support for Istio service mesh operator, Envoy, and control plane
* Support for Telegraf telemetry collection
* Support for additional container runtime: CRI-O
* Updated default network plugin: Calico
* Support Intel® device plugins operator (Intel® Data Streaming Accelerator (Intel® DSA)) for 4th Generation Intel® Xeon® Scalable processor
* Support Intel® Speed Select Technology - Performance Profile (Intel® SST-PP)
* Support for rendering profile config files from template
* Updated Intel® Ethernet 700 and 800 Network Adapter firmware
* Updated Intel® Software Guard Extensions (Intel® SGX) Software Development Kit (SDK)
* Updated Data Plane Development Kit (DPDK) and Open vSwitch (OVS) DPDK for use of AVX-512 instruction sets
* Updated Prometheus, Grafana, and Node Exporter telemetry packages
* Updated Node Feature Discovery (NFD)
* Updated Multus container network interface (CNI)
* Updated OpenSSL toolkit
* Updated Intel® QuickAssist Technology Engine for OpenSSL (Intel® QAT Engine for OpenSSL)
* Updated Intel® Multi-Buffer Crypto for IPSec (intel-ipsec-mb)
* Fixed inventory groups for inclusive terminology
* Fixed kubelet –cpu-cfs-quota to eliminate performance throttling
* Fixed QAT driver VF binding issue on RHEL 8.4
* Fixed inadvertent Intel SST-CP frequency throttling with proportional settings

v21.08

Bare Metal Reference Archetecture 21.08

The following features have been updated or added in this release:

* Updated Intel® Ethernet 700 and 800 Network Adapter drivers
* Updated Intel® Ethernet 800 Dynamic Device Personalization (DDP) profiles
* Updated Intel device plugins (Intel QAT, Intel® Software Guard Extensions (Intel® SGX), Intel® Server GPU)
* Support additional operating system versions: RHEL 8.4 and Ubuntu 21.04
* Support additional container runtime: containerd
* Support Kubernetes version 1.21
* Updated Kubernetes features: Node Feature Discovery (NFD), Telemetry Aware Scheduling (TAS), and SR-IOV Device Plugin
* Support Kubernetes Operators for: Intel® device plugin operator (Intel SGX, Intel Server GPU) and SR-IOV network
* Support Intel® QuickAssist Technology Engine for OpenSSL (Intel® QAT Engine for OpenSSL)
* Support containerized Intel® SGX Key Management Service (KMS) including integration of Key Management Reference Application (KMRA) version 1.2.1
* Updated Collectd, Prometheus, and Grafana components
* Support for DPDK 21.05 and OvS 2.15
* Support Ansible Cluster Removal Playbook for cluster teardown and redeployment

v21.03

    Bare Metal Reference Architecture 21.03

    Two new configuration profiles are defined with this release and will continue to evolve in future releases to meet workload needs.
    * Regional Datacenter Configuration Profile - Central-office deployments currently tailored exclusively for Media Visual Processing workloads such as CDN Transcoding
    * Access Configuration Profile - Far Edge deployments supporting access point or gateway type workloads

    In addition, BMRA Release v21.03 has been updated to include support for the following technologies:
    * Support for 3rd Generation Intel® Xeon® Scalable processor
    * Updated Intel® Ethernet 800 Series Network Adapters with Dynamic Device Personalization (DDP) profiles
    * Support for Intel® Server GPU
    * Support for Intel® Quick Assist Technology (Intel® QAT) drivers and services
    * Added security with Intel® SGX Key Management
    * Updated Power Management including support for Intel® Speed Select Technology – Core Power (Intel® SST-CP) and Intel® Speed Select Technology – Base Frequency (Intel® SST-BF)
    * Updated Kubernetes version to 1.19.x
    * Added Kubernetes Device Plugin for Intel® Software Guard Extensions (Intel® SGX) 
    * Added Collectd plugins (unixsock, network) 
    * Added Grafana dashboards (cpu, disk, intel, ipmi, netlink, ovs, power, numa, hugepages, ethstats) 
    * Support for multiple Data Plane Development Kit (DPDK) version options with custom patch support 
    * Support for Multiple SR-IOV driver assignments per physical function (PF)

v2.1

Bare Metal Reference Architecture 2.1

Point Release containing bug fixes for the following:

* Intel® Ice driver fails to load on Ubuntu 18.04
* Updated Intel Network Adapter drivers to resolve driver compilation issues
* Increased driver download timeouts
* Removed duplicate collectd install in remote_fp profile
* Added missing NFD role to remote_fp profile
* Fixed CMK container template to include cmk binary
* Cleaned up data plane interface and IOMMU terminology
* Defaulted data plane interface examples as empty lists
* Updated Intel Network Adapter DDP profile URL
* Updated Intel Network Adapter FW package URL
* Fixed data plane interface selection on DDP profile loading
* Updated Helm stable repo URL
* Fixed dpdk vfio-pci binding for Ubuntu 18.04 and 20.04
* Updated default DPDK version to fix compilation on CentOS/RHEL 8
* Fixed PowerTools repository names for CentOS 8
* Updated sriov-dp version to build behind proxy
* Pinned google-auth python package version for legacy python2 compatibility

v2.0.0

Bare Metal Reference Architecture 2.0

Four configuration profiles are defined for BMRA 2.0 and more will be introduced in upcoming releases. Two of the configuration profiles are network location-specific:

• On-Premises Edge Configuration Profile – Typical Customer Premises deployment supporting, for example, Content Delivery Network (CDN) and Smart City scenarios.
• Remote Central Office-Forwarding Configuration Profile – Near Edge deployments supporting fast packet forwarding workloads such as Cable Modem Termination System (CMTS), Virtual Broadband Network Gateway (vBNG), and User Plane Function (UPF).

Two additional non location specific configuration profiles enable flexible deployments per need:

• Basic Configuration Profile – Minimum BMRA Kubernetes cluster setup.
• Full Configuration Profile – Complete BMRA setup supporting all available software features.

In addition, BMRA Release v2.0 has been updated to include support for the following technologies:

• Intel 800 series NICs
• Multiple Operating Systems (OSes): CentOS, Red Hat Enterprise Linux (RHEL), and Ubuntu
• Installation of Real Time Linux kernel packages
• Dynamic Device Personalization (DDP) on Intel NICs
• Intel® SST-BF support
• Improved cluster security
• Telemetry stack based on collectd, Prometheus, and Grafana
• Updated versions of many Kubernetes capabilities to pull in recent features and fixes
• Ansible playbook modifications (e.g. native CPU Manager configuration, NIC firmware updates)

v1.4.1

Bare Metal Reference Architecture 1.8

This version includes Kubernetes 1.16.

This release includes the following changes:

• added Telemetry Aware Scheduling
• added DDP profile loading capability for Intel® Ethernet Network Adapter 700 Series
• added Bond CNI plugin
• added SST-BF configuration utility
• updated DPDK to 19.11 and OVS to 2.13.0
• bug fixes and improvements