24.01.1

Reference Architecture 24.01.1

New Components/Features:
- Support Intel® Edge AI Box 4.0 on 4th/5th Gen Intel® Xeon® Scalable processor for the Edge/IoT
- Support Intel® In-Band Manageability framework software deployment with ThingsBoard server for remote device management operations for Intel® Edge AI box

New Hardware (Platforms/CPUs/GPUs/Accelerators):
- Support Intel® Edge AI Box 4.0 on Intel® Core™ Ultra PS Processor for the Edge/IoT

Updates/Changes:
- Notable version updates for RA components:
  OpenVINO™ v2024.1
  Intel® Deep Learning Streamer (Intel® DL Streamer) v2023.0.1
  Intel® AI Boost public driver v1.2
  Intel® GPU driver 20240430 release
  Intel® In-Band Manageability framework v4.2.1
  Intel® Base Containers sets v4.0 for Intel® Edge AI Box 4.0
  Ubuntu mainline kernel v6.8.1 for Intel® Core™ Ultra / Ultra PS Processor
- Support to specify GitHub mirror and DockerHub mirror before deployment
- Notable bug fixes
  Fixed OpenCV python binding not work issue
  Fixed Intel® Deep Learning Streamer python binding not work issue
  Fixed CBR video encoding not work on kernel 6.5 issue
  Fixed OpenCV building error on low system memory system

Removed Support:
- Removed support for CloudRA

Known Limitations/Restrictions:
- Only install GPU out of tree driver for kernel version <= 6.5.0-35 on on_prem_aibox profile, higher version kernel will use GPU in tree driver.
- XPU Manager metrics only partially supported on GPU in tree driver
- VAAPI HEVC video encoding not work on Intel® Core™ Ultra / Ultra PS Processor, QSV HEVC video encoding is ok.

24.01

Reference Architecture 24.01

New Components/Features:
- Support 5th Gen Intel® Xeon® Scalable processor for the Edge/IoT 
- Support Intel® Edge AI Box 4.0 on 12th Gen Intel® Core™ desktop processors, 12th Gen Intel® Core™ processor for IoT Edge, and Intel® Core™ Ultra Processor for Edge Deployments
- Support Intel® In-Band Manageability framework software to enable an administrator to perform critical device management operations for Intel® Edge AI box
- Support Intel® Workload Services Framework version of the Istio* Envoy* workload
- Support VMs provision via KubeVirt* (v1.1.0)
- Support external access to services via Ingress controller for Kubernetes using NGINX* as a reverse proxy and load balancer (ngress-nginx v4.8.3)
- Support automated detection and coniguration of SR-IOV and Intel® QuickAssist Technology (Intel® QAT) devices for VMRA

Updates/Changes:
- Versions upgraded for the vast majority of RA components (See Section 4.1 for all supported versions)
- Notable updates:
    FlexRAN™ to v23.11 [under NDA]
    Kubernetes* to v1.28.3
    Rancher to 2.7.9
    Data Plane Development Kit (DPDK) to v23.11
    Open vSwitch with DPDK to 3.2.1
    Intel® Multi-Buffer Crypto for IPsec library to v1.5
    Intel® QuickAssist Technology Engine for OpenSSL* (Intel® QAT Engine for OpenSSL*) to v1.4
    OpenSSL* to 3.1.4
    Intel® Power Manager to v2.3.1
    SR-IOV FEC Operator to v2.8
    OpenVINO™ v2023.1
    FFmpeg to v2023q3
    Intel® Deep Learning Streamer (Intel® DL Streamer) 2023.0
    OpenCV 4.8.0
    VPP to 23.10
    Intel® Dynamic Load Balancer (Intel® DLB) software release v8.7.0
    Intel® Data Streaming Accelerator (Intel® DSA) software release v4.14
    Istio 1.20.1
    Intel® Node Feature Discovery (NFD) to 0.14.3
    Linux overlay kernel 6.3.0-x 
    Intel® Core™ Ultra processor GPU driver and X11/Wayland* UI framework 
    Intel® AI Boost driver for Intel® Core™ Ultra processor [under NDA]
    Intel® Core™ Ultra processor audio firmware and topology [under NDA]

Updates/Changes made for the Reference System 23.10.1 minor release:
- Support Intel® Core™ Ultra processor with NDA packages
- Update Base Container components supported by On Prem Edge AI Box Configuration Profile

New Hardware (Platforms/CPUs/GPUs/Accelerators):
- Intel® Core™ Ultra processor for Edge Deployments 
- 5th Gen Intel® Xeon® Scalable processor for the Edge/IoT

Removed Support:
- Officially discontinued support for 3rd Gen Intel® Xeon® Scalable processors

Known Limitations/Restrictions:
- CPU Control Plane Plugin for Kubernetes* is only supported on Ubuntu OS with single node deployment
- MinIO* is supported only with CRI-O runtime
- Only in-tree Intel® QuickAssist Technology (Intel® QAT) drivers supported on RHEL 9.2 and Rocky 9.2 
- UserSpace CNI with VPP is not supported
- Intel® Trust Domain Extensions (Intel® TDX) on VMRA does not support Intel® Dynamic Load Balancer (Intel® DLB), Intel® DSA, Intel® QAT, or network adapter device passthrough due to Intel® TDX driver security concerns 
- Intel® Ethernet Operator DDP update feature might not work in rare cases; Legacy DDP update feature does not work
- KubeVirt and GPU_DP do not support Docker runtime
- Intel® Media Transport Library v23.08 only supports ICE versions between 1.9.11 and 1.11.14
- OneAPI ITEX sample does not support the GPU path on RHEL 9.2

23.10.1

Reference Architecture 23.10.1

New Components/Features:
- Support Intel® Ultra Processor (Meteor Lake) with NDA packages, including:
    Linux overlay kernel 6.3.0-x
    Meteor Lake GPU driver and X11/Wayland UI framework
    Meteor Lake NPU driver [under NDA]
    Meteor Lake audio firmware and topology [under NDA]

Updates/Changes:
- Update Base Container components supported by On Prem Edge AI Box Configuration Profile
    aibox-base: from ubuntu 22.04
      gpu 20230912,  openvino runtime 2023.1
    aibox-base-devel: from aibox-base
      openvino dev tool 2023.1
    aibox-dlstreamer: from ubuntu 22.04
      gpu 20230912
      openvino runtime 2023.0, openvino dev tool 2023.0
      dlstreamer 2023.0
    aibox-opencv-ffmpeg: from aibox-base
      opencv 4.8.0, ffmpeg 2023q2

New Hardware (Platforms/CPUs/GPUs/Accelerators): 
- Intel Ultra Processor (Meteor Lake)

Known Limitations/Restrictions:
- AIBox Base Containers only work for CPU + GPU platforms earlier than Meteor Lake

23.10

Reference Architecture 23.10

New Components/Features:
- Extended support for 5th Gen Intel® Xeon® Scalable processors to VMRA, which requires the NDA version of Intel® QuickAssist Technology (Intel® QAT) drivers
- Support for Intel® Trust Domain Extensions (Intel® TDX) on 5th Gen Intel® Xeon® Scalable processors with limitations
- Support for 12th Gen Intel® Core™ processor for IoT Edge 
- Support for Edge AI Box 3.1 on 12th Gen Intel® Core™ desktop processors, 12th Gen Intel® Core™ processor for IoT Edge
- Extended support for Software Defined Factory use-case on Intel® Core™ and Intel Atom® platforms to include an additional platform to manage the deployment of controller node via Kubernetes
- Support deployment of a Machine Controller use case via VMRA 
- Support ECDSA keys for 5G O-RAN security with NETCONF server/client authentication (Intel® Software Guard Extensions (Intel® SGX)) 
- Added boot guard and secure boot to 5G O-RAN security
- Use of Vector Packet Processing (VPP) (23.06) dataplane within Calico
- Integrated Intent-Driven Orchestration Release v0.2 with BMRA
- Support for Intel® Infrastructure Processing Unit (Intel® IPU) ASIC E2000 card and Infrastructure Programmer Development Kit (IPDK) Networking Recipe
- Support for virtual machine and bare metal mixed Kubernetes deployment 
- Support for generic virtual machine type 
- Integrated Intel® Media Transport Library library
- Integrated Intel® XPU Manager (Intel® XPUM) for GPU health and telemetry monitoring 
- Extend telemetry stack support to container only environment for AIBox 
- Initial support of Collection, other consumers can directly import and use just roles from this collection 
- Added Rocky Linux 9.2 as base OS

Updates/Changes:
- Version upgraded for the vast majority of RA components
- Notable updates:
    FlexRAN™ to v23.07
    Kubernetes* to v1.27.1
    Intel® Containerized Telegraf to 1.3
    Service Mesh Istio to v1.19.0
    Intel® Managed Distribution of Istio* Service Mesh to v1.19.0-intel.0
    Open vSwitch with DPDK to 3.2
    Intel Multi-Buffer Crypto for IPsec library to v1.4
    Intel® QAT Engine for OpenSSL to v1.3.1
    Intel® Power Manager to v2.3.1
    Data Plane Development Kit (DPDK) to v23.07
    Intel® Ethernet Operator v23.08
    SR-IOV FEC Operator to v2.7.1
    OpenVINO™ v2023.1
    Key Management Reference Application (KMRA) v2.4
    Kubernetes to v1.27 for Cloud RA on Microsoft* Azure Kubernetes Service (AKS) and Amazon Elastic Kubernetes Service* (Amazon EKS)

New Hardware (Platforms/CPUs/GPUs/Accelerators):
- 12th Gen Intel® Core™ desktop processors with Intel® Arc™ Discrete Graphics GPU A380 
- 12th Gen Intel® Core™ processor for IoT Edge with Intel® Iris® Xe Integrated Graphics
- 13th Gen Intel® Core™ mobile processor with Intel® Iris® Xe Integrated Graphics

Removed Support:
- Discontinued support for Rocky 9.1 as base OS

Known Limitations/Restrictions:
- Intel® Data Center GPU Flex Series, CPU Control Plane Plugin for Kubernetes*, Intel® Media SDK (only Docker runtime) are only supported on Ubuntu OS
- MinIO* is supported only with CRI-O runtime 
- Only in-tree Intel® QuickAssist Technology (Intel® QAT) drivers supported on RHEL 9.2 and Rocky 9.2 
- Intel® Data Streaming Accelerator (Intel® DSA) may not work on some older (earlier stepping) CPUs on RHEL 9.2 and Rocky Linux 9.2 
- Work-around applied to fix default Intel® QAT driver 4.23.0 install fail in VMRA mode on 3rd Gen Intel® Xeon® Scalable processors
- UserSpace CNI with VPP is not supported 
- Intel® Trust Domain Extensions (Intel® TDX) on VMRA does not support Intel® Dynamic Load Balancer (Intel® DLB), Intel® DSA, Intel QAT, or network adapter device passthrough due to Intel® TDX driver security concerns 
- Intel Ethernet Operator DDP update feature might not work in rare cases; use legacy DDP update feature instead
- Mixed deployment on_prem_sw_defined_factory profile with 12th Gen Intel® Core™ or later CPU should disable hyper thread 
- Power Manager fail to set target frequency with frequency_scaling_driver set as "acpi_cpufreq"

23.07.1

Reference Architecture 23.07.1

New Components/Features:
- FlexRAN™ Docker container support (timer and XRAN mode test) on 5th Gen Intel® Xeon® Scalable processors (MCC)
- Support for Intel® Video AI Box through new configuration profile - on_prem_aibox
- Integrated Docker Compose 2.12
- Generated Base Containers Dockerfiles for Intel® Video AI Box, including
    aibox-base
      GPU user mode drivers and runtimes (202230714)
      OpenVINO runtime (2022.3)
    aibox-base-dev
      OpenVINO developer tools (2022.3)
    aibox-dlstreamer
      DLStreamer (2022.3)
    aibox-opencv-ffmpeg
      OpenCV (4.7)
      FFMPEG (cartwheel 2023q1)

Updates/Changes:
- Bug fix in Project Sylva validation suite to revise device ID map and readme and config update

New Hardware (Platforms/CPUs/GPUs/Accelerators):
- ADL-S with Arc dGPU
- RPL-P with integrated GPU

Known Limitations/Restrictions:
- N/A (same as RA23.07)

23.07

Reference Architecture 23.07

New Components/Features:
- Support for 5th Gen Intel® Xeon® Scalable processors requires the NDA version of the Intel® QuickAssist Technology (Intel® QAT) drivers
- Support for 12th Gen Intel® Core™ processors
- Support for Intel Atom® x6000e series processors
- Support for 5G O-RAN security with NETCONF server/client authentication (Intel® Software Guard Extensions (Intel® SGX))
- Support Rancher RKE2 Kubernetes distribution for the access edge configuration profile with Ubuntu. (v1.26.2+rke2r1)
- Support for Intel® FPGA SmartNIC WSN6050 Platform for video production
- Enabled Intel® Infrastructure Processing Unit (Intel® IPU) ASIC E2000 card and made it available in the host machine, which requires the NDA version of image
- Support for Intel® SGX in VMRA by upgrading QEMU and libvirt
- Support for Key Management Reference Application (KMRA) in Virtual Machine Reference Architecture (VMRA)
- Support for KMRA on 4th Gen Intel® Xeon® Scalable processors on production SKUs
- Implemented Intel® SGX-enabled Istio in VMs
- Support for Cilium eBPF Dataplane on Microsoft Azure Kubernetes Service (AKS)
- Updates to Kubernetes version and tools used to deploy on Microsoft Azure Kubernetes Service (AKS) and Amazon Web Service (AWS) Elastic Kubernetes Service (EKS) in Cloud Reference System Architecture
- Implemented support and option for Intel® QuickAssist Technology (Intel® QAT) in-tree versus out-of-tree drivers and libraries
- Integrated Intel® oneAPI Base Toolkit (Base Kit) 2023.1 and Intel® AI Analytics Toolkit (AI Kit) 2023.1.1
- Integrated FFmpeg with cartwheel (Intel GitHub 2023q1 release)
- Added two Configuration Profiles:
    On-Premises SW Defined Factory Configuration Profile for industrial use cases
    On-Premises VSS Configuration Profile for Video Structuring Server (VSS) use cases
- RHEL 9.2 as base OS
- RHEL 9.2 RT as base OS
- Ubuntu 22.04.2 as base OS

Updates/Changes:
- Version upgraded for the vast majority of Reference System components (See elsewhere in this document for complete BOM and versions)
- Notable updates:
    FlexRAN™ to v23.03
    Kubernetes* to v1.26.3
    CPU Control Plane Plugin for Kubernetes* to v0.1.2
    Telemetry Aware Scheduling to v0.5.0
    GPU Aware Scheduling to v0.5.2
    Intel® Power Manager to v2.2
    Service Mesh Istio to v1.18.1
    Intel® Managed Distribution of Istio* Service Mesh to v1.18.0-intel.0
    Data Plane Development Kit (DPDK) to v23.05
    Open vSwitch with DPDK to 3.11
    Traffic Analytics Development Kit (TADK) to 23.03
    OpenSSL to openssl-3.1.0
    Intel® Data Center GPU Flex Series driver to 20230519 release
    SR-IOV FEC Operator to 2.7
    Intel® Platform Telemetry Insights to 23.07 (with license)
- Kubespray* is provided via ansible-galaxy collection instead of git submodule

Updates/Changes made for the Reference System 23.02.1 minor release:
- Stack Validation:
    Test cases created for Anuket Reference Architecture Kubernetes Component Level Architecture specifications, to be used for Anuket Reference Conformance Kubernetes and Project Sylva Stack validation
    Test cases created for Device Plugins Single Root IO Virtualization (SR-IOV) Data Plane Development Kit and Multus Container Network Interface
    Cloud Native Network Function (CNF) Validation:
    Test case to check CNF allocation of SR-IOV devices like virtual functions of network adapters or accelerators, to be used for Project Sylva CNF validation
- Added workaround for building the Intel® Ethernet Operator and SR-IOV FEC (Forward-Error Correction) Operator
- Resolved issue regarding the user-space CNI by disabling Vector Packet Processing (VPP)
- Removed dependency of Intel® QuickAssist Technology (Intel® QAT) on OpenSSL to allow independent deployment of Crypto libraries

New Hardware (Platforms/CPUs/GPUs/Accelerators):
- 5th Gen Intel® Xeon® Scalable processors (XCC, MCC)
- 4th Gen Intel® Xeon® Scalable processor with Intel® vRAN Boost up to 32 cores
- Intel® FPGA SmartNIC WSN6050 Platform
- 12th Gen Intel® Core™ processors
- Intel Atom® x6000e series processors
- Intel® Infrastructure Processing Unit (Intel® IPU) ASIC E2000 card

Removed Support:
- Discontinued supporting Cloud Native Data Plane (CNDP)
- Discontinued supporting RHEL 9.0 as base OS
- Discontinued supporting RHEL 8.6 RT as base OS

Known Limitations/Restrictions:
- Intel® Data Center GPU Flex Series, CPU Control Plane Plugin for Kubernetes, Intel® Media SDK (only Docker runtime) are only supported on Ubuntu OS
- FlexRAN™ container support is limited to v22.07, Ubuntu 22.04 base OS, and only on 3rd Gen Intel® Xeon® Scalable processors
- MinIO is supported only with CRI-O runtime
- Only in-tree Intel® QuickAssist Technology (Intel® QAT) and Intel® Ethernet Network Adapter E810 drivers supported on RHEL 9.2
- Intel® Ethernet Network Adapter E810 in-tree driver does not support VF function on RHEL 9.2, which impacts XRAN mode test in FlexRAN™ application
- Intel® QuickAssist Technology (Intel® QAT) is not supported on Rocky Linux 9.1 on 5th Gen Intel® Xeon® Scalable processors
- Intel® Data Streaming Accelerator (Intel® DSA) may not work on some older (earlier stepping) CPUs on RHEL 9.2
- UserSpace CNI with VPP is not supported
- Rancher only supported for containerd.
- CAdvisor not supported on CRI-O runtime

23.02.1

Reference Architecture 23.02.1

New Components/Features:
- Stack Validation: 
    Test cases created for Anuket Reference Architecture Kubernetes Component Level Architecture specifications, to be used for Anuket Reference Conformance Kubernetes and Project Sylva Stack validation 
    Test cases created for Device Plugins Single Root IO Virtualization (SR-IOV) Data Plane Development Kit and Multus Container Network Interface
- Cloud Native Network Function (CNF) Validation: 
    Test case to check CNF allocation of SR-IOV devices like virtual functions of network adapters or accelerators, to be used for Project Sylva CNF validation

Updates/Changes:
- Added workaround for building the Intel® Ethernet Operator and SR-IOV FEC (Forward-Error Correction) Operator
- Resolved issue regarding the user-space CNI by disabling Vector Packet Processing (VPP)
- Removed dependency of Intel® Quick Assist Technology (Intel® QAT) on OpenSSL to allow independent deployment of Crypto libraries

New Hardware (Platforms/CPUs/GPUs/Accelerators):
- N/A (same as RA23.02)

Removed Support: 
- CNDP (Cloud Native Data Plane)

Known Limitations/Restrictions:
- N/A (same as RA23.02)

23.02

Reference Architecture 23.02

New Components/Features:
- Media Analytics Libraries
	Intel® Deep Learning Streamer (Intel® DL Streamer), GStreamer, OpenVINO™ toolkit
	OpenCL™ software, Level zero GPU, DPC++, and VAAPI from the Intel® GPU toolkit
- FlexRAN™ software running as a Docker container (now available without NDA)
- Rook/Ceph as a storage-related component
- Rocky Linux 9.1 as base operating system (with some limitations mentioned below)
- Non-root user deployment of Virtual Machine Reference System Architecture (VMRA)
- Custom cluster naming in VMRA
- Support for using Amazon Web Services (AWS) and Azure “Cloud” CLIs as an alternative to Terraform
- Azure Kubernetes Service (AKS) support for static CPU Management Policy and Intel® CPU Control Plane Plugin for Kubernetes
- Intel® Software Guard Extensions (Intel® SGX) on AKS

Updates/Changes:
- Software versions upgraded for the majority of RA components (See User Guide for complete BOM and versions)
  Notable updates:
	Kubernetes to v1.26.1
	MinIO to v4.5.8
	DPDK to v22.11.1
	Service Mesh to v1.17.1
	VPP to v2302
	KMRA to v2.3
- Eliminated the BMRA for Object Storage Setup deployment model. The storage-related features (MinIO, LPVSP, and Rook/Ceph) are now provided as optional components in select configuration profiles
- Support of geo-specific mirrors for Kubespray (for example, in the People’s Republic of China)
- Supported Kubernetes versions updated for AKS and Amazon EKS
- Ubuntu images updated for AKS and Amazon EKS
- Ability to deploy more RA software components on Azure and AWS
	Elasticsearch
	Kibana

New Hardware (Platforms/CPUs/GPUs/Accelerators):
- N/A

Removed Support:
- full_nfv profile
- Ubuntu 20.04 as base operating system
- Rocky Linux 9.0 as base operating system

Known Limitations/Restrictions:
- When using the Cilium CNI, secondary interfaces are not supported
- Intel® Dynamic Load Balancer (Intel® DLB) is not fully supported on Rocky Linux 9.1
- FlexRAN container support is limited to FlexRAN v22.07, Ubuntu 22.04 base operating system, and only on 3rd Gen Intel® Xeon® Scalable processors
- Media Analytics is supported only with Docker runtime
- MinIO is supported only with CRI-O runtime
- VMRA cluster expansion with additional VM nodes might fail
- Trusted Certificate Attestation (TCA) is not fully functional in VMRA

22.11.1

Reference Architecture 22.11.1

New Components/Features:
- N/A (same as RA22.11)
Updates/Changes:
- Intel(R) QAT 2.0 drivers for the 4th Gen Intel(R) Xeon(R) Scalable processors (formerly code named Sapphire Rapids [SPR]) are sourced from public repo. No longer under NDA; please ignore Guide requirement to provide the "QAT20.L.0.9.9-00019.tar.gz" driver package file.
- Resolved issue regarding downloading CPUID for Rocky Linux 9 and RHEL 9
New Hardware (Platforms/CPUs/GPUs/Accelerators):
- N/A (same as RA22.11)
Removed Support: 
- N/A (same as RA22.11)
Known Limitations/Restrictions:
- N/A (same as RA22.11)

22.11

Reference Architecture 22.11

New Components/Features:
- Intel® Software Guard Extensions (Intel® SGX) support on 4th Gen Intel Xeon Scalable processors (SPR)
- Enabled select features (SGX and QAT) to be deployed through Ansible tags to facilitate interoperability with Intel’s Workload
Services Framework (WSF)
- Support deployment via hostname or FQDN
- Support for clean-up /re-deploy of the ‘Basic’ profile
- Support for the Content Delivery Network (CDN) use case
- New observability stack including Opentelemetry and Kibana (and expanding Jaeger support to VMRA)
- Intel’s CPU Control Plane Management
- Support for Local Volume Provisioner
- Support for Cilium
- Cloud RA: Support for Azure AKS deployments on top of previous support for AWS EKS
- Cloud RA: Support for generating and deploying configuration profiles and using the generated host/group_vars during
deployment
- Cloud RA: Support for Cilium with kube-proxy and eBPF CNI on Azure
- Cloud RA: Proximity Placement Groups for Azure

Updates/Changes:
- Version updates for the vast majority of RA components (See User Guide for complete list and versions)
- Autodetection of the QAT and FEC ACC devices
- Extended Linkerd as a service mesh option for VMRA
- Support for Cluster Flow Config with the updated Intel Ethernet Operator (IEO)
- Enhanced discovery mechanism for Cloud RA

New Hardware (Platforms/CPUs/GPUs/Accelerators):
- Intel® Data Center GPU, code named Arctic Sound-M (ATS-M)
- FEC Accelerator (ACC200) embedded into 4th Gen Intel® Xeon® Scalable processors (Sapphire Rapids) with vRAN Boost
- 3rd Gen Intel® Xeon® Scalable processor CPU (Icelake) SKU: 6348
- 4th Gen Intel® Xeon® Scalable processor CPU (Sapphire Rapids) SKUs: 6421N, 6438N, 8480+, 8487C

Removed Support:
- SG1 Graphics card
- Visual Cloud Accelerator Card for Analytics (VCAC-A)

Known Limitations/Restrictions:
- Key Management Reference Application (KMRA) is NOT supported on 4th Gen Intel Xeon Scalable processors (SPR)
- KMRA is NOT supported on CRI-O runtime with 3rd Gen Intel Xeon Scalable processors (ICX)
- CRI-O runtime is NOT supported on Ubuntu 20.04
- NIC Firmware update is NOT supported through Intel Ethernet Operator (IEO)
- VMRA support with containerd runtime environment is limited (unstable) and might exhibit failures of some pods