From a148aebe3a704a6c48df0792274c74f60946c113 Mon Sep 17 00:00:00 2001 From: Sujana-M <52772661+Sujana-M@users.noreply.github.com> Date: Fri, 29 Sep 2023 19:04:12 +0530 Subject: [PATCH] Exerciser documents and README updates (#402) - Added the README which provides information on where to download the RDN2 SW stack - Any changes required for running SBSA exerciser on RDN2 reference - All the documentation and README updates related to PCIe exerciser functionality Signed-off-by: Sujana M --- README.md | 25 +- docs/PCIe_Exerciser/ErrorInjection.md | 127 +++ docs/PCIe_Exerciser/Exerciser.md | 475 +++++++++++ .../Exerciser_API_porting_guide.md} | 6 +- .../PCIeConfigurableHierarchy.md | 786 ++++++++++++++++++ docs/PCIe_Exerciser/README.md | 76 ++ .../example_pcie_hierarchy_0.json | 157 ++++ platform/pal_baremetal/README.md | 2 +- 8 files changed, 1647 insertions(+), 7 deletions(-) create mode 100644 docs/PCIe_Exerciser/ErrorInjection.md create mode 100644 docs/PCIe_Exerciser/Exerciser.md rename docs/{Exerciser_porting_guide.md => PCIe_Exerciser/Exerciser_API_porting_guide.md} (96%) create mode 100644 docs/PCIe_Exerciser/PCIeConfigurableHierarchy.md create mode 100644 docs/PCIe_Exerciser/README.md create mode 100644 docs/PCIe_Exerciser/example_pcie_hierarchy_0.json diff --git a/README.md b/README.md index 4130ef54..b18e1374 100644 --- a/README.md +++ b/README.md @@ -14,12 +14,18 @@ For more information, download the [SBSA specification](https://developer.arm.co SBSA **Architecture Compliance Suite** (ACS) is a collection of self-checking, portable C-based tests. This suite includes a set of examples of the invariant behaviors that are provided by the [SBSA specification](https://developer.arm.com/documentation/den0029/h/?lang=en), so that implementers can verify if these behaviours have been interpreted correctly. + Most of the tests are executed from UEFI Shell by executing the SBSA UEFI shell application. A few tests are executed by running the SBSA ACS Linux application which in turn depends on the SBSA ACS Linux kernel module. +The tests can also be executed in a Bare-metal environment. The initialization of the Bare-metal environment is specific to the environment and is out of scope of this document. ## Release details - Code Quality: REL v7.1.2 - The tests are written for version 7.1 of the SBSA specification. + - For complete coverage of the SBSA rules, availability of an Exerciser is required for Exerciser tests to be run during verficiation at Pre-Silicon level. + - For complete coverage, both SBSA and BSA ACS should be run. + - The SBSA specification compliments the [BSA specification](https://developer.arm.com/documentation/den0094/c/?lang=en). + - The tests can be run at both the Pre-Silicon and Silicon level. - The compliance suite is not a substitute for design verification. - To review the SBSA ACS logs, Arm licensees can contact Arm directly through their partner managers. - To know about the SBSA rules not implemented in this release, see [Test Scenario Document](docs/Arm_SBSA_Architecture_Compliance_Test_Scenario.pdf). @@ -31,13 +37,19 @@ A few tests are executed by running the SBSA ACS Linux application which in turn ## Additional reading - For information about the implementable SBSA rules test algorithm and for unimplemented SBSA rules, see the [Test Scenario Document](docs/Arm_SBSA_Architecture_Compliance_Test_Scenario.pdf). - - For information on test category(UEFI, Linux, BM) and applicable systems(SR,PreSilicon), see [Test Checklist](docs/Arm_SBSA_testcase-checklist.rst) + - For information on test category(UEFI, Linux, Bare-metal) and applicable systems(SR,Pre-Silicon), see [Test Checklist](docs/Arm_SBSA_testcase-checklist.rst) - For details on the design of the SBSA ACS, see the [Arm SBSA Validation Methodology Document](docs/Arm_SBSA_Architecture_Compliance_Validation_Methodology.pdf). - For details on the SBSA ACS UEFI Shell Application, Linux Application and PMU Linux Application see the [Arm SBSA ACS User Guide](docs/Arm_SBSA_Architecture_Compliance_User_Guide.pdf). - - For details on the SBSA ACS Baremetal support, see the + - For details on the SBSA ACS Bare-metal support, see the - [Arm SBSA ACS Bare-metal User Guide](docs/Arm_SBSA_ACS_Bare-metal_User_Guide.pdf). - [Bare-metal Code](platform/pal_baremetal/).
-Note: The Baremetal PCIe enumeration code provided as part of the SBSA ACS should be used and should not be replaced. This code is vital in analyzing of the test result. +Note: The Bare-metal PCIe enumeration code provided as part of the SBSA ACS should be used and should not be replaced. This code is vital in analyzing of the test result. + +### Running Exerciser tests for complete coverage + +Exerciser is a client device wrapped up by PCIe Endpoint. This device is created to meet SBSA requirements for various PCIe capability validation tests. Running the Exerciser tests provides additional test coverage on the platform. + +Note: To run the exerciser tests on a UEFI Based platform with Exerciser, the Exerciser PAL API's need to be implemented. For details on the reference Exerciser implementation and support, see the [Exerciser.md](docs/PCIe_Exerciser/Exerciser.md) and [Exerciser_API_porting_guide.md](docs/PCIe_Exerciser/Exerciser_API_porting_guide.md) ## SBSA ACS Linux kernel module To enable the export of a few kernel APIs that are necessary for PCIe and IOMMU tests, Linux kernel module and a kernel patch file are required. These files are available at [linux-acs](https://gitlab.arm.com/linux-arm/linux-acs). @@ -214,6 +226,13 @@ shell> ./sbsa ``` - For information on the SBSA Linux application parameters, see the [SBSA ACS User Guide](docs/Arm_SBSA_Architecture_Compliance_User_Guide.pdf). +## ACS build steps - Bare-metal abstraction + +The Bare-metal build environment is platform specific. + +To execute the Bare-metal code from UEFI Shell, checkout to [bare-metal](https://github.com/ARM-software/sbsa-acs/tree/baremetal_reference) branch of SBSA and the build steps to integrate and run the same from UEFI shell are provided in the [README.md](https://github.com/ARM-software/sbsa-acs/blob/baremetal_reference/README.md) + +For details on generating the binaries to run on Bare-metal environment, refer [README.md](platform/pal_baremetal/README.md) ## Security implication Arm Enterprise ACS test suite may run at higher privilege level. An attacker may utilize these tests as a means to elevate privilege which can potentially reveal the platform security assets. To prevent the leakage of secure information, it is strongly recommended that the ACS test suite is run only on development platforms. If it is run on production systems, the system should be scrubbed after running the test suite. diff --git a/docs/PCIe_Exerciser/ErrorInjection.md b/docs/PCIe_Exerciser/ErrorInjection.md new file mode 100644 index 00000000..f5488758 --- /dev/null +++ b/docs/PCIe_Exerciser/ErrorInjection.md @@ -0,0 +1,127 @@ +# PCIe Error Injection user documentation +This document gives details of the error injection mechanism for PCIe endpoints and how it can be used to inject errors and check for error reporting inside PCIe subsystem. + +## Introduction to PCIe error injection mechanism + +A new Error injection extended capability, implemented as a DVSEC, has been added to enable user to inject errors in an PCIe endpoint. This capability can be enabled using a new parameter : `error_injection_supported`. The layout of this capability is as follows: + +``` + Register Name Description Address offset + ------------- ----------- -------------- + + Extended Capability Header Standard DVSEC header. 0x0 + + DVSEC Header 1 Error injection DVSEC details. 0x4 + + Control Register Configure and inject errors (ERROR_CTL_REG) 0x8 + +``` + +## Register description + +``` + Extended Capability Header Bits Description r/w Value at reset + -------------------------- ----- ----------- --- -------------- + + extended_capability_id 15:0 Identifies which extended capability RO 0x0023 + structure this is. + + capability_version 19:16 Identifies the version of this RO 0x1 + structure. + + next_capability_offset 31:20 Provides the byte offset from the top of RO Depends on + config space to the next extended capability position of + structure. A value of 000h ends the linked next capability + list of extended capability structures. + + + DVSEC Header 1 Bits Description r/w Value at reset + -------------------------- ----- ----------- --- -------------- + + dvsec_vendor_id 15:0 Holds a designated VendorID assigned by the RO 0x13B5 + PCI-SIG. This VendorID would be assigned to + a group of companies collaborating on a + common set of register definitions which + would like in this structure in PCI config + space. + + dvsec_rev 19:16 Holds a vendor-defined version number RO 0x0 + + dvsec_length 31:20 Indicates the size (in bytes) of this RO 0xC + extended capability structure, including + the Extended Capability Header, DVSEC + Header1 and DVSEC Header 2. + + + Control Register Bits Description r/w Value at reset + -------------------------- ----- ----------- --- -------------- + + dvsec_id 15:0 Holds a vendor-defined identification RO 0x1 + number to help determine the nature and + format of this structure. + + inject_error_on_dma 16 Put endpoint in Corrupt DMA mode. See RW 0x0 + Corrupt DMA section for more details. + + inject_error_immediately 17 Inject error in this endpoint configured RW 0x0 + using the error_code field. + + reserved 19:18 Reserved RO 0x0 + + error_code 30:20 Error code configuration for corrupt DMA RW 0x0 + mode and inject_error_immediately + + treat_uncorrectable_as_ 31 If error code is an uncorrectable error, RW 0x0 + fatal then this bit configures severity of the + error. This bit has no effect if endpoint + supports Advanced Error Reporting (AER). + If AER is supported, this bit is + overridden by AER uncorrectable severity + register. +``` + +## How to inject errors + +There are 2 ways to inject errors: + * Inject immediately : Using the `inject_error_immediately` bit set, the user can inject an error at that endpoint with the error configured using the `error_code` field in control register. The error_codes are defined in Error Codes section. This bit is cleared once the error has been injected. + + * Corrupt DMA : With the `inject_error_on_dma` bit set, the endpoint is put in Corrupt DMA mode. Any peer-to-peer DMAs generated in corrupt DMA mode, will lead to an error injection in destination endpoint. All DMAs will fail by default in this mode, so this bit will need to be cleared for normal functioning of DMAs for this endpoint. The injected error in destination endpoint will be as configured by `error_code` field in control register. The error_codes are defined in Error Codes section. + +## Error Codes +``` + Error Name Error Code + ---------- ---------- + Correctable Receiver Error 0x00 + Correctable Bad TLP 0x01 + Correctable Bad DLLP 0x02 + Correctable Replay Num Rollover 0x03 + Correctable Replay Timer Timeout 0x04 + Correctable Advisory Non-Fatal Error 0x05 + Correctable Internal Error 0x06 + Correctable Header Log OverFlow 0x07 + Uncorrectable Data Link Error 0x08 + Uncorrectable Surprise Down Error 0x09 + Uncorrectable Poisoned TLP Received 0x0A + Uncorrectable Flow Control Error 0x0B + Uncorrectable Completion Timeout 0x0C + Uncorrectable Completer Abort 0x0D + Uncorrectable Unexpected Completion 0x0E + Uncorrectable Receiver Overflow 0x0F + Uncorrectable Malformed TLP 0x10 + Uncorrectable ECRC Error 0x11 + Uncorrectable Unsupported Request 0x12 + Uncorrectable ACS Violation 0x13 + Uncorrectable Internal Error 0x14 + Uncorrectable MultiCast Blocked TLP 0x15 + Uncorrectable Atomic Op Egress Blocked 0x16 + Uncorrectable TLP Prefix Blocked Egress 0x17 + Uncorrectable Poisoned TLP Egress Blocked 0x18 + Invalid configuration 0x19 onwards +``` + +## Error reporting in intermediate components: +When an corrupt DMA passes through an intermediate component such as a switch, it can detect and report an uncorrectable error as an advisory non-fatal error. The detecting bridges (inside the switch) will, in this case, log a correctable advisory non-fatal error and optionally report an correctable error message to its parent root port. This detection and reporting of intermediate errors can be enabled using a new parameter `report_advisory_non_fatal_errors` for a switch. + +-------------- + +*Copyright (c) 2023, Arm Limited and Contributors. All rights reserved.* diff --git a/docs/PCIe_Exerciser/Exerciser.md b/docs/PCIe_Exerciser/Exerciser.md new file mode 100644 index 00000000..a41a928f --- /dev/null +++ b/docs/PCIe_Exerciser/Exerciser.md @@ -0,0 +1,475 @@ +# PCIe EXERCISER DOCUMENTATION +This document gives details of the various PCIe capabilities that exerciser device supports and how the exerciser is supposed to behave. + +## Contents + +* Introduction to PCIe Exerciser Endpoint Device +* Execution model and flow control for following PCIe Features: + + * DMA Transaction + * MSI Generation and Service + * Legacy Interrupt Generation and Service + * ATS Transaction + * Transaction Monitoring + +## Introduction to PCIe Exerciser Endpoint Device + +PCIe Exerciser is a client device wrapped up by PCIe Endpoint. This device was created to meet SBSA (Server Based System Architecture) requirements for various PCIe +capability validation tests. Exerciser's BAR0 space contains following control and support registers to support various PCIe capabilities in exerciser: + +``` + Register Name Description Address offset + ------------- ----------- -------------- + + MSI Control Register Configure and trigger MSIs. 0x0 + + Legacy Interrupt Control Configure and trigger legacy interrupts. 0x4 + Register + + DMA Control Register Configure and trigger DMA transactions(DMA_CTL_REG) 0x8 + + DMA Offset Register Configure DMA transaction access offset (in bytes) 0xc + from base of exerciser memory. + + Bus Address Register Configure bus address that is destination/source 0x10 + for DMA or untranslated input address for ATSRequest. + + DMA Length Register Configure DMA transaction length. 0x18 + + DMA Status Register Status of last DMA transaction. 0x1c + + PASID Value Register Store the pasid value passed for initiating DMA/ATS 0x20 + transaction. + + ATS Control Register Configure and trigger ATS transaction(ATS_CTL_REG) 0x24 + + ATS Reply Translated Address Hold the translated address returned from an 0x28 + Register ATSRequest. + + ATS Translated Range Size Size of the ATS translated address region in bytes. 0x30 + Register + + ATS Reply Permission Permissions returned for translated memory address 0x38 + Register region as a result of ATSRequest. + + Illegal Requestor ID This register is DANGEROUS, as it will cause illegal 0x3c + Control Register transactions to be generated by exerciser. + Use case is when SW wants exerciser to issue a + transaction with custom/illegal requestor id, + rather than the one determined through PCI enumeration. + + Transaction Trace Database Holds information about the config and memory 0x40 + Register transactions. Each read will return one 32 bit entry + previously recorded. + + Transaction Trace Control Control bit to start/stop transaction monitoring. 0x44 + Register + + +``` + +## Execution model and flow control + +**DMA Transaction** + +``` + DMA Control Register Bit Description r/w Value at reset + -------------------- --- ----------- --- -------------- + + dmatxntrig 3:0 Trigger DMA for given configuration RW 0h + Set to 1h to trigger DMA. This is reset + to 0h after transaction completes. + values 2h – Fh are reserved. + + Dmatxndir 4 DMA transaction type 0b : Read, 1b: Write RW 0b + + dmatxnsnoop 5 Snoop attribute for DMA transaction RW 0b + 0: Snoop + 1: No-snoop + + dmapasiden 6 PASID Enable bit RW 0b + 0: PASID disabled + 1: PASID enabled + + dmaIsPrivileged 7 Is Privileged access mode bit RW 0b + 0: User access mode + 1: Privileged access mode + (dmapasiden bit must be 1) + + dmaIsInstruction 8 Is Instruction access bit RW 0b + 0: Data access + 1: Instruction access (dmapasiden bit + must be 1) + + dmaUseATCforTranslation 9 Use ATC for input address translation RW 0b + 0: do no translation mapping. + 1: If true then, if DMA input address is + in ATC input address range, then DMA address + will be mapped to output translated address + as per translation response of last ATSRequest + + dmaAddressType 11:10 DMA bus address type RW 0h + 0: Default (treated as Untranslated) + 1: Untranslated + 2: Translated + 3: Reserved + Note: Any transaction with Reserved address + type will fault by logging error in UR bit in + device status register in upstream rootport. + DMA status register will also reflect internal + error. + + dmactlres 31:12 Reserved RO 0h + + + DMA Status Register Bit Description r/w Value at reset + ------------------- -- - ----------- --- -------------- + + dmastatus 1:0 DMA error codes RO 0h + 0h: No error. DMA successful + 1h: DMA range out of bounds + 2h: DMA internal error + 3h: Reserved + + dmaStatusClear 2 Write 1 to clear DMA status WO 0b + + dmastatres 31:3 Reserved RO 0h + + + Pasid Value Register Bit Description r/w Value at reset + -------------------- --- ----------- --- -------------- + + PASIDValue 19:0 PASID value for DMA/ATSRequest RW 0h + + PASID_Reserved 31:20 Reserved RO 0h + + + DMA Requestor ID Register Bit Description r/w Value at reset + ------------------------ --- ----------- --- -------------- + + RequestorID 15:0 Custom Requestor ID RW 0h + for DMA(ReqIDValid must be 1) + + ReqIDReserved 30:16 Reserved RO 0h + + ReqIDValid 31 Requestor ID Valid bit RW 0b + 0: Requestor ID is not valid + 1: Requestor ID passed is valid +``` + + **HOW TO USE DMA** + + * Before triggering DMA all the required DMA attribute fields like DMA bus address, + dmatxnsnoop etc., fields should be correctly set in the following registers: + * DMA Control Register + * Bus Address Register + * DMA Length Register + * DMA Requestor ID Register + * DMA Transaction Offset Register + * PASID Value Register + + * If user wants to do a Pure Translated transaction for a virtual address which is in ATC's input + address range then dmaUseATCforTranslation bit must be set to true, so that input virtual address + will be mapped to translated address as per ATC fields and translated address will go on the bus for DMA. + + * User can set custom requestor ID for an DMA transaction by using Illegal Requestor ID Control Register. + Note: This is illegal as per PCIe specification as requestor ID should be assigned during PCIe enumeration. + + * If passed address for DMA transaction is already translated one, then dmaAddressType must be set to translated. + Note: This bypasses the ATC and so is also illegal as per PCIe specification as only a transaction translated + by the ATC is allowed to go out on the bus as a Translated Transaction. Also, if dmaAddressType is set to translated + and dmaUseATCforTranslation bit is set, then exerciser will fault by logging an error message while initiating DMA, + as translation mapping of an already translated address might go wrong. + + * DMAPasidEn bit must be set to 1 when a valid PASID is passed for DMA transaction. If dmaIsPrivileged + or dmaIsInstruction is 1, then DMAPasidEn bit must be set to 1 for correct DMA response. + + * Status of the last DMA request can be known from DMA Status Register. + +**MSI Generation and Service** + +``` + MSI Control Register Bit Description r/w Value at reset + -------------------- --- ----------- --- -------------- + + MSICTLID 10:0 Index of MSI to be triggered RW 0h + (0 to 2047) + + MSICTLRES 30:11 Reserved RO 0h + + MSICTLTRG 31 Set to 1 to trigger MSI. RW 0b + This bit is reset after MSI + is sent. +``` + + **HOW TO TRIGGER MSI:** + + * Poll until (MSI Control Register).MSICTLTRG == 0. + * Write MSI Control Register with MSICTLTRG == 1 and MSICTLID == desired MSI Index to initiate + MSI from exerciser device. + +**Legacy Interrupt Generation and Service** + +``` + Legacy Interrupt Control Register Bit Description r/w Value at reset + --------------------------------- ---- ----------- --- -------------- + + intxctltrg 0 Trigger legacy interrupt RW 0b + Set to 1 to trigger + interrupt Reset this bit + to clear interrupt + + intxctlres 31:1 Reserved RO 0h + +``` + + **HOW TO TRIGGER LEGACY/PIN INTERRUPT:** + + * Write 1 into intxctltrg to trigger legacy pin interrupt from exerciser device. + +**ATS Transaction** + +``` + ATS_CTL_REG Bit Description r/w Value at reset + ----------- --- ----------- --- -------------- + + ATSRequestTrigger 0 Trigger ATS Request for given WO 0b + configuration Set to 1h to + trigger DMA. This gets reset to + 0h after transaction completes. + + ATSRequestIsPrivileged 1 Access mode for ATS Request is RW 0b + privileged or not. + 0: user access + 1: Privileged access + (ATSPasidenabled bit must be 1) + + ATSRequestNoWriteRequested 2 Read permission requested for RW 0b + ATS translated address region + 0: Read & Write permission + requested + 1: Only Read permission requested + + ATSPasidEnabled 3 PASID Enable bit for ATSRequest RW 0b + 0: PASID disabled + 1: PASID enabled + + ATSExcutePermissionRequested 4 Is Execute permission Requested RW 0b + or not: + 0: execute permission not + requested + 1: execute permission Requested + (ATSPasidenabled bit must be 1) + + ATSInvalidate/ClearATC 5 Write 1 to clear ATC and W1C 0b + last ATS translated Results. + + ATSIsInFlight 6 ATS Request is in flight or not RO 0b + 0: ATS Request is not in flight + 1: ATS Request is in flight + + ATSTranslationStatus 7 ATS Translation status RO 0b + 0: Translation failed + 1: Translation Successful + + ATSTranslationCacheable 8 ATS Translation is cacheable RO 0b + or not. + 0: Not Cacheable, means + RW permission is 0 + 1: Cacheable, RW!=2’b00. + + ATCInvalidated 9 Whether ATC was invalidated RO 0b + or not. + 0: ATC not invalidated + 1: ATC was invalidated + + ATSReserved 31:10 Reserved RO 0h + + + + ATSPermissionReg Bit Description r/w Value at reset + ---------------- --- ----------- --- -------------- + + ExecutePermission 0 Whether execute permission is RO 0b + given for translated address or + not + 0: Execute permission not given + 1: Execute permission given + + WritePermission 1 Whether write permission is RO 0b + given for translated address + or not. + 0: Write permission not given + 1: Write permission given + + ReadPermission 2 Whether Read permission is RO 0b + given for translated address + or not + 0: Read permission not given + 1: Read permission given + + ExcutePrivileged 3 If 1 Execute permission was RO 0b + given when ATSRequest was + privileged. + + WritePrivileged 4 If 1 Write permission was RO 0b + given when ATSRequest was + privileged. + + ReadPrivileged 5 If 1 Read permission was given RO 0b + when ATSRequest was privileged. + + ReplyPermissionReserved 31:6 Reserved RO 0h + +``` + + **HOW TO USE ATS:** + + * Before starting an ATS request, untranslated input address for ATSRequest must be written onto + Bus Address Register, with all related bit (is_read, etc.) set in ATS_CTL_REG. + SW should enable ATS and PASID capability for exerciser (which will set exerciser_ats_enabled and + exerciser_ats_pasid_enabled, where exerciser_ats_enabled will enable client side ATS Transaction + capability for exerciser and exerciser_ats_pasid_enabled will enable PASID based ATSRequest for exerciser). + Then ATS request can be triggered. Registers to be used for ATS: + * ATS Control Register + * PASID Value Register + * ATS Reply Translated Address Register + * ATS Reply Permission Register + * ATS Translated Range Size Register + + * If there is any fault occurred during the translation, then error log will get reflected in + command line as MSG_INFO and status of ATSRequest remains as 0 (failed - by default). User can + detect whether error has occurred during the translation when following condition is true: + + * ATSRequest is not in flight (ATS_CTL_REG[6] == 0) and ATS transaction status bit (ATS_CTL_REG[7]) is 0. + + * If ATSRequest response result is cacheable (RW!= 2'b00 : Read/Write permission returned as ATSRequest result), + then ATS response cacheable bit (ATS_CTL_REG[8]) is set to true. + + * If a matching ATC invalidation comes while ATS request is in flight, translation will be retried, discarding + the current translation result. + + * If there is no error and translation response is cacheable, then translation result will be stored in ATC + along with permissions returned. + + * Translation result will also get reflected in ATS Reply Translated Address Register, + ATS Reply Permission Register, ATS Translated Range Size Register + + * DMA transaction's range must be fully covered by the ATC entry or it will fail. + + * Whenever any DMA transaction want to use ATC based translated address, it must set + dmaUseATCforTranslation flag (DMA_CTL_REG[11]) to true, so that translated address will go into the bus + after being remapped to translated output address if it has all the necessary read/write permissions. + + * Currently exerciser can store maximum 1 ATS request result, mappings are stored in ATC till invalidated by SMMU or + user writes 1 to clear ATC bit (ATS_CTL_REG[5]) which will clear/invalidate all ATC mappings and fields. + + * ATC invalidation happens in following ways when an ATCInvalidate Request received by the exerciser: + + * If ATC is empty, then there is nothing to be invalidated, ATC invalidation call gets returned immediately. + + * Some form of invalidation will be done when invalidation range contains ATC's untranslated input address + and either of the following is true: + * Invalidation range contains address for which any ongoing ATSRequest is in progress. + * SubstreamID is valid for both atc_invalidate request and atc_entry and + either ATC invalidate's substreamID matches with atc_entry's substreamID + or G-Global PASID bit is set. + + * If there is any ongoing ATS translation request with the address that is in atc_invalidation_address_range, + then current translation result will be discarded, ATC will be invalidated and then current ATS translation request + will be retried again. + + * If any DMA PCIe Translated transaction is using any address which is in atc_invalidation_address_range, + then we wait for the DMA transaction to complete and then ATC invalidation is done. + + * If no PCIe Translated transaction is using any address which is in atc_invalidation_address_range, + then we invalidate the ATC immediately if invalidation_range intersects/matches with atc_entry's untranslated + address. + + * ATC can be invalidated by writing 1 to clear ATC bit (ATS_CTL_REG[5]), which will eventually invalidate ATC and discard + all the mappings for last ATSRequest. + + * Note: the following ATS behaviours are illegal under the PCIe specification, but implemented in the model for validation purposes only: + * reading the ATC entry + * issuing a translated transaction without using the ATC + +**Transaction Monitoring** + +Transaction monitoring capabilities in the exerciser provides the ability to record the incoming transactions, for both config +and memory transactions. This includes, + +* config read and write transactions serviced in PCIe endpoints. + +* memory transactions serviced in PCIe endpoint BARs. + +* memory transactions which are forwarded from PCIe endpoint to device(like exerciser). + + +Each recorded transaction has the following information: + + * a 32 bit entry indicating transaction attributes. It has bitfields for request + type, R/W transaction, cfg/mem transaction and transaction byte size(bitfields given below) + + * a 32 bit entry indicating lower 32 bits of config or memory address + + * a 32 bit entry indicating upper 32 bits of config or memory address + + * a 32 bit entry indicating lower 32 bits of transaction data read or written to + + * a 32 bit entry indicating upper 32 bits of transaction data read or written to + +Note: Each beat in a burst transaction is recorded as a single separate transaction. +``` + ________________________________ + | TX ATTRIBUTES | + | CFG/MEM ADDRESS_LO | + | CFG/MEM ADDRESS_HI | + | DATA_LO | + | DATA_HI | + |________________________________| + + Recorded Information Format +``` + +**Registers specific to transaction monitoring** + +``` + TransactionTraceReg Bit Description R/W Value at reset + ------------------- --- ----------- --- -------------- + Transaction Info. 31:0 Returns one 32 bit entry + mentioned above on each RO 0xFFFFffff + read. + + For transaction attributes (TX_ATTRIBUTES), lower 16 bits indicates the type, R/W and cfg/mem information. + Upper 16 bits of transaction attributes indicates transaction byte size. The bit indicating the + transaction byte size is calculated as log2(bytes). For example, if size is 8, bit3 of the upper 16 bits + will be set. + ------------------------------------------------------------------------------------------------ + 31:16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 + ------------------------------------------------------------------------------------------------ + | Transaction byte size | Reserved C/M R/W Type + + Bit0: Request type(defaults to zero). + Bit1: 1 for Read and 0 for Write. + Bit2: 1 for CFG transaction and 0 for MEM transaction. + Bits 31-16: Log2 of transaction size, in bytes. + + + TransactionTraceControlReg Bit Description R/W Value at reset + -------------------------- --- ----------- --- -------------- + Start/stop monitoring 0 Control bit to start or stop + transaction monitoring. RW 0x0 +``` + + + **HOW TO USE TRANSACTION MONITORING:** + * Number of transactions recorded depends on the exerciser parameter 'max_transaction_trace_entries'. The maximum + entries supported is 32, defaults to 16, minimum 1. + * To start transaction monitoring, write 0x1 to transaction trace control register(offset 0x44). + * To stop transaction monitoring, write 0x0 to transaction trace control register(offset 0x44). + * Read the transaction trace database register(offset 0x40). Each read returns one 32 bit entry mentioned above, starting + from first transaction. A value of 0xFFFFffff indicates invalid entry. + +-------------- + +*Copyright (c) 2023, Arm Limited and Contributors. All rights reserved.* diff --git a/docs/Exerciser_porting_guide.md b/docs/PCIe_Exerciser/Exerciser_API_porting_guide.md similarity index 96% rename from docs/Exerciser_porting_guide.md rename to docs/PCIe_Exerciser/Exerciser_API_porting_guide.md index 25f54798..8670d138 100644 --- a/docs/Exerciser_porting_guide.md +++ b/docs/PCIe_Exerciser/Exerciser_API_porting_guide.md @@ -1,8 +1,8 @@ -# Exerciser Porting Guide -This document gives details of the various PCIe capabilities that exerciser device supports and how the exerciser is supposed to behave. +# Exerciser API Porting Guide +This document gives details of the various PCIe capabilities that exerciser device supports and how the exerciser is supposed to behave. It targets the Silicon partners and/or EDA vendors who are porting Exerciser API's on their platform ## Introduction to PCIe Exerciser Endpoint Device -PCIe Exerciser is a client device wrapped up by PCIe Endpoint. This device was created to meet SBSA (Server Based System Architecture) requirements for various PCIe capability validation tests. +PCIe Exerciser is a client device wrapped up by PCIe Endpoint. This device was created to generate custom stimuli to meet SBSA (Server Based System Architecture) requirements for various PCIe capability validation tests. ### Generating DMA - Before triggering DMA all the required DMA attribute fields like DMA bus address, DMA length, exerciser instance fields should be correctly set
diff --git a/docs/PCIe_Exerciser/PCIeConfigurableHierarchy.md b/docs/PCIe_Exerciser/PCIeConfigurableHierarchy.md new file mode 100644 index 00000000..ed0b4b2a --- /dev/null +++ b/docs/PCIe_Exerciser/PCIeConfigurableHierarchy.md @@ -0,0 +1,786 @@ +# PCIe Configurable Hierarchy +## User Guide + +### Introduction +Configurable PCIe hierarchy allows creation of a customized PCIe hierarchy based on a JSON formatted file. + +### How To +Add the following parameter to the command line before running the model, with the path to the PCIe hierarchy file. +``` + -C pci.hierarchy_file_name=/path/to/hierarchy.json +``` +### JSON format for hierarchy +An endpoint device is represented in the following format +``` +{ + "/": { + "parameter1": + "parameter2": + " ... ":< ... > + } +} +``` + +A root port is represented in the following format. A `__downstream__` parameter denotes a device connected downstream of the port. +``` +{ + "rootport/": { + "parameter1": + "parameter2": + " ... ":< ... > + "__downstream__":{ + "/":{ + "parameter3": + "parameter4": + " ... ":< ... > + + } + } + } +} +``` +A switch is represented in the following format. A `__downstream__` parameter denotes the device connected downstream of the switch, with the downstream port at device number `` on the switch's internal bus. +``` +{ + "switch/": { + "parameter1": + "parameter2": + " ... ":< ... > + "__downstream__": { + "/":{ + "parameter3": + "parameter4": + " ... ":< ... > + } + }, + "__downstream__"{ + "/":{ + "parameter5": + "parameter6": + " ... ":< ... > + } + } + } +} +``` + +### Device Parameters +#### PCIe Endpoints +PCIe endpoint is the common PCIe frontend for all endpoint type devices in the hierarchy, which are described in later sections. The parameters for PCIe related features of endpoints are described below. +##### Common endpoint parameters +- **device** + Device number of the endpoint. + Type: int, Min:0, Max:31 +- **function** + Function number for the endpoint. + Type: int, Min: 0, Max: 7 +- **vendor_id** + PCI Vendor ID. + Type: int, Min:0, Max:0xFFFE +- **device_id** + PCI Device ID. + Type: int, Min:0, Max:0xFFFE +- **base_class** + PCI Base Class. + Type: int, Min:0, Max:255 +- **sub_class** + PCI Sub Class. + Type: int, Min:0, Max:255 +- **bar0_64bit** + If BAR 0 is 64 bits wide, if region size is non zero. + Type: bool +- **bar1_64bit** + If BAR 1 is 64 bits wide, if region size is non zero. + Type: bool +- **bar2_64bit** + If BAR 2 is 64 bits wide, if region size is non zero. + Type: bool +- **bar3_64bit** + If BAR 3 is 64 bits wide, if region size is non zero. + Type: bool +- **bar4_64bit** + If BAR 4 is 64 bits wide, if region size is non zero. + Type: bool +- **bar0_log2_size** + Log2 of the size of the region pointed to by BAR 0, zero is reserved which means bar is not used*. + Type: int, Min:0, Max:63 +- **bar1_log2_size** + Log2 of the size of the region pointed to by BAR 1, zero is reserved which means bar is not used. + Type: int, Min:0, Max:63 +- **bar2_log2_size** + Log2 of the size of the region pointed to by BAR 2, zero is reserved which means bar is not used. + Type: int, Min:0, Max:63 +- **bar3_log2_size** + Log2 of the size of the region pointed to by BAR 3, zero is reserved which means bar is not used. + Type: int, Min:0, Max:63 +- **bar4_log2_size** + Log2 of the size of the region pointed to by BAR 4, zero is reserved which means bar is not used. + Type: int, Min:0, Max:63 +- **bar5_log2_size** + Log2 of the size of the region pointed to by BAR 5, zero is reserved which means bar is not used. + Type: int, Min:0, Max:63 +- **uses_interrupt** + Enable support for legacy interrupts. + Type: bool +- **interrupt_pin_index** + Interrupt pin used by this function 1:INTA, 2:INTB, 3:INTC, 4:INTD. + Type: int, Min:1, Max:4 +- **multi_function** + Set to true if this endpoint is part of a multi function device. + Type: bool +- **pcie_version** + PCIe version , bits[3:0] in capabilities register. 1 is PCIe3.0. 2 is PCIe4.0. + Type: int, Min:1, Max:15 +- **prog_iface** + PCI Programming Interface. + Type: int, Min:0, Max:15 +- **rev_id** + PCI Revision ID. + Type: int, Min:0, Max:15 +- **subsys_id** + PCI Subsystem ID. + Type: int, Min:0, Max:0xFFFE +- **subsys_vendor_id** + PCI Subsystem Vendor ID. + Type: int, Min:0, Max:0xFFFE +- **express_capability_device_type** + PCI Express Capabilities Device Type, bits[7:4] in capabilities register. 0 is PCIe EndPoint, 9 is RCiEP. + Type: int, Min:0, Max:15 +- **msix_support** + Enable device support for MSI-X. + Type: bool +- **msix_pba_bar** + BAR used by MSI-X pending bit array. + Type: int, Min:0, Max:5 +- **msix_table_bar** + BAR used by MSI-X table. + Type: int, Min:0, Max:5 +- **msix_table_size** + Size of tables for MSI-X. + Type: int, Min:0, Max:2048 +- **power_mgmt_capability** + Device supports Power-Management capabilities. + Type: bool +- **pasid_supported** + If set then the PCIe device can emit PASID (SubstreamIDs). + Type: bool +- **ext_fmt_field_supported** + Enable extended format field support. + Type: bool +- **extended_tag_supported** + Extended Tag field support. + Type: bool +- **link_port_number** + Port number for PCIe link. + Type: int, Min:0, Max:255 +- **pri_supported** + If set then the PCIe function supports Page Request Interface(requires ATS). + Type: bool +- **rber_supported** + Enable role-based error reporting. + Type: bool +- **slot_and_root_status_msi_idx** + MSI index for reporting slot and root status changes. + Type: int, Min:0, Max:2047 +- **tag_10bit_completer_supported** + Enable 10 bit tag completer support. + Type: bool +- **tag_10bit_requester_supported** + Enable 10 bit tag requester support. + Type: bool +- **aspm_optionality_compliant** + Enable ASPM optionality compliance. + Type: bool +- **ats_supported** + If set then the PCIe function supports Address Translation Services(ATS). + Type: bool +- **error_injection_supported** + If set then the PCIe function supports DVSEC Error Injection Capability. + Type: bool +- **aer_supported** + If set then the PCIe function supports Advanced Error Reporting Capability(AER). + Type: bool + +#### Exerciser +Exerciser is a PCIe device used to generate various stimuli that can help test PCIe/SMMU integration in the system. +##### Endpoint parameters default values +- **device**: 0 +- **function**: 0 +- **vendor_id**: 0x13B5 +- **device_id**: 0xED01 +- **base_class**: 0xED +- **sub_class**: 0 +- **bar0_64bit**: false +- **bar1_64bit**: false +- **bar2_64bit**: false +- **bar3_64bit**: false +- **bar4_64bit**: false +- **bar0_log2_size**: 12 +- **bar1_log2_size**: 14 +- **bar2_log2_size**: 15 +- **bar3_log2_size**: 0 +- **bar4_log2_size**: 0 +- **bar5_log2_size**: 12 +- **uses_interrupt**: true +- **interrupt_pin_index**: 1 +- **multi_function**: false +- **pcie_version**: 2 +- **prog_iface**: 0 +- **rev_id**: 0 +- **subsys_id**: 0 +- **subsys_vendor_id**: 0x13B5 +- **express_capability_device_type**: 0 +- **msix_support**: true +- **msix_pba_bar**: 4 +- **msix_table_bar**: 2 +- **msix_table_size**: 2048 +- **power_mgmt_capability**: true +- **pasid_supported**: false +- **ext_fmt_field_supported**: true +- **extended_tag_supported**: true +- **link_port_number**: 0 +- **pri_supported**: true +- **rber_supported**: true +- **slot_and_root_status_msi_idx**: 0 +- **tag_10bit_completer_supported**: true +- **tag_10bit_requester_supported**: true +- **aspm_optionality_compliant**: true +- **ats_supported**: true +- **error_injection_supported**: false +- **aer_supported**: false +##### Exerciser parameters and default values +- **memory_bar** + BAR index to be used for exerciser memory access. + Type: int, Min:0, Max: 5 + Default: 1 +##### Example +``` + "exerciser/exerciser0": { + "device": 1, + "function": 0, + "interrupt_pin_index": 1, + "memory_bar": 3, + "express_capability_device_type": 0 + } +``` + +#### AHCI Controller +Advanced Host Controller Interface, or AHCI, is a technical standard for an interface that enables software to communicate with Serial ATA (SATA) devices. +##### Endpoint parameters default values +- **device**: 0 +- **function**: 0 +- **vendor_id**: 0xABC* +- **device_id**: 0xACED* +- **base_class**: 0x1* (Device Class for Mass Storage) +- **sub_class**: 0x6* (Sub Class for SATA) +- **bar0_64bit**: false +- **bar1_64bit**: false +- **bar2_64bit**: false +- **bar3_64bit**: false +- **bar4_64bit**: false +- **bar0_log2_size**: 13 +- **bar1_log2_size**: 13 +- **bar2_log2_size**: 12 +- **bar3_log2_size**: 13 +- **bar4_log2_size**: 12 +- **bar5_log2_size**: 13 +- **uses_interrupt**: false* +- **interrupt_pin_index**: 1 +- **multi_function**: false +- **pcie_version**: 2 +- **prog_iface**: 0x1* +- **rev_id**: 0x1* +- **subsys_id**: 0x2* +- **subsys_vendor_id**: 0x13B5* +- **express_capability_device_type**: 0 +- **msix_support**: true* +- **msix_pba_bar**: 4* +- **msix_table_bar**: 2* +- **msix_table_size**: 1* +- **power_mgmt_capability**: true +- **pasid_supported**: false +- **ext_fmt_field_supported**: true +- **extended_tag_supported**: true +- **link_port_number**: 0 +- **pri_supported**: false +- **rber_supported**: true +- **slot_and_root_status_msi_idx**: 0 +- **tag_10bit_completer_supported**: true +- **tag_10bit_requester_supported**: true +- **aspm_optionality_compliant**: true +- **ats_supported**: false +- **error_injection_supported**: false +- **aer_supported**: false +##### AHCI Parameters and Default values +- **force_mode** + Force disk to report support for at most PIO/DMA/NCQ mode (only for testing/bring-up purposes). PIO mode is always supported. Use NCQ for maximum performance. + Default: "NCQ" +- **image_path** + Comma-separated list of zero or more disk images (up to 32). Each image represents one SATA disk which is connected to one port of the AHCI controller. Empty list elements are allowed and result in a SATA port which has no disk attached. An empty string (default) means one SATA port with no disk attached. + Default: "" +- **run_async** + Do host I/O in a background thread asynchronously. Enabling this makes the simulation non-deterministic and may or may not improve performance. + Default: false + +\* *Parameter values that are fixed and cannot be overridden.* +##### Example +``` +"ahci/ahci0": { + "device": 10, + "function": 0, + "image_path": "/path/to/image", + "express_capability_device_type": 9 + } +``` + +#### Host Bridge +On real systems, Host Bridge is what connects the tree of PCI buses (which are +internally connected with PCI-to-PCI Bridges) to the rest of the +system. On FVPs, this is merely a placeholder representation of the concept, and does not provide any functionality. +##### Endpoint parameters default values +- **device**: 0 +- **function**: 0 +- **vendor_id**: 0x13B5 +- **device_id**: 0 +- **base_class**: 0x6* +- **sub_class**: 0 +- **bar0_64bit**: false +- **bar1_64bit**: false +- **bar2_64bit**: false +- **bar3_64bit**: false +- **bar4_64bit**: false +- **bar0_log2_size**: 12 +- **bar1_log2_size**: 0 +- **bar2_log2_size**: 0 +- **bar3_log2_size**: 0 +- **bar4_log2_size**: 0 +- **bar5_log2_size**: 0 +- **uses_interrupt**: false +- **interrupt_pin_index**: 1 +- **multi_function**: false +- **pcie_version**: 2 +- **prog_iface**: 0xf +- **rev_id**: 0xf +- **subsys_id**: 0xf +- **subsys_vendor_id**: 0x13B5 +- **express_capability_device_type**: 9 +- **msix_support**: false +- **msix_pba_bar**: 6 +- **msix_table_bar**: 6 +- **msix_table_size**: 1 +- **power_mgmt_capability**: true +- **pasid_supported**: false +- **ext_fmt_field_supported**: true +- **extended_tag_supported**: true +- **link_port_number**: 0 +- **pri_supported**: true +- **rber_supported**: true +- **slot_and_root_status_msi_idx**: 0 +- **tag_10bit_completer_supported**: true +- **tag_10bit_requester_supported**: true +- **aspm_optionality_compliant**: true + +##### Example +``` +"hostbridge/hb": { + "device": 0, + "function": 0, + } +``` + +#### SMMU Test Engine +SMMUv3TestEngine is a PCIe device used to generate various stimuli that can help test SMMU integration in the system. +##### Endpoint parameters default values +- **device**: 0 +- **function**: 0 +- **vendor_id**: 0x13B5* +- **device_id**: FF80 +- **base_class**: 0xFF* +- **sub_class**: 0 +- **bar0_64bit**: true +- **bar1_64bit**: false +- **bar2_64bit**: true +- **bar3_64bit**: false +- **bar4_64bit**: true +- **bar0_log2_size**: 18 +- **bar1_log2_size**: 0 +- **bar2_log2_size**: 15* +- **bar3_log2_size**: 0 +- **bar4_log2_size**: 12* +- **bar5_log2_size**: 0 +- **uses_interrupt**: false +- **multi_function**: false +- **pcie_version**: 2 +- **prog_iface**: 0x0 +- **rev_id**: 0xf +- **subsys_id**: 0x0 +- **subsys_vendor_id**: 0x13B5* +- **express_capability_device_type**: 9 +- **msix_support**: true +- **msix_pba_bar**: 4 +- **msix_table_bar**: 2 +- **msix_table_size**: 2048 +- **power_mgmt_capability**: true +- **pasid_supported**: false +- **ext_fmt_field_supported**: true +- **extended_tag_supported**: true +- **link_port_number**: 0 +- **pri_supported**: true +- **rber_supported**: true +- **slot_and_root_status_msi_idx**: 0 +- **tag_10bit_completer_supported**: true +- **tag_10bit_requester_supported**: true +- **aspm_optionality_compliant**: true +- **error_injection_supported**: false +- **aer_supported**: false + +##### Example +``` +"smmuv3testengine/smmute": { + "device": 0, + "function": 0, + } +``` + +#### RootComplex Event Collector (RCEC) +RCEC is a PCIe Type 0 device which signals Error/PME messages received from associated RCiEPs to the upstream PVBus2PCI/RootComplex. +## Endpoint parameters default values +- **device**: 0 +- **function**: 0* +- **vendor_id**: 0x13B5* +- **device_id**: 47B1* +- **base_class**: 0x8* +- **sub_class**: 0x7* +- **bar0_log2_size**: 12 +- **uses_interrupt**: false +- **pcie_version**: 2 +- **prog_iface**: 0x0 +- **rev_id**: 0x0 +- **subsys_id**: 0x0* +- **subsys_vendor_id**: 0x13B5 +- **express_capability_device_type**: 9* +- **msix_support**: false +- **msix_pba_bar**: 6 +- **msix_table_bar**: 6 +- **msix_table_size**: 2048 +- **power_mgmt_capability**: true +- **pasid_supported**: false +- **ext_fmt_field_supported**: true +- **extended_tag_supported**: true +- **link_port_number**: 0 +- **rber_supported**: true +- **slot_and_root_status_msi_idx**: 0 +- **tag_10bit_completer_supported**: true +- **tag_10bit_requester_supported**: true +- **rcec_associated_device_function_info** : "" +- Above parameter takes a string of device-function numbers of RCiEPs associated with this RCEC. + Note that an empty string("") represent that no RCiEP is associated with the RCEC. + As an example, the user should set this string parameter as per the instructions below: +``` + params.set("rcec_associated_device_function_info", "8 16 24"); +``` + String information is interpreted as stream of associated device-function numbers each separated by a space. + Note that for current PCIe RCEC-RCiEP interface implementation, the model doesn't include support for RCEC & RCiEP + interfacing between different root bus devices. So all the RCiEPs and their RCECs are connected to the same + RootComplex and hence have the same root bus(ecam_start_bus_number). + Each device_function number is an 8 bit number with bits [7:3] as device number and bit [2:0] as function number. + The input string must contain device-function number only in decimal format. +- **error_injection_supported**: false +- **aer_supported**: false + +##### Example +``` +"rcec/rcec0": { + "device": 0, + "function": 0, + "msix_support" : true, + "rciep_associated_device_function_info": "8 16" + } +``` + +#### Root Port +##### Parameters and Default values +- **device_number** + Device number on this bus. + Type: int, Min:0, Max:31 + Default: 0 +- **device_id** + PCI Device ID. + Type: int, Min:0, Max:0xFFFE + Default: 0xdef +- **vendor_id** + PCI Vendor ID. + Type: int, Min:0, Max:0xFFFE + Default: 0x13B5 +- **acs_supported** + Access control services supported. + Type: bool + Default: false +- **dpc_supported** + Downstream Port Containment supported. + Type: bool + Default: false +- **aspm_optionality_compliant** + Enable ASPM optionality compliance. + Type: bool + Default: true +- **ext_fmt_field_supported** + Enable extended format field support. + Type: bool + Default: true +- **extended_tag_supported** + Extended Tag field support. + Type: bool + Default: true +- **link_port_number** + Port number for PCIe link. + Type: int, Min:0, Max:255 + Default: 0 +- **pcie_version** + PCIe version , bits[3:0] in capabilities register. 1 is PCIe3.0. 2 is PCIe4.0. + Type: int, Min:1, Max:15 + Default: 2 +- **rber_supported** + Enable role-based error reporting. + Type: bool + Default: true +- **slot_and_root_status_msi_idx** + MSI index for reporting slot and root status changes. + Type: int, Min:0, Max:2047 + Default: 0 +- **tag_10bit_completer_supported** + Enable 10-bit tag completer support. + Type: bool + Default: true +- **tag_10bit_requester_supported** + Enable 10-bit tag requester support. + Type: bool + Default: true +- **error_injection_supported** + If set then the PCIe function supports DVSEC Error Injection Capability. + Type: bool + Default: false +- **aer_supported** + If set then the PCIe function supports Advanced Error Reporting Capability(AER). + Type: bool + Default: false +- **report_advisory_non_fatal_errors** + If set, then the PCIe function supports Advisory non-fatal error reporting. + Type: bool + Defaut: false +##### Example +``` +"rootport/rootport0": { + "device_number": 5, + "__downstream__": { + "ahci/ahci1": { + "device": 0, + "function": 0, + "image_path": "/path/to/image" + } + } + } +``` +#### Switch +##### Parameters and Default values +- **device_number** + Device number on this bus. + Type: int, Min:0, Max:31 + Default: 0 +- **acs_supported** + Access control services supported. + Type: bool + Default: false +- **report_advisory_non_fatal_errors** + If set, then the PCIe function supports Advisory non-fatal error reporting. + Type: bool + Defaut: false +##### Example +``` +"switch/switch0": { + "device_number": 0, + "__downstream__10": { + "ahci/ahci4": {...} + }, + "__downstream__20": { + "ahci/ahci2": {...} + } + } +``` +#### Root Bridge +The term root bridge is taken from the PCI Firmware specification Rev 3.x. It refers to an abstraction for the platform's PCI config and memory I/O access mechanism. A root bridge can be used to create multiple ECAM and MMIO regions within the system's ECAM and MMIO limits. +##### Parameters and Default values +- **ecam_start** + ECAM base address. + Type:int, Min:``, Max:`` +- **ecam_end_incl** + ECAM end address. + Type:int, Min:``, Max:`` +- **mem32_start** + 32-bit memory window base address. + Type:int, Min:``, Max:`` +- **mem32_end_incl** + 32-bit memory window end address. + Type:int, Min:``, Max:`` +- **mem64_start** + 64-bit memory window base address. + Type:int, Min:``, Max:`` +- **mem64_end_incl** + 64-bit memory window base address. + Type:int, Min:``, Max:`` +- **ecam_start_bus_number** + ECAM start bus number. + Type:int, Min:``, Max:`` + Note: Each bus consumes 1MB of memory, so each ecam should have enough space (ecam_start - ecam_end_incl) to accomodate the total number of buses (start_bus - end_bus) allocated to it. + +##### Example +``` +"rootbridge/rb0": { + "ecam_start": 0x60000000, + "ecam_end_incl": 0x67ffffff, + "mem32_start": 0x70000000, + "mem32_end_incl": 0x73ffffff, + "mem64_start": 0x500000000, + "mem64_end_incl": 0x67fffffff, + "ecam_start_bus_number": 0x0, + + "__downstream__": { + "ahci/ahci0": { + "device": 30, + "function": 0, + "image_path": "", + "express_capability_device_type": 9 + }, + "ahci/ahci1": { + "device": 31, + "function": 0, + "express_capability_device_type": 9, + "image_path": "" + } + } + } +``` + +### Sample hierarchies + +#### Example 1: Single ECAM + +An absence of rootbridge before the beginning of the device hierarchy implies a single ECAM configuration. +``` +{ + "ahci/ahci0": { + "device": 30, + "function": 0, + "image_path": "", + "express_capability_device_type": 9 + }, + "smmuv3testengine/smmuv3testengine3": { + "device": 15, + "function": 0, + "express_capability_device_type": 9 + }, + "rootport/rootport0": { + "device_number": 2, + "__downstream__": { + "ahci/ahci1": { + "device": 0, + "function": 0, + "image_path": "", + "multi_function": true, + }, + "ahci/ahci2": { + "device": 0, + "function": 1, + "image_path": "" + } + } + }, + "rootport/rootport1": { + "device_number": 5, + "__downstream__": { + "ahci/ahci3": { + "device": 0, + "function": 0, + "image_path": "" + } + } + }, + "rootport/rootport2": { + "device_number": 6, + "__downstream__": { + "switch/switch0": { + "device_number": 0, + "__downstream__10": { + "ahci/ahci4": {} + }, + "__downstream__20": { + "ahci/ahci5": {} + } + } + } + } +} +``` + +#### Example 2: 2 ECAMs +``` +{ + "rootbridge/rb0": { + "ecam_start": 0x60000000, + "ecam_end_incl": 0x67ffffff, + "mem32_start": 0x70000000, + "mem32_end_incl": 0x73ffffff, + "mem64_start": 0x500000000, + "mem64_end_incl": 0x67fffffff, + "ecam_start_bus_number": 0x0, + + "__downstream__": { + "ahci/ahci0": { + "device": 30, + "function": 0, + "image_path": "", + "express_capability_device_type": 9 + }, + "ahci/ahci1": { + "device": 31, + "function": 0, + "express_capability_device_type": 9, + "image_path": "" + }, + "rcec/rcec0": { + "device": 1, + "msix_support": true, + "bar0_log2_size": 16, + "bar2_log2_size": 16, + "bar4_log2_size": 12, + "msix_table_bar": 2, + "msix_pba_bar": 4, + "msix_table_size": 2048, + "rciep_associated_device_function_info": "240 248" + } + } + }, + "rootbridge/rb1": { + "ecam_start": 0x68000000, + "ecam_end_incl": 0x6fffffff, + "mem32_start": 0x74000000, + "mem32_end_incl": 0x77ffffff, + "mem64_start": 0x680000000, + "mem64_end_incl": 0x7ffffffff, + "ecam_start_bus_number": 0x1, + + "__downstream__": { + "ahci/ahci0": { + "device": 2, + "function": 0, + "image_path": "/path/to/image", + "express_capability_device_type": 9 + } + } + } +} +``` + +-------------- + +*Copyright (c) 2023, Arm Limited and Contributors. All rights reserved.* diff --git a/docs/PCIe_Exerciser/README.md b/docs/PCIe_Exerciser/README.md new file mode 100644 index 00000000..b2c46635 --- /dev/null +++ b/docs/PCIe_Exerciser/README.md @@ -0,0 +1,76 @@ + +# Running SBSA ACS tests on FVP + +The SBSA ACS tests can be executed on the Reference Design (RDN2) Fixed Virtual Platform (FVP). The FVP also offers enhanced support for running the exerciser tests, thereby expanding test coverage on the platform. The SBSA Exerciser reference implementation is specifically designed for the RDN2 Fixed Virtual Platform. + + +## Required modifications on RDN2 reference platform to run SBSA ACS tests + +This section provides information from where to download the pre-built model for the RDN2 FVP and build the software stack for the same using the source code. It also provides information on how to enable the exerciser so all the SBSA tests, including the exerciser, can be run. + +### Build Steps + +To build the RDN2 software stack and download the pre-built model for the FVP, follow the below steps. + +1. Create RDN2 directory and cd into it + ``` + cd rdn2_path + ``` + +2. Download the required sources for setting up the RDN2 software stack by following the steps outlined in the [Setup Workspace](https://neoverse-reference-design.docs.arm.com/en/latest/platforms/common/setup-workspace.html#setup-workspace-label) documentation. + +3. Download the latest version of RDN2 FVP from [Arm Ecosystem FVPs](https://developer.arm.com/downloads/-/arm-ecosystem-fvps) and set the MODEL PATH + ``` + export MODEL=model_path_to_Linux64_GCC-9.3/FVP_RD_N2 + ``` + +4. Modify the following in build-scripts/build-tf-a.sh + ```diff + - make -j $PARALLELISM PLAT=$plat ARCH=$TF_A_ARCH DEBUG=$TF_A_DEBUG_ENABLED ${tfa_build_flags} ${targets} + + make -j $PARALLELISM PLAT=$plat ARCH=$TF_A_ARCH ENABLE_MPAM_FOR_LOWER_ELS=1 RAS_TRAP_LOWER_EL_ERR_ACCESS=0 DEBUG=$TF_A_DEBUG_ENABLED ${tfa_build_flags} ${targets} + ``` + +5. Modify the following in uefi/edk2/ArmPkg/Drivers/ArmGic/GicV3/ArmGicV3Dxe.c + ```diff + - mGicNumInterrupts = ArmGicGetMaxNumInterrupts (mGicDistributorBase); + + mGicNumInterrupts = 16384; + ``` + +6. Build the platform using the command + ``` + ./build-scripts/rdinfra/build-test-acs.sh -p rdn2 all + ``` + + +**Note**: The platform name for the build will be **RD-N2**. For more details on software stack and obtaining the latest version of RD-N2 FVP, see [RDN2](https://gitlab.arm.com/infra-solutions/reference-design/docs/infra-refdesign-docs/-/tree/main/platforms/rdn2) + + +### Modifications to enable the Exerciser in PCIe hierarchy. + +This sections provides information on the modifications required to integrate and enable the Exerciser in the PCIe hierarchy as a programmable endpoint. + +A reference of the example_pcie_hierarchy_0.json is present [here](example_pcie_hierarchy_0.json) + +To run the SBSA exerciser tests make the following change in ${rdn2_path}/model-scripts/rdinfra/platforms/rdn2/run_model.sh + +```diff +- -C pcie_group_0.pciex16.hierarchy_file_name= \ +- -C pcie_group_1.pciex16.hierarchy_file_name="example_pcie_hierarchy_2.json" \ +- -C pcie_group_2.pciex16.hierarchy_file_name="example_pcie_hierarchy_3.json" \ +- -C pcie_group_3.pciex16.hierarchy_file_name="example_pcie_hierarchy_4.json" \ ++ -C pcie_group_0.pciex16.hierarchy_file_name="example_pcie_hierarchy_0.json" \ +``` + +To run Exerciser test 905 - PE 2/4/8B writes to PCIe as 2/4/8B, please make the following change to all exerciser enteries in the example_pcie_hierarchy_0.json file located at ${rdn2_path}/model-scripts/rdinfra/platforms/rdn2/ + +```diff +- "max_transaction_trace_entries": 8, ++ "max_transaction_trace_entries": 1, +``` + +## Useful links + +- For information on configuring the [PCIe Hierarchy](PCIeConfigurableHierarchy.md) +- For more information on PCIe capabilities that exerciser supports, refer [Exerciser.md](Exerciser.md) + +*Copyright (c) 2023 Arm Limited and Contributors. All rights reserved.* diff --git a/docs/PCIe_Exerciser/example_pcie_hierarchy_0.json b/docs/PCIe_Exerciser/example_pcie_hierarchy_0.json new file mode 100644 index 00000000..2617b1bf --- /dev/null +++ b/docs/PCIe_Exerciser/example_pcie_hierarchy_0.json @@ -0,0 +1,157 @@ +{ + "hostbridge/hostbridge0": { + }, + "ahci/ahci0": { + "device": 30, + "function": 0, + "image_path": "", + "express_capability_device_type": 9, + "ats_supported": true, + "pasid_supported": true, + "aer_supported": true + }, + "exerciser/exerciser0": { + "device": 31, + "function": 0, + "interrupt_pin_index": 4, + "express_capability_device_type": 9, + "uses_client_based_atc": true, + "ats_supported": true, + "max_transaction_trace_entries": 8, + "pasid_supported": true, + "error_injection_supported": true, + "aer_supported": true + }, + "exerciser/exerciser1": { + "device": 15, + "function": 0, + "interrupt_pin_index": 4, + "express_capability_device_type": 9, + "uses_client_based_atc": true, + "ats_supported": true, + "max_transaction_trace_entries":8, + "pasid_supported": true, + "error_injection_supported": true, + "aer_supported": true + + }, + "rootport/rootport0": { + "device_number": 1, + "dpc_supported": true, + "acs_supported": true, + "aer_supported": true, + "__downstream__": { + "ahci/ahci1": { + "device": 0, + "function": 0, + "image_path": "" + } + } + }, + "rootport/rootport1": { + "device_number": 2, + "dpc_supported": true, + "aer_supported": true, + "acs_supported": true, + "error_injection_supported": true, + "msix_support": true, + "__downstream__": { + "exerciser/exerciser2": { + "interrupt_pin_index": 4, + "device": 0, + "function": 0, + "multi_function": true, + "msix_support": true, + "uses_client_based_atc": true, + "ats_supported": true, + "error_injection_supported": true, + "max_transaction_trace_entries": 8, + "pasid_supported": true, + "aer_supported": true + }, + "exerciser/exerciser3": { + "device": 0, + "function": 1, + "interrupt_pin_index": 4, + "msix_support": true, + "uses_client_based_atc": true, + "ats_supported": true, + "error_injection_supported": true, + "max_transaction_trace_entries": 8, + "pasid_supported": true, + "aer_supported": true + } + } + }, + "rootport/rootport2": { + "device_number": 3, + "acs_supported": true, + "dpc_supported": true, + "aer_supported": true, + "error_injection_supported": true, + "msix_support": true, + "__downstream__": { + "switch/switch0": { + "device_number": 0, + "__downstream__0": { + "ahci/ahci2": {} + }, + "__downstream__1": { + "exerciser/exerciser4": { + "device": 0, + "function": 0, + "multi_function": true , + "interrupt_pin_index": 4, + "uses_client_based_atc": true, + "msix_support": true, + "ats_supported": true, + "error_injection_supported": true, + "max_transaction_trace_entries": 8, + "pasid_supported": true, + "aer_supported": true + }, + "exerciser/exerciser5": { + "device": 0, + "function": 7, + "interrupt_pin_index": 4, + "error_injection_supported": true, + "msix_support": true, + "uses_client_based_atc": true, + "ats_supported": true, + "max_transaction_trace_entries": 8, + "pasid_supported": true, + "aer_supported": true + } + }, + "__downstream__2": { + "smmuv3testengine/smmuv3testengine1": { + "device": 0, + "function": 0, + "multi_function": true + }, + "smmuv3testengine/smmuv3testengine2": { + "device": 0, + "function": 7 + } + } + } + } + }, + "rootport/rootport3": { + "device_number": 4, + "acs_supported": true, + "dpc_supported": true, + "aer_supported": true, + "__downstream__": { + "smmuv3testengine/smmuv3testengine3": { + "device": 0, + "function": 0, + "multi_function": true + }, + "smmuv3testengine/smmuv3testengine4": { + "device": 0, + "function": 1 + } + } + } +} diff --git a/platform/pal_baremetal/README.md b/platform/pal_baremetal/README.md index e4638cca..1d7ab89b 100644 --- a/platform/pal_baremetal/README.md +++ b/platform/pal_baremetal/README.md @@ -20,7 +20,7 @@ Reference Makefile is present at [Makefile](../../Makefile). To compile SBSA, pe The output library files will be generated at /build/lib/ -Note: Any platform specific changes can be done by using TARGET_EMULATION macro defintion +Note: Any platform specific changes can be done by using TARGET_EMULATION macro defintion. The pal_baremetal reference code is located in [pal_baremetal](.). To customize the bare-metal code for different platforms, create a directory in [pal_baremetal](.) folder and copy the reference code from [source](src) folder and [RDN2](FVP/RDN2) folder to . For more details on how to port the reference code to a specific platform and for further customisation please refer to the [User Guide](../../docs/Arm_SBSA_ACS_Bare-metal_User_Guide.pdf)