Hardware capability enhancements

[hungry-foolish]

Current hardware capability in the kernel is not sufficient. There are many hardware operations, which can be divided into the categories listed below, and in the long run we may need support for all these:
WARNING:VERY LONG LIST

Page Table Related Calls

• Flush the CPU-local Cache of a Page Table
  This operation should be implemented as a total flush of all caches of the designated page table on the current CPU. If the processor supports ASID, then all the TLB entries that corresponds to the page table’s ASID must be invalidated (including the locked-down entries); if the processor does not support ASID, and the current page table is the designated one, a full TLB flush is necessary; if the processor does not support ASID and the page table is not the current one, no operation is needed and a success can be returned directly.
• Invalidate One Line of the CPU-local Cache of a Page Table
  This operation should be implemented as an invalidation of one cache line of the designated page table on the current CPU. If the processor supports ASID, then one of the TLB entries that corresponds to the page table’s ASID must be invalidated (including the locked-down entries); if the processor does not support ASID, and the current page table is the designated one, one of the TLB entries should be invalidated; if the processor does not support ASID and the page table is not the current one, no operation is needed and a success can be returned directly.
• Set the ASID of a Page Table
  This operation should set a particular page table’s ASID (or we can designate it when we create the top-level page directory). The storage of ASID is decided by the designer of the system. What if we run out of ASID?
• Lockdown a Page into the TLB
  This operation should be implemented as a TLB lockdown of a particular entry.
• Get or Set the Page Attributes of a Certain Page
  This operation should be implemented as a lookup or modification to the properties of a certain page. The most common property is the “dirty” (a.k.a. “modified” & “accessed”) flag, which can be used to determine whether the page is written to; some processors may have other page properties that can be consulted or modified.

Interrupt Controller Related Calls

• Get or Set the State of a Local Interrupt Source
  This operation should be implemented as a read or write to the status of one local interrupt source on the current CPU. Common local interrupt statuses include their activity, preemptive and non-preemptive priority, and the kernel endpoint that they correspond to. Usually this will modify the settings of the CPU’s local interrupt controller (e.g. LAPIC of x86-64).
• Get or Set the State of a Global Interrupt Source
  This operation should be implemented as a read or write to the status of one global interrupt source. Common global interrupt statuses include their activity, the CPU to route them to, and the CPU’s local interrupt source to match them with. Usually this will modify the settings of the system’s global interrupt controller (e.g. IOAPIC of x86-64).
• Trigger the Local Interrupt Source of a Certain CPU
  This operation should be implemented as a triggering operation on a local interrupt source of a CPU. Regular applications include triggering local software interrupt and sending Inter-Processor Interrupts (IPIs). This will usually trigger such interrupts through the local interrupt controller of the CPU (e.g. LAPIC of x86-64), sometimes it can also trigger such interrupts through global interrupt controllers (e.g. IOAPIC of x86-64). The functionality of this operation may include triggering one interrupt source of one CPU, triggering multiple interrupts of one CPU, triggering one interrupt of multiple CPUs or triggering multiple interrupts of multiple CPUs; the function combination implemented is decided by the implementer.

Cache and Prefetching Related Calls

• Enable Cache
  This operation should enable a certain cache on a certain CPU. Usually this just involves a single register write. The cache being configured can be Level 1 (L1), Level 2 (L2), etc. Cache can further be divided into instruction cache and data cache.
• Disable Cache
  This operation should disable a certain cache on a certain CPU. It is the implementer’s decision to make whether the modified cache lines are flushed to memory before a certain cache is turned off.
  Should we flush the cache right before disabling it?
• Configure Cache
  Some processors support dynamic configuration of cache organization. This may involve allowing using a portion of cache as TCM, allowing cache organization reconfiguration, allowing configuring cache as instruction or data dedicated, allowing cache incoherency between CPUs, or allowing reserving cache lines for a certain CPU.
• Invalidate Cache
  In many scenarios we need to invalidate a specific cache line or a portion of a cache line. This is for flushing the data in cache into memory so that some peripherals may use them. Typical such peripherals include DMAs, networking cards and video cards.
• Lockdown Cache
  In many scenarios we wish some data can retain in the cache and never be evicted, and we need cache lockdown to implement this. There are many implementation variants of cache lockdown; a single line, a single set, a single way, or some area.
• Enable Prefetcher
  This operation enables the prefetcher or the CPU. Usually this only involves a prefetch engine register configuration.
• Disable Prefetcher
  This operation disables the prefetcher of the CPU. Usually this only involves a prefetch engine register configuration.

Hardware Hotplug Related Calls

• Get or Set the State of a Physical Processor Package
  This operation reads or writes the status of a physical processor package. “Physical processor” refers to a processor package, or a circuit board that contains multiple processor packages. The common statuses that needs to be get or set include power on/off, contact detection, etc.
• Get or Set the State of a Logical Processor Package
  This operation reads or writes the status of a logical processor. “Logical processor” refers to cores that software can independently identify, and each core may run only one thread at a time. It is worth noting that a logical processor does not necessarily corresponds to a single physical core or a package. Multi-core processors may integrate multiple physical processors in the same package, while multithreading may emulate multiple logical processors with one physical processor. Common statuses that needs to be get or set includes processor initialization and shutdown, etc.
• Get or Set the State of a Physical Memory Package
  This operation reads or writes the status of a physical memory package. “Physical memory package” refers to a single memory device, or a memory module that includes multiple physical memory device. The common statuses that needs to be get or set include power on/off, contact detection, etc.

Clock and Power Related Calls

• Get or Set Voltage Parameters
  This operation reads or writes the supply voltage of CPU, memory and other peripherals. The voltage unit can be milliVolts (mV) or microVolts (uV).
• Get or Set Frequency Parameters
  This operation reads or writes the operating frequency of CPU, memory and other peripherals. The frequency can be milliHertz (mHz), Hertz (Hz), kiloHertz (kHz), MegaHertz (MHz) or GigaHertz (GHz).
• Get or Set Power States
  This operation reads or writes the power state of CPU, memory and other peripherals. Power state usually involves the switching of power domains and clock domains. On processors that supports Dynamic Voltage Frequency Scaling (DVFS), this operation is also responsible for setting the status of the functionality.
• Get or Set Safety Protection States
  This operation reads or writes the safety lock of voltage and frequency. When the safety lock is effective, some voltage and frequency settings are not allowed, because this may cause a system crash, or even cause permanent damage to the system (especially voltage adjustments). When the safety lock is overridden, these restrictions will be removed and more extreme system settings are allowed.

System Monitoring Related Calls

• Get CPU Functionality Support Statuses
  CPUs of the same architecture can support different subsets of functionality for many reasons. This operation allows the user to consult the CPU functionality flags to confirm its software compatibility.
• Get or Set Performance Monitor Configurations
  Usually every CPU have a set of performance counters that can register the events in the system. There are many events in the system, while the number of counters is usually limited, thus we can only choose a subset of events to register. This allows configuring the state of performance counters and assigning an event to a counter.
• Get or Set Counting Monitor Values
  This operation gets or sets the value of a counting performance monitor. Counting monitors are responsible for recording the number of certain counting events in the systems. Common counting events include:
  1. Cache eviction count
  2. Cache hit or miss count
  3. Upper-level cache miss and lower-level cache hit or miss count
  4. Upper-level cache miss and lower-level cache hit, count of whether the hit involves coherency operation or not
  5. In a NUMA architecture, the access count of local or remote memory
  6. Processor total (or some) interrupt count
  7. Processor branch prediction correct or incorrect count
  8. Processor atomic instruction execution success or failure count, and serialized operations’ count
  9. Processor fetch or retire total (or some, e.g. branch, memory access, I/O, multiplication/division, etc.) instruction count.
• Get or Set Cycle Monitor Values
  This operation gets or sets the value of a cycle performance monitor. Cycle monitors are responsible for recording the number of clock cycles associated with certain events in the system. Common cycle-accounting events include:
  1. Number of cycles since power-up
  2. Number of cycles wasted due to cache misses
  3. Number of cycles wasted due to TLB misses
  4. Number of cycles wasted due to branch prediction failure, instruction delay slot or other reasons which stall the instruction issuing stage
• Get or Set Data Monitor Values
  This operation gets or sets the value of data performance counters. Data performance counters are responsible for recording the amount of data exchange caused by certain events or on a certain channel. Common data-accounting events are:
  1. The total amount of a certain type of data (packet size, direction, category) on a certain bus
  2. The total amount of data exchange caused by certain instructions
• Get or Set Physical Monitor Values
  This operation gets or sets the value of physical performance counters. Physical performance counters are responsible for registering certain physical quantities or related informantion. Common physical quantities include:
  1. The temperature of a certain part of the circuit or ambient temperature
     2.The revolution speed of the cooling fan
     3.The current strength of a certain clock or power domain
     4.The total energy consumed by a certain clock or power domain
• Get or Set Cumulative Monitor Values
  This operation gets or sets the value of cumulative performance counters. Cumulative performance counters are responsible for registering certain cumulative quantities. Common cumulative quantities include:
  1. The total uptime or number of power cycles
  2. The uptime of current session
  3. The health status and damage control status

Hardware Virtualization Related Calls

• We need more discussion on the API design for this. The VMCS is with component or a standalone capability? This is undecided.

Secure Monitor Related Calls

• SGX and Tustzone support. We may find an API that accomodates the both. We'll see.

Debug and Trace Related Calls

• Debug Printing
  Usually, this will print a character to serial, and will not return before the printing finises. This can also be implemented as multi-character printing; in this case, additional characters needs to be passed in by extra registers of implementer’s choice.
• Get or Set the Register Set of a Thread
  Getting or setting the current register set of a certain thread is necessary for analyzing the execution state of the program. This operation may also be used to change the execution flow of the program,or modify its coprocessor state. It is worth noting that the Composite's system calls only implement 5 parameters and 3 return values by default, and the threads’ register set is usually very large. To increase read/write efficiency, the implementer should use more registers of their choice to facilitate argument and return value passing.
• Get or Set The Invocation Register Set of a Thread
  Getting or setting the invocation register set of a certain thread can force the invocation return of the thread to other addresses than where the invocation started. This can be useful for implementing some functionality such as paravirtualization. It is worth noting that even if invocation register set of a certain thread is only a small subset of its full register set, the implementer should use other registers of their choice for passing parameters and return values as needed.
• Get or Set Debug Engine Mode
  The debug engine usually contains some registers through which you can set the processor’s debug mode and state.
• Get or Set Instruction Breakpoint Statuses
  Some processors may support hardware instruction breakpoints. When the processor executes to the address of a hardware instruction breakpoint, a debug interrupt will be triggered, and the debugger can intercept this interrupt and know where the program ran to. Some processors may support breakpoint ASID, in which case a top-level page directory capability must be passed in.
• Get or Set Data Breakpoint Statuses
  Some processors may support hardware data breakpoints. When the processor reads or writes the address of a hardware data breakpoint, a debug interrupt will be triggered, and the debugger can intercept this interrupt and know what variable is accessed by this program . Some processors may support breakpoint ASID, in which case a top-level page directory capability must be passed in.

@gparmer @WenyuanShao @phanikishoreg @ryuxin