虛擬化技術虛擬化技術Virtualization TechniquesHardware Support VirtualizationSR-IOVSR-IOVAgendaOverviewIntroductionMemory VirtualizationStorage VirtualizationServers VirtualizationI/O VirtualizationPCIe VirtualizationMotivationDirected I/OPCIe ArchitectureSR-IOVArchitecture Supporting SR-IOV CapabilityARI Alternative Routing ID InterpretationACS Access Control ServicesATS - Address Translation ServiceTheory of OperationsOverviewMemory VirtualizationStorage VirtualizationServers VirtualizationI/O VirtualizationOverviewMemory VirtualizationUses memory more effectivelyWas revolutionary, but now is assumed Storage VirtualizationPresents storage resources in ways not bound to the underlying hardware characteristicsFairly common now Servers VirtualizationIncreases typically under-utilized CPU resourcesBecoming more commonOverviewI/O VirtualizationVirtualizing the I/O path between a server and an external deviceCan apply to anything that uses an adapter in a server, such as:Ethernet Network Interface Cards (NICs)Disk Controllers (including RAID controllers)Fibre Channel Host Bus Adapters (HBAs)Graphics/Video cards or co-processorsSSDs mounted on internal cardsPCIe I/O VirtualizationMotivationDirected I/OPCIe ArchitecutureMotivationI/O Virtualization SolutionsA - Software only B - Directed I/O (enhance performance)C Directed I/O and Device Sharing (resource saving)Virtual MachineVirtual MachineI/O DriverVirtual MachineI/O DriverVirtual Machine MonitorVirtual MachineI/O DriverVirtual MachineI/O DriverVirtual Machine MonitorVirtual MachineI/O DriverVirtual MachineI/O DriverVirtual Machine MonitorVirtual FunctionPhysical FunctionA Software onlyB Directed I/OC Directed I/O & Device SharingPCIe I/O VirtualizationMotivationDirected I/OPCIe ArchitectureDirected I/OSoftware-based sharing adds overhead to each I/O due to emulation layerThis indirection has the additional affect of eliminating the use of hardware acceleration that may be available in the physical device.Directed I/O has added enhancements to facilitate memory translation and ensure protection of memory that enables a device to directly DMA to/form host memory.Bypass the VMMs I/O emulation layerThroughput improvement for the VMsDrawbacks to Directed I/OOne concern with direct assignment is that it has limited scalabilityA physical device can only be assigned to one VM.For example, a dual port NIC allows for direct assignment to two VMs. (one port per VM)Consider for a moment a fairly substantial server of the very near future4 physical CPUs12 cores per CPUIf we use the rule that one VM per core, it would need 48 physical ports.Terminology relating to Directed I/OAcronymExpansionDefined ByWhat is it?I/O MMUI/O Memory Management UnitCommon parlanceTranslation mechanism in the system memory controller (North Bridge) that allows a device or set of devices to use translated addresses when accessing main memory. In many cases, it also translates interrupts coming from the devices as messages.ATPTAddress Translation and Protection TablePCI SIGI/O MMUVT-d, VT-d2Virtualization Technology for Directed I/OIntelI/O MMUDMArDMA RemappingIntel, MicrosoftI/O MMUIOMMUI/O Memory Management UnitAMDI/O MMUPCIe I/O VirtualizationMotivationDirected I/OPCIe ArchitectureGeneric Platform System Image(SI)SW, e.g., a guest OS, to which virtual and physical devices can be assignedSystem Image (SI)ProcessorMemoryRoot Complex (RC)RootPort (RP)RootPort (RP)PCIe DeviceSwitchPCIe DevicePCIe DevicePCIe DeviceSystem Image (SI)System Image (SI)System Image (SI)Virtualization IntermediaryPCIe componentsRoot ComplexA root complex connects the processor and memory subsystem to the PCIe switch fabric composed of one or more switch devicesSimilar to a host bridge in a PCI systemGenerate transaction requests on behalf of the processor, which isinterconnected through a local bus.May contain more than one PCIe portand multiple switch devices.PCIe componentsRoot Port (RP)The portion of the motherboard that contains the host bridge. The host bridge allows the PCIe ports to talk to the rest of the computer PCIe DevicePCIe DeviceUnique PCI Function AddressBus / Dev / FunctionCommand, lspci -v, can get PCI device information on linux DeviceFunction1Function2Example: Multi-Function Device The link and PCIe functionality shared by all functions is managed through Function 0All functions use a single Bus Number captured through the PCI enumeration processEach function can be assigned to an SIFunction 0 ATC1Physical Resources1Function 1 ATC2Physical Resources2Function 2 ATC3Physical Resources3Internal RoutingConfiguration ResourcesPCIePortPCIePortPCIePortPCIe DeviceComponents in PCIe Device Configuration SpaceDevices will allocate resource such as memory and record the address into this configuration spaceReference:PCI Local Bus Specification ver.2.3 Chap 6Configuration ResourcesComponents in PCIe Device ARI Alternative Routing Id InterpretationAlternative Routing ID Interpretation as per the PCIe Base SpecificationPhysical ResourcesMemory which allocated from physical memoryATC - Address Translation CacheA hardware stores recently used address translations.This term is used instead of TLB bufferTo differentiate the TLB used for I/O from the TLB used by the CPUFunction 0 ATC1Physical Resources1Function 1 ATC2Physical Resources2Function 2 ATC3Physical Resources3Internal RoutingPhysical V.S. VirtualFunction 0 ATC1Physical Resources1Function 1 ATC2Physical Resources2Function 2 ATC3Physical Resources3Internal RoutingConfiguration ResourcesPCIePortPCIePortPCIePortPCIe DevicePF 0 ATC1Physical ResourcesVF 0,1 Physical ResourcesVF 0,2Physical ResourcesInternal RoutingPCIePortPCIe SR-IOV Capable DeviceConfiguration ResourcesPhysicalVirtualPCIe SR-IOV Capable DeviceSR-IOVA technique performs and manages PCIe Virtualization.PF physical FunctionProvide full PCIe functionality, including the SR-IOV capabilities Discover the page sizes supported by a PF and its associated VFVF virtual Function A “light-weight” PCIe function that is directly accessible by an SI, including an isolated memory space, a work queue, interrupts and command processing.For data movementCan be optionally migrated form one PF to another PF Can be serially shared by different SIPF 0 ATC1Physical ResourcesVF 0,1 Physical ResourcesVF 0,2Physical ResourcesInternal RoutingPCIePortPCIe SR-IOV Capable DeviceConfiguration ResourcesDirectly and Software SharedFigure from Inter PCI-SIG SR-IOV PrimerExtended CapabilitiesSR-IOV Extended CapabilitiesSR-IOVArchitecture Supporting SR-IOV CapabilityARI Alternative Routing ID Interpretation ACS Access Control ServicesATS Address Translation ServiceData Path for Incoming PacketsPlatform with SR-IOVSR-PCIMConfigure SR-IOV CapabilityManagement of PFs and VFsProcessing of error eventsDevice controlsPower managementHot-plugSystem Image (SI)ProcessorMemoryRoot Complex (RC)RootPort (RP)RootPort (RP)PCIe DeviceSwitchPCIe DevicePCIe DevicePCIe DeviceSystem Image (SI)System Image (SI)System Image (SI)Virtualization IntermediaryTranslation Agent (TA)Address Translation and Protection Table (ATPT)SR-PCIMSR-PCIMComponents of SR-IOVTA Translation AgentTranslate address within a PCIe transaction into the associated platform physical address.Hardware or combination of hardware and softwareA TA may also support to enable a PCIe function to obtain address translations a priori to DMA access to the associated memory.Translation Agent (TA)Address Translation and Protection Table (ATPT)Components of SR-IOVATPT Address Translation and Protection TableContain the set of address translations accessed by a TA to Process PCEe requestsDMA Read/WriteInterrupt requestsDMA Read/Write requests are translated through a combination of the Routing ID and the address contained within a PCIe transactionIn PCIe, interrupts are treated as memory write operations. Though the combination of the Routing ID and the address contained within a PCIe transaction as wellTranslation Agent (TA)Address Translation and Protection Table (ATPT)SR-IOVArchitecture Supporting SR-IOV CapabilityARI Alternative Routing ID Interpretation ACS Access Control ServicesATS Address Translation ServiceData Path for Incoming PacketsARI Alternative Routing ID InterpretationRouting ID is used to forward requests to the corresponding PFs and VFsAll VFs and PFs must have distinct Routing IDsARI provides a mechanism to allow single PCIe component to support up to 256 functions.Originally there are 8 functions at most in a PCIe. Figure from Intel PCI-SIG SR_IOV primARI Alternative Routing ID InterpretationFigure from SR-IOV Specification revision 1.1Figure from Intel PCI-SIG SR_IOV primSR-IOVArchitecture Supporting SR-IOV CapabilityARI Alternative Routing ID Interpretation ACS Access Control ServicesATS Address Translation ServiceData Path for Incoming PacketsACS Access Control ServicesThe PCIe specification allows for P2P transactions.This means that it is possible and even desirable in some cases for one PCIe endpoint to send data directly to another endpoint without having to go through the Root Complex.However, in a virtualized environment it is generally not desirable to have P2P transactions. With both direct assignment and SR-IOV, the PCIe transactions should go through the Root Complex in order for the ATS to be utilized.ACS provides a mechanism by which a P2P PCIe transaction can be forced to go up through the RCFigure from Intel PCI-SIG SR_IOV primSR-IOVArchitecture Supporting SR-IOV CapabilityARI Alternative Routing ID Interpretation ACS Access Control ServicesATS Address Translation ServiceData Path for Incoming PacketsATS Address Translation ServicesATS provides a mechanism allowing a virtual machine to perform DMA transaction directly to and from a PCIe endpoint.ATS Address Translation ServicesATS uses a request-completion protocol between a Device and a Root Complex (RC)ATS Address Translation ServicesUpon receipt of an ATS Translation Request, the TA performs the following Requests1.Validates that the Function has been configured to issue ATS Translation Requests.2.Determines whether the Function may access the memory indicated by the ATS Translation Request and has the associated access rights.3.Determines whether a translation can be provided to the Function. If yes, the TA issues a translation to the Function.4.The TA communicates the success or failure of the request to the RC which generates an ATS Translation Completion and transmits via a Response TLP through a RP to the Function.PathFunction(Request)=TA=RC(Completion)=FunctionATS Address Translation ServicesWhen the Function receives the ATS Translation Completion Either updates its ATC to reflect the translation Or notes that a translation does not exist.The Function generates subsequent requests using either a translated address or an un-translated address based on the results of the Completion.SR-IOVArchitecture Supporting SR-IOV CapabilityARI Alternative Routing ID Interpretation ACS Access Control ServicesATS Address Translation ServiceData Path for Incoming PacketsData Path for incoming packets1.The Ethernet packet arrives at the Ethernet NIC2.The packet is sent to the Layer 2 sorter/switch/classifierThis Layer 2 sorter is configured by the Master Driver. When either the MD or the VF Driver configure a MAC address or VLAN, this Layer 2 sorter is configured.Data Path for incoming packets3. After being sorted by the Layer 2 Switch, the packet is placed into a receive queue dedicated to the target VF.4. The DMA operation is initiated. The target memory address for the DMA operation is defined within the descriptors in the VF, which have been configured by the VF driver within the VM.Data Path for incoming packets5. The DMA Operation has reached the chipset. Intel VT-d, which has been configured by the VMM then remaps the target DMA address from a virtual host address to a physical host address. The DMA operation is completed; the Ethernet packet is now in the memory space of the VM6. The NIC fires interrupt, indicating a packet has arrived. This interrupt is handled by the VMMData Path for incoming packets7. The VMM fires a virtual interrupt to the VM, so that it is informed that the packet has arrivedSummarySR-IOV creates Virtual Function, which records the information of the virtual PCIe device and be directly mapped to a system image.Virtual Function is a “light weight” function just for data movement. The management is controlled by Physical Function.ATC, a hardware stores recently used address translationsARI, a mechanism to allow single PCIe component to support up to 256 functions. And Routing ID is used to forward requests to the corresponding PFs and VFs. ATS, a mechanism allowing a virtual machine to perform DMA transaction directly to and from a PCIe endpointIn the end, a example show up the data path for the incoming packets.ReferenceIntel PCI-SIG SR-IOV Primer“SR-IOV Networking in Xen: Architecture, Design and Implementation” Yaozu Dong, Zhao Yu and Greg RoseSingle Root I/O Virtualization and Sharing Specification Revision 1.1Address Translation Services Revision 1.1“Implementing PCI I/O Virtualization Standards”, Mike Krause and Renato RecioPCI SIG IOV Work Group Co-chairshttp:/www.mindshare.com/files/ebooks/PCI%20System%20Architecture%20(4th%20Edition).pdfpdfhttp:/http:/http:/Q & A