© 2003 Microsoft Corporation. All rights reserved. 1 Matthew D Hendel Software Engineer...

1© 2003 Microsoft Corporation. All rights reserved.

Matthew D HendelSoftware Engineer

BaseOS/Drivers

Advanced Storport Development


Contents

This presentation is for driver writers already familiar with Storport. The material covered will be on advanced Storport topics such as the best usage of Build I/O and Start I/O routines, error handling, queue implementation, as well as dispatch level processing in Storport and implementation of MSI interrupts in Storport.


Overview

Full Duplex Drivers

Build I/O routine

Example usage of Full Duplex and Build I/O routines

Dispatch Level Processing Passive Initialization Routine

DPC Routines

SpinLocks

Message Signaled Interrupts in Storport


Full Duplex Drivers

Storport adds the notion of “Full Duplex” I/O to the miniport model

In Full Duplex mode interrupts may occur while in StartIo routine

Simultaneous access to I/O submission and I/O completion

Miniport authors are responsible for synchronizing access to data structures that are shared between StartIo and Interrupt routines

Storport will not raise IRQL during I/O submission


Full Duplex Example: I2O Example Miniport

I2O has separate submission and completion ports

Inbound Post_List is used for command submission

Outbound Post_List is used for completed commands

Because the command submission and completion use separate lists the miniport’s Start I/O and Interrupt routines do not require any synchronization

Inbound (to IOP)

Free_List

Post_List

Outbound (from IOP)

Free_List

Post_List


Using a Build I/O Routine

Miniports spend a significant amount of time in StartIo routines performing work that does not need to be serialized such as building scatter-gather lists and translating commands

Since StartIo routines are serialized, this represents wasted time

BuildIo routine is called before StartIo allowing miniport to perform per-I/O initialization and translation

No locks are held during BuildIo routine, if a miniport needs to access external data structures during BuildIo routine it must manually synchronize using StorPortSynchronizeAccess or StorPortAcquireSpinLock


Build I/O Example: I2O Example Miniport

I2O uses non-SCSI command set, and different scatter-gather list format

For read and write requests, I2O miniport performs translation between SCSI and I2O commands in Build I/O routine

I2O Build I/O routine performs the following:

If SCSI Read command, translate SCSI Read to I2O read command

If SCSI Write command, translate SCSI Write to I2O write command, also translate SCSI Force Unit Access flag to I2O write-thru flag

Set appropriate I2O callback routine

Translate Storport scatter-gather list to I2O scatter-gather list

I2O command is built in SRB Extension


Example Of Build I/O Routine And Full Duplex

Example illustrates how the use of a Build I/O routine and Full Duplex mode can improve performance

Example parameters Start I/O routine takes 100 time units

Interrupt routine takes 30 time units

Thirty percent of Start I/O can be deserialized form other Start I/O work

Start I/O is completely independent of interrupt

Example shows two I/O commands issued and two interrupts handled


SCSIport Synchronization

Access to miniport synchronized with interrupt spinlock

Simple model is good for many adapters

I/O Request 3I/O Request 3

I/O Request 4I/O Request 4 ISR 2ISR 2

I/O Request 3at time = 0


Interrupt 2at time = 120

100


Time 0 200 300

Total time is 260 units

Completeat time = 260

Processor 1Processor 2

ISR 1ISR 1


Storport Synchronization Using Buildio Routine

Breaking the I/O Request into Start and Build portions reduces time Build routine creates the lower level I/O packet

Start issues the I/O request to the hardware

StartIo 3StartIo 3

StartIo 4StartIo 4 ISR 2ISR 2




100


Time 0 200 300



Processor 1Processor 2 Build 4Build 4

Build 3Build 3 ISR 1ISR 1


Storport Synchronization Using Buildio And Full Duplex

Adding Full Duplex allows StartIo and interrupt routines to be deserialized

StartIo 3StartIo 3 StartIo 4StartIo 4

ISR 2ISR 2




100


Time 0 200 300



Processor 1Processor 2 Build 4Build 4

Build 3Build 3

ISR 1ISR 1


Error Handling

Hierarchical reset instead of bus reset Perform reset of logical unit, failing that, reset of target, failing that reset of bus

Much more efficient on non-SCSI buses

Used by clusters to prevent resetting all targets on a bus


Reset Requests

Logical unit reset Send asynchronously via SCSI REQUEST BLOCK All I/O to logical unit is blocked while handling reset request Default timeout of 30 seconds Bus reset routine invoked if not handled within timeout

Target reset Send asynchronously via SCSI REQUEST BLOCK All I/O to adapter is blocked while handling the reset request Default timeout of 30 seconds Bus reset routine invoked if not handled within timeout

Bus reset Sent synchronously via HwStorReset routine All I/O to adapter is blocked while handling the reset request Must be handled


Storport Queues

No per-adapter queue limitations

Logical unit may have up to 255 requests outstanding

Logical unit Pause and Resume routines

Logical unit Busy and Ready routines

Adapter Pause and Resume routines

Adapter Busy and Ready routines


Direct Access To Scatter-gather List

Direct access to Storport Scatter-Gather lists improves code and prevents traditional get-physical-address loop in port driver

SCSIport Scatter-Gather Loop

Storport Scatter-Gather Call

while (BytesLeft) { PhysicalAddress = ScsiPortGetPhysicalAddress ( Extension,

Srb, DataBuffer, &Length ); BytesLeft -= Length; DataBuffer += Length;}

ScatterGatherList = StorPortGetScatterGatherList ( Extension, Srb );


Dispatch Level Processing

In some early Storport drivers, performance analysis showed large amounts of time spent in ISR. Further analysis showed that some drivers need to perform significant post-processing of requests.

It is very expensive to perform this processing at ISR time. Instead we added a new Storport mechanism to allow miniports to create and issue deferred procedure calls (DPCs).


Windows Driver Model IRQLs

Interrupt Request Levels (IRQLs) represent different levels at which code runs in kernel-mode

Interrupts for lower IRQLs are masked when running at a given IRQL

Following IRQLs are important to device driver writers: PASSIVE_LEVEL – May be interrupted by any interrupt, may synchronously

wait for completion

APC_LEVEL – May be interrupted by DISPATCH_LEVEL or higher level routines

DISPATCH_LEVEL – May be interrupted by any interrupts at device interrupt level (DIRQL); in full duplex mode, the following are all called at DISPATCH level: Start I/O routine, Build I/O routine, DPC routine, timer routine and reset bus routine

DIRQL – Level at which a device interrupts, may be interrupted by a higher level DIRQL


Summary Of Storport Dispatch Level Processing

To support Dispatch Level Processing, Storport adds the following five new routines: StorPortEnablePassiveInitialization – Invoke a callback routine at PASSIVE

level where StorPortInitializeDpc may be called

StorPortInitializeDpc – Initialize a miniport DPC routine

StorPortIssueDpc – Issue a DPC

StorPortAcquireSpinLock – Acquire a Storport spinlock

StorPortReleaseSpinLock – Release a spinlock previously acquired via StorPortAcquireSpinLock

Additional miniport callbacks HwStorPassiveInitialization

HwStorDpcRoutine


Passive Initialization

StorPortEnablePassiveInitialization routine Requests that the port driver call back the miniport at PASSIVE_LEVEL to

initialize DPCs

Passive initialization may be used for other things besides initializing DPCs (e.g., passive level enumeration)

May only be called from within Miniport’s HwInitialize routine

Routine will fail under some circumstances Crashdump and hibernation

Down-level operating systems (prior to SP1)

Driver will still load on older operating systems


Initialization Of DPCs

StorPortInitializeDpc initialized a DPC routine Must be called at PASSIVE_LEVEL, i.e., from HwPassiveInitialize routine

May initialized any number of DPCs necessary

Simple wrapper over KeInitializeDpc


Issuing DPCs

StorPortIssueDpc routine Invoke the DPC routine

DPCs are not queued. This is very important – if you call StorPortIssueDpc multiple times before the DPC executes, the DPC will only be invoked a single time. Miniports who use DPCs must handle this scenario, for example, by queuing pending requests.

Wrapper around KeInsertQueueDpc


Miniport DPC Routine

Miniport HwDpcRoutine Invoked as a result of calling StorPortIssueDpc

No synchronization of DPCs with other portions of the driver is done by Storport, including synchronization between different instances of this DPC routine

Miniports must use StorPortAcquireSpinLock and StorPortReleaseSpinLock to synchronize between DPC routine and other portions of miniport

Use DPC routine to perform operations that would take too long to perform at interrupt level


StorPort Locks

No locks are held when executing the following miniport callbacks: Find Adapter Routine Build I/O Routine Adapter Control Routine Passive Initialization Routine DPC Routine

Interrupt spinlock is held during the execution of the miniport callbacks: Hardware Initialize Routine Hardware Interrupt Routine

Start I/O lock is held during the execution of the following miniport callbacks: Hardware Start I/O routine Hardware Timer routine Hardware Reset Bus routine


DPC Routine Lock

No locking is automatically done when accessing the DPC routine, including locking the DPC against simultaneously running on multiple processors of the machine at the same time

Miniport must protect itself using StorPortAcquire/ReleaseSpinLock routines

Protecting Lock

Same DPC routine on a separate processor

DpcLock

StartIo routine StartIoLock

Interrupt routine InterruptLock

Multiple different DPCs StartIo lock


Message Signaled Interrupts In Storport

Message Signaled Interrupts provide an advanced interrupting mechanism that solves several problems with line based interrupts

No need for interrupt sharing

Multiple interrupt messages for same device

Increased bus throughput by eliminating some reads from devices


Storport InterruptSynchronizationMode

InterruptSynchronizeAll synchronizes all MSI interrupts using the same spinlock In this mode no two MSI interrupts will occur at the same time

Much simpler for driver writers

InterruptSynzhronizePerMessage only acquires the per-message lock when calling an MSI interrupt Multiple MSI interrupts for different messages may occur simultaneously

More complicated for miniport

Cannot synchronize with a single interrupt, so StorPortSynchronizeAccess and StorPortAcquireSpinLock with a parameter of InterruptLock are not allowed

A common MSI case will be with a single MSI interrupt


Message Signaled Interrupt Routine

Miniports MessageSignaledInterrupt routine is called back with device extension and MessageId parameters

MessageId may be used in call to StorPortGetMessageInterruptInformation to obtain the data structure describing the MSI interrupt MessageId

MessageData

MessageAddress

InterruptVector

InterruptLevel

InterruptMode


References

Further information on Storport is available in the Microsoft Windows Server 2003 DDK.

Links http://www.microsoft.com/windowsserversystem/storage/storport.mspx

http://www.microsoft.com/whdc/hwdev/tech/storage/entstor/default.mspx


© 2003 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Date post:	05-Jan-2016
Category:	Documents
Upload:	buck-copeland
View:	214 times
Download:	2 times

© 2003 Microsoft Corporation. All rights reserved. 1 Matthew D Hendel Software Engineer...

Documents