Windows OS Principles

Windows OS Principles

Instructor: Hengming Zou, Ph.D.

In Pursuit of Absolute Simplicity 求于至简，归于永恒

22

Content Outline

2.1 Structuring of the Windows Operating System

2.2 Windows Core System Mechanisms

2.3 Windows on Windows - OS Personalities

2.4 The Windows API - Naming, Conventions, Types

33

2.1 Structuring of Windows OS

Architecture Overview

Program Execution Environment

Kernel Mode Architecture

System Threads

System Processes / Services

44

System supportprocesses

Serviceprocesses

Userapplications

Environmentsubsystems

Subsystem DLLs

Executive

Kernel Device drivers

Hardware Abstraction Layer (HAL)

Windowingand graphics

UserMode

KernelMode

Simplified OS Architecture

55

OS Architecture

Multiple personality OS design

– user applications don't call the native Windows operating system services directly

Subsystem DLLs is to translate a documented function into the appropriate internal (and undocumented) Windows system service calls

66

OS Architecture

Environment subsystem processes

– Manage client processes in their world

– Impose semantics such as process model, security

Windows 2000 shipped with three environment subsystems: Win32, POSIX, and OS/2

– XP has only Windows subsystem

– Win32 only one that matters

– DOS, Win 3.1 emulated by a Win32 program

– Enhanced POSIX subsystem available (Interix)

77

OS Architecture

88

Kernel-Mode Components: Core OS

Core OS contains two components:

– Executive and Kernel

– both contained in file Ntoskrnl.exe

Executive

– base operating system services,

– memory management, process and thread management,

– security, I/O, inter-process communication

99

Kernel-Mode Components: Core OS

Kernel

– low-level operating system functions,

– thread scheduling, interrupt and exception dispatching,

– multiprocessor synchronization.

Kernel provides a set of routines & basic objects

– that the rest of the executive uses to implement higher-level constructs

1010

Kernel-Mode Components: Drivers

Device drivers

– hardware device drivers translate user I/O function calls into specific hardware device I/O requests

– virtual devices - system volumes and network protocols

– Contained in files ending in .Sys

1111

Kernel-Mode Components: Drivers

Windowing and Graphics System

– graphical user interface (GUI) functions (Win32 USER and GDI),

– dealing with windows, user interface controls, and drawing

– [Contained in file Win32k.sys


– isolates kernel, device drivers, and executive from hardware

– Hides platform-specific hardware differences (motherboards)

– Contained in file Hal.dll

1212

Background System Processes

Core system processes,

– logon process, the session manager, etc.

– not started by the service control manager

Service processes

– host Win32 services

– i.e.; Task Scheduler and Spooler services

– Many Windows server apps, such as SQL Server and Exchange Server, also include components that run as services

1313

Portability

When NT was designed, there was no dominant processor architecture

– Therefore it was designed to be portable

How achieved?

1414

Portability

Most NT OS code and device drivers is written in C– HAL and kernel contain some assembly language

Some components are written in C++:– Win32 windowing/graphics subsystem driver

– Win2K volume manager

Hardware-specific code is isolated in low level layers of the OS (such as Kernel and the HAL)– Provides portable interface

1515

Portability

NT 4.0 had support for:

– x86, MIPS, PowerPC, Digital Alpha AXP

PowerPC and MIPS dropped soon after NT 4 release

Alpha AXP dropped in 1999 (supported through SP6)

1616

Reentrant and Asynchronous Operation

Windows kernel is fully reentrant

– Kernel functions can be invoked by multiple threads simultaneously

– No serialization of user threads when performing sys calls

I/O system works fully asynchronously

– Asynchronous I/O improves application’s throughput

– Synchronous wrapper functions provide ease-of-programming

1717

Key Windows System Files

Core OS components:

– NTOSKRNL.EXE**: Executive and kernel

– HAL.DLL: Hardware abstraction layer

– NTDLL.DLL: Internal support functions and system service dispatch stubs to executive functions

Core system processes:

– SMSS.EXE: Session manager process

– WINLOGON.EXE: Logon process

– SERVICES.EXE: Service controller process

– LSASS.EXE : Local Security Authority Subsystem

1818

Key Windows System Files

Windowing subsystem:

– CSRSS.EXE*: Windows subsystem process

– WIN32K.SYS: in32 and GDI kernel-mode components

– KERNEL32/USER32/GDI32.DLL: Win32 subsystem DLLs

1919

Key System Components

OS/2 WindowsPOSIX

Environment Subsystems

UserApplication

Subsystem DLL

WindowsUser/GDIDeviceDriver

Executive

Device Drivers Kernel


UserMode

KernelMode

System& ServiceProcesses

Windows

2020

Multiple OS Personalities

NT was designed to support multiple “personalities”, called environment subsystems

– Programming interface

– File system syntax

– Process semantics

2121


Environment subsystems provide exposed, documented interface between application and Windows native API

Each subsystem defines a different set of APIs and semantics

Subsystems implement these by invoking native APIs

Example: – Windows CreateFile in Kernel32.Dll calls native NtCreateFile

2222


.exes and .dlls you write are associated with a subsystem

– Specified by LINK /SUBSYSTEM option

– Cannot mix calls between subsystems

2323


3 environment subsystems originally provided with NT:

Windows –Windows API

– (originally 32-bit, now also 64-bit)

OS/2 - 1.x character-mode applications only

– Removed in Windows 2000

Posix - only Posix 1003.1

– bare minimum Unix services -no networking, windowing, threads

– Removed in XP/Server 2003 – enhanced version ships with Services For Unix 3.0

2424


Of the three, Windows provides access to the majority of NT native functions

Of the three, Windows is required to be running

– System crashes if Windows subsystem process exits

– POSIX and OS/2 subsystems are actually Windows applications

– POSIX & OS/2 start on demand (first time an app is run)Stay running until system shutdown

2525

App calls Subsystem

Function is entirely implemented in user mode

– No message sent to environment subsystem process

– No Win NT executive system service called

– Examples: PtInRect(), IsRectEmpty()

Function requires one/more calls to NT executive

– Examples: Windows ReadFile()/WriteFile() implemented using I/O system services NtReadFile()/NtWriteFile()

2626

App calls Subsystem

Function requires some work in environment subsystem process (maintain state of client app)

– Client/server request (message) to env. Subsystem (LPC facility)

– Subsystem DLL waits for reply before returning to caller

Combinations of 2/3: CreateProcess() / CreateThread()

2727

Windows Subsystem

Environment subsystem process (CSRSS.EXE): – Console (text) windows

– Creating and deleting processes and threads

– Portions of the support for 16-bit virtual DOS machine (VDM)

– Other func: GetTempFile, DefineDosDevice, ExitWindowsEx

kernel-mode device driver (WIN32K.SYS):– Window manager: manages screen output;

– input from keyboard, mouse, and other devices

– user messages to applications.

– Graphical Device Interface (GDI)

2828

Windows Subsystem (contd.)

Subsystem DLLs (such as USER32.DLL, ADVAPI32.DLL, GDI32.DLL, and KERNEL32.DLL)– Translate Windows API functions into calls to NTOSKRNL.EXE and WIN32K.SYS.

Graphics device drivers– graphics display drivers, printer drivers, video miniport drivers

Prior to NT 4.0, window manager and graphics services were part of the user-mode Win32 subsystem process

Is NT Less Stable with Win32 USER & GDI in Kernel Mode?

2929

2 GB2 GBUserUser

process process spacespace

2 GB2 GBUserUser


2 GB2 GBSystemSystemSpaceSpace

2 GB2 GBSystemSystemSpaceSpace

3 GB3 GBUserUser


3 GB3 GBUserUser


1 GB1 GBSystem SpaceSystem Space

1 GB1 GBSystem SpaceSystem Space

Default 3 GB user space

32-bit x86 Address Space

3030

Fastest way to find CPU hogs

Red=Kernel, Blue=User mode

Double-click on a process to see a per-thread display for that process

Sum of threads’ bars for a process represents all of the process’s time, not all CPU time

Screen snapshot from:Resource Kit | QuckSlice

Kernel vs User-Mode: QuickSlice (qslice.exe)

3131

““Running” means waiting for Running” means waiting for window messageswindow messages

Applications tab: List of top level visible windows

Right-click on a window and Right-click on a window and select “Go to process”select “Go to process”

Task Manager: Processes vs Applications Tabs Processes tab: List of processes

3232

hardware interfaces (buses, I/O devices, interrupts, interval timers, DMA, memory cache control, etc., etc.)

System Service Dispatcher

Task Manager

Explorer

SvcHost.Exe

WinMgt.Exe

SpoolSv.Exe

ServiceControl Mgr.

LSASS

Ob

jectM

gr.

WindowsUSER,

GDI

File

System

Cach

e

I/O Mgr


UserApplication

Subsystem DLLs

System Processes Services Applications

SystemThreads

UserMode

KernelMode

NTDLL.DLL

Device &File Sys.Drivers

WinLogon

Session Manager

Services.Exe POSIX

Windows DLLs

Plu

g an

dP

lay Mg

r.

Po

wer

Mg

r.

Secu

rityR

eference

Mo

nito

r

Virtu

alM

emo

ry

Pro

cesses&

Th

reads

Lo

calP

roced

ure

Call Graphics

Drivers

Kernel


(kernel mode callable interfaces)

Co

nfig

ura-

tion

Mg

r(reg

istry)

OS/2

Windows

Windows NT/2000/XP/2003 Architecture

3333

Microkernel OS?

Is NT/2000/XP/Server 2003 a microkernel-based OS?

No – not using the academic definition

– OS components and drivers run in their own private address spaces, layered on a primitive microkernel

All kernel components live in a shared address space– Therefore no protection between OS and drivers

3434

Microkernel OS?

Why not pure microkernel?

Performance

– separate address spaces would mean context switching to call basic OS services

Most other commercial OS (Unix, Linux, VMS etc.) have the same design

3535

Microkernel OS?

But it does have some attributes of a microkernel OS

OS personalities running in user space as separate processes

Kernel-mode components don't reach into one another’s data structures– Use formal interfaces to pass parameters and access and/or modify data structures

Therefore the term “modified microkernel”

3636

Symmetric Multiprocessing (SMP)

No master processor

– All the processors share just one memory space

– Interrupts can be serviced on any processor

– Any processor can cause another processor to reschedule what it’s running

Maximum # of CPUs stored in registry

– HKLM\System\CurrentControlSet\Control\Session Manager\LicensedProcessors

3737

MemoryMemory I/OI/O

CPUsCPUs

L2L2CacheCache

SMPSMP

Symmetric Multiprocessing (SMP)

Current implementation limit is # of bits in a native word– 32 processors for 32-bit systems

– 64 processors for 64-bit systems

– Not an architectural limit

– just implementation

3838

Hyperthreading

New technology in newer Xeon & Pentium 4 processors

– Makes a single processor appear as a dual processor to OS

– Also called simultaneous multithreading technology (SMT)

Chip maintains two separate CPU states

– “logical processors”

– Execution engine & onboard cache is shared

3939

Hyperthreading

Works with Windows 2000, but only XP & Server 2003 are “hyperthreading aware”

Logical processors don’t count against physical processor limits

Scheduling algorithms take into account logical vs physical processors

–Applications can also optimize for it (new Windows function in Server 2003)

4040

Hyperthreading

1 2 3 4

Four logical processors

Two physical processors

4141

NUMA

NUMA (non uniform memory architecture) systems

Groups of physical processors with local memory– Connected to the larger system through a cache-coherent interconnect bus

Still an SMP system

– e.g. any processor can access all of memory

– But node-local memory is faster

4242

NUMA

Scheduling algorithms take this into account

– Tries to schedule threads on processors within the same node

– Tries to allocate memory from local memory for processes with threads on the node

New Windows APIs to allow applications to optimize

4343

SMP Scalability

Scalability is a function of:

– parallelization and resource contention

Can’t make a general statement

Depends on what you are doing and

– if the code involved scales well

4444

SMP Scalability

Kernel is scalable

OS can run on any available processor and on multiple processors at the same time

Fine-grained synchronization within the kernel as well as within device drivers allows more components to run concurrently on multiple processors

Concurrency has improved with every release

4545

SMP Scalability

Applications can be scalable

Threads can be scheduled on any available CPU

Processes can contain multiple threads that can execute simultaneously on multiple processors

Programming mechanisms provided to facilitate scalable server applications

– Most important is I/O completion ports

4646

Multiple Product Packages…

Windows XP Home Edition

– Licensed for 1 CPU, 4GB RAM

Windows 2000 & XP Professional

– Desktop version (but also is a fully functional server system)

– Licensed for 2 CPUs, 4GB RAM (16GB on Itanium; 32GB on x64)

Windows Server 2003, Web Server

– Reduced functionality Standard Server (no domain controller)

– Licensed for 2 CPUs, 2GB RAM

4747


Windows Server 2003, Standard Edition (formerly Windows 2000 Server)

– Adds server and networking features (active directory-based domains, host-based mirroring and RAID 5, NetWare gateway, DHCP server, WINS, DNS, …)

– Licensed for 4 CPUs, 4GB RAM (32GB on x64)

4848


Windows Server 2003, Enterprise Edition (formerly Windows 2000 Advanced Server )

– 3GB per-process address space option, Clusters (8 nodes)

– Licensed for 8 CPUs, 32GB RAM (64GB on 64-bit editions)

Windows 2000 Datacenter Server & Windows 2003 Server, Datacenter Edition

– Process Control Manager

– Licensed for 32 CPUs, 64GB RAM (64 CPUs & 512GB on Itanium—in the future, 1TB!)

4949

…Built On the Same Core OS

Through W2K, core OS executables were identical– NTOSKRNL.EXE, HAL.DLL, xxxDRIVER.SYS, etc.

– but XP & Server 2003 have different kernel versions

Registry indicates system type (set at install time)– HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\ProductOptions

– ProductType: WinNT=Workstation, ServerNT=Server not a domain controller, LanManNT=Server that is a Domain Controller

– ProductSuite: indicates type of Server (Advanced, Datacenter, or for NT4: Enterprise Edition, Terminal Server, …)

5050

…Built On the Same Core OS

Code in the operating system tests these values and behaves slightly differently in a few places– Licensing limits (number of processors, number of network connections, etc.)

– Boot-time calculations (mostly in the memory manager)

– Default length of time slice

– See DDK: MmIsThisAnNtasSystem

5151

NTOSKRNL.EXE

Core operating system image

Contains Executive and Kernel

Also includes entry points for routines actually implemented in Hal.Dll

Many functions are exposed to user mode via NtDll.Dll and the environment subsystems

– To be discussed

5252

NTOSKRNL.EXE

Four retail variations:

NTOSKRNL.EXE Uniprocessor

NTKRNLMP.EXE Multiprocessor

W2K adds PAE (page address extension) versions

– must boot /PAE (32-bit Windows only)

NTKRNLPA.EXE Uniprocessor w/extended addressing support

NTKRPAMP.EXE Multiprocessor w/extended addressing support

5353

NTOSKRNL.EXE

Two checked build (debug) variations:

NTOSKRNL.EXE,

NTKRNLMP.EXE Debug multiprocessor

NTKRNLPA.EXE,

NTKRPAMP.EXE Debug multiprocessor w/extended addressing

5454

UP vs MP File Differences

These files are updated when moving from UP to MP

Everything else is the same (drivers, EXEs, DLLs)

NT4: Win32k.sys, Ntdll.dll, and Kernel32.dll had uniprocessor versionsName of file on system disk

Name of uniprocessor version on CD-ROM

Name of multiprocessor version on CD-ROM

NTOSKRNL.EXE \I386\NTOSKRNL.EXE \I386\NTKRNLMP.EXE

NTKRNLPA.EXE \I386\NTKRNLMP.EXE \I386\NTKRPAMP.EXE

HAL.DLL Depends on system type

Depends on system type

5555

Debug Version (“Checked Build”)

Special debug version called “Checked Build”

Multiprocessor versions only (runs on UP systems)

– helps catch synchronization bugs that are more visible on MP

Primarily for driver testing, but can be useful for catching timing bugs in multithreaded applications

5656


Built from same source files as “free build”

– aka “retail build”

– But with “DBG” compile-time symbol defined

This enables:

– error tests for “can’t happen” conditions in kernel mode (ASSERTs)

– validity checks on arguments passed from one kernel mode routine to another

5757

#ifdef DBG

if (something that should never happen has happened)

KeBugCheckEx(…)

#endif


5858


Since no checked Windows 2000 Server provided, can copy checked NTOSKRNL, HAL, to a normal Server system

– Select debug kernel & HAL with Boot.ini /KERNEL=, /HAL= switches

Windows Server 2003 has its own checked build

See Knowledge base article 314743 HOWTO:

– Enable Verbose Debug Tracing in Various Drivers & Subsystems

5959

Executive

Upper layer of the operating system

Provides “generic OS” functions (“services”)– Process Manager, Object Manager, Cache Manager

– LPC (local procedure call) Facility

– Configuration Manager, Memory Manager

– Security Reference Monitor

– I/O Manager, Power Manager

– Plug-and-Play Manager

6060

Executive

Almost completely portable C code

Runs in kernel (“privileged”, ring 0) mode

Most interfaces to executive services not documented

6161

Kernel

Lower layers of the operating system

Implements processor-dependent functions

– x86 vs. Itanium etc.

implements many processor-independent functions that

– are closely associated with processor-dependent functions

6262

Kernel

Main services

– Thread waiting, scheduling & context switching

– Exception and interrupt dispatching

– Operating system synchronization primitives (different for MP vs. UP)

– A few of these are exposed to user mode

Not a classic “microkernel”

– shares address space with rest of kernel-mode components

6363

HalGetInterruptVectorHalGetAdapterWRITE_PORT_UCHAR

Sample HAL routines:

HAL-Hardware Abstraction Layer

Responsible for a small part of hardware abstraction

– Components on the motherboard not handled by driversSystem timers, Cache coherency, and flushingSMP support, Hardware interrupt priorities

Subroutine library for the kernel & device drivers

– Isolates Kernel and Executive from platform-specific details

– Presents uniform model of I/O hardware interface to drivers

6464

HAL-Hardware Abstraction Layer

Reduced role as of Windows 2000

– Bus support moved to bus drivers

– Majority of HALs are vendor-independent

HAL also implements some functions that appear to be in the Executive and Kernel

6565

HAL - Hardware Abstraction Layer

Selected at installation time

– See \windows\repair\setup.log to find out which one

– Can select manually at boot time with /HAL= in boot.ini

HAL kit

– Special kit only for vendors that must write custom HALs

– requires approval from Microsoft

– see http://www.microsoft.com/whdc/ddk/HALkit/default.mspx

http://www.microsoft.com/whdc/ddk/HALkit/default.mspx

http://www.microsoft.com/whdc/ddk/HALkit/default.mspx

6666

Kernel-Mode Device Drivers

Separate loadable modules (drivername.SYS)

– Linked like .EXEs

– Typically linked against NTOSKRNL.EXE and HAL.DLL

– Only one version of each driver binary for both uniprocessor (UP) and multiprocessor (MP) systems…

– … but drivers call routines in the kernel that behave differently for UP vs. MP Versions

Defined in registry

– Same area as Windows services (t.b.d.) - differentiated by Type value

6767

Kernel-Mode Device Drivers

Several types:

– “ordinary”, file system, NDIS miniport, SCSI miniport (linked against port drivers), bus drivers

– More information in I/O subsystem section

To view loaded drivers, run drivers.exe

– Also see list at end of output from pstat.exe – includes addresses of each driver

To update & control:

– System properties->Hardware Tab->Device Manager

– Computer Management->Software Environment->Drivers

6868

System Threads

Functions in OS and some drivers that need to run as real threads

– E.g., need to run concurrently with other system activity, wait on timers, perform background “housekeeping” work

– Always run in kernel mode

– Not non-preemptible (unless they raise IRQL to 2 or above)

– For details, see DDK documentation on PsCreateSystemThread

What process do they appear in?

– “System” process (NT4: PID 2, W2K: PID 8, XP: PID 4)

– In Windows 2000 & later, windowing system threads (from Win32k.sys) appear in “csrss.exe” (Windows subsystem process)

6969

Examples of System Threads

Memory Manager

– Modified Page Writer for mapped files

– Modified Page Writer for paging files

– Balance Set Manager

– Swapper (kernel stack, working sets)

– Zero page thread (thread 0, priority 0)

Security Reference Monitor

– Command Server Thread

7070

Examples of System Threads

Network

– Redirector and Server Worker Threads

Threads created by drivers for their exclusive use

– Examples: Floppy driver, parallel port driver

Pool of Executive Worker Threads

– Used by drivers, file systems, …

– Work queued using ExQueueWorkItem

– System thread (ExpWorkerThreadBalanceManager) manages pool

7171

Identifying System Threads: Process Explorer With Process Explorer:

– Double click on System process

– Go to Threads tab – sort by CPU time

As explained before, threads run between clock ticks (or at high IRQL) and thus don’t appear to run

– Sort by context switch delta column

7272

Identifying System Threads: Process Explorer

7373

Process-Based Code

OS components that run in separate executables (.exe’s), in their own processes

– Started by system

– Not tied to a user logon

Three types:

– Environment Subsystems (already described)

– System startup processes“system startup processes” is not an official MS-defined name

– Windows Services

7474

Process-Based Code

Let’s examine the system process “tree”

– Use Tlist /T or Process Explorer

7575

Process-Based NT Code: System Startup Processes First two processes aren’t real processes

not running a user mode .EXE

no user-mode address space

different utilities report them with different names

data structures for these processes (and their initial threads) are “pre-created” in NtosKrnl.Exe

– and loaded along with the code

7676

Process-Based NT Code: Startup Processes (Idle) Process id 0

– Part of the loaded system image

– Home for idle thread(s) (not a real process nor real threads)

– Called “System Process” in many displays

(System) Process id 2 (8 in Windows 2000; 4 in XP)

– Part of the loaded system image

– Home for kernel-defined threads (not a real process)

– Thread 0 (routine name Phase1Initialization) launches the first “real” process, running smss.exe...and then becomes the zero page thread

7777

Process-Based NT Code: Startup Processes

smss.exe Session ManagerThe first “created” process Takes parameters from \HKEY_LOCAL_MACHINE\System\CurrentControlSet

\Control\Session ManagerLaunches required subsystems (csrss) and then winlogon

csrss.exe Windows subsystem

winlogon.exe Logon process: Launches services.exe & lsass.exe; presents first login promptWhen someone logs in, launches apps in \Software\Microsoft\Windows NT\WinLogon\Userinit

services.exe Service Controller; also, home for many NT-supplied servicesStarts processes for services not part of services.exe (driven by \Registry\Machine\System\CurrentControlSet\Services )

lsass.exe Local Security Authentication Server

userinit.exe Started after logon; starts Explorer.exe (see \Software\Microsoft\Windows NT\CurrentVersion\WinLogon\Shell) and exits (hence Explorer appears to be an orphan)

explorer.exe and its children are the creators of all interactive apps

7878

Where are Services Defined?

Defined in the registry:– HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services

– one key per installed service

Mandatory information kept on each service:– Type of service (Windows, Driver, ...)

– Image name of service .EXENote: some .EXEs contain more than one service

– Start type (automatic, manual, or disabled)

7979

Where are Services Defined?

Optional information:– Display Name

– New in W2K: Description

– Dependencies

– Account & password to run under

Can store app-specific configuration parameters– “Parameters” subkey under service key

8080

ServiceController/Manager

(Services.Exe)

SetupApplication

CreateServiceRegistry

ServiceProcesses

ControlPanel

Life of a Service

Install time

– Setup app tells Service Controller about service

System boot/initialization

– SCM reads registry, startsservices as directed

Management/maintenance

– Control panel can start/stop services & change startup parameters

8181

Process Explorer: Service Information

Process Explorer identifies Service Processes

– Click on Options->Highlight Services

8282

Service Processes

A process created & managed by the Service Control Manager (Services.exe)

– Similar in concept to Unix daemon processes

– Typically configured to start at boot time (if started while logged on, survive logoff)

– Typically do not interact with the desktop

Note: Prior to Windows 2000 this was the only way to start a process on a remote machine

– Now you can do it with WMI

8383

Mapping Services to Service Processes

Tlist /S (Debugging Tools) or Tasklist /svc (XP/2003) list internal name of services inside service processes

Process Explorer shows more: external display name and description

8484

Mapping Services to Service Processes

8585

Service Control Tools

Net start/stop – local system only

Sc.exe (built in to XP/2003; also in W2K Resource Kit)

– Command line interface to all service control/configuration functions

– Works on local or remote systems

Psservice (Sysinternals) – similar to SC

8686

Service Control Tools

Other tools in Resource Kit

–Instsrv.exe – install/remove services (command line)

–Srvinstw.exe – install/remove services (GUI)

Why are service creation tools included in Reskit?– Because Reskit comes with several services that are not installed as services when you install the Reskit

8787

Windows 2000 introduced generic Svchost.exe

– Groups services into fewer processesImproves system startup timeConserves system virtual memory

– Not user-configurable as to which services go in which processes

– 3rd parties cannot add services to Svchost.exe processes

Services Infrastructure

8888

XP/2003 have more Svchost processes due to two new less privileged accounts for built-in services

– LOCAL SERVICE, NETWORK SERVICE

– Less rights than SYSTEM accountReduces possibility of damage if system compromised

On XP/2003, four Svchost processes (at least):

– SYSTEM

– SYSTEM (2nd instance – for RPC)

– LOCAL SERVICE

– NETWORK SERVICE

Services Infrastructure

8989

2.2 Windows Core System Mechanisms

Object Manager & Handles

Local Procedure Calls

Exception Handling

Memory Pools

9090

Objects and Handles

Many Windows APIs take arguments that are handles to system-defined data structures, or “objects”

App calls CreateXxx, which creates an object and returns a handle to it

App then uses the handle value in API calls that operate on that object

9191

Three Types of Windows Objects/Handles

Windows “kernel objects”

– events, mutexes, files, processes, threads, etc.

– Objects are managed by the NT “Object Manager”,

– and represent data structures in system address space

– Handle values are private to each process

9292

Three Types of Windows Objects/Handles

Windows “GDI objects”

– pens, brushes, fonts, etc.

– Objects are managed by the Windows subsystem

– Handle values are valid system-wide / session-wide

Windows “User objects” (windows, menus, etc.)– Objects are managed by the Windows subsystem

– Handle values are valid system-wide / session-wide

9393

Handles and Security

Process handle table

– Unique for each process

– In system address space (hence, is trusted)

– Hence cannot be modified from user mode

Security checks are made when handle table entry is created

– i.e. at CreateXxx time

– Handle table entry indicates the “validated” access rights to the object

Read, Write, Delete, Terminate, etc.

9494

Handles and Security

APIs that take an “already-opened” handle look in the handle table entry before performing the function

For example:

– TerminateProcess checks to see if the handle was opened for Terminate access

No need to check file ACL, process or thread access token, etc., on every write request

– checking is done at file handle creation, i.e. “file open”, time

9595

HandleCount = 1ReferenceCount = 1

Event Object

Handles, Pointers, and Objects

Handle Table

Process A

Handle Table

Process B

System Space

handles

index

Handle to a kernel object is an index into the process handle table, and is invalid in any other process

Handle table entry contains the system-space address (8xxxxxxx or above) of the data structure; this address is the same regardless of process context

Although handle table is per-process, it is actually in system address space (hence protected)

9696


Event Object

Handles, Pointers, and Reference Count

Handle Table

Process A

Handle Table

Process B

System Spacehandles

indexHandleCount = 2

ReferenceCount = 0

DuplicateHandle



Thread(in a wait statefor the event)

Thread(in a wait statefor the event)

Note: there is actually another data structure, a “wait block”, “between” the thread and the object it’s waiting for

9797

Object Manager

Executive component

For managing system-defined “objects”

Objects are data structures with optional names

Objects managed here include

– Windows Kernel objects, but not Windows User or GDI objects

Object manager implements user-mode handles and the process handle table

9898

Object Manager

OM is not used for all NT data structures Generally, only those types that need to be shared, named, or exported to user mode

Some data structures are called “objects” but are not managed by the object manager – e.g. “DPC objects”

In part, a heap manager… – Allocates memory for data structure from system-wide, kernel space heaps (pageable or nonpageable)

9999

Object Manager

… with a few extra functions:

– Assigns name to data structure (optional)

– Allows lookup by name

– Objects can be protected by ACL-based security

– Provides uniform naming, sharing, and protection schemeSimplifies C2 security certification by centralizing all object protection in one place

– Maintains counts of handles and references (stored pointers in kernel space) to each objectObject cannot be freed back to the heap until all handles and references are gone

100100

Executive Objects

Object type

Represents

Object directory

Container object for other objects: implement hierarchical namespace to store other object types

Symbolic link

Mechanism for referring to an object name indirectly

Process Virtual address space and control information necessary for execution of thread objects

Thread Executable entity within a process

Section Region of shared memory (file mapping object in Windows API)

File Instance of an opened file or I/O device

Port Mechanism to pass messages between processes

Access token

Security profile (security ID, user rights) of a process or thread

101101

Executive Objects (contd.)

Object type

Represents

Event Object with persistent state (signaled or not) usable for synchronization or notification

Semaphore Counter and resource gate for critical section

Mutant Synchronization construct to serialize resource access

Timer Mechanism to notify a thread when a fixed period of time elapses

Queue Method for threads to enqueue/dequeue notifications of I/O completions (Windows I/O completion port)

Key Reference to registry data – visible in object manager namespace

Profile Mechanism for measuring execution time for a process within an address range

102102

Object Structure

Type object contains static, object-type specific data:- shared among all instances of an object type- link all instances together (enumeration)

Object nameObject directorySecurity descriptorQuota chargesOpen handle countOpen handles listObject typeReference count

Object-specific data

Object header

Object body

Process 1

Process 2

Process 3

Type nameAccess typesSynchronizable? (Y/N)Pageable? (Y/N)Methods: open, close, delete parse, security, query name

Type object





103103

Object Methods

Method When method is called

Open When an object handle is opened

Close When an object handle is closed

Delete Before the object manager deletes an object

Query name When a thread requests the name of an object, such as a file, that exists in a secondary object domain

Parse When the object manager is searching for an object name that exists in a secondary object domain

Security When a process reads/changes protection of an objects, such as a file, that exists in a secondary object domain

Process opens handle to object \Device\Floppy0\docs\resume.doc

Object manager traverses name tree until it reaches Floppy0

Calls parse method for object Floppy0 with arg \docs\resume.doc

104104

\mssdk\bin\winnt\winobj.exe (no shortcut exists by default)

Winobj from Platform SDK

105105

Object Manager Namespace

System and session-wide internal namespace

View with Winobj from www.sysinternals.com

106106

Interesting Objects(visible in WinObj)

in \ObjectTypes– objects that define types of objects

in \BaseNamedObjectsthese will appear when Windows programs use CreateEvent, etc.

– mutant (Windows mutex)

– queue (Windows I/O completion port)

– section (Windows file mapping object)

– Event, Semaphore

In \GLOBAL??– DOS device name mappings for console session

107107


Hierarchical directory structure

– based on file system model

System-wide (not per-process)

– With Terminal Services, Windows objects are per-session by default

– Can override this with “global\” prefix on object names

Volatile (not preserved across boots)

– As of Server 2003, requires SeCreateGlobalPrivilege

can be extended by secondary object managers

– e.g. file system

– Hook mechanism to call external parse routine (method)

108108


Supports case sensitive or case blind

Supports symbolic links

– used to implement drive letters, etc.

Lookup done on object creation or access by name

– Not on access by handle

Not all objects managed by object manager are named

– e.g. file objects are not named

– un-named objects are not visible in WinObj

109109

Kernel Memory Pools (System-Space Heaps)

Nonpaged pool

Has initial size and upper limit

– can be grown dynamically, up to the max

32-bit upper limit: 256 MB on x86 (NT4: 128MB)

– 64-bit: 128 GB

Performance counter displays current total size

– allocated + free

Max size stored in kernel variable

– MmMaximumNonPagedPoolInBytes

110110

Kernel Memory Pools (System-Space Heaps)

Paged pool

32-bit upper limit:

– 650MB (Windows Server 2003),

– 470MB (Windows 2000),

– 192MB (Windows NT 4.0)

64-bit: 128 GB

Max size stored in MmSizeOfPagedPoolInBytes

Pool size performance counters display

– current size, not max

– To display maximums, use “!vm” kernel debugger command

111111

Invoking Kernel-Mode Routines

Code is run in kernel mode for one of three reasons:

Requests from user mode

– Via the system service dispatch mechanism

– Kernel-mode code runs in the context of the requesting thread

112112


Interrupts from external devices

– Interrupts (like all traps) are handled in kernel mode

– NT-supplied interrupt dispatcher invokes the interrupt service routine

– ISR runs in the context of the interrupted thread (so-called “arbitrary thread context”)

– ISR often requests the execution of a “DPC routine”, which also runs in kernel mode

113113


Dedicated kernel-mode threads

– Some threads in the system stay in kernel mode at all times (mostly in the “System” process)

– Scheduled, preempted, etc., like any other threads

114114

Screen snapshot from: Programs | Administrative Tools | Performance Monitorclick on “+” button, or select Edit | Add to chart...

Accounting for Kernel-Mode Time

115115

Accounting for Kernel-Mode Time

“ Processor Time” = total busy time of processor – equal to elapsed real time - idle time

“Processor Time” = “User Time” + “Privileged Time”

“Privileged Time” = time spent in kernel mode

“Privileged Time” includes:– Interrupt Time

– DPC Time

– other kernel-mode time (no separate counter for this)

116116

Kernel Memory Pools (System-Space Heaps) NT provides two system memory pools:

Nonpaged Pool and Paged Pool

Used for systemwide persistent data

– visible from any process context

Nonpaged pool required for memory accessed from

– DPC/dispatch IRQL or above

– Page faults at DPC/dispatch IRQL or above cause a system crash

117117

Kernel Memory Pools (System-Space Heaps) Pool sizes are a function of memory size & Server vs. Workstation

– Can be overidden in Registry:HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management

– But are limited by implementation limits (next slide

118118

Lookaside Lists

Instead of frequently allocating and freeing pool…

…Use lookaside lists instead

Implements a driver private list of preallocated blocks of pool

Routines automatically extend the lookaside list from the systemwide pool

Avoids hitting systemwide pool spinlock for every allocation/release

119119

Lookaside Lists

Optimizes CPU/memory cache behavior (the same driver will use the same list items over and over)

See:

– ExInitialize[N]PagedLookasideList

– ExAllocateFrom[N]PagedLookasideList

– ExFreeTo[N]PagedLookasideList

– ExDelete[N]PagedLookasideList

120120

Increased System Memory Limits

Key system memory limits raised in XP & Server 2003

W2K limit of 200 GB of mapped file data eliminated

– Previously limited size of files that could be backed up

Maximum System Page Table Entries (PTEs) increased

– Can now describe 1.3 GB of system space (960 MB contiguous)

– Windows 2000 limit was 660 MB (220 MB contiguous)

– Increases number of users on Terminal Servers

– Also means maximum device driver size is now 960MB (was 220)

121121

Local Procedure Calls (LPCs)

IPC – high-speed message passing

Not available through Windows API

– Windows OS internal

Application scenarios:

– RPCs on the same machine are implemented as LPCs

– Some Windows APIs result in sending messages to Windows subsyst. proc.

– WinLogon uses LPC to communicate with local security authentication server process (LSASS)

– Security reference monitor uses LPC to communicate with LSASS

122122

Local Procedure Calls (LPCs)

LPC communication:

– Short messages < 256 bytes are copied from sender to receiver

– Larger messages are exchanged via shared memory segment

– Server (kernel) may write directly in client‘s address space

123123

Port Objects

LPC exports port objects to maintain state of communication:

– Server connection port: named port, server connection request point

– Server communication port: unnamed port, one per active client, used for communication

– Client communication port: unnamed port a particular client thread uses to communicate with a particular server

– Unnamed communication port: unnamed port created for use by two threads in the same process

124124

Port Objects

Typical scenario:

– Server creates named connection port

– Client makes connection request

– Two unnamed ports are created, client gets handle to server port, server gets handle to client port

125125

Use of LPC ports

Client addressspace

Kernel addressspace

Server addressspace

Messagequeue

Connection port

Clientcommunication

port

Servercommunication

port

Sharedsection

Client process Server process

Handle

Handle

Server viewof section

Handle

Client viewof section

126126

Exception Dispatching

Exceptions are conditions that result directly from the execution of the program that is running

Windows introduced a facility known as structured exception handling, which allows applications to gain control when exceptions occur

127127

Exception Dispatching

The application can then fix the condition and return to the place the exception occurred,

– unwind the stack (thus terminating execution of the subroutine that raised the exception), or

– declare back to the system that the exception isn’t recognized and the system should continue searching for an exception handler that might process the exception.

128128

Exception Dispatching (contd.)

Trap handler Exception

dispatcher)

Debugger(first chance)

Frame-basedhandlers

Debugger(second chance)

EnvironmentsubsystemKernel default

handler

(Exceptionframe, clientthread ID)

exception

Unhandled exceptions are passed tonext handler

Exception dispatcher sends debug message toDebugger via LPC/excepion port & session manager process

LPC to debugger port

Structured exception handling;

– Accessible from MS VC++ language: __try, __except, __finally

– See Jeffrey Richter, „Advanced Windows“, MS Press

– See Johnson M.Hart,Win32 System Programming, Addison-Wesley

129129

Internal Windows API exception handler

Processes unhandled exceptions

– At top of stack, declared in StartOfProcess()/StartOfThread()

void Win32StartOfProcess(LPTHREAD_START_ROUTINE lpStartAddr,

LPVOID lpvThreadParm) {

__try {

DWORD dwThreadExitCode = lpStartAddr(lpvThreadParm);

ExitThread(dwThreadExitCode);

} __except(UnhandledExceptionFilter(

GetExceptionInformation())) {

ExitProcess(GetExceptionCode());

}

}

130130

2.3 Windows on Windows - OS Personalities Environment Subsystems

System Service Dispatching

Windows on Windows - 16bit

Windows on Windows - 64bit

131131


OS/2 WindowsPOSIX


UserApplication

Subsystem DLL

WindowsUser/GDIDeviceDriver

Executive



UserMode

KernelMode


Windows

132132

What about .NET and WinFX?

.NET/WinFXApplication

Windows API DLLsUserMode

Windows KernelKernelMode

.NET/WinFX Framework

WinFX is the .NET Framework that will ship with Longhorn

Both .NET and WinFX are built on standard Windows APIs

– They are not a subsystem

– They do not call undocumented NT system calls

133133

Subsystem Information in Registry

Subsystems configuration and startup information is in:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems

Values:

– Required - list of value names for subsystems to load at boot time

– Optional - list of value names for subsystems to load when needed

– Windows - value giving filespec of Windows subsystem (csrss.exe)

csrss.exe Windows APIs required - always started when NT boots

– Kmode - value giving filespec of Win32K.Sys (kernel-mode driver portion of Windows subsystem)

– Posix - file name of POSIX subsystem

psxss.exe Posix APIs optional - started when first Posix app is run

134134

Subsystem Information in Registry

Some Windows API DLLs are in “known DLLs” registry entry:

HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\KnownDLLs

– Files are opened as mapped files

– Improves process creation/image startup time

135135

Subsystem Components

1:API DLLs – for Windows: Kernel32.DLL, Gdi32.DLL, User32.DLL, etc.

2:Subsystem process – for Windows: CSRSS.EXE (Client Server Runtime SubSystem)

3:For Windows only: kernel-mode GDI code

– Win32K.SYS - (this code was formerly part of CSRSS)

136136

OS/2Windows

POSIX


UserApplication

Subsystem DLL

WindowsUser/GDI

UserMode

Executive



KernelMode


3

2

1

Subsystem Components

137137

Role of Subsystem Components

1:API DLLs:Export the APIs defined by the subsystem

–Implement them by calling NT “native” services, or by asking the subsystem process to do the work

3:Win32K.Sys– Implements Windows User & GDI functions; calls routines in GDI drivers

– Also used by Posix and OS/2 subsystems to access the display

138138

Role of Subsystem Components

2: Subsystem process:

– Maintains global state of subsystem

–Implements a few APIs that require subsystem-wide state changesProcesses and threads created under a subsystem

Drive letters

Window management for apps with no window code of their own (character-mode apps)

Handle and object tables for subsystem-specific objects

139139

UserMode

KernelMode

NtDll.Dll

OS/2 Windows POSIX

Environment SubsystemsSystem& ServiceProcesses

ExecutiveDevice Drivers Kernel


KiSystemService

LPC

UserApplication

Subsystem DLL

1 2

Simplified Architecture (3.51 and earlier)

most Windows Kernel APIs

all other Windows APIs, including User and GDI APIs

140140

NtDll.Dll

OS/2Windows

POSIX


WindowsUser/GDI

UserMode

Executive



KernelMode


1 3 2

KiSystemService

UserApplication

Subsystem DLL

LPC

1: most Windows Kernel APIs 2: most Windows User and GDI APIs (these were formerly part of

CSRSS) 3: a few Windows APIs

NT4/2000/XP/2003 Simplified Architecture

141141

Role of CSRSS.EXE

Windows Subsystem Process

A few APIs are implemented in this separate process

In 3.51 and earlier:

– Nearly all User and GDI APIs were implemented in CSRSS

– CSRSS had a thread for every app thread that created a window

– GDI drivers (video, printer) were user mode, mapped into this process

– This was done for protection, esp. to keep GDI drivers in user mode

142142

Role of CSRSS.EXE

CSRSS in NT 4.0: role is greatly diminished

– Maintains system-wide state information for all Windows “client” processes

– Several Windows services LPC to CSRSS for “setup and teardown” functionsProcess and thread creation and deletion

Get temporary file name

Drive letters

– Security checks for file system redirector

– Window management for console (character cell) applications …

– … including NTVDM.EXE

143143

IMAGE_SUBSYSTEM_UNKNOWN 0 // Unknown subsystemIMAGE_SUBSYSTEM_NATIVE 1 // Image doesn't require a subsystemIMAGE_SUBSYSTEM_WINDOWS_GUI 2 // Windows subsystem (graphical app)IMAGE_SUBSYSTEM_WINDOWS_CUI 3 // Windows subsystem (character cell)IMAGE_SUBSYSTEM_OS2_CUI 5 // OS/2 subsystemIMAGE_SUBSYSTEM_POSIX_CUI 7 // Posix subsystem

Header of Executable File Specifies Subsystem Type Subsystem for each .exe specified in image header

– see winnt.h

–

144144

Header of Executable File Specifies Subsystem Type

– see Explorer / QuickView (right-click on .exe or .dll file)

– or exetype image.exe (Resource Kit)

145145

Native Images

.EXEs not linked against any subsystem

– Interface to NT executive routines directly via NTDLL.DLL

Two examples:

– smss.exe (Session Manager -- starts before subsystems start)

– csrss.exe (Windows subsystem)

146146

Look at subsystem startup information in registry

Using EXETYPE, look at subsystem types for:

– \windows\system32\notepad.exe,

– cmd.exe, csrss.exe

Lab: Subsytems & Images

147147

POSIX Subsystem

Windows 2000 implements POSIX.1

– ISO/IEC 9945-1:1990 or IEEE POSIX standard 1003.1-1990

– POSIX.1 compliance as specified in Federal Information Processing Standard (FIPS) 151-2 (NIST)

– POSIX Conformance Document in \HELP in Platform SDK

support for impl.of POSIX.1 subsystem was mandatory for NT

– fork service in NT executive

– hard file links in NTFS

148148

POSIX Subsystem

Microsoft supplies a full POSIX subsystem forWindows 2000 under the Interix product name

limited set of services

– such as process control, IPC, simple character cell I/O

– POSIX subsystem alone is not a complete programming environment

POSIX executable cannot

– create a thread or a window

– use remote procedure calls (RPCs) or sockets

149149

Porting UNIX Apps to NT

UNIX-to-Win32 porting libraries

– DataFocus (http://www.datafocus.com/)

– ConsenSys (http://www.consensys.com/)

– Cygnus - CygWin GNU tools (http://www.cygnus.com/)

POSIX subsystem with complete UNIX system service and utilities environment

– Interix from Microsoft (used to be SoftWay (http://www.opennt.com/)

– Bought by Microsoft as of Sept. 99

150150

Porting UNIX Apps to NT

NT Resource Kit includes optional set of POSIX utilities

POSIX executables are linked against PSXDLL.DLL

– Header files in platform SDK

151151

Watching the POSIX Subsystem Start

POSIX subsystem starts on demand:

1. Type tlist /t, check that POSIX subsyst. is not running

2. Run \ntreskit\POSIX\LS.EXE

3. Run tlist /t again, PSXSS.EXE is child of SMSS.EXE

152152

ApplicationProcess

CSRSS.EXE(Windowssubsystemprocess)

Subsystem DLL

ExecutiveLocalProcedureCall

LPC is an undocumented system service in NtDll.Dll(not exposed

directly in API)U

K

Invoking (a few) Windows Services

Some Windows services are still implemented in the Windows subsystem process

153153

System Call Dispatching

NTDLL.DLL provides interface for NT native system calls

154154

call WriteFile(…)

call NtWriteFilereturn to caller

do the operationreturn to caller

Int 2E or SYSCALL or SYSENTERreturn to caller

call NtWriteFiledismiss interrupt

Windows application

WriteFile in Kernel32.Dll

NtWriteFilein NtDll.Dll

KiSystemServicein NtosKrnl.Exe

NtWriteFilein NtosKrnl.Exe

Windows-specific

used by all subsystems

software interrupt

U

K

Example: Invoking a Windows Kernel API

155155

Invoking System Functions from User Mode Kernel-mode functions (“services”) are invoked from user mode via a protected mechanism

– x86: INT 2E (as of XP, SYSENTER is used where available); AMD32 and x64: SYSCALL

– i.e., on a call to an OS service from user mode, the last thing that happens in user mode is this “change mode to kernel” instruction

– Causes an exception or interrupt, handled by the system service dispatcher (KiSystemService) in kernel mode

– Return to user mode is done by dismissing the interrupt or exception

156156

Invoking System Functions from User Mode The desired system function is selected by the “system service number”

– Every NT function exported to user mode has a unique number

– This number is stored in a register just before the “change mode” instruction (after pushing the arguments to the service)

– This number is an index into the system service dispatch table

– Table gives kernel-mode entry point address and argument list length for each exported function

157157

Invoking System Functions from User Mode All validity checks are done after the user to kernel transition– KiSystemService probes argument list, copies it to kernel-mode stack, and calls the executive or kernel routine pointed to by the table

– Service-specific routine checks argument values, probes pointed-to buffers, etc.

– Once past that point, everything is “trusted”

158158

Invoking System Functions from User Mode This is safe, because:

– The system service table is in kernel-protected memory; and

– The kernel mode routines pointed to by the system service table are in kernel-protected memory; therefore:

– User mode code can’t supply the code to be run in kernel mode; it can only select from among a predefined list

– Arguments are copied to the kernel mode stack before validation; therefore:

– Other threads in the process can’t corrupt the arguments “out from under” the service

159159

call write(…)

call NtWriteFilereturn to caller


Int 2Ereturn to caller


Posix application

write in psxdll.dll




Posix-specific


software interrupt

U

K

Example: Invoking a Posix API

160160

Ntdll.dll

Interface to NT system calls (285 calls starting with “Nt”)

These user-mode routines have the same function names and arguments as the kernel mode routines they invoke

– e.g. NtWriteFile in NtDll.Dll invokes NtWriteFile in NtosKrnl.Exe

Majority are not supported or documented

161161

Ntdll.dll

7 are (partially) documented in the Platform SDK:

– NtQuerySystemInformation, NtQuerySystemTime,

– NtQueryInformationProcess, NtQueryInformationThread,

– NtCreateFile, NtOpenFile, NtWaitForSingleObject

The DDK describes 25 of them as “Zw” routines

– such as ZwReadFile

– These entry points call the corresponding “Nt” interface via the system call interfaceThus, “previous mode” is kernel mode (means no security checks)

– Kernel mode code could also call NtReadFile directly

162162

Ntdll.dll

Other user-mode support routines

– Image loader (“Ldr”)

– Debug infrastructure (“Dbg”)

– Csrss support routines (“Csr”)

– RTL routines (“Rtl”)

– Tracing routines (“Etw”) [new as of Windows Server 2003]

163163

call NtWriteFile(…)




native application(smss.exe, csrss.exe)





software interrupt

U

K

Calling a “Native” API from User Mode

164164

Invoking (most) User and GDI Services

call user or GDI service(…)


call Windows routinedismiss interrupt

application

Gdi32.Dllor User32.Dll


service entry pointin Win32K.Sys

Windows-specific

software interrupt

U

K


165165

16-bit MS-DOS emulation

MS-DOS application

16-bit

Virtual device drivers (COM, LPT, keyboard)

32-bit MS-DOS emulation 32-bit

640K

0

16MB

1MBMS-DOS ext. memory

Windows Subsystem DLLs

OS/2Windows

POSIX


UserApplication

Subsystem DLL

WindowsUser/GDI

UserMode

Executive



KernelMode


7FFFFFFF

16-bit Applications on 32-bit Windows

NT runs NTVDM.EXE

– NT Virtual Dos Machine

NTVDM is a Windows image

– No “DOS subsystem” or “Win16 subsystem”

166166

16-bit MS-DOS emulation

MS-DOS application

16-bit

Virtual device drivers COM, LPT, keyboard

32-bit MS-DOS emulation 32-bit

WindowsExample: three DOS apps running in three NTVDM processes

640K

0

16MB

1MB


extended memory

7FFFFFFF

DOS 16-bit Applications

e.g. command.com, edit.com (NT4 had qbasic.exe)

NT runs NTVDM.EXE (NT Virtual DOS Machine)

– See \System\CurrentControlSet\Control\WOW\cmdline

167167

DOS 16-bit Applications

Each DOS app has a separate process running NTVDM

– DOS & Windows 16-bit drivers not supported

Note: NT command prompt is not a “DOS box”, despite icon;

– it’s a Windows console application (CMD.EXE)

168168

Windows 16-Bit Applications

e.g. sysedit.exe, winhelp.exe

NT also runs NTVDM.EXE

– See \CurrentControlSet\Control\WOW\wowcmdline

NTVDM loads wowexec.exe

– WOW = “Windows on Windows”

– Win16 calls are translated to Win32

169169

WindowsWin16 app

Win16 app

Win16 appexample: three Win16 apps (and wowexec.exe) running in one NTVDM process

Win16 API


Virtual Device Drivers

16-bit

32-bit

7FFFFFFF

16MB

0

Windows 16-Bit Applications

170170

Windows 16-bit Apps Multitasking Details Each Win16 app runs in a separate thread in the common NTVDM process

They cooperatively multitask among themselves

– Win16 Yield API...

…and the one (if any) that wants to run, preemptively multitasks with all other threads on NT

necessary to meet serialization assumptions of some Win16 apps

171171

NT4 only:

Windows 16-bit Apps Multitasking Details

172172

Windows 16-bit Apps Multitasking Details Option to run Win16 apps in separate VDMs

– Run in Separate Memory Space = run in separate process

– default set by \CurrentControlSet\Control\WOW\DefaultSeparateVDM

– Win16 apps run this way preemptively multitask with all other threads, including the un-Yield’ed thread in a shared Win16 NTVDM (if any)

173173

Monitoring 16-bit Applications

To most of NT, an NTVDM process is just another process

Task Manager

– tasks are simply the names of top-level windows Win16 windows included

– processes display identifies Win16 apps within NTVDM processesby reading the NTVDM process’s private memory (undocumented interface)

– does not identify the DOS apps within each NTVDM process

174174

Monitoring 16-bit Applications

TLIST (resource kit)

– identify DOS apps within each NTVDM process (by window title)

– for a shared Win16 NTVDM process, only shows one window title

QuickView, exetype

– identifies DOS, Win16, etc., application .exe’s

175175

DOS applications:

Run qbasic.exe and then edit.com

look at process list in Task Manager Process tab

– cannot differentiate which NTVDM.EXE is which

From Applications tab, right click on window

– goto process (now can map which NTVDM.EXE process is which)

Lab: 16-bit Applications

176176

Windows 3.1 applications:

Run winhelp.exe twice

– do not check “run in separate memory space”

Run winhelp.exe once and

– check “run in separate memory space”

Bring up Task Manager Process tab

– make sure “Show 16-bit Tasks” is checked on the View menu

Look at Task Manager Process tab and see 16-bit applications identified inside the two NTVDMs

Lab: 16-bit Applications

177177

Wow64

Allows execution of Win32 binaries on 64-bit Windows

– Wow64 intercepts system calls from the 32-bit applicationConverts 32-bit data structure into 64-bit aligned structures

Issues the native 64-bit system call

Returns any data from the 64-bit system call

IsWow64Process() function can tell a 32-bit process if it is running under Wow64

178178

Wow64

Performance

– On x64, instructions executed by hardware

– On IA64, instructions have to be emulatedNew Intel IA-32 EL (Execution Layer) does binary translation of Itanium to x86 to speed performance

Downloadable now – bundled with Server 2003 SP1

179179

Wow64 Components

Wow64.dll:

– provides core emulation infrastructure and thunks for Ntoskrnl.exe entry-point functions; exception dispatching

Wow64win.dll:

– provides thunks for Win32k.sys entry-point functions

Wow64cpu.dll:

– manages thread contexts, supports mode-switch instructions

180180

64-bit ntdll.dll64-bit ntdll.dll

Wow64.dllWow64.dll Wow64win.dllWow64win.dll

Wow64cpu.dllWow64cpu.dll

Win32k.sysWin32k.sysExecutiveExecutive

Kernel Kernel ModeModeUser ModeUser Mode32-bit ntdll.dll32-bit ntdll.dll

32-bit EXE, DLLs32-bit EXE, DLLs

Wow64 Components

181181

Wow64 Limitations

Cannot load 32-bit DLLs in 64-bit process and vice versa

Does not support 32-bit kernel mode device drivers

– Drivers must be ported to 64-bits

– Special support required to support 32-bit applications using DeviceIoControl to driverDriver must convert 32-bit structures to 64-bit

182182

Wow64 Feature Support on 64-bit Windows

Platforms

IA64 x64

16-bit Virtual DOS Machine (VDM) support

N/A N/A

Physical Address Extension (PAE) APIs

N/A Yes

GetWriteWatch() API N/A Yes

Scatter/Gather I/O APIs N/A Yes

Hardware accelerated with DirectX version 7,8 and 9

Software-

Emulation Only

Yes

Wow64 Limitations

183183

Wow64 File Locations

Location of system files– 64-bit system files are in \windows\system32

– 32-bit system files are in \windows\syswow64

– 32-bit applications live in “\Program Files (x86)”

– 64-bit applications live in “\Program Files”

File access to %windir%\system32 redirected to %windir%\syswow64

Two areas of the registry redirected (see next slide)

184184

Wow64 Registry Redirection

Two registry keys have 32-bit sections:– HKEY_LOCAL_MACHINE\Software

– HKEY_CLASSES_ROOT

– Everything else is shared

32-bit data lives under \Wow6432Node

– When a Wow64 process opens/creates a key, it is redirected to be under Wow6432Node

185185

Example: Cmd.exe on 64-bit System

32-bit Cmd.exe process:

186186

Example: Cmd.exe on 64-bit System

64-bit Cmd.exe process:

187187

Four Contexts for Executing Code

Full process and thread context:

User applications

Windows Services

Environment subsystem processes

System startup processes

Have thread context but no “real” process:

Threads in “System” process

188188

Four Contexts for Executing Code

Routines called by other threads/processes:

Subsystem DLLs

Executive system services (NtReadFile, etc.)

GDI32 and User32 APIs implemented in Win32K.Sys (and graphics drivers)

No process or thread context

(“arbitrary thread context”)

Interrupt dispatching

Device drivers

hardware interfaces (buses, I/O devices, interrupts, interval timers, DMA, memory cache control, etc., etc.)

System Service Dispatcher

Task Manager

Explorer

SvcHost.Exe

WinMgt.Exe

SpoolSv.Exe

ServiceControl Mgr.

LSASS

Ob

jec

tM

gr.

WindowsUSER,

GDI

File

Sy

ste

m C

ac

he

I/O Mgr


UserApplication

Subsystem DLLs

System Processes Services Applications

SystemThreads

UserMode

KernelMode

NTDLL.DLL

Device &File Sys.Drivers

WinLogon

Session Manager

Services.Exe POSIX

Windows DLLs

Plu

g a

nd

Pla

y M

gr.

Po

we

rM

gr.

Se

cu

rityR

efe

ren

ce

Mo

nito

r

Virtu

al

Me

mo

ry

Pro

ce

ss

es

&T

hre

ad

s

Lo

ca

lP

roc

ed

ure

Ca

ll GraphicsDrivers

Kernel


(kernel mode callable interfaces)

Co

nfig

ura

-tio

n M

gr

(reg

istry

)

OS/2

Windows

Windows NT/2000/XP/2003 Architecture

190190

Where is the Code?

Kernel32.Dll, Gdi32.Dll, User32.Dll

– export Windows entry points

NtDll.Dll

provides user-mode access to system-space routines

also contains heap manager, image loader, thread startup routine

NtosKrnl.Exe (or NtkrnlMp.Exe)

executive and kernel

includes most routines that run as threads in “system” process

191191

Where is the Code?

Win32K.Sys

– the loadable module that includes the now-kernel-mode Windows code (formerly in csrss.exe)

Hal.Dll

– Hardware Abstraction Library

drivername.Sys

– loadable kernel drivers

192192

The Windows API - Naming, Conventions, Types Windows API principles

Portable programming - the standard C library

Comparing UNIX and Windows programming styles:

– an example program

193193

Windows API - Overview

APIs to Windows systems evolved over time:

– Win16 - introduced with Windows 2.0

– Win32 - introduced with Windows NT, Windows 95

– Win64 (?) - some additions to Win32 introduced with Windows XP 64bit Edition

“Windows API” summarizes all of the above

194194

Windows API - major functionality

File System and Character I/O

Direct File Access and File Attributes

Structured Exception Handling

Memory Management and Memory-Mapped Files

Security

Process Management

Inter-process Communication

Threads and Scheduling, Windows Synchronization

195195

Windows API Principles

System resources are kernel objects referenced by a handle (handle vs. UNIX file descriptors & PIDs)

Kernel objects must be manipulated via Windows API

Objects – files, processes, threads, IPC pipes, memory mappings, events – have security attributes

Windows API is rich & flexible:

– convenience functions often combine common sequences of function calls

Windows API offers numerous synchronization and communication mechanisms

196196

Windows API principles (contd.)

Thread is unit of executions (instead of UNIX process)

– A process can contain one or more threads

Function names are long and descriptive (as in VMS)

- WaitForSingleObject()

- WaitForMultipleObjects()

197197

Windows API Naming Conventions

Predefined data types are in uppercase

– BOOL (32 bit object to store single logical value)

– HANDLE

– DWORD (32 bit unsigned integer)

– LPTSTR

– LPSECURITY_ATTRIBUTE

198198

Windows API Naming Conventions

Prefix to identify pointer & const pointer

– LPTSTR (defined as TCHAR *)

– LPCTSTR (defined as const TCHAR *)

(Unicode: TCHAR may be 1-byte char or 2-byte wchar_t)

– See \$MSDEV\INCLUDE\WINDOWS.H, WINNT.H, WINBASE.H

(MSDEV=C:\Program Files\Microsoft Visual Studio\VC98\)

199199

Differences from UNIX

HANDLEs are opaque (no short integers)– No analogy to file descriptors 0,1,2 in Windows

No distinctions between HANDLE and process ID– Most functions treat file, process, event, pipe identically

Windows API processes have no parent-child relationship– Although the NT kernel maintains this relationship

Windows text files have CR-LF instead of LF (UNIX)

Anachronisms: “long pointer“ (32 bit)– LPSTR, LPVOID

200200

Portability: The Standard C Library

Included in the Windows API

C library contains functions with limited capability to manage OS resources (e.g.; files)

Often adequate for simple programs

Possible to write portable programs

Include files:

– <stdlib.h>, <stdio.h>, <string.h>

201201

Example Application

Sequential file copy:

– The simplest, most common, and most essential capability of any file system

– Common form of sequential processing

Comparing programs:

– Quick way to introduce Windows API essentials

– Contrast different approaches

– Minimal error processing

202202

Sequential File Copy

UNIX:

File descriptors are integers; error value: -1

read()/write() return number of bytes processed,

– 0 indicates EOF

– Positive return value indicates success

close() works only for I/O objects

I/O is synchronous

Error processing depends on perror() & errno (global)

203203

#include <sys/types.h>#include <sys/stat.h>#include <fcntl.h>#include <stdio.h>#include <errno.h>#define BUF_SIZE 256

int main (int argc, char *argv []) {int input_fd, output_fd;ssize_t bytes_in, bytes_out;char rec [BUF_SIZE];if (argc != 3) {

printf ("Usage: cp file1 file2\n");return 1;

}input_fd = open (argv [1], O_RDONLY);if (input_fd == -1) {

perror (argv [1]); return 2;}output_fd =

open(argv[2],O_WRONLY|O_CREAT,0666);if (output_fd == -1) {

perror (argv [2]); return 3;}

/* Process the input file a record at atime. */

while ((bytes_in = read (input_fd, &rec, BUF_SIZE)) > 0) {

bytes_out = write (output_fd, &rec, bytes_in);

if (bytes_out != bytes_in) {perror ("Fatal write error.");return 4;

}}close (input_fd);close (output_fd);return 0;

}

CP Program: UNIX Implementation

204204

File Copy with Standard C Library

Open files identified by pointers to FILE structures– NULL indicates invalid value

– Pointers are „handles“ to open file objects

Call to fopen() specifies whether file is text or binary

Errors are diagnosed with perror() of ferror() Portable between UNIX and Windows Competitive performance Still constrained to synchronous I/O No control of file security via C library

205205

#include <windows.h>#include <stdio.h>#include <errno.h>#define BUF_SIZE 256

int main (int argc, char *argv []) {FILE *in_file, *out_file;char rec [BUF_SIZE];size_t bytes_in, bytes_out;if (argc != 3) {

printf ("Usage: cp file1 file2\n");return 1;

}in_file = fopen (argv [1], "rb");if (in_file == NULL) {

perror (argv [1]);return 2;

}out_file = fopen (argv [2], "wb");if (out_file == NULL) {

perror (argv [2]);return 3;

}

/* Process the input file a record at a time. */

while ((bytes_in = fread (rec,1,BUF_SIZE,in_file)) > 0) {bytes_out =

fwrite (rec, 1, bytes_in, out_file);if (bytes_out != bytes_in) {

perror ("Fatal write error.");return 4;

}}

fclose (in_file);fclose (out_file);return 0;

}

CP Program: C library Implementation

206206

File Copying with Windows API

<windows.h> imports all Windows API function definitions and data types

Access Windows objects via variables of type HANDLE

Generic CloseHandle() function works for most objects

Symbolic constants and flags

– INVALID_HANDLE_VALUE, GENERIC_READ

Functions return boolean values

System error codes obtained via GetLastError()

Windows security is complex and difficult to program

207207

#include <windows.h>#include <stdio.h>#define BUF_SIZE 256

int main (int argc, LPTSTR argv []) {HANDLE hIn, hOut;DWORD nIn, nOut;CHAR Buffer [BUF_SIZE];if (argc != 3) {

printf("Usage: cp file1 file2\n");return 1;

}hIn = CreateFile (argv [1],

GENERIC_READ, FILE_SHARE_READ, NULL,OPEN_EXISTING,FILE_ATTRIBUTE_NORMAL,NULL);

if (hIn == INVALID_HANDLE_VALUE) {printf ("Input file error:%x\n",

GetLastError ());return 2;

}

hOut = CreateFile (argv [2],GENERIC_WRITE, 0, NULL,CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);

if (hOut == INVALID_HANDLE_VALUE) {printf("Output file error: %x\n",

GetLastError ());return 3;

}while (ReadFile (hIn, Buffer,

BUF_SIZE, &nIn, NULL) && nIn > 0) {

WriteFile (hOut, Buffer,nIn,&nOut,NULL);if (nIn != nOut) {

printf ("Fatal write error: %x\n", GetLastError ());

return 4;}

}CloseHandle (hIn);CloseHandle (hOut);return 0;

}

CP Program - Windows API Implementation

208208

CP with Windows API Convenience Functions

#include <windows.h>#include <stdio.h>

int main (int argc, LPTSTR argv []){

if (argc != 3) {printf ("Usage: cp file1 file2\n"); return 1;

}if (!CopyFile (argv [1], argv [2], FALSE)) {

printf ("CopyFile Error: %x\n", GetLastError ()); return 2;

}return 0;

}

Convenience functions may improve performance

– Programmer does not need to be concerned about arbitrary buffer sizes

– OS manages speed vs. space tradeoffs at runtime

Thoughts Change Life意念改变生活

Date post:	10-Jan-2016
Category:	Documents
Upload:	johnna
View:	39 times
Download:	0 times

Windows OS Principles

Documents