1
University of Pittsburgh Manas Saksena 1
Real-Time Systems
Manas SaksenaUniversity of Pittsburgh
University of Pittsburgh Manas Saksena 2
What is a real-time system?
♦ A system that maintains a continuous andtimely interaction with its environment
Throttle Value
Cruise -ControlSystem
BrakeAcceleratorWheel RotationDesired SpeedCommands
University of Pittsburgh Manas Saksena 3
Real-Time Systems
♦ Real-time computer system�corrects depends on
− when the results are produced (timeliness)�in addition to
− what results are produced
♦ Embedded System�The computer system is part of a larger system�Examples:
− automobile control system(s)
− air-traffic control
University of Pittsburgh Manas Saksena 4
Timing Constraints
♦ Speed Control�sample sensor value (speed) every ‘100’ ms,
− periodic activity
− inter-arrival time (period)�output actuator command within ‘60’ ms
− deadline for each arrival
♦ Brake Press�stop cruise control within ‘50’ ms
− arrival rate: irregular
− deadline
University of Pittsburgh Manas Saksena 5
Hard vs Soft Real-Time
♦ Hard Real-Time�Missing deadlines means system failure
♦ Safety Critical�Missing deadlines can cause serious loss
♦ Soft Real-Time�Deadlines may be missed occasionally
♦ Most real systems have a mix of hard and soft real -time components♦ The distinction between hard and soft real -time is somewhat subjective
♦ Soft real-time is not non real -time
University of Pittsburgh Manas Saksena 6
Value Functions
deadline
Hard real-time
Soft real-timevalue
2
University of Pittsburgh Manas Saksena 7
Traditional Real-Time Applications
♦ Avionics
♦ Data Acquisition
♦ Digital Controllers
♦ Defense Systems
♦ Industrial Process Control
University of Pittsburgh Manas Saksena 8
Emerging Embedded Real -Time Applications
♦ Cell-phones, VoIP phone, PDA’s
♦ MP3 players
♦ Set-top boxes, Game Consoles
♦ Automotive Systems
♦ Network Boxes
♦ On Demand Servers
♦ Embedded Devices
University of Pittsburgh Manas Saksena 9
Plantsensor
Control-rawcomputation
A/D
A/D
D/AReference input
actuator
Controller
Real-time Control Systems
University of Pittsburgh Manas Saksena 10
Real-Time Software DesignBasics
University of Pittsburgh Manas Saksena 11
Software Structure: State Transition Diagrams
initialization
external trigger?
Take actions
Change system state
ISR: to set/clearevents
University of Pittsburgh Manas Saksena 12
State Machines
manual
automaticHandleEvent
InitializeObject
TerminateObject
Wait forEvent cruise/startControl()
timeout/doControl()
shutdown/
initial state
final state
state machine
state
trigger
action expression
transition
3
University of Pittsburgh Manas Saksena 13
State Transition Diagrams: Design
manual
automatic
University of Pittsburgh Manas Saksena 14
Schedulability Analysis
♦ Are my timing requirements being met?♦ Real-Time System Model�
A set of events− Maximum arrival rate for each event�
A handler for each event
− Maximum execution time for each handler�A deadline for each event
− Finish time (e) – Arrival time (e) <= Deadline
Response Time
University of Pittsburgh Manas Saksena 15
Software Structure: Cyclic Executives
♦ Replace Interrupts by Polling�More predictable behavior
♦ Static Cyclic Schedule�Repeated at run-time
♦ Implemented by a single loop
A B C D A B C D
��������� �������������������� ���������� ������� �! �������#"$ ���%
University of Pittsburgh Manas Saksena 16
Real-Time System Software
♦ Concurrency�real-world events occur concurrently
− multi-tasking simplifies software structure
♦ Timed Events�need timers to trigger timed events
− build timer support
♦ Timeliness of Response�Different events have different requirements�How to structure software to satisfy all the requirements
University of Pittsburgh Manas Saksena 17
Real-time System Development
Target system
Application tasks
Real-time OS
HardwareInputoutput
Development host
Pentium PC stationSUN workstation
NT OS(Solaris)
Compiler, debugger,loader, simulator,
shell monitor
RS-232Ethernet
University of Pittsburgh Manas Saksena 18
Cross-Development for Embedded Systems
♦ Use the host to�edit, compile, and build application programs�configure the target
♦ At the target embedded system, use tools to�load, execute, debug, and monitor (performance and timing)
♦ Iterative process
RS-232 (console)
Ethernet (load, debug, monitor)
Target (MBX860)
RTOS (vxWorks)
application tasks
PC Host + development tools
4
University of Pittsburgh Manas Saksena 19
Real-Time Operating Systems
andReal-Time Software Design
Manas Saksena
University of Pittsburgh Manas Saksena 20
Traditional Real-Time OS Requirements
♦ Multi-Tasking kernel�Fixed priority, preemptive scheduling
♦ Inter-task communication�Semaphores, Message Queues, etc.
− Priority Inheritance
♦ Timers♦ Device Drivers♦ Interrupt Management
Do this with low and predictable overheads.
University of Pittsburgh Manas Saksena 21
Real-time OS
● Functions:&task management,
− scheduling, dispatcher− communication (pipe, queue)− synchronization (semaphore,
event)
'memory management'time management 'device driver'interrupt service
Interruptdispatch
Interruptservice
Time service &events
Services (create thread, sleep, notify, send,…)
Scheduling &
dispatcher
System calls(trap)
External interrupt
Timer interrupt
kernel
Task execution
University of Pittsburgh Manas Saksena 22
Schedulable Entities
Scheduler Task Queue Signal Queue
Event Queue
Task’s Thread of Control
Kernel
Application
University of Pittsburgh Manas Saksena 23
Real-Time OS: Examples
♦ Traditional Real-Time OSes�VxWorks, pSOS, QNX, LynxOS
♦ Minimal Real-Time OSes�ThreadX, Ercos, Nucleus
♦ General Purpose OS Made Real -Time�Solaris, NT, Linux
♦ There are many other real -time operating systems.♦ Many developers roll out their own real -time operating system
University of Pittsburgh Manas Saksena 24
Response Time Factors
♦ Execution Time�Amount of time taken in the absence of resource contention
♦ Higher Priority Non Schedulable Entities�Interrupt Handlers
♦ Scheduling�Time taken by higher priority work
♦ Shared Resources�Time taken by lower priority work
− Often, due to contention on some shared resource
5
University of Pittsburgh Manas Saksena 25
Scheduling
♦ Tasks�Primary scheduling entities�Scheduled by the kernel
♦ Events or Messages�Done within the task�Usually makes sense with “event -driven” tasks
♦ Signals�Interrupts to tasks�Useful for activating tasks (e.g., timer expiry)
University of Pittsburgh Manas Saksena 26
Real-Time Software Design
♦ Task Centric Design�Tasks are primary units for scheduling�Application = Set of Tasks�Task Activation by signals
♦ Event Centric Design�Events are primary units for scheduling�Application = Set of Events and their Responses�Single Task or Multiple Tasks
University of Pittsburgh Manas Saksena 27
Task Centric Design
♦ Task Scheduling�
Preemptive or Non-Preemptive�Often, priority based mechanism
♦ Task Priorities�
Application Controlled
Read Inputs
Initialize Task
Terminate Task
Wait Control
Perform Work
Write Outputs
University of Pittsburgh Manas Saksena 28
Event Centric Design - Single Task
♦ Event Scheduling�
Non-Preemptive�Priority Based
♦ Event Priorities�
Application Controlled
ProcessEvent
Initialize
Terminate
Wait forEvent
University of Pittsburgh Manas Saksena 29
Event Centric Design - Multiple Tasks
♦ Task Priorities and Scheduling�Fixed, or Dynamic�Preemptive Scheduling
ProcessEvent
Initialize
Terminate
Wait forEvent
ProcessEvent
Initialize
Terminate
Wait forEvent
University of Pittsburgh Manas Saksena 30
Task Centric Design
Manas Saksena
6
University of Pittsburgh Manas Saksena 31
Abstract Design Model
♦ Identify Events�Interrupts�Timers
♦ Event Handlers�Code Segments
− call these actions
eventaction
task
University of Pittsburgh Manas Saksena 32
Implementing Tasks
♦ Two Common Ways�Interrupt Handlers�Threads
− Need an RTOS that supports threads (also called tasks in many RTOSes)
♦ Interrupt Handlers�fast response times�difficult to program correctly
♦ Typically, mix of both�often IH will do some minimal work and then wake up a task to the real work
University of Pittsburgh Manas Saksena 33
Multi-Tasking
♦ Communications�Shared Memory
− Semaphores/Mutexes to Control Access�Messages
− Message Queues
♦ Scheduling�Priorities
− IH may be viewed as higher priority tasks�Preemptive Priority Scheduling
− kernel is usually fully preemptive
University of Pittsburgh Manas Saksena 34
Key Concepts
♦ Tasks�Encapsulate thread of control�Schedulable Unit�Each task implements “one” function
♦ Protected Objects�Encapsulate Shared Data (Resource)�Mutually exclusive operation of Object Methods
University of Pittsburgh Manas Saksena 35
Response Time Components
♦ Execution�Execution time of the task itself, running alone
♦ Preemption�Execution delayed by higher priority tasks
♦ Blocking (Priority inversion)�Execution delayed by lower priority tasks
HIGHER PRIORITY TASK
MY TASK
LOWER PRIORITY TASK
P B
PREEMPTIONTIME BLOCKING
TIME
University of Pittsburgh Manas Saksena 36
Execution and Preemption
time
prio
rity
7
University of Pittsburgh Manas Saksena 37
Preemption Analysis
♦ Each higher priority task has a preemption effect
♦ Depends on Number of Arrivals and
Execution Time of the Higher Priority Task
jjj CRsNumArrivalRI ×= )()(
Number of Arrivals of Task “j” in a window of interval R
Response Time
Preemption Effect from Task “j”
University of Pittsburgh Manas Saksena 38
Unbounded Priority Inversion
High and low priority tasks share a resource – This can lead to UNBOUNDED Blocking
s1 s1 s1
s1
s1Normal execution Execution in Critical Section
Unbounded Priority Inversion
τhigh
τlow
S1
τhigh
τmedium
τlow
τhighestAttempts to lock S 1
University of Pittsburgh Manas Saksena 39
Highest Locker Protocol♦ High and low priority tasks share a
critical section♦ Ceiling Priority of CS = Highest
Priority Task that can use the CS♦ Execute CS at Ceiling priority
s1Normal execution Execution in Critical Section
τhigh
τlow
S1
s1
s1
Ready
Ready
s1
τhigh
τmedium
τlow
τhighest
University of Pittsburgh Manas Saksena 40
Priority Inheritance Protocol
♦ High and low priority tasks share a critical section
♦ Inherit priority of blocked task in critical section
s1
s1Normal execution Execution in Critical Section
τhigh
τlow
S1
s1 s1
Attempts to lock S 1
Ready
s1
τhigh
τmedium
τlow
τhighest
University of Pittsburgh Manas Saksena 41
τ1
S1
S2
τ2
τ3
• Same task set as sample problem• Assumes priority inheritance used• May need to wait for all possible critical sections
S2
Normal execution Execution in Critical Section using S1
Execution in Critical Section using S2S2
S1
S1
Attempts to lock S 1
S1
Attempts to lock S 2
S2
S2S1τ1
τ2
τ3
Example of Chained Blocking
University of Pittsburgh Manas Saksena 42
Blocking Analysis
♦ Common Property�A lower priority task can cause blocking only while it is in a critical section
♦ Highest Locker�The critical section must have a ceiling priority that is higher (or equal) than task priority�Only one such CS can cause blocking
♦ Priority Inheritance Protocol�Multiple CS can cause blocking in a “chained manner”�The effect is sum of the largest such chain
8
University of Pittsburgh Manas Saksena 43
Response Time Analysis
(∈
++=)(
)(ihpj
ijiii RIBCR
Execution time of task
Blocking time from lower priority tasks (priority inversion)
Response Time
Preemption from higher priority tasks
Assumptions
♦ No Self Interference
♦ Bounded Blockingii TR ≤
University of Pittsburgh Manas Saksena 44
Design/Implementation Choices
♦ Non-Preemptive Scheduling�Similar Analysis, but consider that
− No preemption effect once task is given CPU− Blocking effect due to non -preemptive execution
♦ Resource Contention Protocols�Different protocols have different blocking effects
♦ Priority Assignment�Preemption and Blocking Effects Depend on Task Priorities
)Task Response times depend on all these three factors.)These represent “design/implementation” choices. )Functional Correctness does (should) not depend on the choice.
University of Pittsburgh Manas Saksena 45
Linux as a Real -Time Operating System
Manas Saksena
University of Pittsburgh Manas Saksena 46
Why Linux?
♦ Reliable, Full-featured Operating System�Rich multi-tasking support�Security, Protection�Networking Support
− TCP/IP, RSVP, SIP, MPLS, H.323�Multimedia Support
− JPEG, MPEG, GSM
♦ Standard, Known Environment and API’s�Unix Lineage
− Familiar environment for many users/developers�POSIX Compliance
University of Pittsburgh Manas Saksena 47
Why Linux?
♦ The Cost Factor�Free run -time royalties
♦ The Open Source Factor �A global team of programmers enhancing the environment literally all the time�Availability of libraries, tools, and device drivers�Source Code Access allowing “peeking inside the hood” (and customizing as necessary)
♦ The Popularity Factor�Excellent textbooks and documentation
University of Pittsburgh Manas Saksena 48
Why Linux?
♦ Small Embedded Systems�Modular Kernel, possible to configure the kernel to suitable size�Customizable Root File System�BusyBox Utilities
♦ High-End Embedded Systems�High-Availability�Clustering�SMP Support
9
University of Pittsburgh Manas Saksena 49
Linux API: Tasking
♦ Process�Encapsulates a thread of control and an address space
− Address space may be shared giving threads in effect�
Schedulable Entity
♦ Threads�Are processes to the Linux kernel
− Scheduled by the Linux kernel�Can be created such that they share the address space with the parent process, effectively giving threads
University of Pittsburgh Manas Saksena 50
Linux API: POSIX, SVR4, BSD
♦ POSIX 1003.1.b (Real-Time Extensions)�Priority Scheduling�Memory Locking�Clocks and Timers�Real-Time Signals
♦ POSIX 1003.1.c (Thread Extensions)�Using pthreads library�Thread creation, destruction, etc.�Mutexes, Condition Variables
♦ SVR4 IPC�Shared Memory�Semaphores
♦ Networking: �BSD Sockets
University of Pittsburgh Manas Saksena 51
Linux Internals Architecture
DeviceDrivers
Modules
Core Mechanisms
Process Scheduler
vfsmm
ipc net
University of Pittsburgh Manas Saksena 52
Linux Internals: Scheduling
♦ Schedulable Entities�Processes
− Real-Time Class: SCHED_FIFO or SCHED_RR− TimeSharing Class: SCHED_OTHER�
Real-Time processes have− Application defined priority
− Higher priority than time-sharing processes
♦ Non Schedulable Entities�Interrupt Handlers
− Have priorities, and can be nested�Bottom Halves & Task Queues
− Run on schedule, ret from system call, ret from interrupt
University of Pittsburgh Manas Saksena 53
Linux and Real-Time: Problems
♦ Timer Granularity�Many real-time tasks are driven by timer interrupts�In Standard Linux, the timer is set to expire at 10 ms intervals
♦ Scheduler Predictability�Linux scheduler keeps tasks in an unsorted list�Requires a scan of all tasks to make a scheduling decision�Scales poorly as number of tasks increases, and is especially poor for real -time performance
University of Pittsburgh Manas Saksena 54
Linux Internals: Priority Inversion
♦ Non Preemptible Kernel�A process running in the kernel cannot be preempted by another (higher) priority process�Introduces, unwanted priority inversion
♦ Non Schedulable Entities�Interrupt Handlers�Bottom Half Handlers
♦ Resource Sharing�No mechanism to bound priority inversion in the presence of resource sharing
10
University of Pittsburgh Manas Saksena 55
The Real-Time Linux Challenge
How to leverage the advantages of Linux,while making it suitable for real -time systems?
University of Pittsburgh Manas Saksena 56
Approaches to Real-Time Linux
♦ Approaches limiting Real -time and Non Real -time Task Interactions�
Compliant Kernel Approach
− LynxOS/Blue Cat Linux�Thin Kernel Approach
− RTLinux/RTAI
♦ Approaches that integrate Real -time and Non Real -time tasks�
Core Kernel Approach
− TimeSys Linux/RT, Hard Hat Linux�Resource Kernel Approach
− TimeSys Linux/RT
University of Pittsburgh Manas Saksena 57
Approaches to Real-Time Linux
�Compliant Kernel Approach
�Dual Kernel Approach
�Core Kernel Approach
�Resource Kernel Approach
University of Pittsburgh Manas Saksena 58
Compliant Kernel Approach
Linux Kernel(Embedded Applications)
Real-Time Kernel(Real-Time Applications)
Linux System Call API Linux System Call API
Linux Development ToolsAnd Environment
Linux Development ToolsAnd Environment
University of Pittsburgh Manas Saksena 59
Compliant Kernel Approach
♦ Basic Claim�Linux is defined by its API and not by its internal implementation�The real-time kernel is a non Linux kernel
♦ Implications�No benefits from the Linux kernel�Not possible to benefit from the Linux kernel evolution�Not possible to use Linux hardware support�Not possible to use Linux device drivers
University of Pittsburgh Manas Saksena 60
Compliance
♦ 100% Linux API�Support all of Linux kernel API
♦ Implications�Any Linux application can run on real -time kernel
− Development can be done on Linux Host, with rich set of host tools for development�
All Linux libraries are trivially available to run on real-time kernel
− Third party software �Achieving 100% Linux API is non -trivial
− Consider the amount of effort put on Linux kernel development
11
University of Pittsburgh Manas Saksena 61
Approaches to Real-Time Linux
�Dual Kernel Approach
�Compliant Kernel Approach
�Core Kernel Approach
�Resource Kernel Approach
University of Pittsburgh Manas Saksena 62
The Thin Kernel Approach
Hardware
Real-Time Kernel (RT-Linux or RTAI)
Real-TimeTask
Real-TimeTask
Real-TimeTask
Linux Kernel
LinuxProcess
LinuxProcess
User-Level
Kernel-Level
University of Pittsburgh Manas Saksena 63
Linux Preemptability
♦ Linux kernel is run as the lowest priority task in the real-time kernel�
Requires small changes to the Linux kernel
♦ Linux kernel can be preempted at any time by the real-time kernel
♦ Real-Time kernel is a small multi -tasking kernel providing multi -tasking, inter-task communication and synchronization, and a fixed priority scheduler
Very tight response times for rt-tasks
University of Pittsburgh Manas Saksena 64
Programming Model
♦ Real-Time Tasks�Run in kernel�Limited API, restricted to what is provided by the real-time kernel
♦ Non Real-Time Tasks�Run in user-space under Linux
Real-TimeTasks
Non Real-TimeTasks
University of Pittsburgh Manas Saksena 65
Programming Model
♦ Memory Protection�Not available for real -time tasks�A single errant real -time task can bring the system down
♦ Duplication of Functionality�Currently:
− Multi-tasking, task scheduling, inter -task communication and synchronization�
Future Extensibility− E.g., IP protocol stack, file system, …
♦ Starvation of non real -time tasks
University of Pittsburgh Manas Saksena 66
Soft and Dynamic Real -Time
♦ Static Decomposition into�Real-time parts, and non real -time parts
♦ Consider�Real-time often involves many shades of gray (soft real -time)�The soft/hardness of real -time may be dynamically decided
♦ Implications�Support soft real-time tasks in real-time kernel
− Limited API�Support soft real-time tasks in Linux kernel
− No performance guarantees
12
University of Pittsburgh Manas Saksena 67
Approaches to Real-Time Linux
�Compliant Kernel Approach
�Dual Kernel Approach
�Core Kernel Approach
�Resource Kernel Approach
University of Pittsburgh Manas Saksena 68
Core Kernel Approach
♦ Basic Ideas�Make the kernel more suitable for real -time�Ensure that the impact of changes is localized so that
− Kernel upgrades can be easily incorporated− Kernel reliability and scalability is not
compromised
♦ Mechanisms�Static Configuration
− Can be configured at compile time�Dynamic Configuration
− Using loadable kernel modules
University of Pittsburgh Manas Saksena 69
Preemptive Kernel
♦ Linux kernel is non -preemptive�Simplifies synchronization in the kernel since kernel data structures that are not touched by interrupts need not be locked. �Introduces potentially large priority inversions
♦ But, Linux is already SMP safe�That is, the hooks needed to make it preemptive are already there.
♦ Solution�Use SMP lock hooks to make the kernel preemptible
University of Pittsburgh Manas Saksena 70
Minimize Non-Schedulable Work
♦ Schedulable Entities�Processes or Threads
♦ Non Schedulable Entities�Interrupt Handlers�Bottom Halves
♦ Solution�Use schedulable entities (interrupt threads) to do the bulk of the work�These can be prioritized according to application requirements
University of Pittsburgh Manas Saksena 71
Priority Inheritance
♦ A lower priority process holds a resource needed by a higher priority process�
Can lead to unbounded priority inversion, unless some form of priority inheritance is used.
♦ Solution�Provide a kernel mutex implementation that supports priority inheritance�Use the mutex for locks in the kernel�Export the mutex to user level processes using system calls
University of Pittsburgh Manas Saksena 72
High Accuracy Timers
♦ Linux uses a periodic heartbeat timer (by default 10 ms) to implement timers.�
Provides poor accuracy�Increasing the frequency improves accuracy, but increases overheads
♦ Solution�Use a one -shot “hardware” timer to support all software timers
• Use a 10 ms periodic timer to do regular Linux timer work�
Can provide high accuracy at a low overhead
13
University of Pittsburgh Manas Saksena 73
Predictable Scheduler
♦ Current Linux Scheduler�Does not scale well as the number of processes increases
♦ Real-Time Scheduler�Predictable, Low Overhead
♦ Solution(s)�Separate real-time and non real -time
− Non real-time processes do not effect real -time scheduling behavior�
Implement a better scheduler
University of Pittsburgh Manas Saksena 74
Core Kernel Approach
♦ Allows the use of most if not all existing Linux primitives, applications, and tools. �
Need to avoid primitives that can take extended time in the kernel
♦ Allows the use of most existing device drivers written to support Linux. �
Need to avoid poorly written drivers that unfairly hog system resources
♦ Robustness and Reliability�Core kernel modifications can effect robustness, but source is available
University of Pittsburgh Manas Saksena 75
URLs
♦ www.embedded.com�Embedded Systems Conference
− papers available online
♦ http://cs-www.bu.edu/pub/ieee-rts/Home.html�IEEE Real-Time Technical Committee Page
− many useful links