IBM Labs in Haifa
Automatic Noise Simulation in UDP
Eitan Farchi, Yoel Krasny, Yarden Nir
IBM Labs in Haifa
Outline
� UDP transport
� Difficulties in testing UDP based application
� ConTest/UDP – Automatic noise simulation in UDP
� Comparing ConTest/UDP to white box testing in testing of group communication
IBM Labs in Haifa
� Transport protocols over IP
� TCP� Connection based� Reliable� Analogous to making a telephone call� Usage examples: FTP, HTTP
� UDP� Not connection based� Unreliable� Analogous to sending a letter� Usage examples: Ping, Multicast, Stocks Quote server
� Java programs use TCP or UDP through the java.net package
UDP VS TCP
Java.netTransport(TCP/UDP)Network
(IP)Link
(device driver)
IBM Labs in Haifa
Link
IP
UDP
Application
Java
Link
IP
UDP
Application
Java
NetworkNetwork
1
Unreliable flow in UDP – Losing messages
AB
IBM Labs in Haifa
Link
IP
UDP
Application
Java
Link
IP
UDP
Application
Java
NetworkNetwork
Unreliable flow in UDP – Duplicating messages
22
2 2
2
AB
IBM Labs in Haifa
Link
IP
UDP
Application
Java
Link
IP
UDP
Application
Java
NetworkNetwork
Unreliable flow in UDP – Changing the order
44
2 42 3
3
4334
3
AB
IBM Labs in Haifa
Link
IP
UDP
Application
Java
Link
IP
UDP
Application
Java
NetworkNetwork
Unreliable flow in UDP – Delaying messages
55
2 42 3 5
AB
IBM Labs in Haifa
Testing UDP based applications
� A UDP based application must handle network noises�For example – Any reliable component on top of UDP
� Testing scenarios of network noise is compulsory
� The probability to encounter such scenarios during testing in normal environment is low
IBM Labs in Haifa
Automatic Simulation of Noise by ConTest/UDP
� ConTest/UDP addresses the difficulties in testing UDP applications
� ConTest/UDP simulates UDP noise and network failures
� ConTest/UDP creates an intermediary layer above the Java API
� ConTest/UDP is part of the ConTest suite, that aims to increase the probability of exposing concurrent bugs
IBM Labs in Haifa
ConTest/UDP Options
� Remote Nodes – apply noise on messages from/to certain nodes
� Direction – incoming and/or outgoing messages
� Mode �No noise�Delay – packets are accumulated by the tool�Block – packets are lost�Random noise – packets are tampered randomly
� Strength – determines the random noise level
IBM Labs in Haifa
Mode = DelayDirection = Out
Link
IP
UDP
Application
Java
Tool
Link
IP
UDP
Application
Java
ConTest/UDP
Link
IP
UDP
Java
Tool
Link
IP
UDP
Application
Java
ConTest/UDP
Delaying messages with ConTest/UDP
NetworkNetwork
123
AB
IBM Labs in Haifa
Mode = No-Noise
Link
IP
UDP
Application
Java
Tool
Link
IP
UDP
Application
Java
Link
IP
UDP
Java
Link
IP
UDP
Application
Java
Delaying messages with ConTest/UDP
NetworkNetwork
123
1 2 3 4
4
AB
ConTest/UDP ConTest/UDP
IBM Labs in Haifa
Mode = BlockDirection = Out
Link
IP
UDP
Application
Java
Tool
Link
IP
UDP
Application
Java
Link
IP
UDP
Java
Link
IP
UDP
Application
Java
Losing messages with ConTest/UDP
NetworkNetwork
123 XXX
AB
ConTest/UDP ConTest/UDP
IBM Labs in Haifa
Mode = No-Noise
Link
IP
UDP
Application
Java
Tool
Link
IP
UDP
Application
Java
Link
IP
UDP
Java
Link
IP
UDP
Application
Java
Losing messages with ConTest/UDP
NetworkNetwork
123 XXX
4
4
AB
ConTest/UDP ConTest/UDP
IBM Labs in Haifa
Outline
� UDP transport
� Difficulties in testing UDP based application
� ConTest/UDP – Automatic noise simulation in UDP
� Comparing ConTest/UDP to white box testing of a group communication application
IBM Labs in Haifa
� Multi-point to multi-point communication for a dynamic group of processes
� Supplies a group membership service that tracks the set of groupmembers – view
� Supports various guarantees for its messages delivery:� FIFO delivery� Virtual Synchrony� Total Order
Group Communication
IBM Labs in Haifa
� Multi-point to multi-point communication for a dynamic group of processes.
� Supplies a group membership service that tracks the set of groupmembers – view
� Supports various guarantees for its messages delivery:� FIFO delivery� Virtual Synchrony� Total Order
Group Communication
A
BC
View Id 1{A}View Id 2{A, B}View Id 3{A, B, C}
IBM Labs in Haifa
Transport layer
Membership Layer� VRI – Java implementation based on Clue (AS400)
� All layers support a generic interface
� Every layer is responsible for a certain taskVirtual Synchrony
Application Layer
Versatile Replication Infrastructure
IBM Labs in Haifa
� The Transport Layer provides reliable Multicast delivery over UDP
� Testing the reliability is possible using Contest/UDP and a Verification Layer
� Running with the Random mode provides powerful automatic testing
Testing the reliability of the Transport Layer
Transport Layer
Virtual Synchrony
Application Layer
Membership Layer
ConTest/UDP
Verification Layer
IBM Labs in Haifa
VRI
UDP
D
VRI
UDP
C
VRI
UDP
B
VRI
UDP
A
{A, B, C, D}
{A, B, C, D} {A, B, C, D}
{A, B, C, D}
Network connection is broken
X
Testing partitions – required scenario
IBM Labs in Haifa
VRI
UDP
D
VRI
UDP
C
VRI
UDP
B
VRI
UDP
A
{A, B, C, D}
{A, B, C, D} {A, B, C, D}
{A, B, C, D}
X
After some time the group should be
partitioned
Testing partitions – required scenario
{A, B}
{A, B}
{C, D}
{C, D}
IBM Labs in Haifa
VRI
ConTest
UDP
B
VRI
ConTest
UDP
C
VRI
ConTest
UDP
D
VRI
ConTest
UDP
A
{A, B, C, D}
{A, B, C, D}
{A, B, C, D}
{A, B, C, D}
Connection is brokenBlock
(C, D)
Block(C, D)
Block(A, B)
Block(A, B)
Testing partitions – simulated scenario
IBM Labs in Haifa
{A, B, C, D} {A, B, C, D}{A, B} {C, D}
After some time the group should be
partitioned
{A, B, C, D} {A, B, C, D}{A, B} {C, D}
Testing partitions – simulated scenario
VRI
ConTest
UDP
A
VRI
ConTest
UDP
B
VRI
ConTest
UDP
C
VRI
ConTest
UDP
D
IBM Labs in Haifa
� Follows the white box testing strategy
� Responsible for injecting noise into the VRI stack
� Implements the generic interface of a layer in the stack
The Manipulator Layer
Transport Layer
Virtual Synchrony
Application Layer
Membership Layer
Manipulator
Manipulator
IBM Labs in Haifa
� This scenario illustrates that sometimes it is hard to create certain scenario with ConTest/UDP.
� When a new view is established in a certain member C before another member B, C should queue view messages from B until the new view is established
View message synchronization
IBM Labs in Haifa
NewView
VRI
Transport
A
NewView
View N{A, B, C, D}
View N{A, B, C, D}
View N{A, B, C, D}
View N{A, B, C, D}
View message synchronization – required scenario
VRI
Transport
B
VRI
Transport
C
VRI
Transport
DX
IBM Labs in Haifa
View N{A, B, C, D}
View N{A, B, C, D}
View N+1{A, B, C}
View N+1{A, B, C}
VRI
Transport
A
VRI
Transport
B
VRI
Transport
D
View N{A, B, C, D}
View N{A, B, C, D}
NewView
View message synchronization – required scenario
VRI
Transport
C
X
IBM Labs in Haifa
VRI
Transport
C
View N{A, B, C, D}
View N{A, B, C, D}
View N+1{A, B, C}
View N+1{A, B, C}
VRI
Transport
A
VRI
Transport
B
NewView
View N{A, B, C, D}
11
1
1
View message synchronization – required scenario
VRI
Transport
D
View N{A, B, C, D}
X
IBM Labs in Haifa
View N{A, B, C, D}
View N+1{A, B, C}
VRI
Transport
C
View N+1{A, B, C}
View N+1{A, B, C}
NewView
NewView
1
1
View message synchronization – required scenario
VRI
Transport
A1
VRI
Transport
B
VRI
Transport
D
View N{A, B, C, D}
X
IBM Labs in Haifa
VRI
Transport
B
VRI
Transport
C
NewView
NewView
View N{A, B, C, D}
View N{A, B, C, D}
View N{A, B, C, D}
View message synchronization – Simulated scenario
Manip
View N{A, B, C, D}
VRI
Transport
DX
VRI
Transport
A
IBM Labs in Haifa
View N{A, B, C, D}
View N+1{A, B, C}
VRI
Transport
B
View N{A, B, C, D}
View N+1{A, B, C}
VRI
Transport
A
VRI
Transport
C
View N{A, B, C, D}
View message synchronization – Simulated scenario
Manip
Delay New View Call
View N{A, B, C, D}
VRI
Transport
DX
IBM Labs in Haifa
VRI
Transport
C
Manip
View N{A, B, C, D}
View N+1{A, B, C}
VRI
Transport
A
View N{A, B, C, D}
View N+1{A, B, C}
View N{A, B, C, D}
11
1
1
View message synchronization – Simulated scenario
VRI
Transport
B
View N{A, B, C, D}
VRI
Transport
DX
IBM Labs in Haifa
View N{A, B, C, D}
View N{A, B, C, D}
View N+1{A, B, C}
View N+1{A, B, C}
VRI
Transport
A
VRI
Transport
B
Transport
C
View N{A, B, C, D}
1
1
1
View message synchronization – Simulated scenario
Manip
VRIDelay Over
View N{A, B, C, D}
VRI
Transport
DX
IBM Labs in Haifa
� ConTest/UDP - provides a powerful utility to test UDP-based programs
� ConTest/UDP - easy to use with no effect on the application code
� ConTest/UDP - not always suitable to test application specific scenarios�Such scenarios require specifically tailored testing code
Conclusions
