Post on 19-Jan-2016
transcript
NETWORK PROGRAMMINGINTRO TO DISTRIBUTED SYSTEMSFALL 2015
L3-socketDongsu Han
Some material taken from publicly available lec-ture slides including Srini Seshan’s and David Anderson’s
Today’s Lecture
Lecture Transport protocols: UDP and TCP Version control using SVN
TA-led programming session Editor, socket programming Debugging with gdb
Announcement Programming Assignment 1 is out. (Due in two weeks-- 추석 )
Review of Previous Lectures
What is a distributed system? (L1) Definition Example
What are some of the important decisions made in the In-ternet architecture? (L2) Narrow waist: Network provides minimal functionality (best-ef-
fort) Why?: to connect existing networks that are heterogeneous
Stateless architecture: no per-flow state inside network Why? For survivability
Principles
P1: IP over all network (narrow waist) P2: Layering Principle 3: Fate Sharing Principle 4: Soft-state Principle 5: End-to-End Argument Next lecture Principle 6: network layer provides one simple
service: best effort datagram (packet) delivery
Principle 4: Soft-state5
Soft-state (as oppose to hard-state) Announce state Refresh state Timeout state
Penalty for timeout – poor performance Robust way to identify communication flows (firewall)
Possible mechanism to provide non-best effort service Helps survivability
Goal 2: Types of Service6
Principle 6: network layer provides one simple service: best effort datagram (packet) delivery All packets are treated the same
Network layer (IP) provides “best effort” service. Relatively simple core network elements
Other functions are implemented at the end-host at the transport layer (TCP/UDP) Building block from which other services (such as reliable data stream)
can be built Contributes to scalability of network
Issues: No QoS support assumed from below Hard to implement without network support QoS is an ongoing debate…
Summary
Successes: IP on every-thing!
Drawbacks…
but perhaps they’re to-tally worth it in the con-text of the original Inter-net. Might not have worked without them!
“This set of goals might seem to be nothing more than a checklist of all the desirable network features. It is impor-tant to understand that these goals are in order of impor-tance, and an entirely dif-ferent network architecture would result if the order were changed.”
7
8
Transport Protocols
Lowest level end-to-end proto-col. Header generated by sender is in-
terpreted only by the destination Routers view transport header as
part of the payload; they are not supposed to see the transport header.
7
6
5
7
6
5
Transport
IP
Datalink
Physical
Transport
IP
Datalink
Physical
IP
router
2 2
1 1
What does the transport layer do?
No per-flow (connection) state in the network Transport layer maintains the connection state. Transport
layer identifies an end-to-end connection.
How does the transport layer identify an end-point (ap-plication)?
De-multiplexing: Port numbers
Network layer gets you to the host. A port identifies the sending or receiving socket (appli-
cation). A socket is one endpoint of a two-way communication
between two programs running on the network. A socket is bound to a port number so that the transport layer can identify the application that data is destined to be sent.
Identifying two end-points: An endpoint = <IP address, Port #> Source endpoint = <SrcIPAddr, SrcPort> destination endponit = <DestIPAddr, DestPort>
11
Protocol and Port Demultiplexing
Multiple choices at each layer Applications open a socket (identified by the port
number)
Port 80 Port 22 Port 22Port 80
TCP UDP
IP
NET1 NET2 NETn…
TCP/UDPIP
Port Number= 80
Network
Protocol Field= TCP
Type Field=IP
12
Server and Client
TCP/UDP
IP
Ethernet Adapter
Server
TCP/UDP
IP
Ethernet Adapter
Clients
Server and Client exchange messages over the net-work through a common Socket API
Socket API
hardware
kernel space
user spaceports
Transport protocols
UDP Provides only integrity and de-multiplexing
TCP adds…
14
UDP: User Datagram Protocol [RFC 768]
“bare bones” Internet transport protocol, “Best effort” service
UDP segments may be: Lost, duplicated (bug, spurious re-
transmission at L2) Delivered out of order to app
Connectionless: No handshaking between UDP
sender, receiver Each UDP segment handled inde-
pendently
Why is there a UDP?• No connection establishment
(which can add delay)• Simple: sender, receiver
state is minimal• Small header• No congestion control: UDP
can blast away as fast as desired
15
UDP, cont.
Often used for streaming multimedia apps (e.g., Skype) Loss tolerant Rate sensitive
Other UDP uses (why?): DNS
Reliable transfer over UDP Must be at application layer Application-specific error re-
covery
Source port # Dest port #
32 bits
Applicationdata
(message)
UDP segment format
Length ChecksumLength, in
bytes of UDPsegment,includingheader
Transport protocols
UDP Provides only integrity and de-multiplexing
TCP adds… Connection-oriented Reliable (error control) Ordered byte stream (in-order delivery) Bi-directional byte-stream Flow and congestion controlled
TCP
Protocol implemented entirely at the ends Fate sharing
Abstraction provided Connection oriented protocol (hand-shake or connection establish-
ment) In-order byte-stream
Implemented functionality (next lecture) Flow control Reliability (error control) Reassembly (in-order delivery) Congestion control
18
/* get the server host entry */hp = gethostbyname(argv[1]);if (hp == NULL) {perror("gethostbyname");return -1;}
/* create stream socket */sockfd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);if (sockfd < 0) {perror("socket");return -1;}
/* set the destination address */daddr.sin_family = AF_INET;memcpy(&daddr.sin_addr.s_addr, hp->h_addr, hp->h_length);daddr.sin_port = htons(PORT);
TCP Client Example
19
/* connect to the destination */
if (connect(sockfd, (struct sockaddr *)&daddr, sizeof(struct sockaddr_in))) {
close(sockfd);
perror("connect");
return -1;
}
inet_ntop(AF_INET, &daddr.sin_addr, s, sizeof(s));
printf("client: connecting to %s\n", s);
/* receive message sent from the server */
if ((numbytes = recv(sockfd, buf, MAXDATASIZE-1, 0)) == -1) {
perror("recv"); exit(1);
}
/* print the received message */
buf[numbytes] = '\0';
printf("client: received '%s'\n", buf);
close(sockfd);
TCP Client Example
Difference in abstraction
UDP Recv(len) call gives you a packet (the payload part)
TCP Connection establishment (hand-shake) Will see in the TA
led session. Recv(len) call give you a sequence of bytes
UDP segment (packet)
Byte stream
UDP segment (packet)
len len
Incoming stream
ApplicationRecv()
len len
Byte stream
Applicationsend()
TCP– What goes on underneath
Byte stream Packet (1~7) Packet (8~10)
Receive buffer
Packet (9~11)
Packet (8~10)
I need seq# 8.
22
User Datagram Protocol(UDP): An Analogy
Example UDP applicationsMultimedia, voice over IP
UDP Single socket to receive mes-
sages No guarantee of delivery Not necessarily in-order delivery Datagram – independent packets Must address each packet
Postal Mail Single mailbox to receive letters Unreliable Not necessarily in-order delivery Letters sent independently Must address each reply
Transmission Control Pro-tocol (TCP): An Analogy
TCP Reliable – guarantee delivery Byte stream – in-order delivery Connection-oriented – single
socket per connection Setup connection followed by
data transfer
Telephone Call Guaranteed delivery In-order delivery Connection-oriented Setup connection followed by
conversation
Example TCP applicationsWeb, Email, Telnet
REVISION CONTROL & MAKE
Overview
Revision control systems Motivation Features Subversion primer
Make Simple gcc Make basics and variables Testing
Useful Unix commands
Dealing with large codebases
Complications Code synchronization Backups Concurrent modifications
Solution: Revision Control System (RCS) Store all of your code in a repository on a server Metadata to track changes, revisions, etc… Also useful for writing books, papers, etc…
Basic RCS features (1/2)
Infinite undo go back to previous revisions
Automatic backups all your code ever, forever saved
Changes tracking see differences between revisions
Basic RCS features (2/2)
Concurrency Multiple people working at the same time
Snapshots Save stable versions on the side (i.e. handin)
Branching Create diverging code paths for experimentation
Typical RCS workflow
1. Create repository on RCS server2. Checkout the repository to your local machine3. Work locally: create, delete and modify files4. Update the repository to check for changes other
people might have made5. Resolve any existing conflicts6. Commit your changes to the repository
RCS implementations
Concurrent Versions System (CVS) Concurrency, versioning of single files
Subversion (SVN) File and directory moving and renaming, atomic commits
Git Modern, de-facto standard
Concurrent edits (1/2)
John DavidFile v1
Both check out file
Edits (lines 1-20)
File v2
Edits (lines 30-40)
commit
Update
Merged automatically!
Concurrent edits (2/2)
John DavidFile v1
Both check out file
Edits (lines 1-20)
File v2
Edits (lines 10-20)
commit
Update
Conflict needing manual in-tervention!
Resolving conflicts
When changes can’t be merged automatically SVN gives you 3 files:
file.mine : your file file.rx : the remote file file : original file with marked conflicts
You can Discard your changes Discard others’ changes Go over each conflict and arbitrate as appropriate
Interacting with SVN
Command line Readily available on your lab machines
Graphical tools Easier to use Subclipse for Eclipse gives IDE/SVN integration tortoiseSVN
실습 server
IP: 143.248.141.40, 143.248.141.42~143.248.141.44 ID: ee<your student id> (e.g., ee20110111) Password: ee324EE#@$
Svn password
20100456 6390020100667 2683420100967 4801320110198 81020110227 3961720110319 1641120110468 5770020110548 6430220110968 3086020120184 1942420120277 1297620120508 1727820120677 4002920120973 61034
20121092 42821
20130098 4280720130185 3079120130234 1344820130363 1442120130394 2079620130451 1300820130477 5370520130478 4512220130505 480520130842 74220140055 2038920140244 4533620140544 2379320153347 5044
Command line SVN example (1/2)
$ svn co –-username=<<student id>> https://ina.kaist.ac.kr/svn/test
$ cd test
$ vim <<student id>>.txt$ vim <<student id>>.$ svn add <<student id>>.txt
$ svn commit -m 'adding a text file with my student id‘$ svn update
$ cd ../$ svn cp trunk tags/<<student id>>_submission$ svn commit -m 'making a tag of the trunk for submission!‘
$ svn up$ echo -e ”<<student id>>" >> student.txt$ svn commit –m “added my student id”
Command line SVN example (2/2)
Revision control lets you note (and then see) what you changed:
> svn log student.txtr986 | ntolia | 2006-08-01 17:13:38 -0400 (Tue, 01 Aug 2006) | 6 lines
This allows the sp to get rid of chunks early before a transfer is complete.Useful when a file is requested in-order and the file size > mem cache size
> svn diff -r 1:2 student.txtIndex: file===================================================================--- file (revision 1)+++ file (revision 2)@@-1,2+1,3@@ This isatestfile -It startedwithtwolines +It nolongerhastwolines +it hasthree
General SVN tips
Update, make, test, only then commit Merge often (svn up) Comment commits Avoid commit races Modular design avoids conflicts
Know more
subversion.tigris.org for SVN software & info svnbook.red-bean.com for SVN book
Makefile
You are on your own.
Make
Utility for executable building automation Saves you time and frustration Helps you test more and better
Simple gcc
If we have files:• prog.c: The main program file• lib.c: Library .c file• lib.h: Library header file
% gcc -c prog.c -o prog.o
% gcc -c lib.c -o lib.o
% gcc lib.o prog.o -o binary
gcc flags
• Useful flags1. -g: debugging hook
2. -Wall: show all warnings
3. -Werror: treat warning as errors
4. -O0, -O1, -O2, -O3: optimization level
5. -DDEBUG: macro for DEBUG (#define DEBUG)
• Avoid using dangerous optimizations that could af-fect correctness
More gcc
% gcc -g -Wall -Werror -c prog.c -o prog.o
% gcc -g -Wall -Werror -c lib.c -o lib.o
% gcc -g -Wall -Werror lib.o prog.o -o binary
This gets boring, fast!
Makefile basics
• Build targetstarget: dependency1 dependency2 ...
unix command (start line with TAB)unix command
binary: lib.o prog.ogcc -g -Wall lib.o prog.o -o binary
lib.o: lib.cgcc -g -Wall -c lib.c -o lib.o
prog.o: prog.cgcc -g -Wall -c prog.c -o prog.o
clean:rm *.o binary
Makefile example
Makefile variables (1/7)
• Variables
CC = gcc
CFLAGS = -g -Wall -Werror
OUTPUT = binary
binary: lib.o prog.ogcc -g -Wall lib.o prog.o -o binary
lib.o: lib.cgcc -g -Wall -c lib.c -o lib.o
prog.o: prog.cgcc -g -Wall -c prog.c -o prog.o
clean:rm *.o binary
Makefile variables (2/7)
CC = gccCFLAGS = -g -WallOUTPUT = binary
$(OUTPUT): lib.o prog.o$(CC) $(CFLAGS) lib.o prog.o -o binary
lib.o: lib.c$(CC) $(CFLAGS) -c lib.c -o lib.o
prog.o: prog.c$(CC) $(CFLAGS) -c prog.c -o prog.o
clean:rm *.o $(OUTPUT)
Makefile variables (3/7)
CC = gccCFLAGS = -g -WallOUTPUT = binary
$(OUTPUT): lib.o prog.o$(CC) $(CFLAGS) lib.o prog.o -o binary
lib.o: lib.c$(CC) $(CFLAGS) -c lib.c -o lib.o
prog.o: prog.c$(CC) $(CFLAGS) -c prog.c -o prog.o
clean:rm *.o $(OUTPUT)
Makefile variables (4/7)
CC = gccCFLAGS = -g -WallOUTPUT = binaryOBJFILES = lib.o prog.o
$(OUTPUT): $(OBJFILES)$(CC) $(CFLAGS) $(OBJFILES) -o binary
lib.o: lib.c$(CC) $(CFLAGS) -c lib.c -o lib.o
prog.o: prog.c$(CC) $(CFLAGS) -c prog.c -o prog.o
clean:rm *.o $(OUTPUT)
Makefile variables (5/7)
CC = gccCFLAGS = -g -WallOUTPUT = binaryOBJFILES = lib.o prog.o
$(OUTPUT): $(OBJFILES)$(CC) $(CFLAGS) $(OBJFILES) -o binary
lib.o: lib.c$(CC) $(CFLAGS) -c lib.c -o lib.o
prog.o: prog.c$(CC) $(CFLAGS) -c prog.c -o prog.o
clean:rm *.o $(OUTPUT)
Makefile variables (6/7)
CC = gccCFLAGS = -g -WallOUTPUT = binaryOBJFILES = lib.o prog.o
$(OUTPUT): $(OBJFILES)$(CC) $(CFLAGS) $(OBJFILES) -o binary
%.o: %.c# $<: dependency (%.c)# $@: target (%.o)$(CC) $(CFLAGS) -c $< -o $@
clean:rm *.o $(OUTPUT)
Makefile variables (7/7)
Simple Test Script (1/2)
% ./server 6667 &
% cat testfile.01 | ./testscript.py
% cat testfile.02 | ./testscript.py
% killall -9 server
Simple Test Script (2/2)#/bin/sh
echo “Starting server on port 6667.”./server 6667 &SERVERPID = $!
echo “Running test files.”cat testfile.01 | ./testscript.pycat testfile.02 | ./testscript.py
echo “Killing server process.”kill $(SERVERPID)
CC = gccCFLAGS = -g -WallOUTPUT = binaryOBJFILES = lib.o prog.o
all: $(OUTPUT)
$(OUTPUT): $(OBJFILES)$(CC) $(CFLAGS) $(OBJFILES) -o binary
%.o: %.c# $<: dependencies (%.c)# $@: target (%.o)$(CC) $(CFLAGS) -c $< -o $@
clean:rm *.o $(OUTPUT)
Augmenting the Makefile for testing (1/2)
CC = gccCFLAGS = -g -WallOUTPUT = binaryOBJFILES = lib.o prog.o
all: $(OUTPUT) test
$(OUTPUT): $(OBJFILES)$(CC) $(CFLAGS) $(OBJFILES) -o binary
%.o: %.c# $<: dependencies (%.c)# $@: target (%.o)$(CC) $(CFLAGS) -c $< -o $@
test: $(OUTPUT)sh ./testscript.sh
clean:rm *.o $(OUTPUT)
Augmenting the Makefile for testing (2/2)
Using the Makefile
% make
% make test
% make clean
• Know more
Google– “makefile example”– “makefile template”– “make tutorial”– Chapter 3 of Dave Andersen’s notes
Extra: useful Unix commands (1/2)
• find “func_name” in files% grep -r func_name .
• replace “bad_func_name” to “good_func_name”% sed -e “s/bad_func_name/good_func_name/g”\
prog.c > prog.c.new
Extra: useful Unix commands (2/2)
• find a file named “prog.c”% find -name prog.c
• download files from the Internet% wget http://address/to/file.tar.gz
• untar and unzip a file% tar xzvf file.tar.gz
Assignment
You have access to svn repository at https://ina.kaist.ac.kr/svn/<student id>
However, we will change your password (to some un-known integer).
Luckily, we made a server that gives you a hint. When your client asks whether the username and the
password match, the server can tell you whether the password provided is less than, equal to, or greater than your real password.
Assignment I
Protocol is defined in the document. (See course web page).
Query
Response
Server (provided by the in-structor)Client
Part I: Assume the network is reliable. (due 9/14) Part II: The server may drop a packet. (due 9/24)