IPsec Used as a Protocol for a VPN
Michael Evans
December 16, 2013
CNT 4104 Software Project in Computer Networks
Instructor: Dr. Janusz Zalewski
Computer Science & Software Engineering Programs
Florida Gulf Coast University
Ft. Myers, FL 33965
© Copyright 2013 by Florida Gulf Coast University
Evans
2
1. Introduction
IPSec is a protocol suite used in Virtual Private Networks (VPN). IPSec can encrypt
traffic between two nodes using a Transport Mode. This encrypts the upper levels of the
datagram. The entire datagram can also be encrypted using Tunnel Mode, in the form of a VPN.
Its subprotocols consist of Authentication Headers (AH) and Encapsulating Security Payloads
(ESP).
ESP provides packet authentication, data integrity, and anti-replay features, plus
confidentiality. AH provides the above, except for confidentiality. Any fields in an AH header are
unencrypted, authenticating the outer header of an IPSec packet. ESP headers are also
unencrypted, but it also has a partially encrypted trailer, thus encapsulating the data. AH is used
with ESP because it provides an assurance of authenticity and integrity. Also, ESP must support
DES encryption and some policies have approved only AH for that purpose [1].
Figure 1.1 – A sample of a construction and interpretation of an AH packet
Figure 1.1 shows the (Security Parameters Index (SPI) and the Sequence Number.
The SPI is used by the receiver to determine which Security Association (SA) authenticated the
packet. The Sequence Number is what provides the aforementioned anti-replay protection. This
is to prevent malicious hackers from injecting their own packets int to the datastream. If the
recipient receives a second packet with the same Sequence Number, it is rejected. An ESP
packet, as shown in Figure 1.2, has the above features, as well as a padding section, used to tell
the receiver how much padding has been added to the packet.
Evans
3
Figure 1.2 – A sample of construction and interpretation of an ESP packet
Internet Key Exchange (IKE) is a third subprotocol in the IPSec suite. IKE builds upon
the Internet Security Association and Key Management Protocol ISAKMP) and Oakley protocol.
Cryptographic keys are traded over IKE using Diffie-Hellman key exchange. Diffie-Hellman
exchange operates using a public key system with numbers from a pre-agreed upon group of
prime numbers p, generated by a pre-agreed upon generator g. The products of gx mod p and gy
mod p are exchanged as public values and then generate the shared private key value of gxy mod
p, thus making it possible to secure an insecure channel. The drawback to this approach is
vulnerability to a man-in-the-middle attack [1].
IPSec implementation can be done within an end user’s computer or in the routers
between end points of the network connection. Native implementation, for which the logic is
roughly sketched in Figure 1.3, can be integrated into the operating system. A replacement stack
can be written by a third party to be used in place of the operating system vendor’s native
implementation. A third possibility is that it can be configured as a compatibility layer between
the network and the link layer of the protocol stack, also known as the “Bump in the Stack”
(BITS). Both host implementation and BITS have their advantages and disadvantages [2].
Evans
4
Figure 1.3 - An example hardware layout of host implemented IPSec
Router implementation, for which the logic is roughly sketched in Figure 1.4, has similar
options. The first is a native implementation that is integrated into the router’s embedded
operating system. The second is a “Bump in the Wire” approach, similar to BITS. It is less
frequently used as it requires dedicated hardware to be added to each router in the network.
Figure 1.4 - An example hardware layout of router implementation
Evans
5
As IPSec is a widely used method of sending encrypted communications over the
network layer, and it is of great importance for IT professionals everywhere. Few entities these
days rely entirely on in-house networks. Usually their networks are connected through an
internetwork operated by a third party. Therefore, many VPNs use IPSec for private and secure
corporate and personal communications.
The existence of VPN technology, which uses protocols like IPSec, provides great utility
to even those working outside the world of information technology. Encryption is a guarantor of
privacy to society, something desirable in itself. In a time in which nation-state governments
routinely monitor vast amounts of Internet traffic, IPSec becomes a valuable tool in maintaining
personal privacy.
Evans
6
2. Definition of the Problem
Florida Gulf Coast University has Wi-Fi hotspots located throughout the campus. The
problem that arises from this is that personal messages can be snooped on or stolen. The
objective of this project is set up a custom VPN server, which uses the IPSec protocol, that can
be accessed over the school Wi-Fi.
The original plan was to use one of FGCU’s existing Linux Fedora-based servers and
configure a virtual private network which uses IPSec. This was to be followed up by testing the
security of the VPN. When a reasonable degree of security is reached, the server can be beta
tested by setting up user accounts and interfaces. Ideally, the accounts can be based on students’
Eagle mail accounts and passwords, though time constraints may not allow that phase to be
reached.
Evans
7
3. Design Solutions
The design solution involves configuring two machines to run IPSec and being able to
exchange, at minimum, IPSec encrypted packets. One is a server running Linux Fedora 9 at
sanibel.cs.fgcu.edu. The other is a laptop running Ubuntu 12.04. Both will have Openswan, an
open source IPSec implementation [3].
Once the needed IPSec tools are installed and configured on both systems, theywill
exchange their public keys. Both will be connected to the Internet and a virtual private network
connection will be established. They can then exchange packets, encrypted according to the
flowchart in Figure 3.1.
Figure 3.1 - IPSec encrypting packets. © 1999, 2011 Oracle
Evans
8
4. Implementation of the Server
In theory, all that is needed to get IPSec running is to install a few packages and change a
few settings. This is more complicated in practice.
The initial intention was to deploy implementation on a virtual machine running the
server edition of Ubuntu 12.04. After first attempting to use a Fedora based server, work returned
to the desktop edition of Ubuntu 12.04. While this sounds simple enough, doing so from scratch,
without a step-by-step guide, took many hours of research and debugging. However, the script in
Appendix A gives an administrator with root privileges a good start installing and configuring
IPSec. It creates files throughout /proc/sys/net/ipv4/conf/ directories needed to pass
ipsec verify’s checks, then sets IPSec to run on start up.
The administrator will need to manually turn on IP forwarding by changing
"net.ipv4.ip_forward = 0" to "net.ipv4.ip_forward = 1" in /etc/sysctl.conf,
as shown in Figure 4.1. The sysctl must then be reloaded with the command sysctl -p
sysctl.conf. Finally, ipsec pluto starts the pluto daemon.
Figure 4.1 - Enabling IP forwarding
Evans
9
Key to encryption is a pre-shared key. ipsec showhostkey --left revealed that
ipsec.secrets was empty, as expected on a machine not previously configured. ipsec
newhostkey --output /etc/ipsec.secrets created an RSA key to be used for IPSec
communication. The /etc/ipsec.secrets file was also updated to include a pre-shared key
to be used between client and server. Figure 4.2 shows the RSA key, the left and right addresses
the PSK is to be shared between, and a commented out PEM (Privacy-enhanced Electronic Mail)
certificate, created for, but not used in, this project.
Figure 4.2 - Contents of /etc/ipsec.secrets
To authenticate users, a list of users and passwords were added to /etc/ppp/chap-
secrets. The format, shown in Figure 4.3, is username, method of connecting, password, and
IP addresses that a given user may connect from. For example, zalewski can connect using
L2TP from any IP address while mevans can connect using any method from any IP address.
Evans
10
Figure 4.3 - Sample formatting for /etc/ppp/chap-secret’s user list
Appendix B has the contents of the server’s /etc/ipsec.conf file, which can easily be
adapted for any IPSec server using Transport Mode instead of Tunnel Mode and pre-shared keys
instead of certificates.
Over the course of troubleshooting, openswan-2.6.39 was compiled from source and
installed to try to minimize bugs in the program. This did not prove successful. Openswan was
removed and replaced with strongSwan using apt. However, strongSwan also did not resolve the
issues and appeared to introduce new issues. Openswan was reinstalled using apt, which created
the problem of having to manually edit ipsec verify located in
/usr/local/libexec/ipsec/verify. The methods udp500check() and
udp4500check() were fixed to look for ss in /bin/ instead of /usr/sbin/. This is not
necessary for configuration, but is mentioned in the event that using a later version of Openswan
for ipsec verify becomes relevant to future troubleshooting.
Evans
11
5. Experiments with the Client
Initial tests of the client used a computer running Mac OS X, version 10.8.5.
Unfortunately, it repeatedly failed to connect to the server. Researching the problem suggested
that difficulties in connecting Mac OS X to a VPN over IPSec is often due to Apple’s
implementation of IPSec. Testing was then switched to computers running Windows 7 and
Windows 8. A new connection was configured from the Network and Sharing Center, shown in
Figure 5.1.
Figure 5.1 – Windows Network and Sharing Center
VPN configuration is listed under “Connect to a workplace”, as shown in Figure 5.2.
Connections are made over the local Internet connection instead of directly dialing the server,
shown in Figure 5.3. Then the setup wizard requires the IP address, in this case, 10.100.40.173,
and a name for the connection, shown as IPSecZal in Figure 5.4. Finally, a username and
password that match the one from /etc/ppp/chap-secrets is entered, shown in Figure 5.5.
Evans
12
Figure 5.2 – Connecting to a workplace (VPN)
Figure 5.3 – Choosing a means of connecting
Evans
13
Figure 5.4 – IPSecZal
Figure 5.5 – Credentials window
Evans
14
To advance further, in the network connection window, “Connect IPSecZal”, click
properties, as shown in Figure 5.6. In the security tab, change type of VPN to Layer 2 Tunneling
Protocol with IPSec (L2TP/IPSec), shown in Figure 5.7. Then, click Advanced Properties to
ender the PSK as shown in Figure 5.8.
Figure 5.6 – IPSecZal login window
Evans
15
Figure 5.7 – IPSecZal Properties
Figure 5.8 – IPSecZal Advanced Properties
Evans
16
Under normal circumstances, the client would connect to the server. However, the
network being tested on is behind a NAT (Network Address Translation), giving errors like the
one shown in Figure 5.9. Network Address Translation allows multiple computers to use a single
IP address. This is completely opposed to IPSec’s packet header authentication, and will often
break it. Further testing showed that it was not possible to telnet into the server on ports 500 and
4500 while the firewall was disabled, confirming the presence of a NAT.
Figure 5.9 – Error Connecting to IPSecZal
Evans
17
6. Conclusion The scope of this project proved that a longer period than one semester could be
involved. While configuring a computer to run IPSec was possible within the timeframe, a fully
encrypted virtual network was not. The bash script in Appendix A will configure IPSec on the
Sanibel server. For ease of use, it can also be found in /home/mevans/cnt/ at
sanibel.cs.fgcu.edu.
Several attempts were made to establish a working IPSec connection. The first attempt to
run the network virtually as a proof of concept failed. The size of the virtual machines, as well as
VirtualBox requiring an Ethernet connection not present on the developer's laptop, made this
impossible. Additionally, any virtual network would eventually have to be implemented on
physical machines.
An all-Linux host-to-host connection proved unfeasible as the Ubuntu laptop did not have
a static connection. A Linux/Windows mixed host-to-host connection using the server at
69.88.163.11 (Sanibel) also did not pan out. The network-to-network connection could not be
established due to the lack of access to the Sanibel server and the aforementioned incompatibility
between VirtualBox and bridged adapter mode.
Limited success was achieved with the server running the desktop edition of Ubuntu
12.04. A working IPSec server was configured and the logs from that server confirm the
beginnings of an IPSec connection. Unfortunately, the NAT prevents any real connection from
being established at this time.
Finally, an alternative for using Openswan may be a Freeswan [4], but this option was not
tired in this project.
Evans
18
7. References
[1] Kaufman, E. and A. Newman, Implementing IPsec: Making Security Work on VPNs,
Intranets, and Extranets. John Wiley and Sons. 1999.
[2] Doraswamy, N. and D. Harkins, IPSec: The New Security Standard for the Internet, Intranets,
and Virtual Private Networks. Prentice Hall PTR. 2003
[3] Openswan: An IPsec implementation for Linux. URL: https://www.openswan.org/
[4] Freeswan: An implementation of IPSEC & IKE for Linux. URL:
http://www.freeswan.org/intro.html
Evans
19
Appendix A. Script for Installing and Configuring IPSec
#!/bin/bash
apt-get install xl2tpd openswan ipsec-tools;
# fix redirect errors
for f in /proc/sys/net/ipv4/conf/*/accept_redirects;
do echo 0 > $f;
done;
for f in /proc/sys/net/ipv4/conf/*/send_redirects;
do echo 0 > $f;
done;
chkconfig ipsec on; # make sure ipsec is running on start up
Evans
20
Appendix B. Content of Server’s ipsec.conf file
version 2.0
config setup
nat_traversal=yes
virtual_private=%v4:10.0.0.0/8,%v4:192.168.0.0/16,%v4:172.16.0.0/12
oe=off
protostack=netkey
conn L2TP-PSK-NAT
rightsubnet=vhost:%priv
also=L2TP-PSK-noNAT
conn L2TP-PSK-noNAT
authby=secret
pfs=no
auto=add
keyingtries=3
rekey=no
ikelifetime=8h
keylife=1h
type=transport
left=10.100.40.173 # Your server's IP address here
leftnexthop=%defaultroute
leftprotoport=17/1701
right=%any
rightprotoport=17/%any