White Paper: Nothing but Net : Applications, a Networking ... · PDF fileThe majority of...

Copyright 2000 by R.Azzopardi, Oracle Corp Canada

Nothing but Net : Applications, a Networking Perspective

Robert AzzopardiOracle Canada

Introduction

With the advent of Oracle Applications' internet computing architecture and the increasing demand forWide Area Network (WAN) access to key business systems, both the importance and complexity ofnetworking is increasing. Once relegated to the "backroom", Networking now fundamentally affects theway people conduct and construct their business.

What is REALLY happening with Oracle Applications and my network ?" This is the question we will beanswering in this paper as well as looking at hints on improving performance and hot topics such asQuality of Service.

We will examine some network "buzz" words : Bandwidth, Latency, Virtual Private Networks (VPNs) andQoS (Quality of Service) and relate them back to running Oracle Applications 10.7 NCA and R11 in a liveenvironment. Measurement techniques will be discussed and results of operation under different networkconditions presented.

As networks grow in both importance and complexity, understanding the impact of systems such asOracle Applications becomes a business issue rather than a purely technical consideration. The aim ofthis presentation is to provide both theory and practice to enable implementers and administrators to askthe right questions, make informed decisions and help ensure success.

Networking 101

There are literally thousands of books covering the many aspects of networks and networking. As thispaper will take an empirical approach, some more detailed references are provided in the bibliography.

Starting with some definitions :using the metaphor of the Information Super Highway, Bandwidth can beregarded as the posted speed limit. Latency as the time it takes to get on the on-ramp, onto the highway,to your destination and back.

Relating back to a computer network, the bandwidth of a data link is the maximum amount of data thatcan be sent between two computers in a period of time over that link. It is measured in bits per second(bps) or multiples of thousand bps such as Kbps or Mbps. Another analogy for visualizing bandwidth iswater flowing in a pipe. The wider the pipe, the more water can flow through it at any given instant. This isshown in Figure 1. When the same amount of water flows through a narrower pipe, the peak is flattenedout and spreads. Applying this analogy to Oracle Applications traffic, as the bandwidth of the link isdecreased, the same volume of data still has to be moved, it just takes longer. This behavior will be shownin network traces later in this paper. With bandwidth, in general : “More is better, but expensive”.

W ide P ipe

Narrow P ipe

W ater W aves

W ater W aves

Direction

Direction

P eak is spread ou t bylim ited capacity o f p ipe

W ave is una ffec ted

Figure 1 : Behavior of water in a pipe.

For this discussion, latency is the time it takes for a packet of data to travel from your PC to the server. Ona Local Area Network (LAN) it is measured in milliseconds while on a WAN (Wide Area Network) it can beof the order of 100s of milliseconds. In the case of latency : the smaller the better. The PING commandfound on Unix and in Windows measures round trip latency : the time taken for a packet to go from sourceto destination and back. Figure 2 breaks network latency down into components :

TotalLatency

SenderOverhead

(Netw ork cards,Hubs etc)

Tim e ofFlight

ReceiverOverhead

(Netw ork cards,Hubs etc)

Message S ize

Bandw idth= + + +

Network

Speed of Light

Distance

Propagation VelocityX

~ 0.5

PING (round trip)

Figure 2 : Network Latency and ping

From Figure 2, latency is directly affected by the distance covered by the network link. While thiscomponent is insignificant in local area network, it starts to become significant for global WAN links andfor satellite links. Depending on the media, signals do not actually travel at the speed of light, rather thespeed is governed by the propagation velocity in that medium as shown.

Network latency is not a fixed quantity. It varied with traffic load, message size etc.The following traces were obtained between two computers, one in Calgary Canada and the other inAustralia :

40 byte ping 190/212/295 (from The Cloud) 500 byte ping 210/234/295 (from The Cloud)

Figure 3 : Effect of packet size on ping

Two things to note in Figure 3. Firstly that link latency is variable and secondly, that it is affected bypacket size. (size of the message being sent)

Within certain limits, the application of money can change the bandwidth of a link. This may be throughfaster network cards, bigger leased lines, Fast Ethernet. Unfortunately in the case of network latency, thelaws of physics are not as easily influenced by the dollar and in very long links the contribution of distanceand propagation speed becomes significant and inescapable.

Finally, the fact that a network is in place, probably means that it is use. Even on a seemingly quietnetwork there is traffic: netbios traffic generated by Windows PC, traffic generated by routers and variousnetwork broadcasts. In addition, the cost of WAN links between remote offices means that they are keptas small as possible.

And with a busy network comes the potential for congestion. With a medium like Ethernet, when there arelots of nodes (computers) trying to transmit data at the same time, there will invariably be “traffic jams” inwhich case the affected nodes will have to wait a small amount of time and try to retransmit their data.In short, adding a network application to a network in trouble or even a busy network never makes thesituation better !

In summary, before we even look at Oracle Applications, it is important to consider the intrinsic behavior ofnetworks themselves and then look at an application in terms of• Bandwidth• Latency• Network Congestion Measuring Network Traffic and Behavior Before delving into results, it is useful to talk about the tools used. One definition of a Wide Area Network is a data communications network that serves users across abroad geographic area and may utilize transmission devices provided by common carriers. It is oftendepicted by a picture of a cloud.

Deployment of applications across a Wide area network is arguably one of the biggest challenges facedby network people during an implementation. In addition, the costs and challenges do not go away afterimplementation. The same questions come up : “What will the response be like for users across the country ?” “What sort of connection do we need between HQ and branch offices for our applications users ?” There are essentially two ways to answer these questions : • Install it and see how it works• Simulate and obtain indicative metrics

While the first method is the most common, simulation and testing prior to deployment can mitigate riskand help build business cases for different strategies if the existing network infrastructure is foundwanting.A WAN can be emulated with hardware, software or a mixture of both. Two main aspects need to beconsidered : Bandwidth and Latency.Hardware delay emulators such as those produced by Adtech are an elegant solution but cost puts themout of reach of many implementations. Similarly for the use of back-to-back routers to produce differentbandwidth connections - you need to have the routers available. Software WAN emulators can provide avery cost effective alternative.

The majority of testing done for this paper was carried out using “The Cloud” - a WAN emulator developedby Shunra Software LTD (www.shunra.com). Many of the screen shots appearing in this paper are fromthis tool. The Cloud runs on NT and can emulate virtually any type of WAN combination of latency,bandwidth and even packet loss. It acts as a virtual device driver that sits between the PC’s TCP/IP stackand the network and delays packets according to the user’s desired settings. Figure 4 shows the mainconfiguration screen of “The Cloud”.

Figure 4: The Cloud Configuration Screen

A range of different Latency options can be configured within the Cloud as shown in Figure 5.

Figure 5 : The Cloud - Latency Configuration optionsFor simplicity’s sake, a fixed latency was used in the testing carried out for this paper. As discussedpreviously, in the real world latency is not a constant. The issue becomes one of tractability andminimizing the variables introduced when studying the performance of the Applications.

While it can reside on a dedicated NT PC acting effectively as a router, for the purposes of this paper, TheCloud was installed on the same machine that ran the Oracle Applications Client. The test setup used isshown in Figure 6. ��

��

D ua l P e n tiu m III5 00 M h z2 56 M B R A MA p p lica tio ns 1 10 3

The C loudW A N

Em ulator

3 C O M M in i H ub

Network Sniffer

2 66 M H z N oteb oo k2 56 M B R A M

Figure 6 : Test setup for Latency testing

A freeware alternative for WAN emulation : NISTnet was also investigated(http://www.antd.nist.gov/itg/nistnet/) . Nistnet runs on Linux acting as a router and is a variant of the“Hitbox” developed for Sun Solaris some years ago. It is implemented as a kernel module extension tothe Linux operating system and an X Window System-based user interface application. While many usershave reported success in using NISTnet, it only runs on a 2.0.3x Linux kernel (circa Redhat 5.1) and itsinstallation was found to be quite time consuming and was not actively pursued for this testing. TheRedhat 6.x version is expected soon.

For verification purposes, Bandwidth throttling was also studied using two back-to-back Cisco Routers asshown in Figure 7. The two routers are connected via a serial cable. By varying the clock rate on the 1601router (acting as a piece of Data Communications Equipment or DCE) different WAN bandwidths can beachieved.

C isco 16 0 1(D C E )

C isco 25 2 2 (D T E )

S e ria l C ab le

A p p lica tio n s S e rver

A p p lica tio n s C lien t

1 40 .8 7 .9 6 .62

1 40 .8 7 .9 7 .30

1 40 .8 7 .1 0 1 .3 0

1 40 .8 7 .1 0 0 .2 0

Figure 7 : Emulating WAN bandwidth with back-to-back routers

In order to compare the behavior of Oracle Applications under different link conditions, it is important tomeasure the same transaction’s performance. Two strategies were used to achieve this. In the first,Mercury Interactive’s LoadRunner was used to record a virtual user. The virtual user script was thenplayed back under different network conditions.

To use LoadRunner to record an Oracle Applications transaction, the Applications are started from withinLoadRunner and user activities are captured as they occur. LoadRunner 6 was used and which has theability to natively record Oracle Applications (NCA) clients. Once the client operations have been recorded,wait times can be adjusted, individual transaction markers added and parameters included. Wait times areto ensure that during playback the transaction is as realistic as possible and also does not run too fast forthe Server.The syntax is “C” -like and is shown in Figure 8. Additionally, actual client GUI interface activity can berecorded using WinRunner.

Figure 8 : Mercury LoadRunner Virtual User Creation.

The second strategy was to use Oracle Developer InterceptServer. It is important to note that this is no t aproduct and not available through Oracle Shipping, but rather a tool set written in Java for recording andplaying back Oracle Developer Server activity. It is available on a limited basis for use by OracleConsulting and Pre-Sales. One of the biggest engagements it was used for was the Norwegian TaxAdministration (SKARP) benchmark.The InterceptServer is run entirely from the command line. The operation of recording with InterceptServer is show in Figure 9.

InterceptServer listens to and records redirected Developer Messaging. For this testing theInterceptServer and the Applications client were on the same machine. This does not have to be the case.Note that JAR files are used locally and are no longer referred to in the HTML startup file. It is necessaryto explicitly specify the JAR files in the classpath on the InterceptServer machine. Alternatively, the JARfiles can be all unzipped into the classpath. (this second approach was used for this testing - notrecommended, but it does work).

P O R T 9 0 02

P O R T 8 9 00

<PAR AM nam e="serverPort" va lue="8900"><PAR AM nam e="serverH ost" va lue="ca l1449.ca.oracle .com "><PAR AM nam e="serverApp" va lue="O rac leApplications"><PAR AM nam e="regis tryPath" va lue="/O A_JAVA/orac le /apps/fnd/form sC lient">

set c lasspath=% C LA S S P A TH% ;d:\javajava In terceptS erver server=140.87.100.20:9002 port=8900 m ode= record debug=3 file=g l_run1

ca l14 4 9 .ca .o ra c le .co m

1 40 .87 .10 0 .20In te rcep t S erve r + C lie n to n sa m e P C

M od if ied H T M L sta rtu p file

Figure 9 : InterceptServer setup

Playback is slightly different. The Applications client used for recording is no longer required. The setup isshown in Figure 10.

P O R T 9002

P O R T 8900

set c lasspath=% C LAS SP ATH % ;d:\javajava -m s10M -m x20M InterceptS erver server=bunyip.ca.oracle.com :9002 m ode=play file=gl_run1 processes=1 debug=3

ca l1449.ca.orac le .com

bunyip .ca.orac le .com

Figure 10 : InterceptServer Playback

While paramaterization is possible it is a manual and error prone process and not for the faint hearted. Auseful trick found during testing was to actually record the InterceptServer session using LotusScreencam. This allows the tester to go back after the fact and correlate InterceptServer events withApplications actions.

The tests for this paper were run on an unoptimized R11.0.3 installation on NT. (built using the “1 HrInstall” ). The server was a 2 way 500 Mhz Pentium III with 256 MB of RAM.

Two basic transactions were used :1. General Ledger Journal Entry

LoginNavigatorNew Journal EntryJournal Entry 1 - 4 linesJournal Entry 2 - 10 lines (JE1, JE2, JE3 ) Repeated 3 timesJournal Entry 3 - 2 linesLogout

The same transactions were also recorded and replayed using InterceptServer.

2. Accounts Payable InvoicesLoginNavigatorAP Invoice 1AP Invoice 2AP Invoice 3AP Invoice 4AP Invoice 5Logout

While not definitive, these transactions provide a snapshot of how Oracle Applications behaves. Everyimplementation is different and different setups and processes can make a dramatic difference toperformance. To meet this need Oracle has available its Standard Benchmark kit. This is a FREE toolkitfor Applications customers wishing to carry out testing. It provides a starting point for performance testing.

Oracle Applications

Over the past 5 years, Oracle Applications has evolved through a number of different network transportmechanisms :

• Character Mode : Telnet traffic• Smart Client : SQL*net traffic• internet computing architecture : Developer Messaging

Each of these behaves differently over network links. Of the three, the internet computing architecture(formerly known as NCA) is the least susceptible to high latency conditions and has a bandwidthrequirement similar to character mode. Because of its traditional client server architecture, Smart Clienthad higher bandwidth requirements and is more susceptible to poor performance over high latency links.In this paper we will be concentrating on the internet computing architecture deployment of OracleApplications. TCP/IP : a brief diversion and a security consideration TCP/IP stands for Transmission Control Protocol / Internet Protocol. It is an industry standard that formsthe heart of the Internet and is the protocol over which Oracle Applications operates. While TCP/IP operates over a couple of different mediums, we will only be examining Ethernet. Ethernet belongs to the CSMA/CD (carrier sensing multiple access/collision detection) family of LANS. Allnodes (computers) on a CMSA/CD LAN can access the network at any time. Before sending data, nodes“listen” to the network to see if some other node is broadcasting. If the network is in use, the node wishingto transmit waits. A collision occurs when two nodes listen for network traffic, hear none and then bothtransmit at the same time. When this happens both transmissions are effectively lost and both stationsmust retransmit.

Data is transmitted in frames which contain information about which node the data originated from andwhere it destination is (as shown in Figure 11).

7bytes 1 byte 6 bytes 6 bytes 2 bytes 46-1500 bytes 4 Preamble S

O F

Destination Address

Source Address

Type Data FCS

Figure 11 : TCP/IP Frame For the purposes of this paper we are only concerned with the Destination, Source and Data portions ofthe frame. This information comes into play when we are analyzing Apps network performance. BecauseEthernet is a broadcast protocol it is possible to “sniff” the ethernet and capture packets, display andanalyze them. The equipment is shown in the Figure 5.

Ethernet

A pps S erve r

SNIFFERANALYZER

Figure 12 : Ethernet Sniffing The network card in the Sniffer PC is placed in “promiscuous mode” and listens to all traffic. This is thenprocessed by the Sniffer/Analyzer software. It is important to note that unauthorized use of networksniffers is frowned upon by System Administrators and fellow employees alike and is at the very least abreach of privacy and against policy in most companies. These tests were carried out on a privatenetwork ! The Figure 13 shows a public domain sniffer called Analyzer (http://netgroup-serv.polito.it/analyzer/)capturing Oracle Applications traffic :

Figure 13 : Oracle Applications session sniffer output This trace is from the communications between an Apps Client and Apps Server with data encryptionturned OFF. In a production system encryption MUST be turned on ! No “ifs ands or buts” !! Under NT on your Forms Server machine it is a registry setting HKEY_LOCAL_MACHINE

SOFTWARE ORACLE

FORMS45_MESSAGE_ENCRYPTION=TRUE

Under Unix it is an environment variable. FORMS45_MESSAGE_ENCRYPTION=TRUE (or FORMS60_MESSAGE_ENCRYPTION=TRUE) If this is not set correctly, plain text information is being broadcast ! By default encryption is turned on, BUT it is worth checking ! Forms message encryption was turned off for all of the testing carried out for this paper. Oracle Applications : The Connection Process The mechanics of connection for Oracle Applications (NCA and R11x) is summarized in Figure 15. Firstthe client sends the Oracle Applications Startup URL, which is then processed by the Webserver (OracleApplication Server) in step 2. In Step 3 the client request the Java Applet required by Oracle Applications.Once this is completed, the Applet is started and a connection to the Forms Listener is made (step 5). TheForms Listener spawns off an instance of the Forms Runtime Engine (step 6) which then continues theconnection to the Applet running on the client. Communication between Client and Server then continuesover a TCP/IP socket.

Data

��

��

��

C lien t sendsU R L

H T M Lpage

providedto

clien t

D own loadreques tfo r Ja vaA pp le t

A pp le tre tu rnedto c lien t

co nnect toForm s

L istener

hand o ffco nnection

to ap p le t

Form s R untim e E ng ine

H T TP L iste ner(W e b S erver)

Form s L istene r

C lien t P C

O racle A pp lica tionsS erver

1

2

3

4

5

6Spawn

Runtim e

7

Figure 15 : Oracle Applications connection mechanics. In essence the Web Server is being used to broker the initial connection to Oracle Applications. It has aroles to play in providing HTML Help and also providing report viewing and attachment functionality. Thelatter are outside the scope of this paper. The Java Applet on the client communicates with the Forms Runtime Engine on the Server throughTCP/IP “sockets”. A socket is a combination of IP Address and port number, a port being a 16 bit numberthat defines an application running over a network. For example Telnet runs on port 23, FTP runs on port21. A crude analogy is a Post Office Box - the IP Address is the Post Office, the port is the mailbox. In thecase of Oracle Applications Forms Server, the default port is 9000. Socket communications of this typewill not pass through a firewall. This effectively prevents Oracle Applications versions prior to R11i fromoperating across a firewall. However, in Forms 6i, (as used for Applications 11i) the internal messaging between the Forms Serverand Java client is encapsulated in HTTP packets that can pass through a firewall. The first step we shall examine is the download of the Applet and JAR files, steps 3 and 4. The Joys of Jars When Oracle Applications loads, a variety of different “class” files are required. These may be thought ofas library files . Rather than have a large number of small files which would be extremely inefficient, theindividual class files are bundled in container files with a .jar extension after their name. JAR stands forJava Archive file. Compartmentalizing the code provides the dual advantages of more flexible patchingand updating as well as the ability to operate on a need to use basis.

10.7 NCA uses a single appscore.jar file. R11 loads a number of jar files on startup :

• fndewt.jar• fndforms.jar• fndaol.jar• fndctx.jar• fndtcf.jar• fndhier.jar• akobjnav.jar

is a basic set, with additional files such as• cctclient.jar• csclient.jar• performs.jar• azwizard.jar

being called depending on the modules installed and patches applied. Structurally, JAR files are a variant on the zip compressed archive format and can actually be viewedusing winzip. While modifying JAR file contents is totally unsupported, it is useful to know from the point ofview of debugging and understanding how things works :

Figure 16 : Inside JAR files

Preliminary releases of R11i used the following complement of jar files :

• fndewt.jar• fndforms.jar• fndaol.jar• fndctx.jar• fndtcf.jar• fndhier.jar• akobjnav.jar• azwizard.jar• xlatacct.jar• fndfsec.jar• performs.jar• bomjar.jar• fndutil.jar

This is subject to change, but in general as new functionality is added to Oracle Applications, new orchanged jar files will find their way into the distribution. One of the features of R11i is the reorganization ofJAR files so that they are downloaded on a need to use basis - that is, if BOM is not installed, theassociated JAR files are neither required, not loaded.

What happens on the Network the first time you start Oracle Applications on a Client ?

In a nutshell, the jar files are downloaded ONCE and cached. This why the very first startup of Apps canbe slow, particularly over dialup lines. The initial network trace for startup is shown below :

Figure 17 : Oracle Applications R11, initial JAR file download.

This represents starting the apps from appletviewer (or from jinitiator within a browser) for the FIRST timeand getting to the sign on screen.The jar files are now stored in c:\jdk\cache or c:\Program Files\Oracle\Jinitiator\cache and do not need tobe downloaded again, unless they change. In fact, you can see the files being downloaded into theappropriate directory by simply opening it up in Explorer and watching.

Things change dramatically for subsequent logins :

Figure 18 : Oracle Applications Startup - JAR file cached

Two important things to note :1. The amount of network traffic is orders of magnitude smaller, Caching means that files are

downloaded once !2. The same behavior is seen in 10.7 NCA, R11 and R11i.

Once a JAR file has been downloaded, it is stored in a modified form in the jcache directory. Extrainformation is stored inside the file which is then given a seemingly cryptic name. The additionalinformation allows Jinitiator to uniquely identify the file and compare it to the JAR stored on theWebserver.JAR files on a web server are uniquely identified by a URL as shown in the excerpt from an R11 startuphtml script :

<OBJECT classid="clsid:9F77A997-F0F3-11d1-9195-00C04FC990DC" width="250" height="40" codebase="http://BUNYIP.ca.oracle.com/jinit115211.exe#Version=1,1,5,21,1"> <PARAM name="type" value="application/x-jinit-applet;version=1.1.5.21.1"> <PARAM name="code" value="oracle.forms.uiClient.v1_4.engine.Main"> <PARAM name="codebase" value="/OA_JAVA/"> <PARAM name="archive" value=" /OA_JAVA/oracle/apps/ fnd /jar/ fndewt.jar, /OA_JAVA/oracle/apps/ fnd /jar/ fndforms.jar, /OA_JAVA/oracle/apps/ fnd /jar/ fndaol.jar,

In Windows, a URL would not be a legal filename, so Jinitiator produces a hash of the URL to produce avalid, unique filename which can be compared during startup. Figure 19 shows a cached JAR file viewedin a hex editor. The URL is encoded into the cached file and it is no longer a standard zip archive.

Figure 19 . Cache JAR file internals

Every time a JAR file is used from cache, its timestamp is updated.

Prior to the introduction of caching within the appletviewer, JAR files were sometimes copied to the localPC drive.Does downloading JAR files directly to a PC and storing locally help now that caching is available ?

Not significantly. A number of test runs were carried out loading Applications JAR files from the WebServer, cache and from local disk. The times measured were from initial selection of Applications (frombookmark) through to appearance of the sign on screen. Timing was carried out using an “on-screen”stopwatch.

The results are summarized in the table below :

First Time Second Timeseconds seconds

Webserver Run 1 48 40Webserver Run 2 46 38Webserver Run 3 46 38Local (C:\) Run 1 40 38Local (C:\) Run 2 38 38

Other than initial loading, no significant improvement in subsequent Applications’ startup time was noted.Results would indicate that the serious version control problems created (and the unsupportedconfiguration) and the insignificant performance gain mean that loading JAR files locally does not warrantconsideration.

Of course, to every “rule” there is always and exception …In order to minimize network traffic for applications startup and initial install in situations where bandwidthis at a premium, it is possible to place jar files on a local webserver. This still centralizes files but savesWAN bandwidth during setup and ongoing maintenance.This setup is Figure 20 :

W AN

Local Web Server

<EM BE D w idth=250 height=40 p luginurl="http ://BU N YIP.ca.oracle.com /jin it115211.exe " type="application/x-jin it-applet;vers ion=1 .1.5.21.1"

code="oracle .form s.uiC lient.v1_4.eng ine.M ain" codebase="/O A_JAVA /" java_a rchive="http ://BU N YIP.ca.oracle.com /fnd ew t.ja r, h ttp ://B U NY IP.ca.o racle.co m /fn dfo rm s.jar, h ttp ://B U NY IP.ca.o racle.co m /fn daol.jar, h ttp ://B U NY IP.ca.o racle.co m /fn dc tx .jar, h ttp ://B U NY IP.ca.o racle.co m /fn dtcf.jar, h ttp ://B U NY IP.ca.o racle.co m /fn dh ie r.jar, h ttp ://BU N YIP.ca.oracle.com /ako bjnav.ja r" serverPort="9002" serverApp="O racleApplications" reg istryPath="/O A_JAVA /orac le/apps /fnd /form sC lient" serverArgs="m odule=E:\oa\applts t\fnd\11.0.28\fo rm s\U S\FN DS CS G Nuserid=app lsyspub/pub@ VD 11 fndnam =apps"

c lien tBrowser="netscape">

<N O EM BED >

B U N Y IP .ca .o ra c le .co m

b ig foo t.u s .orac le .co m

h ttp ://b ig foo t.u s .orac le .co m :6 6 6 6 /O A _ H T M L /U S /a p p s_s ta rt.h tm

apps_start.htm

JAR files served fromlocal w ebserver

Client

Figure 20 : Setting up JAR files on local WebServer

The local webserver can be Oracle Applications Server or another webserver : Xitami was used for thistest configuration.

Having downloaded the JAR files, the next step in the process is Developer Server and Forms.

Developer Server

Oracle Developer Server communicates with the Java Client using messages. These messages form acollection of name-value pairs that tell the client which object to act upon and how. The use of this Meta-data reduces network traffic.

Message StructureObject Property = Value Pairs

eg. Obj001: x=10, y=23Message structure can be seen more clearly from the following extract of a “translated” InterceptServertrace :

<message type=2 handlerClassId=257 handlerId=490> <property id=137 datatype="java.lang.String">Intercept Server Journal</property> </message>

The HTML style tags are added by InterceptServer convert utility for ease of reading. The actual traffic isan undecipherable hex stream. The action above is setting a text field in a Journal Entry.

The name-value pair structure is very efficient. The 317 Kbyte (fmx) GL Journal Entry form produces onlyabout 15 KB of network traffic. The Accounts Payable Invoice workbench (1,956 Kbyte fmx) translates toless than 40 Kbytes of network traffic. To put this into context, traffic generated by Developer serverloading a form across the network is of the same order of magnitude as a typical “gif” image file found ona website such as the OAUG website : www.oaug.org .

Even when there is no Applications traffic, a small amount of traffic between client and server is produced.This is the “heart beat” generated by Developer server. Every two minutes the client sends a 6 bytesequence that tells the server that it is alive and well. If this is not received, the server process shutsdown. The heart beat functionality ensures that dead connections are cleaned up and do not consumesystem resources.

Oracle Applications Network Behavior

The first thing to note about Oracle Applications network traffic is that it is bursty. A source that sendsnetwork traffic at markedly different rates at different times is described as being “bursty”.This is an important consideration for deployment. The network traffic fingerprint of an Oracle Applicationsuser will not be steady even flow of data. It will vary over the course of a session depending on the type ofactivity - Navigation or Data Entry.

Figure 21 shows the network traffic generated during the course of the GL Journal Entry Transactions.Peaks occur when the Applications are downloading form description meta-data (ie loading a form).

Figure 21. Client Traffic for GL Journal Entry -using NT Performance Monitor

CAVEAT : THINGS ARE NOT ALWAYS AS THEY APPEAR !

During testing it was noted that peak rates were often not captured or displayed by the tool used (NTperformance monitor sampling at 1 second intervals). The Cloud was found to be more accurate indetermining peak rates because it captures and processes every packet. This discovery brought to light asecond issue. When measuring network traffic produced as a result of LoadRunner or InterceptServer,artificially high spikes for form drawing activities were observed. The traces were markedly different fromnetwork traces taken directly from the application with actions executed manually. Two factors were foundto be coming into play :1. During a “human” Oracle Applications session, there is an interaction between the GUI (Graphical

User Interface), the user and the underlying Developer messaging. When playing back a recordedtransaction, the GUI and human interaction is no longer present and individual messaging eventsoccur much more rapidly. The inclusion of “wait times” is not sufficient. This is the reason that MercuryWinRunner records actual GUI activities and is used during playback scenarios.

2. An application running on a LAN with effectively unrestricted bandwidth can and will grab as muchdata as it can at any one time - hence the spikes. WANS are self limiting in this respect sinceunlimited bandwidth is not available.

Despite the differences in peak shapes between the two types of traces, the area under the curve (totalbytes) is the same - as it should be given the same amount of data is being transmitted and received.This effect was not noticed during the data entry phase, where in comparison, the average traffic isapproximately 1Kbps. If one considers that the majority of time is spent in a data entry phase, theefficiency of the architecture becomes apparent.

Different modules have different forms of varying size and varying complexity. A more complex case froma networking point of view is Accounts Payable Invoice entry. The AP workbench form is much larger

than the GL forms. In addition during the course of a transaction, the user will be calling the invoicedistributions form. As expected the traffic profile is different as shown in Figure 22.

Figure 22 : Accounts Payable Transaction

These traces were taken for transactions taken over a 10BT LAN. We are interested in not only the peakbut also the area under the curve - how much data is actually transferred. When this is calculated, theactual amount of data transferred is of the same scale as a typical small “gif” image downloaded from aweb site.

As bandwidth is throttled, the peaks will reduce dramatically and broaden as shown below in Figure 23aand 23b for the same GL transaction over an unrestricted link and over a link restricted to 28.8 Kbps.The important points to note are :• For the same transaction, the same amount of data is transferred.

• When bandwidth is unrestricted ie on a Local Area Network, peaks are narrow and “high”• When bandwidth is restricted (28.8 Kbps) peaks broaden and shorten

• The same transaction will take longer. This phenomenon is addressed in more detail later in this paper.

Figure 23a : GL Transaction with no bandwidth throttling

Figure 23b : GL Transaction with bandwidth throttled down to 28.8 Kbps

Latency In order to describe the effects of latency, it is useful to have a working definition of time :

“Time is what stops things happening all at once” As the distance between client and server becomes longer as occurs in a WAN, it takes more time fortransactions to take place. This is a direct function of the number of packets that have to be exchangedbetween client and server AS WELL as the design of the user interface. A properly designed interfaceactually absorbs link latency thereby reducing the “spongy keyboard” effect observed in telnet (charactermode) applications operating over long data links. Oracle Applications has been optimized to reduce the number of round trips that need to take placebetween operations such as moving between fields. The following results were obtained using MercuryLoadRunner, that is the user and user interface were not present and only the network effects on thedeveloper server messaging were being measured. In a live situation, human interaction, think times andthe user interface come into play and latency effects are minimized.

Effect of Latency on GL JE transactions(using Mercury Load Runner)

0

50

100

150

200

250

0 20 40 60 80 100 120

Latency in milli seconds

Tim

e to

Com

plet

e R

un

(sec

onds

)

JE 1

JE 2

JE 3

Total

Figure 24. Effect of Latency on GL transactions Higher Latency test run :

Effect of Latency on GL JE Transactions(using Mercury LoadRunner)

0200400600800

100012001400

0 50 100 150 200 250 300 350 400

Link Latency ms

Tim

e to

Com

plet

e R

un(s

econ

ds)

JE1

JE2

JE3

Total

Figure 25. Effect of Latency on GL transactions The difference between the three JE transactions is the number of GL lines entered. The Total transactionincludes : startup, form load and additional sleep times. As a result of the larger volume of data required

by activities shown in the Total transaction, it naturally shows a greater (but still linear) latencydependency. Within the bounds of experimental error, the application behaves in a linear fashion. There is no “dog-leg”where performance degrades dramatically. Similarly for AP :

Effect of Latency on AP Invoice Entry Transactions(using Mercury Interactive)

0

100

200

300

400

500

600

0 50 100 150 200 250 300 350 400

Latency ms

Tim

e to

Com

plet

e R

un

(sec

onds

)

Invoice 1

Invoice 2

Invoice 3

Invoice 4

Invoice 5

Total

Figure 26. Effect of Latency on AP transactions So in summary, if a given transaction is measured from two points in a network with different latencies,simple statistics can infer the performance at another point with an appropriate level of confidence. Bandwidth Throttling As link bandwidth is reduced, peaks broaden since less data can be transferred in a given time period.The following data was obtained using Mercury LoadRunner to drive a series of transactions through TheCloud to emulate different data links. It does not take into account User Interface response nor useractivity. It merely reflects the movement of packets across a link. The reality is that the Oracle Application User Interface absorbs a large proportion of this effect. Significant bandwidth effects appear to disappear at around 128kbps. In “real life” a number of factors need to be considered :• Think Time• Navigation Time : tab-ing or mousing between fields From the graph bellow it can be seen that there significant difference between transaction run times at128Kbps and on a 10BT LAN.

Effect of Bandwidth on GL Transactions

0

10

20

30

40

50

60

70

28kbps 56kbps 64kbps 128kbps 10BT

Bandwidth

Tim

e to

Com

ple

te R

un

JE 1

JE 2

JE 3

D ecreasing T ransaction tim e

= C heck w ith C isco R outer D C E/D T E clock ra te

Figure 27. Effect of Bandwidth on GL transactions Decreasing bandwidth effectively has the same overall effect as increased link latency in terms ofthroughput. However, its effects are manifested differently - screen drawing and when relatively “large”amounts of data are being transferred between client and server. Latency tends to be an overall effect. During different phases of a session, Oracle Applications exhibits different packet sizes. The packet sizedistribution for a series GL transactions is as shown in Figure 28.

Packet Size Distribution

0

200

400

600

800

1000

1200

1400

1600

1800

2000

1 to75

76to

150

151to

225

226to

300

301to

375

376to

450

451to

525

526to

600

Number of Packets

Figure 28 : GL Transaction packet distribution. The majority of the packets are small with larger packets used during startup (and JAR file download)Form definition/drawing. The actual size of the larger packets is essentially dictated by the MTU(Maximum Transmission Unit) for TCP/IP. There is a large amount of information about setting

parameters such MTU and TCP/IP Window sizes. Much of it relates to improving Windows Dial-upperformance and Cable Modem performance. The danger with modifying these settings on a LAN /WANis that there is as much (if not more) chance of degrading performance as there is of gainingimprovements. For example setting MTU to a value greater than what network equipment (routers andPCs) can result in performance degradation due to packet fragmentation. Tread carefully and researchbefore changing things !

Packet Size Distribution

0

10

20

30

40

50

60

70

80

Per

cent

age

1-64

65-127

128-255

256-511

512-1023

Figure 29 : Packet size distribution of the course of a transaction. Figure 29 shows how different packet sizes are present at different stages of a Oracle Applicationssession. From left to right, during startup/login and navigation there is a higher percentage of the largersize packets. The larger packets appear whenever form meta-data is transferred. This tails off during thedata entry phase. A more detailed analysis of this behavior is outside of the scope of this paper, butpacket (frame) size distribution is an important consideration when carrying out detailed analysis of anetwork and when implementing technologies such as VPNs and QoS. Bandwidth and Latency act upon the network traffic associated with a transaction in different waysalthough both have the overall effect of lengthening the time it takes to complete a transaction. The following traces (Figures 30 and 31) show the network behavior of Oracle Applications underdifferent bandwidth conditions :

Figure 30 : 28.8 Kbps and 56 Kbps data link (vertical scale 8Kbps) Despite the different traces, the same amount of data is being transferred for each individual action. Notethat irregular shaped peaks are in most cases a function of the sampling interval chosen for graphing.(verification and calculation of areas is set as an exercise for the reader!). Bandwidth throttling limits the instantaneous rate of transfer, so at lower bandwidths the peaks arebroader. In other words it takes more time to download a given form. Data entry is not as affectedbecause data transfer rates during this type of activity are much smaller than those experienced duringform drawing. The twin peaks during the Logon sequence are caused by the delay when entering username andpassword.

Figure 31a : 64 Kbps, 128 Kbps, 10 BT (Vertical Scales 8 Kbps, 20 Kbps, 20 Kbps)

Figure 31b : 64 Kbps, 128 Kbps, 10 BT (Vertical Scales 8 Kbps, 20 Kbps, 20 Kbps) As bandwidth increases peaks become sharper.

Figure 32 : Effect of 100ms and 200 ms Latency (Vertical Scale 20 Kbps) Studying Figure 32, it is reasonable to say that while Latency does have an affect on activities such asform drawing, it’s effect is not as great as bandwidth restriction. This stands to reason since during thecourse of data transfer, receipt of packets is acknowledged and this is taking fractionally more time.Latency is more noticeable when doing data entry and navigating within a form. Bandwidth restriction ismore evident during navigation. The Effect of multiple users on a Data Link How many users ? The answer is always : “it depends….” Why ? Going back to the analogy of water flowing through a pipe in Figure 33, if there is already water in the pipe,there is obviously less room for additional flow. In a network the existing water in a pipe is other businessand administrative traffic.

W ate r W aves

Direction

Direction

W ater W aves

Exis ting W ate r in P ipe

Less o f p ipe is ava ilab le

Figure 33 : Water flowing through a partially filled pipe. So starting with a single GL user being emulated using InterceptServer we obtain the baseline shown inFigure 34.

Figure 34 : Single GL User InterceptServer As more users are added, the effect is additive and the average level in the pipe increases. In Figure 35 the effect of 5 and 10 users over a link is illustrated. For these tests additional users wereadded every 30 seconds till the target numbers were reached. Simplistically it is a if the level of water inthe pipe is rising. What works in Oracle Applications’ favor is the fact that the traffic is bursty - there is nota constant stream from any one client.

Figure 35 : 5 and 10 GL Users driven by InterceptServer From earlier results, we have seen how the effect of reduced bandwidth is to spread out the peaksbecause the application has a finite amount of data it needs to transmit and the data link is limited in thespeed at which it can pass it. So as more users are placed on a limited capacity datalink (where there willmost likely be existing non- applications traffic…), the time taken for a user to complete a giventransaction will increase.

Every module and transaction type has its own network fingerprint. An AP user entering invoices will havea slightly higher bandwidth requirement than a user entering Journals. The main factor in this is therequirement to draw intermediate windows such as the AP Invoice Distributions window. So the question “How many remote users can I put over my 64K ISDN line” is not strictly correct. It shouldbe “How many users can I accommodate on my data link given a certain level of response and existingtraffic level”. The best way to answer this is to profile the transactions the user will be carrying out and derive andaverage from this “transaction fingerprint” rather than assuming a single network numbers. Also allow forthe fact that users will be all starting work over a given time period and logging on.(see Figure 35). Oncethis peak is over the data entry phase will exhibit much lower average traffic. Application Desktop Integrator Network Behavior Oracle Applications Desktop Integrator (ADI) is a client-server tool that allows access to OracleApplications through Microsoft Excel. Unlike Oracle Applications internet architecture, ADI uses SQL*netfor communications between the desktop and the server. This is shown in Figure 36.

PC

ApplicationsDesktop

Integrator

SQL*net

SQL*net

TC P/IP

SERVER

ADI Com munications

Figure 36 : Applications Desktop Integrator Network Architecture It is no secret that SQL*net is not as efficient as the messaging protocol used by Developer Server and asa result is more susceptible to sluggish response over Wide Area Network links. To examine ADI’s network behavior, a routine spreadsheet journal entry was measured under differentnetwork conditions. The steps taken were as follows : • Start ADI• Login as OPERATIONS/WELCOME• Allow ADI + Request Center to start• With Excel already running in the background, Select Journal Entry• Set up 101 Journal lines• Set date from List of Values• Upload - Selecting All Rows and either Full Validation or None• Exit

These steps were carried out at different emulated network latency and with the profile option ServerFlexfield Validation turned both on and off. In addition client Pre-Validation was set to either Full or Noneas shown in Figure 37 :

Figure 37 : Client Pre Validation settings in ADI

In addition, Logging was also turned on for ADI. (Please refer to ADI User Guide Appendix A.5). Thiscreates two log files in the PC’s TEMP directory : GLJUMETR.TXT, which details execution times andSQL , and GLSOMETR.TXT which lists API calls. These files are invaluable aids to tuning and monitoringJournal Entry operations.

The first scenario examined is a LAN link with 0ms latency and Full Client Validation

Figure 38 : ADI 0ms Latency Full Client Validation

As the latency of the link increases, the network picture changes :

Figure 39 : ADI 100ms Latency Full Client Validation

A couple of things to note from the trace are :1. Traffic peaks are smaller2. What took 30 seconds on a LAN, now takes approximately 128 seconds.

The different trace profiles may be explained by our working definition of time :“TIME IS WHAT STOPS THINGS HAPPENING ALL AT ONCE”

Because of the 100ms latency, things are simply more spread out and effectively less data is being sent atany given instant.

Excerpts from GLJUMETR.TXT :

0 ms Latency Full Client Validation 100 ms Latency Full Client ValidationTIMING STATISTICS -- VALUES SHOWN INMILLISECONDSGENERAL PROCESSING 1023GET VALUES INTO MEMORY 310BUILD SQL STATEMENT 140BIND VALUES INTO SQL STATEMENT 220EXECUTE SQL STATEMENT 19958VALUE VALIDATION 3697PASTE RESULTS INTO SHEET 420TOTAL MILLISECONDS 25768TOTAL MINUTES 0.43ROWS PER SECOND 3.92ROWS PER MINUTE 235.18

VALUE VALIDATION IS COMPRISED OF:DATE VALIDATION 80CONVERSION VALIDATION 0BUDGET VALIDATION 0ENCUMBRANCE VALIDATION 0REVERSAL VALIDATION 40AMOUNTS VALIDATION 20NULL VALIDATION 101LIST OF VALUES VALIDATION 100DATA TYPE VALIDATION 40

TIMING STATISTICS -- VALUES SHOWN INMILLISECONDSGENERAL PROCESSING 1403GET VALUES INTO MEMORY 290BUILD SQL STATEMENT 160BIND VALUES INTO SQL STATEMENT 260EXECUTE SQL STATEMENT 44674VALUE VALIDATION 57093PASTE RESULTS INTO SHEET 430TOTAL MILLISECONDS 104310TOTAL MINUTES 1.74ROWS PER SECOND 0.97ROWS PER MINUTE 58.10

VALUE VALIDATION IS COMPRISED OF:DATE VALIDATION 851CONVERSION VALIDATION 0BUDGET VALIDATION 0ENCUMBRANCE VALIDATION 0REVERSAL VALIDATION 10AMOUNTS VALIDATION 10NULL VALIDATION 110LIST OF VALUES VALIDATION 1723DATA TYPE VALIDATION 90

ACCOUNT VALIDATION PROCESSING 601ACCOUNT VALIDATION EXECUTE SQL 2274ACCOUNT VALIDATION TOTAL 3176

VERSION AND ATTRIBUTE INFORMATIONAPPLICATIONS VERSION 11FLEXFIELDS WERE VALIDATED ON THE CLIENTSVFLAG 0

ROW COUNTSROWS SCANNED 101ROWS UPLOADED 101VALID ROWS 101INVALID ROWS 0

TOTAL VIRTUAL MEMORY (MB) 2047.88VIRTUAL MEMORY IN USE BY ALL APPLICATIONS(MB)BEFORE UPLOAD 71.10DURING UPLOAD 71.14 UP BY 0.04END OF UPLOAD 71.14 UP BY 0.04

TOTAL PHYSICAL MEMORY (MB) 255.36PHYSICAL MEMORY IN USE BY ALL APPLICATIONS(MB)BEFORE UPLOAD 69.16DURING UPLOAD 69.20 UP BY 0.05END OF UPLOAD 69.93 UP BY 0.78

TOTAL EXCEL MEMORY (MB) 1.32MEMORY IN USE BY EXCEL (MB)BEFORE UPLOAD 0.31DURING UPLOAD 0.31 UP BY 0.00END OF UPLOAD 0.32 UP BY 0.01

ACCOUNT VALIDATION PROCESSING 511ACCOUNT VALIDATION EXECUTE SQL 52566ACCOUNT VALIDATION TOTAL 53338






Comparing the results for the WAN and LAN links it is obvious that validation is affected by the linklatency. The GLJUMETR.TXT log files contain a wealth of information and unfortunately a detaileddiscussion of GL metrics and timing differences is outside the scope of this paper.

So the problem we are faced with is that client side validation on journal upload is very susceptible to longlink latencies.Solution : Change to Server side Flexfield validation with profile option “Flexfields:Validate On Server” setto “YES”

Figure 40 : ADI 0ms Latency, Flexfields Validate on Server, Full Client Validation

At 100 ms Latency

Figure 41 : ADI 100ms Latency, Flexfields Validate on Server, Full Client Validation

Using Server Side Flexfield Validation, excerpts from GLJUMETR.TXT

0 ms Latency Full Client Validation 100 ms Latency Full Client Validation


VALUE VALIDATION IS COMPRISED OF:DATE VALIDATION 110CONVERSION VALIDATION 0BUDGET VALIDATION 0ENCUMBRANCE VALIDATION 0REVERSAL VALIDATION 50AMOUNTS VALIDATION 10NULL VALIDATION 70LIST OF VALUES VALIDATION 71DATA TYPE VALIDATION 30ACCOUNT VALIDATION PROCESSING 662ACCOUNT VALIDATION EXECUTE SQL 2363ACCOUNT VALIDATION TOTAL 3265







VALUE VALIDATION IS COMPRISED OF:DATE VALIDATION 891CONVERSION VALIDATION 0BUDGET VALIDATION 0ENCUMBRANCE VALIDATION 0REVERSAL VALIDATION 60AMOUNTS VALIDATION 10NULL VALIDATION 121LIST OF VALUES VALIDATION 1672DATA TYPE VALIDATION 50ACCOUNT VALIDATION PROCESSING 531ACCOUNT VALIDATION EXECUTE SQL 52366ACCOUNT VALIDATION TOTAL 53167






It is also possible to use no validation during upload. This further reduces the effect of latency by reducingclient-server communications to a bare minimum.

The results obtained are summarized in the following table :

Upload Performance in SecondsLatency

Option 0ms 100msFlexfields:Validate on Server = NO

Full Client Validation on Upload 30 128None 28 69

Flexfields:Validate on Server = YESFull Client Validation on Upload 39 (32) 82

None 27 73

This table describes the results of Journal Uploads under different validation and latency conditions. Somevariability on multiple runs under the same conditions was observed. When uploading with Validation onServer and Full Client validation, the first test yielded 39 seconds - the second run 32 seconds. At thispoint no explanation for the higher than expected result can be offered. Ideally multiple runs would becarried out and the results would be analyzed statistically.

A performance improvement of approximately 35% was obtained by using Server side flexfield validation.This was a reduction in time from 128 seconds to 82 seconds. The magnitude of this reduction is sufficientto conclude that Server based Flexfield Validation is worth using under WAN conditions.

It should be noted that ADI version 4 and above has a number of internal optimizations for WANoperation. Firstly, in order to minimize the number of round trips, batch validation is used - information iscached and then passed onto the server in one lot rather multiple exchanges. The use of this caching canbe monitored using the gljumetr.txt log file. If a client is doing large journal entries, it may be advisable toincrease the RAM on the machine to ensure swapping is not taking place. In addition, during validation,the same data is never validated twice in order to keep network traffic down.Finally, the GLJUMTR.TXT file also captures executed SQL so that it can be examined and the ifnecessary the database can be tuned with additional indexes etc.

Virtual Private NetworksA Virtual Private Network simulates a private network over a public medium such as the Internet. The termVirtual comes about because connections have no physical presence but rather are formed by packetssent from source to destination which encapsulate the original information. This encapsulation processeffectively forms a secure tunnel between source and destination.

Oracle Applications encrypts conversations between the client and middle tier server and transmits themover TCP/IP. While this is more than adequate for intranet use, Applications internet architecture is not asuitable vehicle for communications when the client is somewhere in the Internet due to the need forfirewalls which generally operate by rejecting inbound packets from foreign hosts. This protocol limitationwill be removed in the future with the use of HTTP-s, but for the moment it remains a stumbling block totrue Internet deployment of Applications across firewalls.

The secure tunnels created by VPNs provide the solution to this problem. By encrypting and encapsulatingthe traffic between the client and the middle tier server, VPNs, which are designed to work with firewalls,safely bridge the chasm between the internet and the corporate intranet.

There are a number of vendors providing VPN solutions, including Microsoft, Sun and RedcreekIn addition there is an emerging standard IPsec. VPNs can be implemented in software or hardware (or amixture). The chief drawbacks are the processing overhead of encryption and the network overhead theycreate due to encapsulation.

Edmonton Public School Board (EPSB) is a long-standing customer of Oracle Applications based inAlberta Canada. They are currently in the process rolling out 10.7 using internet computing architecture totheir school district. With the proliferation of the internet within the school environment, EPSB is leveraging

the ISP connections used by some of its schools to provide Oracle Applications access. This is donethrough a series of Virtual Private Networks (VPN).

EPSB chose to implement a solution using Free BSD servers and SKIP tunneling software from SunMicrosystems. While this is does not conform the IPsec standard, it was adopted for functionality and costreasons. The architecture is shown in Figure 41 :

IBM Compatible

INTERNET VIA ISP

IBM Compatible

D O W N T O W NINTRANET

Da tabase andMiddle Tier

Servers

School Intranet

Clients runningWindows 95

Free BSD2.2 .8

F ree BSD2.2 .8

FIREWALL +SKIP VPN

SOFTWARE

FIREWALL +SKIP VPN

SOFTWARE

Edmonton Publ ic School Board VPN Implementat ion

Figure 42 : EPSB VPN architecture

Operation of the VPN is transparent to the clients on the school intranet. The Free BSD PCs at theindividual schools are responsible for all firewall, VPN and Network Address Translation at the schools.

The VPN gateway machine does the routing, tunneling, encryption and authentication. Traffic that isInternet bound goes to their ISP. Traffic that is bound for the EPS WAN goes through the encryptedtunnel. Network connectivity is seamless and transparent at the IP layer. No changes are needed on anyclient or server machinery.

The VPN gateway machines do NAT (Network Address Translation) rather than IP masquerading. Theresult is the same but it is much more functional and simpler to implement. (IP Masquerading is peculiarto Linux and is not widely used. NAT is an Internet standard defined by several RFCs and is widely usedby many vendors such as Cisco, etc.)

The tunnel is built using an IP encapsulation daemon. This daemon handles the tunneling and routing tothe private portion of the network. The tunnel is encrypted (and each packet authenticated) with SKIP(Secure Key Management for IP - See http://www.skip.org) SKIP is a mature and robust technology.While it is not a true "standard", it was the most widely deployed VPN solution in the world (at the time).The emerging IPsec standard (EPSB’s first choice) was not suitable at the time.

A packet filtering firewall (in the BSD kernel) protects the VPN gateway itself. The VPN gatewayprovides whatever services are needed by the LAN behind it. (DNS, DHCP, proxy, HTTP, FTP, SMTP,

POP, etc.) Due to this architecture the machines on the school LAN do not need to be modified in anyway whatsoever.

The protocol overhead is probably about 10%. (SKIP reduces the MTU by 136 bytes. Normally the MTUwill be 1500 bytes).

The above configuration has been working well other than some early problems due to the default settingused by Microsoft TCP/IP on the client PCs. This was fixed by increasing registry parameters related toconnection retries as detailed below :

[HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\VxD\MSTCP]

"MaxDataRetries"="100""MaxConnectRetries"="100"

Quality of Service

Figure 43 shows different types of traffic and presents one viewpoint of their relative importance inbusiness:

Mission CriticalCasual

High

Low

Del

ayS

ensi

tivity

Im portance of Traffic toBusiness

��

��FTP

��

��

Em ail

��

O ther Applications

Intranet W eb Traffic

External W eb Traffic

Relative Im portance andDelivery Sensitivity of

Netw ork Traffic

Figure 43 : Relative Importance of different types of network traffic within a business

This type of prioritization will change from organization to organization and even from department todepartment. Particularly where WAN links are involved, these different types of traffic may compete forbandwidth with the result that performance of mission critical systems may suffer. A classic example of

this is FTP (File Transfer Protocol) traffic affecting other users on a network : In one test, downloading a200 MB file on the same subnet as users using Oracle Applications downgraded response times within theApplications by approximately 20%. This is not a result of Oracle Applications network behavior, thenetwork itself essentially does not differentiate, it is simply network contention and congestion.

Quality of Service or QoS is a group of technologies that aim to make optimum use of an existing networkby allowing administrators to specify appropriate carrying capacity and priority for different traffic andapplication types. A number of vendors have QoS offerings and even Windows 2000 has QoSfunctionality.

Cisco Systems is an Oracle Applications customer and has carried out a number of tests using its QoSproduct and Oracle Applications.

A typical WAN architecture in Figure 44. The links between the different sites are often of differentcapacities. Prioritization of this traffic according to business requirements is highly desirable.

W ANH ub

H ub

Server

R oute rR oute r

Remote Site

Main SiteH ub

Remote Site

R oute r

��

��

��

FTP

te lnet

��

��

W AN Traffic

Figure 44 : Typical WAN architecture

The main focus of QoS is WAN links where bandwidth is at a premium. Figure 45 is a simplified view ofCustom Queueing mechanism employed by Cisco IOS :

��

��

FTP

��

��

��

��

FTP

Traffic IN Traffic OUT

ClassifyTraffic

Traffic is classifiedaccording to Protocol,port etc

out-go ing packe ts

Queues

��

��FTP

��

��FTP

��

��FTP

��

Out-going traffic isscheduled such that nosingle application orgroup of applicationstakes m ore than apredeterm inedproportion of linkcapacity and isprioritized

O racle A pp lica tions T ra fficG iven P rio rity

Figure 45 : Custom Queuing

Tests in Cisco/Oracle test labs demonstrated significant performance improvement when using QoStechnology with Oracle Applications running over low speed WAN links.

There are a number of different queuing strategies and tools available. The difficulty is the evolution ofpolicies that provide the expected levels of service without over-penalizing certain types of traffic.

When Things Go Wrong …

Networks are a complex beast. A network is not unlike an Apps database (a miss-applied patch, orperhaps a strange file layout inherited from the previous Sys Admin) : it has its quirks and idiosyncrasies.The only effective way to track a network problem is to be systematic and prepared. In order to do thisyour network must be :

• Documented• Monitored• Baselined Documentation speaks for itself. When an equipment failure occurs during the middle of month endprocessing, knowing what connects to what is absolutely vital. The sad reality is that networks “mutate”,configurations are tweaked, hubs are re-patched. It is very easy to lose track. Monitoring and Baselining go hand in hand. A baseline is simply a known, documented level of usage fortypical business operations. Knowing what your usage patterns serves a couple of purposes : • Preventative. Sometimes you can see if things are “Heading South” - a broadcasting network card,

user carrying out unauthorized snooping, bandwidth hogs.• Knowing when in the day your utilization is highest - when to expect the phone calls.• Helps build business cases when new equipment is required or data links need bolstering.

When the phone is ringing off the hook with users reporting performance problems it is often too late tostart getting a baseline. The first questions you will be asked during the course of troubleshooting is “Whatis the typical performance like ? Got any graphs ?”. Not having them wastes time when it is most precious. Finally get a small “toolkit” together. It doesn’t need to be a WG analyzer. At the end of this paper anumber of useful FREE tools are listed. Get to know how your network behaves and how Applicationsinteracts with it : Be prepared ! Conclusions This paper has presented a brief empirical overview of a number issues related to Oracle Applications’behavior in a network. My hope is that it will help provide the basis for both qualitative discussions and“back of envelope calculations” and help focus attention on the importance of networking. The more one delves into the world of networking, the more it becomes obvious how complex andinterconnected things become. At best this paper is a snapshot of a work in progress. The major points and lessons learnt are • Make sure that FORMS45_MESSAGE_ENCRYPTION is set to TRUE• Oracle Applications traffic is bursty in nature• Behavior over a latent link is linear• Throttling bandwidth effectively acts in the same way as increased link latency but is manifested in

different parts of the application.• The Oracle Applications GUI absorbs latency.• Using Flexfield: Validate on Server = TRUE can result in dramatic performance improvements for ADI

over WAN links.• VPNs can solve security issues but incur additional network overhead.• Quality of Service technologies can help prioritize network traffic like Oracle Applications and ensure

that it is not swamped out by less critical traffic.

Thankyou

A number of people helped with this paper. Special thanks to

Rob Close : Cisco Systems

Robin Garrison : Mercury Interactive

Steve Holder : Shunra

Brett HookerPeter HellerDave Bodnarchuk

And last but by no means least, Shelly Popescul for her phenomenal patience.

Bibliography + Resources

Cisco “Cisco IOS 12.0 Quality of Service”Cisco Press 1999 ISBN 1-57870-161-9

Clewette, A. “Network Resource Planning for SAP R/3, BAAN IV and Peoplesoft”McGraw-Hill 1998 ISBN 0-07-913647-8

McDysan, D. “QoS & Traffic Management in IP & ATM Network”McGraw-Hill 2000 ISBN 0-07-134959-6

Hennessy & Patterson “Computer Architecture A Quantitative Approach 2nd Ed”Morgan Kaufmann 1996 ISBN 1-55860-329-8

Oppenheimer, P. “Cisco Top Down Network Design”Cisco Press 1999 ISBN 1-57870-069-8

Useful Network Tools (FREE!!)

Analyzer : http://netgroup-serv.polito.it/analyzer/) protocol analyzer (win32)Qcheck : http://www.qcheck.net Ganymede Network testerEthereal : http://ethereal.zing.org/ LinuxEthernet AnalyzerNistnet : http://www.antd.nist.gov/itg/nistnet/ Linux WAN emulator

About the Author

Robert Azzopardi started using Oracle technology in 1991 and has been with Oracle since 1995 workingexclusively with Oracle Applications. He is a Principal Sales Consultant with Oracle Canada based out ofCalgary and is involved in pre-sales, consulting, technical support and custom development with OracleApplications and related technologies.

Prior to Oracle Canada, he was with Oracle Australia in a similar role. Before that he was an ElectricalEngineer working with Distributed Control Systems and Instrumentation, but that was a long time ago andthe only thing he remembers about that period of his life is Ohm’s Law.

Date post:	03-Feb-2018
Category:	Documents
Upload:	duongthuan
View:	217 times
Download:	2 times

White Paper: Nothing but Net : Applications, a Networking ... · PDF fileThe majority of...

Documents