1.WMS Performance ShootoutAndrea Aime (GeoServer)Jeff McKenna (MapServer) Open Source Geospatial Foundation 1

2. Executive summary Compare the performance of WMS servers GeoServer MapServer In a number of different workloads: Vector: plain polygon, point with graphical icons, roadnetwork with cased lines, road network with labels Raster: global imagery Against different data backends: Vector: shapefiles, PostGIS, Oracle spatial, SDE overOracle spatial Raster: ECW and GeoTiff mosaics Open Source Geospatial Foundation 2 3. Benchmarking History 3rd FOSS4G benchmarking exercise. Past exercises included: FOSS4G 2007: Refractions Research run and published the first comparison with the help of GeoServer and MapServer developers. Focus on big shapefiles, postgis, minimal styling FOSS4G 2008: OpenGeo run and published the second comparison with some review from the MapServer developers. Focus on simple thematic mapping, raster data access, WFS and tile caching Friendly competition: goal is to improve all software Open Source Geospatial Foundation 3 4. Past MapServer improvements improvements in large shapefile indexing raster read optimization (single pass for multiple bands) enhancing polygon labelling placement EPSG codes caching PostGIS improvements Oracle improvementsOpen Source Geospatial Foundation4 5. Past GeoServer improvements Overall rendering pipeline improvements GML writing speed-ups Raster data access improvements PostGIS data store optimizations Open Source Geospatial Foundation 5 6. Rules of engagement Each server is tested in its latest version Each server performs exactly the same workload Same set of WMS requests Same data backends Same image output format All modifications made to improve performance are to be included in a future release of the software Data used cannot be modified for display, other than indexing All testing to be done on the same benchmarking machines Open Source Geospatial Foundation6 7. Hardware Configuration WMSSDE GeoServer JMeterOracle MapServer PostGISShapefiles Database BenchECWGeoTIFFOpen Source Geospatial Foundation7 8. Software Specs MapServer: 5.6.0-beta3, GDAL SVN Apache 2.2 FastCGI - 8 static processes GeoServer: 2.0-RC2, Java 6 update18, GDAL 1.4.5 Tomcat 6.0.20 - 8 threads Database: Oracle 11.1.0.7 SDE 9.3.1 PostGIS 1.4.0 Open Source Geospatial Foundation 8 9. Hardware specs Bench (2003):Dell PowerEdge 1750 1 Xeon Nocona 3Ghz, 1 GB RAM, OS: RedHat Enterprise 5.3 WMS (2004): Dell PowerwEdge 2850 4 Xeon Nocona 3.4Ghz, 8 GB RAM 6 73 GB, 15K RPM hard drives, OS: RedHat Enterprise 5.3 Database (2007): Dell Optiplex 755 tower 1 x Intel Core2 Duo CPU E6750 @ 2.66GHz / Dual Core, 4Gb RAM 100Gb SATA 3.0Gb/s 7200 rpm HD, OS: RedHat Enterprise 5.3 Open Source Geospatial Foundation 9 10. Hot vs Cold Benchmarks Hot benchmark The file system caches are fully loaded The software has had the occasion to perform whateverpre-processing is necessary (e.g., open connections to thedata source) Cold benchmark The system has been just started There is no existing cache Both are unrealistic, production is usually a mix of both Hot benchmarks are more practical to run Open Source Geospatial Foundation 10 11. Methodology Each test run performs requests with 1, 10, 20 and 40 parallel clients (for a total of 1500 requests) Each test uses a random set of requests stored in a CSV file: no two requests in the same run are equal, but all servers perform the same workload For each request the random factors are: The image size (between 640x480 and 1024x768) The geographic envelope (extent) Each test is run three times in a row, the results of the third run are used for the comparison: this benchmark assumes full file system caches (hot benchmark) The other GIS server is shut down while the tests are run Open Source Geospatial Foundation 11 12. Datasets Used Polygon layer: areawater_merge: the TIGER 2008 set of polygons describing water surfaces for the state of Texas. 380000 polygons, 195MB shapefile, EPSG:4269 Point layer: gnis_names09: all location and points of interest names for the state of Texas in the GNIS database. 103000 points, 2.8 MB shapefile, EPSG:4269 Line layer, edges_merge: all line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. over 5M lines, 1.2GB shapefile, 1.4GB dbf, EPSG:4269 Raster layer: Bluemarble TNG, 86400 x 43200 pixels, 7 overview layers (TIF, ECW, tiled 512&8, GeoRaster) Open Source Geospatial Foundation 12 13. Equivalent traffic table The presentation charts results as throughput in requests per second. This table translates hits per second at sustained peak load in terms of requests per hour and per dayHits/sHits/hour Hits/day 5 18,000 432,00010 36,000 864,00015 54,000 1,296,00020 72,000 1,728,00025 90,000 2,160,000 Open Source Geospatial Foundation13 14. T1: plain polygon rendering TIGER 2008 areawater layer, merged over Texas Simply fill the polygon in blue, no outline Image scales roughly between 1:10.000 and 1:100.000Open Source Geospatial Foundation14 15. T1: Shapefiles GeoServerMapServer CGI MapServer FCGI3025201510 5 0 1 10 20 40 ThreadsGeoServerMapServer CGIMapServer FCGI1 116.5 12 10 25 14.9 25.8 20 25.2 15.3 26.2 40 25.1 15.5 26.8 Open Source Geospatial Foundation15 16. T1: PostGISGeoServer MapServer FCGI 3025201510 5 0 11020 40 ThreadsGeoServer MapServer FCGI1 10.7 10.1 10 24.4 23.7 20 25.2 24.2 40 25.7 24.5Open Source Geospatial Foundation 16 17. T1: Oracle GeoServer MapServer FCGI 25 20 15 1050 11020 40 ThreadsGeoServer MapServer FCGI1 10.29.2 10 23.1 21.1 20 23.8 21.7 40 23 22Open Source Geospatial Foundation 17 18. T1: SDE GeoServerMapServer FCGI25201510 5 0 110 20 40 ThreadsGeoServerMapServer FCGI19.5 8.9 10 23.322.3 20 23.623.2 40 23.323.6Open Source Geospatial Foundation18 19. T1: observations GeoServer is between 25% and 50% faster reading shapefiles out of the box, MapServer with FastCGI is slightly faster PostGIS performs roughly 5% better than Oracle on higher loads SDE seems to introduce no overhead compared to direct Oracle access for GeoServer, and a slight overhead for MapServer GeoServer and MapServer have exactly the same performance when the backend is PostGIS and OracleOpen Source Geospatial Foundation 19 20. T2: point rendering GNIS names dataset over Texas Each point is rendered using a little GIF/PNG graphics Open Source Geospatial Foundation20 21. T2: shapefiles GeoServerMapServer CGI MapServer FCGI3025201510 5 0 1 102040 ThreadsGeoServer MapServer CGI MapServer FCGI19.3 6.612.2 10 24.314.926.5 20 2515.326.6 40 25.215.727.1 Open Source Geospatial Foundation21 22. T2: PostGISGeoServer MapServer FCGI3025201510 5 0 110 2040 ThreadsGeoServer MapServer FCGI1 11 10.8 10 25.1 24.3 20 25.3 24.6 40 25.825 Open Source Geospatial Foundation22 23. T2: Oracle GeoServer MapServer FCGI 30 25 20 15 1050 11020 40 ThreadsGeoServer MapServer FCGI1 10.59.8 10 25 23.3 20 25.2 23.4 40 24.6 23.9 Open Source Geospatial Foundation23 24. T2: SDE GeoServer MapServer FCGI 30 25 20 15 1050 11020 40 ThreadsGeoServer MapServer FCGI1 10.19.6 10 24.5 23.6 20 24.6 23.9 40 23.9 24.5 Open Source Geospatial Foundation24 25. T3: roads map TIGER 2008 edges, just roads (rivers filtered out), three different line styles based on the road classOpen Source Geospatial Foundation25 26. T3: shapefilesGeoServerMapServer CGI MapServer FCGI 201816141210 8 6 4 2 0 110 20 40 LabelGeoServer MapServer CGI MapServer FCGI17.6 58.2 10 1511 17.1 20 13.111 16.8 40 15.111.5 17.6 Open Source Geospatial Foundation 26 27. T3: PostGIS GeoServer MapServer FCGI18 16 14 12 1086420 11020 40 LabelGeoServer MapServer FCGI17.26.6 10 16.3 15.8 20 14.8 15.1 40 16.6 16.4 Open Source Geospatial Foundation27 28. T3: OracleGeoServer MapServer FCGI 14 12 1086420 11020 40 LabelGeoServer MapServer FCGI15.95.6 10 12.5 12.6 20 11.5 11.6 40 12.7 12.8 Open Source Geospatial Foundation28 29. T3: SDEGeoServer MapServer FCGI14 12 1086420 110 2040 LabelGeoServer MapServer FCGI16.14.7 10 13.3 11.1 20 11.69.7 40 13 11 Open Source Geospatial Foundation29 30. T4: labelled roads Same as T3, but higher scale (between 1:10k and 1:3k) and with labels following lines Open Source Geospatial Foundation30 31. T4: shapefilesGeoServerMapServer CGI MapServer FCGI18 16 14 12 1086420 110 20 40 LabelGeoServer MapServer CGI MapServer FCGI16.35.4 6.6 10 13.4 11.914.5 20 13.9 12.715.5 40 13.2 12.615.3 Open Source Geospatial Foundation 31 32. T4: PostGIS GeoServer MapServer FCGI 25201510 5 0 11020 40 LabelGeoServer MapServer FCGI16.68 10 14.3 18.6 20 14.9 19.4 40 14.7 19.1 Open Source Geospatial Foundation32 33. T4: OracleGeoServerMapServer FCGI201816141210 8 6 4 2 0 110 2040 LabelGeoServerMapServer FCGI16 6.8 10 13.216.8 20 13.917.5 40 13.517.3 Open Source Geospatial Foundation33 34. T4: SDEGeoServer MapServer FCGI201816141210 8 6 4 2 0 110 2040 LabelGeoServer MapServer FCGI15.66.4 10 12.8 16.4 20 13.3 17.4 40 13.1 17.2 Open Source Geospatial Foundation34 35. T5: raster Bluemarble TNG, as single ECW file, 512 mosaic TIFF, and BigTIFF (MapServer) vs 8 tiles mosaic TIFF (GeoServer) ECW file is 260MB TIFFs are uncompressed, tiled, with overviews. Around 16GB for each mosaicOpen Source Geospatial Foundation 35 36. T5: ECW GeoServer MapServer CGI MapServer FCGI25201510 5 0 1102040 ThreadsGeoServerMapServer CGI MapServer FCGI15 4 7.4 10 11.112.620 20 11.212.819.8 40 10.512.619.1 Open Source Geospatial Foundation 36 37. T5: 512 tiles mosaic GeoServerMapServer CGI MapServer FCGI18 16 14 12 1086420 1 102040 ThreadsGeoServerMapServer CGI MapServer FCGI14.8 2.7 6.5 109.71114.9 209.611.416.4 409.911.316.9 Open Source Geospatial Foundation37 38. T5: BigTiff vs 8 tiles tiffGeoServerMapServer CGI MapServer FCGI3025201510 5 0 1102040 ThreadsGeoServer MapServer CGI MapServer FCGIGeoServer cannot do1 6.4 4.8 11.1 1013.7 13.326.9BigTiff, so a 8 tiles 2013.315 28.1mosaic was used 4013.6 15.227.3(almost 2G B per tile)Open Source Geospatial Foundation 38 39. FOSS4G 2009 improvements Several code enhancements were made during this year's exercise: MapServer Fixed problem with labels following features Fixed problem with 5.6.0-beta for SDE layers Found problem with 5.6.0-beta for Oracle fastcgi and memory leak GeoServer Fixed problem with Oracle connection pooling and index scanning Fixed problem with ECW crashes under high loadOpen Source Geospatial Foundation39 40. Hopeful Changes for Next Shootout Involvement of more WMS servers Put the servers under a faster internet connection for quicker data transfers and tests setups Modify the number of parallel client requests performed so that the 2-4-8 parallel clients case gets exposed Phases/cycles for benchmarking to allow for teams to interpret results and make software enhancements Reprojection in both vector and raster data Audience suggestions?Open Source Geospatial Foundation40 41. More Information Scripts, configuration files, results stored in OSGeo SVN: http://svn.osgeo.org/osgeo/foss4g/benchmarking/ Wiki home: http://wiki.osgeo.org/wiki/Benchmarking_2009 Mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarkingOpen Source Geospatial Foundation 41 42. Additional MapServer Results (not presented) Open Source Geospatial Foundation 42 43. MapServer Roads/Lines Test ResultsLine Drawing2018161412 Throughput - 1 Maps/sec10 Throughput - 108Throughput - 206Throughput - 40 420 shape 54 fcgi pg oracle oracle ogr sde fcgi shape shape 56 fcgipg fcgioracle fcgisde Open Source Geospatial Foundation 43 44. MapServer Labelled Roads/Lines Test Results Lines with Labels252015 Throughput - 1 Maps/sec Throughput - 1010 Throughput - 20 Throughput - 405 0 shape 54 fcgi pg oracle oracle ogr sde fcgi shape shape 56 fcgipg fcgioracle fcgisdeOpen Source Geospatial Foundation44 45. MapServer Polygon Test Results Polygon Drawing 302520 Throughput - 1Maps/sec 15Throughput - 10 Throughput - 20 10Throughput - 4050Shape 54 fcgis pg oracle oracle ogr sde fcgishapeshape 56 fcgipg fcgioracle fcgisdeOpen Source Geospatial Foundation45 46. MapServer Point Test ResultsPoint Drawing 302520Throughput - 1Maps/sec 15 Throughput - 10Throughput - 20 10 Throughput - 4050shape 54 fcgi pg oracle oracle ogr sde fcgishape shape 56 fcgipg fcgioracle fcgisdeOpen Source Geospatial Foundation 46 47. MapServer Raster Test Results Raster30 25 20 Throughput - 1 Maps/sec15 Throughput - 10 Throughput - 2010 Throughput - 4050bigtiff fcgi 54ecw cgi 56 ecw fcgi 56tiled tiff fcgi 54 bigtiff cgi 56bigtiff fcgi 56ecw fcgi 54tiled tiff cgi 56tiled tiff fcgi 56 Open Source Geospatial Foundation 47 48. FastCGI Configuration Used for MapServer Tests FastCgiServer /var/www/cgi-bin/mapserv.fcgi -processes 8 -initial_env LD_LIBRARY_PATH=/usr/local/lib:/opt/build/instant_client_11_1/li b:/opt/build/arcsde_c_sdk/lib Open Source Geospatial Foundation 48