MapServer and SDE

Howard Butler

Center for Survey Statistics & Methodology

Iowa State UniversityBeginning

Middle

End

National Resources Inventory

• National longitudinal survey of multiple natural resources

• Estimates of status and change over time on nonfederal lands

– Land cover/use changes– Soil erosion trends– Urbanization and loss of prime farmland– Wetland dynamics

• http://www.nrcs.usda.gov/technical/NRI/Beginning

Middle

End

National Resources Inventory

• > 300,000 area segments

– 160 acres each– 70,000 samples / year– Low altitude photography

Beginning

Middle

End

National Resources Inventory

• 1/3 digital -- 2/3 analog

• > 18 gb of support data

– Federal land– Water– Sample location layers– Support imagery

Beginning

Middle

End

NRI GIS Data History

• Recently migrated to ESRI SDE

• Oracle database for tabular data

• Changes/edits tracked with

ArcGIS editing tools

Beginning

Middle

End

Use of MapServer in NRI

• The need for speed

– ArcIMS doesn’t even compare on

the same hardware– We can get by with cheaper

hardware because of improved

performance– Lower software licensing costs

Beginning

Middle

End

Use of MapServer in NRI

• Still need SDE features, though

– Versioned queries• Edits to data are tracked through a

versioned editing workflow. Web

mapping applications must be able to

see these various versions

– Joins• Spatial and tabular joinsBeginning

Middle

End

2001

Survey year Survey edits Revision edits Frozen edit state

1997

2000

2002

2003

2004

N

2002 version of the 2000control data

NRI Versioned Data Tree

Use of MapServer in NRI

• MapServer brings features, too

– Fluent WMS allows us to leverage other

data sources that we don’t have to

manage– Bulletproof projection on-the-fly support– MapScript is simpler, more robust, more

flexible (supports many languages), and

faster than many of the ArcIMS connector

technologies such as ActiveX and JSPBeginning

Middle

End

Recent improvements in MapServer SDE Support

• Support for Versioned Queries

– Can specify the version on the

CONNECTION string to query from

• Auto-sensing of SDE table structure

– SDE maintains a unique ID for each

table. The user can change the name of

this column at will, and many

applications within the ESRI family are

not consistent.Beginning

Middle

End

Recent improvements in MapServer SDE Support

• New internal query syntax

– ArcSDE 8.x changed the query syntax in

its C API. The new methods allow the

user to pre-join feature and business

tables, making the queries much faster. – This change also will allow us to support

user-specified joins similar to XBASE joins

in the future.

Beginning

Middle

End

Compiling in SDE Support

• Linux

– Easy. Just turn on the configure line parameters related to SDE (--with-sde, --with-sde-version)

• Solaris

– Same as Linux, except that make sure everything is compiled with the same compilerBeginning

Middle

End

Compiling in SDE Support

• Windows

– Harder. Configure all of your

settings in nmake.opt– Ensure that the .def table for all of

the C API functions is available to

map the function names to the

Windows calling conventionBeginning

Middle

End

How to Connect to SDE

• Define a Layer in your MapFile

Beginning

Middle

End

LAYER NAME states TYPE POLYGON CONNECTION "sdemachine.iastate.edu,port:5151,sde,username,password,SDE.DEFAULT" CONNECTIONTYPE SDE DATA "HOBU.STATES_LAYER,SHAPE" TEMPLATE '/where/the/template/file/is/located' CLASS

STYLESYMBOL 'circle'SIZE 3COLOR -1 -1 -1OUTLINECOLOR 0 0 0

END ENDEND

More detail at:http://mapserver.gis.umn.edu/cgi-bin/wiki.pl?MapServerSDE

How to Connect to SDE

Beginning

Middle

End

CONNECTIONTYPE SDE

• MapServer will report an error of

SDE Support not available if SDE

has not been properly compiled-

in

How to Connect to SDE

Beginning

Middle

End

CONNECTION "sdemachine.iastate.edu,port:5151,sde,username,password,SDE.DEFAULT"

-Machine Name

-Can be an IP Address

-Often needs to be the FQDN

How to Connect to SDE

Beginning

Middle

End

CONNECTION "sdemachine.iastate.edu,port:5151,sde,username,password,SDE.DEFAULT"

-SDE Service location

-port: is important

-esri_sde for systems specified in /etc/services

How to Connect to SDE

Beginning

Middle

End

CONNECTION "sdemachine.iastate.edu,port:5151,sde,username,password,SDE.DEFAULT"

-The database name in which SDE is stored

-This is often “SDE”

-Not required by all DBMS, but some text is required by MapServer

How to Connect to SDE

Beginning

Middle

End

CONNECTION "sdemachine.iastate.edu,port:5151,sde,username,password,SDE.DEFAULT"

-User must be granted rights to select the table in the DATA directive using either ArcCatalog or the SDE command-line utilities

How to Connect to SDE

Beginning

Middle

End

CONNECTION "sdemachine.iastate.edu,port:5151,sde,username,password,SDE.DEFAULT"

-The user’s password

-CASE SENSITIVE

How to Connect to SDE

Beginning

Middle

End

CONNECTION "sdemachine.iastate.edu,port:5151,sde,username,password,SDE.DEFAULT" -The version to

connect to

-SDE.DEFUALT is used if none is specified

-CASE SENSITIVE

How to Connect to SDE

Beginning

Middle

End

DATA "HOBU.STATES_LAYER,SHAPE"

Schema Table Name Spatial Column

• No spaces!

• Oracle and MSSQL usually require the

schema name – use the full name

shown in ArcCatalog if not sure

How to Connect to SDE

Beginning

Middle

End

TEMPLATE '/where/the/template/file/is/located'

• Must be specified to query attributes

• Unnecessary performance hog if you

are only drawing layers – especially if

the layer has a lot of attributes

An Example Application

• NRI DataView

• ID-Based

• Python MapScript

• TerraServer WMS

• pyTerra for image dates

• PIL for post-processing image manipulationBeginning

Middle

End

An Example Application

• 300,000 PSUs– QueryByAttribute would require a sequential scan

of the entire table– The ID has natural hierarchy

• 19001_010101R

– First two digits are the state FIPS code

– Second three digits are the county FIPS code

• Use QueryByAttribute on the USSTATES table to limit the extent of the query of the USCOUNTIES table

• Use QueryByAttribute on the USCOUNTIES table to limit the extent of the query on the PSU table

Beginning

Middle

End

An Example Application

Beginning

Middle

End

An Example Application

Beginning

Middle

End

Upcoming MapServer SDE Improvements

• QueryByShape

– Currently only extents are

supported– SDE has support for advanced

operations• Buffer select, touching, common edge

Beginning

Middle

End

Upcoming MapServer SDE Improvements

• Joins

– Will be implemented similar to

XBASE and MySQL – As fast or as slow as your database– We get around this limitation with

database views right now

Beginning

Middle

End

Upcoming MapServer SDE Improvements

• SDE Raster Support

– Been talked about and requested

for a long time– No one has the time or desperate

need for it to fund development– Could be implemented through

GDALBeginning

Middle

End

The End

• Questions?

Beginning

Middle

End