Data Conversion & Integration

Workshop on Census Cartography and Management, Lusaka, Zambia, 08–12 October 2007

Data Conversion &

Integration

Workshop on Census Cartography and Management, Bangkok, Thailand, 15–19 October 2007

Data Conversion/Integration Process

• Data Inventory• Existing hard-copy maps / digital data

• Data Collection (additional )• Satellite Imagery, Aerial Photo, etc.• Field Collection (hand-held devices-GPS,

etc.)• Data Input/Conversion

• Keyboard entry of coordinates• Digitizing/Scanning/Raster-to-Vector• Editing/Building Topology

• Data Integration• Georeferencing/Geocoding


About Geographic Data

• Conversion of hardcopy to digital maps is the most time-consuming task in GIS

• Up to 80% of project costs

• Example: estimated to be a US $10 billion annual market

• Labor intensive, tedious and error-prone


Data Inventory

• National overview maps

• 1:250,000 and 1:5,000,000 (small scale)

• show major civil divisions, urban areas, physical features such as roads, rivers, lakes, elevation, etc.

• used for planning purposes


• Topographic maps- scales range from 1:25,000 to 250,000 (mid-scale)

• Town and city maps at large cartographic scales, showing roads, city blocks, parks, etc. (1:1,000 to 1:5,000)

• Maps of administrative units at all levels of civil division

• Thematic maps showing population distribution for previous census dates, or any features that may be useful for census mapping

Data Inventory (cont.)


Existing Digital Data

• Digital maps• Satellite imagery• GPS coordinates• Etc.


GPS

Remote SensingAerial Photography

Maps

Census & Surveys

Data Collection

GDB

Surveying.

GIS

Capture

Management


Aerial photography

• Aerial photography is obtained using specialized cameras on-board low-flying planes. The camera captures the image digitally or on photographic film.

• Aerial photography is the method of choice for mapping applications that require high accuracy and a fast completion of the tasks.

• Photogrammetry—the science of obtaining measurements from photographic images.


• Traditional end product: printed photos• Today: digital image (scanned from photo) in standard graphics

format (TIFF, JPEG) that can be integrated in a GIS or desktop mapping package

• Trend: fully digital process• digital orthophotos

• corrected for camera angle, atmospheric distortions and terrain elevation

• georeferenced in a standard projection (e.g. UTM)• geometric accuracy of a map• large detail of a photograph

Aerial photography (cont.)



Remote sensing process

Earth Surface

Sources of Energy

Sensing System

Receiving station


GPS• Collection of point data• Stored as “waypoints”• Accuracy dependent on device and environmental variables

Surveying

• Paper Based• Manual recording of information

• Electronic Based• Handheld device


Geographic data input/conversion

• Keyboard entry of coordinates• Digitizing• Scanning and raster to vector conversion• Field work data collection using• Global positioning systems• Air photos and remote sensing


Keyboard entry

• keyboard entry of coordinate data• e.g., point lat/long coordinates

• from a gazetteer (a listing of place names and their coordinates)

• from locations recorded on a map


Latitude/Longitude coordinate conversion• Latitude is y-coo, Longitude is x-coo

• Common format is degrees, minutes, seconds 113º 15’ 23” W 21º 56’ 07” N

• To represent lat/long in a GIS, we need to convert to decimal degrees

-113.25639 21.93528

• DD = D + (M + S / 60) / 60


Data Conversion

• Conversion is often the easiest form to import digital spatial data into a GIS

• Data transfer often rely on the exchange of data in mostly proprietary file formats using the import/export functions of commercial GIS packages

• Open source data Conversion software becoming widely available


Conversion of hardcopy maps to digital data

• Turning features that are visible on a hardcopy map into digital point, line, polygon, and attribute information

• In many GIS projects this is the step that requires by far the largest time and resources

• Newer methods are arising to minimize this arduous step


• Digitizing• Manual digitizing• Heads-up digitizing

• Scanning

• Raster-to-Vector

Conversion of hardcopy maps to digital data (cont.)


Manual DigitizingMost common form of coordinate

data input

• Requires a digitizing table • Ranging in size (25x25 cm to

150x200cm)

• Ideally the map should be flat and not torn or folded

• Cost: hundreds (300) to thousands (5000)


Digitizing steps (how points are recorded)

• trace features to be digitized with pointing device (cursor)

• point mode: click at positions where direction changes

• stream mode: digitizer automatically records position at regular intervals or when cursor moved a fixed distance


Control Points

• If a large map is digitized in several stages and the map has to be removed from the digitizing table occasionally, the control points allow the exact re-registration of the map on the digitizing board.

• Control points are chosen for which the real-world coordinates in the base map’s projection system are known.


Digitizing table

y

x

• Grid of wires in the table creates a magnetic field which is detected by the cursor

• X/Y coordinates in digitizing units are• fed directly into GIS

• High precision in coordinate recording


Heads-Up Digitizing I

• Features are traced from a map drawn on a transparent sheet attached to the screen

• Option, if no digitizer is available; but: accuracy very low


Heads-Up Digitizing II

• Common today is heads-up digitizing, where the operator uses a scanned map, air photo or satellite image as a backdrop and traces features with a mouse

• This method yields more accurate results

• Quicker and easier to retrace and save steps


• Raster-scanned image on the computer screen• Operator follows lines on-screen in vector mode

Heads-Up Digitizing II


Digitizing Errors

• Undershoots

• Dangles

• Spurious Polygons


Digitizing errors

• Any digitized map requires considerable post-processing• Check for missing features• Connect lines• Remove spurious polygons• Some of these steps can be automated


Fixing Errors

• Some of the common digitizing errors shown in the figure can be avoided by using the digitizing software’s snap tolerances that are defined by the user

• For example, the user might specify that all endpoints of a line that are closer than 1 mm from another line will automatically be connected (snapped) to that line

• Small sliver polygons that are created when a line is digitized twice can also be automatically removed


Advantages and Disadvantages of DigitizingAdvantages• It is easy to learn and thus does not require expensive

skilled labor

• Attribute information can be added during digitizing process

• High accuracy can be achieved through manual digitizing; i.e., there is usually minimal loss of accuracy compared to the source map


Advantages and Disadvantages of Digitizing

Disadvantages• It is a tedious activity, possibly leading to operator

fatigue and resulting quality problems which may require considerable post-processing

• It is slow. Large-scale data conversion projects may thus require a large number of operators and digitizing tables

• The accuracy of digitized maps is limited by the quality of the source material


Scanning A viable alternative to digitizing• The map is placed onto the scanning surface where light

is directed at the map at an angle• A photosensitive device records the intensity of light

reflected for each cell or pixel in a very fine raster grid• In gray scale mode, the light intensity is converted

directly into a numeric value, for example into a number between 0 (black) and 255 (white)

• In binary mode, the light intensity is converted into white or black (0/1) cell values according to a threshold light intensity


Scanning• Electronic detector moves across map and records light

intensity for regularly shaped pixels• Flat-bed scanner• Drum-scanner (pictured)


Scanning (cont.)

Types of scanners• Flat

• small format, low cost, good for small tasks

• Drum• high precision but expensive and slow

• Feed• fast, good precision, lower cost than

drum

R G B

pixelwidth

opticalsensor

colorsplicing

computer


• direct use of scanned images

• e.g., scanned air-photos

• digital topographic maps in raster format

Scanning (cont.)


• Scanner output is a raster data set usually needs to be converted into a

• Vector representation- manually (on-screen digitizing)- automated (raster-vector conversion)

line-tracing - e.g., MapScan

• Often requires considerable editing

Scanning (cont.)


Advantages and Disadvantages of Scanning

Advantages• Scanned maps can be used as image backdrops for

vector information• Scanned topographic maps can be used in combination

with digitized EA boundaries for the production of enumerator maps

• Clear base maps or original color separations can be vectorized relatively easily using raster-to-vector conversion software

• Small-format scanners are relatively inexpensive and provide quick data capture


Advantages and Disadvantages of Scanning

Disadvantages• Converting large maps with a small format scanners

requires tedious re-assembly of the individual parts

• Large format, high-throughput scanners are expensive

• Despite recent advances in vectorization software associated with scanning, considerable manual editing and attribute labeling may still be required


Raster to Vector ConversionGets scanned/image data into vector format

• Automatic mode: the system converts all lines on the raster image into sequences of coordinates automatically. automated raster to vector process starts with a line thinning algorithm

• Semi-automatic mode, the operator clicks on each line that needs to be converted; system then traces that line to the nearest intersections and converts it into a vector representation


OBIA Raster to Vector Conversion• Object-Based Image Analysis (OBIA) is a tentative name

for a sub-discipline of GIScience devoted to partitioning remote sensing (RS) imagery into meaningful image-objects, and assessing their characteristics through spatial, spectral and temporal scale. At its most fundamental level, OBIA requires image segmentation,

• attribution, classification and the ability to query and link individual objects (a.k.a. segments) in space and time. In order to achieve this, OBIA incorporates knowledge from a vast array of disciplines involved in the generation and use of geographic information (GI).


Object-Based Image Analysis


OBIA Dwelling Identification

• Segmentation Segmentation basedbased

• Pixel basedPixel based

• Automated Automated DigitizingDigitizing


Object-Based Image Analysis• Increasing demand for updated geo-spatial information,

rapid information extraction• Complex image content of VHSR data needs to be

structured and understood• Huge amount of data can only be utilized by automated

analysis and interpretation• New target classes and high variety of instances• Monitoring systems and update cycles• Transferability, objectivity, transparency, flexibility


Editing• Manual digitizing is error prone• Objective is to produce an accurate representation of the

original map data• This means that all lines that connect on the map must

also connect in the digital database• There should be no missing features and no duplicate

lines• The most common types of errors

• Reconnect disconnected line segments, etc


Some common digitizing errors

overshoot

line digitizedtwice

missingline

spike

undershoot


Building Topology• GIS determines relationships between features in the database

• System will determine intersections between two or more roads and will create nodes

• For polygon data, the system will determine which lines define the border of each polygon

• After the completed digital database has been verified to be error-free

• The final step is adding additional attributes


• The building of relationships between objects• Feature topology describes the spatial relationships between

connecting or adjacent geographic features such as roads connecting at intersections

• The user typically does not have to worry about how the GIS stores topological information

• Feature topology describes the spatial relationships between connecting or adjacent geographic features such as roads connecting at intersections

• The user typically does not have to worry about how the GIS stores topological information

Building Topology


Converting Between Different Digital Formats • All software systems provide links to other formats

• But the number and functionality of import routines varies between packages

• Problems often occur because software developers are reluctant to publish the exact file formats that their systems use -> instability of information (ex. file-geodatabase [.gdb])

• Option of using a third data format• Example: Autocad’s DXF format


Georeferencing/Geocoding • Georeferencing

• Converting map coordinates to the real world coordinates corresponding to the source map’s cartographic projection.

• Attaching codes to the digitized features (geocoded feature)

• each line representing a road would obtain a code that refers to the road status (dirt road, one lane road, two lane highway, etc.)

• Or a unique code that can be linked to a list of street names.


For attribute data:

• spreadsheets

• links to external database

• management systems (DBMS)

• tabulation programs (IMPS, Redatam)


Sample components of a digital EA map

1

2

3

4 56

7

12 13

891011

14

151617

18

19

20

2122

23

2425

26

272829

30

3132

33

34

35 36

37

38

39

4041

42

43

44

454647

48 49 50 51 52

5354

55

56

5758

59

6061

62

636465

66

67

68

69

70

71

7273

74 75

76

777879

80

81 82

83

848586

8788

1 2 3 4

9 10

1 2

7

891011

12

13

14

15

16

21

22

23

24

25 26 27

323334

19

20

21

2223

28

2930

3132

33

41

42

43

50

51

5254

58

59

27

28

31

37

38

42

43

4445

51

4340

4142

61 57

585960

65 62

6364

19

2021

22 31

32

3334

35

41424344

1

2

34

5

6

10

11

12 13

18

1920 21

2728

29

358

Enumeration Area Map Symbols

National Statistical Office -July 1998

Census2000

EA

Locality

District EA-Code

Hospital

Church

School

Buildingnumber

Province:District:Locality:EA-Code:

CartaniaChartesMaptown

14032

022100361

Approximate scale

N

45

17

Street Network Buildings

Boundaries Annotation and symbols

Building numbers Neatlines and legend

361

378

374

349350

358

377

362

358

Lambe r t Av en u e

Mercator Avenue

Cassini DriveCassini Drive

Im h of Drive

Eckert Drive

Miller

DriveBon n e Stree t

M o llw e id e S tre et

Grinten S

treet

Good e

Str e et

Bes se l S tre et

Stree t

Ro bins on S treet

Tisso t S treet

Ga ll S tree tPt olem

y Stree t

Orte l ius S

tr eet

Clar k e S

tr eet

To b le r Stre etSnyder Stre et

Krassowsk ij S

treet361

378

374

349 350

358

377

362

Enum eration Area Map Sym bols

Nationa l S tatistical O ffice - Ju ly 1998Census 2000

EA

Locality

D istrict EA-Code

Hospital

Church

School

Buildingnum ber

Province:D istrict:Locality:EA-Code:

CartaniaChartesMaptown

14032

022100361

500 100Approxim ate scale

200m

N

4 5

1

2

3

4 56

7

1 2 1 3

891011

1 4

1 51 61 7

1 81 9

2 0

2 12 2

2 3

2 4 2 5 2 6

2 72 82 9

3 0

3 1 3 233

3 4

35 3 6

37

3 8

3 9

4 04 1

4 2

43

44

454 647

48 4 9 5 0 51 52

5 35 4

55

5 6

5 75 8

5 9

6 06 1

6 2

6 36 46 5

6 6

6 7

6 86 9

7 0

71

7 273

74 7 5

7 6

7 77879

8 081 8 2

8 3

848 58 6

8 78 8

1 2 3 4

9 1 0

1 2

7

891 011

1 2

1 3

14

1 5

1 6

21

2 2

2 3

2 4

2 5 26 2 7

3 23 33 4

1 92 0

2 1

2 22 3

2 82 9

3 0

3 13 2

3 3

4 1

4 24 3

50

5 1

525 4

5 85 9

2 7

2 8

3 1

3 7

3 8

4 2

4 3

4 44 5

5 1

434 0

4 14 2

6 1 57

58596 0

6 5 62

6 364

1 9

2 02 1

22 3 1

3 23 33 4

35

4 14 24 34 4

1

2

34

5

6

10

11

12 1 3

1 8

1 9 20 2 1

2 728

2 9

17


A Simpler Alternative• In many countries, EA map design may be simpler than in

this example

• Instead of a fully integrated digital base map in vector format, rasterized images of topographic maps may be used as a backdrop for EA boundaries

• What is available already!


• In some instances, map features may be more generalized, for instance by using only the centerlines for the streets and polygons for entire city blocks rather than for individual houses

• This can include the use of free data as a baseline or starting point in the creation or updating of census related maps

A Simpler Alternative


Agencies to contact• National geographic institute / mapping agency• Military mapping services• Province, district and municipal governments• Various government or private organizations dealing with

spatial data• Geological or hydrological survey• Environmental protection authority• Transport authority• Utility and communication sector companies• Land titling & surveying agencies• Academic institutions• Donor activities


GPS coordinatecollection

Existing digitalmaps

Scanning

Raster to vectorconversion(automated or semi-automated)

Editinggeographicfeatures

Dataconversion

Sources ofgeographicinformation

Digital mapdata integration

Digitizing

Combine andintegrate digitalmap sheets

Coding (labelling)of digitalgeographicfeatures

Georeferencing(coordinate trans-formation andprojection change)

Constructtopology forgeographicfeatures

Generate linesand polygons

Additionalgeographic datacollection

Identify existingdata sources

Paper maps,existing printedair photos andsatellite images

Field mappingproductssuch assketch maps

Digital air photosand satelliteimages

Additionaldelineation ofEA boundaries

Parallel activity

Developgeographicattributesdatabase

Date post:	25-Feb-2016
Category:	Documents
Upload:	pete
View:	86 times
Download:	5 times

Data Conversion & Integration

Documents