The influence of training on position and attribute accuracy in volunteered geographic information

GEORGE MASON UNIVERSITY, DEPARTMENT OF GEOGRAPHIC AND

CARTOGRAPHIC SCIENCES, COLLEGE OF SCIENCE

Patricia Pease, MS Candidate, GECA /GISESPDr. Matthew Rice, Thesis DirectorDr. Arie Croitoru, Committee MemberDr. Kevin Curtin, Committee MemberDr. Michele Campagna, Committee Member (Italy)

• Geo-crowdsourcing is emerging geospatial data collection technique in fieldwork

• Citizen science benefits – savings/expertise

• Offering of specific and targeted training

• Experiment set up with trained and untrained participants to assess effectiveness of training

• Both treatment groups then reported obstacles found in field

• Trained project moderators previously characterized obstacles

• Detailed quality analysis done by moderators

• Statistical analysis of QA data found no significant difference

• VGI – although traced much earlier term named by Goodchild in

2007 (GIS in the hands of citizens)

• Christmas Bird Count well established long-running examples

• Citizen science taps into advantages of funding, expertise, and

interest of people in an area

• Value of VGI declines based on errors, low accuracy or malicious

data

• Open sources such as OpenStreetMap and many cell phone apps

have made more VGI possible

• Democratization of GIS (Brown) (‘earned rather than learned’)

• Use and speed of GIS changes

• Much CGD deals with changing, non-static data

• Applications continue to be added and change usability

of GIS

• Goodchild’s concept of ‘platial’ relationaship with

volunteer activity in geography

• Summarization of the American Scientist citizen science

sites – approximately 50 (18 selected)

Continued . . .

• FrogWatch

• Wisconsin Bat Monitoring

• Ushahidi

• Secchi

• Marine Debris Tracker

• Astro Drone (European Space Agency)

• Invaders of Texas

• The Quake Catcher Center

• Project Noah

• Meteor Counter

• PING (precipitation)

• Globe at Night

• eBird

• Dark Sky

• NOVA Energy Lab

• Safecast

• Galloway study of training of school age students to identify White

oaks indicated need for sampling methodology training

• Fowler’s study of fleeting weather and atmosphere conditions

required spatial, temporal and altitudinal elements to be included

• Powell also looked into GeoExposures (2011, UK web launch)

where participants can increase geoscience knowledge base with

their data

• Marine Debris Initiative funded project (Martin, 2013) began in

Georgia makes use of VGI to aid environment

• McClendon and Robinson (2013) social media for input of CGD no

training but potential for disaster information

• Began in London in 2004 requires internet accessibility

• Haklay (2009) compares OSM to Ordinance Survey (UK’s)

• High quality product from skilled core contributors + large

core participants lack formal training

• Positional accuracy, attribute accuracy and completeness

• Provide good base map data for other open-source projects

• Key issue was completeness – coverage influenced by

participant choices – errors not randomly distributed

• Citizens organize mapping parties to give hands-on experience

• Foody (2013) considered land cover identification as important

component of geo-spatial data

• RS generated data followed by analysis = $ costly $

• Unreliability, inconsistencies in accuracy of VGI result in

hesitancy to accept

• Comparisons between volunteer land cover and reference data

• Characterizing performance of ea. volunteer for accuracy can

ID those who could benefit best from training

• Variation in accuracy attributed to background education, skills

and experience of volunteers

• Rice et al (2013) began in 2010 to develop interactive website &

tools to use geo-crowdsourcing on GMU campus

• Path access for disabled members of community considered

(vision and mobility impaired)

• Webmap (geo.gmu.edu/vgi/) modeled after OSM to allow

reports of various types of obstructions

• Gazzetteer all named places, geoparsed to location on campus

• Participants recruited on campus to execute reports of obstacles

using the web interface

• Several updates and revisions to accommodate needs

• Training for end-users was designed to improve overall use of

tools

• Slide presentation images & exercise for student volunteers

• Attribute categories, types and method of location clarified

• Continued testbed environment development

• Moderation tool further developed to allow for Quality Assurance

of incoming reports

• Test site designed to fill gap in assistive geotechnology, more open

and real-time use

• Position of obstacle based on x, y coordinate

• Type of obstacle: obstruction, poor surface, exit/entrance,

crowd/event, construction detour

• Duration: short, medium, long

• Priority: (low, medium, high) based on impact

• Image of obstacle and text description help in moderation of final

report

• All part of final scores developed for Quality Assurance

• Each participant required no more than one hour of time

commitment (including in-class time for training)

How It Works

Identify obstacle

that are in your

path

Go to our websiteor

Mobile app

2

1

Walk around

campus as you

regularly do

3

4

Reporting Obstacles

Create a unique ID - Use letters, numbers, symbols, & others- Do NOT use anything that identifies you (i.e. name, last name, G#)- Use the same ID every time you report

Indicate date & time - When did you observe the obstacle?

Obstacle TypesObstacle Type

Sidewalk obstruction

Construction detour

Entrance/Exit problems

Poor surface conditions

Crowd/Event

Duration & Priority

Duration

Priority

Low (<1 day)

Medium (1-7 days)

Long (>7 days)

Low

Medium

High

Images for training with classification of

obstacle – one example of each type shown

The image is event/ crowd that changes path access, low

duration, < day; low priority (minor inconvenience)

Image such as this might be uploaded with report to give a

higher completion or overall QA score once moderation done

Construction causing sidewalk detour which

could be classified as Medium priority

inconvenience to pedestrians, and Long Duration

(more than 7 days)

other examples from presentation

‘Volunteer outreach’entranceways crowds, events

Other detours and poor surface conditions

Picture 11Classification Exercise:

While viewing 15 different

images, such as this, students

in training marked paper

copies with correct answers

for Obstacle type,

Duration and

Priority

How participants categorized images during training:

Continued . . .

Continued . . .

Continued . . .

Continued . . .

Sidewalk Obstrux32%

Construx5%

Poor Surface63%

Obstacle type: pic. 15

Continued . . .

• Sampling approach limited (students on campus)

• Schedules kept once signed up (not research team!)

• Low incentive – most not altruistic

• Final sample number low (n = 23)

• Wi-fi reliability problematic to students based on surveys

• Slow upload in many cases, students would stop before done

• Low quality reports for moderation – some not usable

• Incoming reports (129) assessed by moderation process

• 23 participants submitted 2 – 8 reports each

• Moderated reports given QA scores based on various criteria:

• Temporal consistency,

• Positional accuracy (both x,y and text description)

• Obstacle Type, Duration and Urgency

• Moderator QA score by 3 moderators, based on all criteria

• QA FINAL SCORE computed as linear combination of all fields

• Tests for both location and attributes done using two-sample

(independent) t-test

• X, Y location as QA interval dependent variable and

Trained/Untrained as categorical independent variable

• Also QA Final Score as dependent variable compared to

Trained/Untrained

• Mann-Whitney test done to compare QA moderator score

ordinal scale dependent variable to Trained/Untrained

• H-α Trained group would score higher QA by moderation as

well as Final and overall scores

• H-ο(null) no significant difference between two groups

• Both Trained/ Untrained groups QA scores 60-65%

• Trained slightly higher (insignificantly) average total quality

score, obstacle type quality score and moderator-assessed QA

• Untrained slightly higher (insignificantly) average score

accuracy of report location

• Tests suggest that training did not significantly influence

participant ability to position object on map

• Positive correlation between Final score and positional

accuracy

Graphed data from scored reports:

Overall scores assigned by moderation process to participants

based on average of all reports, range from 48 to 81 (not

sorted here by Trained /Untrained)

Obstacle type scores based

on agreement with ‘correct’

moderator scoring:

0 - no agreement

1 – partial agreement

2 – perfect agreement

Amount of error or

variance of obstacle

location measured in

meters, from largest

(52 meters!) to

smallest (0.5 meters)

• Finalized moderation reports, scoring and analysis

• Use of obstacle type agreement, location accuracy, variance,

means, other methods of measuring data accuracy

• Statistical analysis indicated test subjects not significantly

different with respect to positional accuracy and for accuracy

of attribute characteristics

• Sample size problematic/ participant incentives low

• n = 23, with uneven distribution of test subjects

• Larger test groups, better planning for reliability of participants and increased incentives – social setting (?)

• Possibly link training to internet sites

• Develop improvements to site for suitability on campus

• Continued student studies to increase usability

• Possible training developed to be included in GIS classes

• Export best training ideas to use for invasive species, bird counts, marine debris, seismic hazards, and many other citizen science applications in crowd-sourcing of geographic data

Abdulmonem, Alabri, and Jane Hunter. "Enhancing the Quality and Trust of Citizen Science Data." 2010 Sixth IEEE International Conference on e-Science.Brisbane: IEEE Computer Society, 2010. 81-88.

Brown, Greg. "An empirical evaluation of the spatial accuracy of public participation GIS (PPGIS) data." Applied Geography 34 (2012a): 289-294.

Brown, Greg. "Public Participation GIS (PPGIS) for Regional and Environmental Planning: Reflections on a Decade of Empirical Research." URISA Journal 24, no. 2 (2012b): 7-18.

Creative Commons. CC Wiki. December 12, 2013. http://www.wiki.creativecommons.org/License_versions (accessed December 14, 2013).

Dunn, Erica H., et al. "Enhancing the Scientific Value of The Christmas Bird Count." The Auk 122, no. 1 (2005): 338-346.

Foody, G. M., et al. "Assessing the Accuracy of Volunteered Geographic Information arising from Multiple Contributors to an Internet Based Collaborative Project." Transactions in GIS (John Wiley & Sons Ltd) 1111 (2013): 1-14.

Fowler, Amy, J. Duncan Whyatt, Gemma Davies, and Rebecca Ellis. "How Reliable are Citizen-Derived Scientific Data? Assessing th Quality of Contrail Observations Made by the General Public." Transactions in GIS 17, no. 4 (2013): 488-506.

Galloway, Aaron W. E., Margaret T. Tudor, and W. Matthew Vander Haegen. "The Reliability of Citizen Sceince: A Case Study of Oregon White Oak Stand Surveys." Wildlife Society Bulletin 34, no. 5 (Dec 2006): 1425-1429.

Goodchild, Michael. "Citizens as Sensors: The World of Volunteered Geography." GeoJournal (National Center for Geographic Information and Analysis) 69, no. 4 (2007): 211-221.

Greenemeier, Larry. "Welcome to Scientific American's Citizen Science Initiative." Scientific American, May 2, 2011.

Haklay, Mordechai. "How good is volunteered geographical information? A comparative study of OpenStreetMap and Ordance Survey datasets." Environment and Planning B: Planning and Design, Sept 13, 2009: 682-703.

Henrich, Joseph Heine, Steven J. and Norenzayan, Ara. "Most people are not WEIRD." Nature, July 1, 2010: 29.

Howe, Jeff. 2006. http://www.wired.com/wiread/archive/14.06/crowds.html(accessed December 8, 2013).

Jackson, Steven P., William Mullen, Peggy Agouris, Andrew Crooks, Arie Croitoru,

and Anthony Stefanidis. "Assessing Completeness and Spatial Error of Features in

Volunteered Geographic Information ." ISPRS Internation Jouinral of Geo-Information 2

(2013): 507-530.

Martin, Jeannie Miller. "Marine debris removal: One year of effort by the Georgia

Sea Turtle-Center-Marine Debris Initiative." Marine Pollution Bulletin 74 (2013): 165-

169.

McClendon, Susan, and Anthony C. Robinson. "Leveraging Geospatially-Oriented

Social Media Communications in Disaster Response." International Journal of

Information Systems for Crisis Response and Management 5, no. 1 (Jan-Mar 2013): 22-40.

Newman, Greg, Don Zimmerman, Alicia Crall, Melinda Laituru, Jim Graham, and

Linda Stapel. "User-freindly web mapping: lessons from a citizen science website."

International Journal of Geograhpical Information Science (Taylor & Francis) 24, no. 12

(Dec 2010): 1851-1869.

Null, Christopher. Tech Hive. PC World. October 4, 2010.

http://www.techhive.com/article/206702/best_online_mapping_service.html?page

=2 (accessed January 20, 2014).

Powell, John, Gemma Nash, and Patrick Bell. "Geo-Exposures: Documenting

temporary geological exposures in Great Britain through a citizen-science web site."

Proceedings of the Geologists' Association. 2013. 638-647.

Rice, Matthew T., Paez, Fanbiana I., Jacobson, R.Daniel, Aburizaiza, Ahmad O.,

Shore, Brandon M. "Supporting Accessibility for Blind and Vision-impaired People

With a Localized Gazetteer and Open Source Geotechnology." Transactions in GIS

(Blackwell Publishing Ltd.) 16, no. 2 (April 2012a): 177-190.

Rice, Matthew T., Caldwell, Douglas R., Paez, Fabiana I., Mulhollen, Aaron, Shore,

Brandon M. Crowdsourced Geospatial Data: A report on the emerging phenomena of

crowdsourced and user-generated geospatial data. contract, Topographic Engineering Center

Technical Report, US Army Corps of Engineers; Engineer Research and

Development Center, Data Level Enterprise Tools Workshop, (2012b), 1-148.

Rice, Matthew T., Curtin, Kevin M., Paez, Fabiana I., Seitz, Christopher R., Qin, Han.

"Crowdsourcing to Support Navigation for the Disabled." Annual, Department of

Geography and Geoinformation Science, George Mason University, Fairfax, (2013),

1-62.

Waters, Nigel. "Social Network Analysis." In Handbook of Regional Science, edited by

Manfred M. Fischer and Peter Nijkamp, 725-740. 2014.

Yang, Xiaolin, and Zhongliang Wu. "Civilian Monitoring video records for

earthquake intensity: a potentially unbiased online information source of macro-

seismology." Natural Hazards, 2013: 1765-1781.