Sidewalk Extraction Using Aerial and Street View Images
Xinyue Ye, Ph.D., Associate Professor
Department of Landscape Architecture and Urban Planning
& Department of Geography
Texas A&M University, Email: [email protected]
Editor-in-Chief, Computational Urban Science
Urban Data Science Lab, TAMU
Acknowledgement
• Ning, H., Ye, X*., Chen, Z., Liu, T., & Cao, T. (2021) Sidewalk Extraction Using Aerial and Street View Images. Environment and Planning B. (in press)
• Ye, X. and Lin, H. (eds.) (2020) Spatial Synthesis: Computational Social Science and Humanities, Springer.
• Jamonnak, S., Zhao, Y., AL-Dohuki, S., Curtis, A., Ye, X., Kamw, F., & Yang J. (2020) GeoVisuals: a visual analytics approach to leverage the potential of spatial videos and associated geonarratives. International Journal of Geographical Information Science. doi: 10.1080/13658816.2020.1737700
• Ye, X. and Liu, X. (eds.) (2019) Cities as Social and Spatial Networks, Springer.
Urban Data Science Lab, TAMU
Outline
• Introduction
• Related Work
• Methodology
• Results
• Discussion
Urban Data Science Lab, TAMU
Outline
• Introduction
• Related Work
• Methodology
• Results
• Discussion
Urban Data Science Lab, TAMU
Walking
Urban Data Science Lab, TAMU
Which information is missed?
Urban Data Science Lab, TAMU
Big Image Data
Urban Data Science Lab, TAMU
http://shadeparadenashville.blogspot.com/2013/12/sidewalk-project-3-walking-from.html
Outline
• Introduction
• Related Work
• Methodology
• Results
• Discussion
Urban Data Science Lab, TAMU
Semantic segmentation based on deep learning
• FCN (Long et al., 2015)
• SegNet (Badrinarayanan et al., 2017)
• U-Net (Ronneberger et al., 2015)
• PSP-Net (Zhao et al., 2016)
• LaU model (He et al., 2019)
• OCR model (Yuan et al., 2019)
Urban Data Science Lab, TAMU
Road extraction from aerial image
• ResUnet on the Massachusetts roads dataset (Mnih and Hinton, 2010)
• SegNet to segment roads in the Massachusetts dataset and the Thailand Earth Observation System (THEOS) dataset (Panboonyuen et al. 2017)
Urban Data Science Lab, TAMU
Sidewalk extraction from street view image
• Senlet and Elgammal (2012): a pixel-wise and a path-wise method to predict and connect the occluded sidewalks covering a university campus.
• Smith et al. (2013): random forest to identify sidewalk after extracting local and global features.
• Mattyus et al. (2016): fully connected deep structured networks and Markov random field.
• Wang et al. (2019):convert street view image into top-view representation.
Urban Data Science Lab, TAMU
Outline
• Introduction
• Related Work
• Methodology
• Results
• Discussion
Urban Data Science Lab, TAMU
sidewalk datasets
• field survey
• deriving from existing land cover data
• extracting from images
Urban Data Science Lab, TAMU
workflow
Urban Data Science Lab, TAMU
Study area (Delaware Valley Regional in Pennsylvania State)
Urban Data Science Lab, TAMU
Training set and test set
Urban Data Science Lab, TAMU
https://medium.com/@abhigoku10/cv2019-papersummary-yolact-real-time-instance-segmentation-e62fa721957f
A sample of the sidewalk training set. The DVRPC sidewalks dataset uses a short line to represent the
conjunction point with crosswalks, which is preserved in the training dataset.
Urban Data Science Lab, TAMU
Missing sidewalks in aerial images between dangles
Urban Data Science Lab, TAMU
Segmentation of street view images by pre-train PSPNet
Urban Data Science Lab, TAMU
A panorama and its depthmap in Equirectangularprojection from Google Street View
Urban Data Science Lab, TAMU
Reproject the segmentation to landcover map
Urban Data Science Lab, TAMU
Extracting sidewalk centerlines from street view images. The centerlines were extracted merely from
landcover map for visual representation
Urban Data Science Lab, TAMU
Outline
• Introduction
• Related Work
• Methodology
• Results
• Discussion
Urban Data Science Lab, TAMU
Sidewalk detection results of YOLACT
Urban Data Science Lab, TAMU
Precision and recall of segmentation results
Urban Data Science Lab, TAMU
An occluded segment of sidewalks was restored using street view images
Urban Data Science Lab, TAMU
A missing sidewalk segment of 85 meters was restored using street view images
Urban Data Science Lab, TAMU
Outline
• Introduction
• Related Work
• Methodology
• Results
• Discussion
Urban Data Science Lab, TAMU
https://www.nwnewsnetwork.org/post/washington-lawmakers-
ponder-rules-sidewalk-driving-delivery-robots
https://www.scientificamerican.com/article/out-of-the-way-human-delivery-robots-want-a-share-of-your-sidewalk
Reality?
Urban Data Science Lab, TAMU
https://www.latimes.com/local/lanow/la-me-ln-fix-broken-sidewalks-20140818-story.html
Urban Data Science Lab, TAMU
https://www.dailynews.com/2018/02/07/la-city-leaders-say-streets-sidewalks-are-broken-and-may-not-be-ready-in-time-
for-olympics/
Urban Data Science Lab, TAMU
Data Challenging Communities
33
Urban Memory
34
query, navigation, and comparison of video, photo, audio, and text
over regions and locations
Urban Memory
35
36
Procedure
37Procedure
GeoVideo
● Location-based video capturer
● Without Internet connection
● Official website
○ http://vis.cs.kent.edu/GeoVisuals/mobile.html
● Appstore○ https://apps.apple.com/us/app/geovideo/id1492419929
● Google Play○ https://play.google.com/store/apps/details?id=com.nlapps.geovideo&hl=en
Apps
39
Human in Cities
40
MethodologicalConceptual
Transformative Outcome
Human-Centered Built Envr. Science
Urban Computing
• Energy Research
• Hazard Reduction & Recovery
• Housing & Land Use
• Smart & Connected Cities/Health
• Spatial Decision Support System
• Transportation & Travel Behavior
Workflow Automation
Urban Systems and Communities in the 21st Century
Text as Data
Emerging AI, ICT, and location-aware technologies
Visual Analytics
Data Fusion
Human Dynamics
Image and Video Analytics
Real-time SystemSpatial Econometrics
Space-time Analysis
Collaborative Research Plan
Turn space to place
from concept to deliverables