SideGuide: A Large-scale Sidewalk Dataset for Guiding Impaired People

Kibaek Park1$\small \dagger$, Youngtaek Oh1$\small \dagger$, Soomin Ham1$\small \dagger$, Kyungdon Joo2$\small \dagger\ast$,
Hyokyoung Kim3$\small$, Hyoyoung Kum3$\small$, In So Kweon1,
1KAIST,    2The Robotics Institute, Carnegie Mellon University,    3TestWorks, Inc.,
$\small \dagger$Equal contributions,  $\ast$Work done at KAIST,  
Paper Download Code
2020 International Conference on Intelligent Robots and Systems (IROS)

An assistance system for impaired people by informing obstacles with distances.
The video is taken on a wheelchair, where many safety-related obstacles are scattered on the way.
Each bounding box is marked with class label and distance.

Abstract

In this paper, we introduce a new large-scale sidewalk dataset called SideGuide that could potentially help impaired people. Unlike most previous datasets, which are focused on road environments, we paid attention to sidewalks, where understanding the environment could provide the potential for improved walking of humans, especially impaired people.

Concretely, we interviewed impaired people and carefully selected target objects from the interviewees' feedback (objects they encounter on sidewalks). We then acquired two different types of data: crowd-sourced data and stereo data. We labeled target objects at instance-level (i.e., bounding box and polygon mask) and generated a ground-truth disparity map for the stereo data. SideGuide consists of 350K images with bounding box annotation, 100K images with a polygon mask, and 180K stereo pairs with the ground-truth disparity.

We analyzed our dataset by performing baseline analysis for object detection, instance segmentation, and stereo matching tasks. In addition, we developed a prototype that recognizes the target objects and measures distances, which could potentially assist people with disabilities. The prototype suggests the possibility of practical application of our dataset in real life.

Introduction

To date, autonomous driving has been one of the most researched topics. Previous studies have largely been focused on self-driving cars and their driving environment, which has led to ample datasets related to roads (e.g., KITTI [1], Cityscapes [2]). In contrast, there has not been enough data investigating the perspectives of pedestrians and their environments, such as sidewalks. Traditionally for cars, fixed lanes separate cars from other vehicles, which makes it easier to detect moving objects. On the other hand, when it comes to a sidewalk, there are no fixed lanes, and there are lots of objects (e.g., pedestrians, personal mobility vehicles, animals, and bollards) that lack directional consistency. Therefore, in the sidewalk environment, objects often occlude each other (i.e., partially blocked by other objects).

Our work is the first large-scale dataset (termed the SideGuide) focused on sidewalks, and which can be deployed in the field of recognition to aid impaired or disabled people. We also used a stereo camera system to capture images of sidewalks, which provides ground-truth disparity as well as instance-level annotation (see Fig. 1).

From the raw data, we annotated 492K images in total for ground-truth (see Table I). Specifically, as instance-level annotation, we generated ground-truth bounding boxes (BB) for 350K images and polygon segmentations for 100K images as a subset of BB. We also generated 180K dense disparity maps from pairs of stereo images. The instance-level annotations and ground-truth disparity provided inference data to the detection model and stereo matching algorithm, respectively, to validate our dataset. The SideGuide was further validated by implementing a prototype that returns the output of object detection and distance of an object from the camera in real-time.

An example image-pair and annotation of a sidewalk. (a) Stereo image pair captured by a ZED stereo camera (top and bottom images indicate left and right images, respectively), (b) Instance-level bounding box and polygon mask annotations in the left image, and (c) The ground-truth dense disparity map.

Our first large-scale sidewalk dataset, the SideGuide will contribute to reducing the gap between technology and real-world deployment for recognition. This dataset could be useful not only for impaired people, but also for other applications like mobile robotics and assistance for vulnerable road users who are not necessarily visually or mobility impaired.

Download Request

In order to access our SideGuide dataset, you must first agree to the terms and conditions by completing a brief survey provided here. You will receive the download link shortly after the approval process is completed.

BibTeX

If you find our work useful for your research, please cite with the following bibtex: 

@inproceedings{park2020sideguide,
    title={Sideguide: a large-scale sidewalk dataset for guiding impaired people},
    author={Park, Kibaek and Oh, Youngtaek and Ham, Soomin and Joo, Kyungdon and Kim, Hyokyoung and Kum, Hyoyoung and Kweon, In So}, 
    booktitle={2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, 
    pages={10022--10029}, 
    year={2020}, 
    organization={IEEE} 
}