Hand-based Activity Recognition Project

Project Overview
We propose a method that recognizes activities of daily living (ADLs) by employing a wrist worn sensor device with such various kinds of sensors as a camera, a microphone, and an accelerometer, and also describes the design of the wrist worn device. Specifically, the device captures a space around the user's hand by the camera to recognize ADLs that involve the manual use of objects such as making tea or coffee and watering plant. Existing wearable sensor devices equipped only with a microphone and an accelerometer cannot recognize these ADLs without object embedded sensors. We also propose an ADL recognition method that takes privacy issues into account because the camera and microphone can capture aspects of a user's private life.

Proposed device
We recognize what a user is doing by using only a wrist mounted sensor device.

Conceptual image of our device Prototype device
ADL recognition method
We capture characteristics of activities by using various sensors mounted on the wrist.
  • Accelerometer: Hand movement and posture
  • Camera: Colors of objects the user is using
  • MIC: Sound of activities, ...
We learn and recognize activities by combining the characteristics.

An example of sensor data that were captured when a person made cocoa.
Instraction movie
Related article
  1. T. Maekawa, Y. Yanagisawa, Y. Kishino, K. Ishiguro, K. Kamei, Y. Sakurai, and T. Okadome: Object-based Activity Recognition with Heterogeneous Sensors on Wrist, Proc. of International Conference on Pervasive Computing (Pervasive 2010), pp. 246-264 (May 2010).
  2. T. Maekawa, Y. Yanagisawa, Y. Kishino, K. Kamei, Y. Sakurai, and T. Okadome: Wristband Type Sensor Device for Recognizing Activities that Involve Object Use, Proc. European Conference on Ambient Intelligence (AmI 2009), Poster paper, (Nov. 2009).
Contact information
Takuya Maekawa
NTT Communication Science Laboratories