Japanese Hand-based Activity Recognition Project Project Overview We propose a method that recognizes activities of daily living (ADLs) by employing a wrist worn sensor device with such various kinds of sensors as a camera, a microphone, and an accelerometer, and also describes the design of the wrist worn device. Specifically, the device captures a space around the user's hand by the camera to recognize ADLs that involve the manual use of objects such as making tea or coffee and watering plant. Existing wearable sensor devices equipped only with a microphone and an accelerometer cannot recognize these ADLs without object embedded sensors. We also propose an ADL recognition method that takes privacy issues into account because the camera and microphone can capture aspects of a user's private life. Proposed device We recognize what a user is doing by using only a wrist mounted sensor device. Conceptual image of our device Prototype device ADL recognition method We capture characteristics of activities by using various sensors mounted on the wrist. Accelerometer: Hand movement and posture Camera: Colors of objects the user is using MIC: Sound of activities, ... We learn and recognize activities by combining the characteristics. An example of sensor data that were captured when a person made cocoa. Instraction movie Related article T. Maekawa, Y. Yanagisawa, Y. Kishino, K. Ishiguro, K. Kamei, Y. Sakurai, and T. Okadome: Object-based Activity Recognition with Heterogeneous Sensors on Wrist, Proc. of International Conference on Pervasive Computing (Pervasive 2010), pp. 246-264 (May 2010). T. Maekawa, Y. Yanagisawa, Y. Kishino, K. Kamei, Y. Sakurai, and T. Okadome: Wristband Type Sensor Device for Recognizing Activities that Involve Object Use, Proc. European Conference on Ambient Intelligence (AmI 2009), Poster paper, (Nov. 2009). Contact information Takuya Maekawa NTT Communication Science Laboratories HOME