Observing Pianist Accuracy and Form with Computer Vision
| Autor: | Bardia Doosti, Christopher Raphael, Yupeng Gu, Jangwon Lee, David J. Crandall, David Cartledge |
|---|---|
| Rok vydání: | 2019 |
| Předmět: |
Computer science
business.industry Piano Feature extraction 02 engineering and technology Convolutional neural network Hand movements Object detection Visualization 030507 speech-language pathology & audiology 03 medical and health sciences Identification (information) 0202 electrical engineering electronic engineering information engineering 020201 artificial intelligence & image processing Computer vision Artificial intelligence 0305 other medical science business |
| Zdroj: | WACV |
| DOI: | 10.1109/wacv.2019.00165 |
| Popis: | We present a first step towards developing an interactive piano tutoring system that can observe a student playing the piano and give feedback about hand movements and musical accuracy. In particular, we have two primary aims: 1) to determine which notes on a piano are being played at any moment in time, 2) to identify which finger is pressing each note. We introduce a novel two-stream convolutional neural network that takes video and audio inputs together for detecting pressed notes and finger presses. We formulate our two problems in terms of multi-task learning and extend a state-of-the-art object detection model to incorporate both audio and visual features. In addition, we introduce a novel finger identification solution based on pressed piano note information. We experimentally confirm that our approach is able to detect pressed piano keys and the piano player's fingers with a high accuracy. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|