Surface-Free Multi-Stroke Trajectory Reconstruction and Word Recognition Using an IMU-Enhanced Digital Pen
Efficient handwriting trajectory reconstruction (TR) requires specific writing surfaces for detecting movements of digital pens. Although several motion-based solutions have been developed to remove the necessity of writing surfaces, most of them are based on classical sensor fusion methods limited,...
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2022-07-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/22/14/5347 |
| _version_ | 1797443867300069376 |
|---|---|
| author | Mohamad Wehbi Daniel Luge Tim Hamann Jens Barth Peter Kaempf Dario Zanca Bjoern M. Eskofier |
| author_facet | Mohamad Wehbi Daniel Luge Tim Hamann Jens Barth Peter Kaempf Dario Zanca Bjoern M. Eskofier |
| author_sort | Mohamad Wehbi |
| collection | DOAJ |
| description | Efficient handwriting trajectory reconstruction (TR) requires specific writing surfaces for detecting movements of digital pens. Although several motion-based solutions have been developed to remove the necessity of writing surfaces, most of them are based on classical sensor fusion methods limited, by sensor error accumulation over time, to tracing only single strokes. In this work, we present an approach to map the movements of an IMU-enhanced digital pen to relative displacement data. Training data is collected by means of a tablet. We propose several pre-processing and data-preparation methods to synchronize data between the pen and the tablet, which are of different sampling rates, and train a convolutional neural network (CNN) to reconstruct multiple strokes without the need of writing segmentation or post-processing correction of the predicted trajectory. The proposed system learns the relative displacement of the pen tip over time from the recorded raw sensor data, achieving a normalized error rate of 0.176 relative to unit-scaled tablet ground truth (GT) trajectory. To test the effectiveness of the approach, we train a neural network for character recognition from the reconstructed trajectories, which achieved a character error rate of 19.51%. Finally, a joint model is implemented that makes use of both the IMU data and the generated trajectories, which outperforms the sensor-only-based recognition approach by 0.75%. |
| first_indexed | 2024-03-09T13:03:18Z |
| format | Article |
| id | doaj.art-f3a4d6618274476f94cbd5b5f45cda8b |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-09T13:03:18Z |
| publishDate | 2022-07-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-f3a4d6618274476f94cbd5b5f45cda8b2023-11-30T21:52:15ZengMDPI AGSensors1424-82202022-07-012214534710.3390/s22145347Surface-Free Multi-Stroke Trajectory Reconstruction and Word Recognition Using an IMU-Enhanced Digital PenMohamad Wehbi0Daniel Luge1Tim Hamann2Jens Barth3Peter Kaempf4Dario Zanca5Bjoern M. Eskofier6Machine Learning and Data Analytics Lab., Department Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), 91052 Erlangen, GermanyMachine Learning and Data Analytics Lab., Department Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), 91052 Erlangen, GermanySTABILO International GmbH, 90562 Heroldsberg, GermanySTABILO International GmbH, 90562 Heroldsberg, GermanySTABILO International GmbH, 90562 Heroldsberg, GermanyMachine Learning and Data Analytics Lab., Department Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), 91052 Erlangen, GermanyMachine Learning and Data Analytics Lab., Department Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), 91052 Erlangen, GermanyEfficient handwriting trajectory reconstruction (TR) requires specific writing surfaces for detecting movements of digital pens. Although several motion-based solutions have been developed to remove the necessity of writing surfaces, most of them are based on classical sensor fusion methods limited, by sensor error accumulation over time, to tracing only single strokes. In this work, we present an approach to map the movements of an IMU-enhanced digital pen to relative displacement data. Training data is collected by means of a tablet. We propose several pre-processing and data-preparation methods to synchronize data between the pen and the tablet, which are of different sampling rates, and train a convolutional neural network (CNN) to reconstruct multiple strokes without the need of writing segmentation or post-processing correction of the predicted trajectory. The proposed system learns the relative displacement of the pen tip over time from the recorded raw sensor data, achieving a normalized error rate of 0.176 relative to unit-scaled tablet ground truth (GT) trajectory. To test the effectiveness of the approach, we train a neural network for character recognition from the reconstructed trajectories, which achieved a character error rate of 19.51%. Finally, a joint model is implemented that makes use of both the IMU data and the generated trajectories, which outperforms the sensor-only-based recognition approach by 0.75%.https://www.mdpi.com/1424-8220/22/14/5347trajectory reconstructioninertial measurement unitsensor-based deep learningconvolutional neural networkhandwriting recognitiondigital pen |
| spellingShingle | Mohamad Wehbi Daniel Luge Tim Hamann Jens Barth Peter Kaempf Dario Zanca Bjoern M. Eskofier Surface-Free Multi-Stroke Trajectory Reconstruction and Word Recognition Using an IMU-Enhanced Digital Pen Sensors trajectory reconstruction inertial measurement unit sensor-based deep learning convolutional neural network handwriting recognition digital pen |
| title | Surface-Free Multi-Stroke Trajectory Reconstruction and Word Recognition Using an IMU-Enhanced Digital Pen |
| title_full | Surface-Free Multi-Stroke Trajectory Reconstruction and Word Recognition Using an IMU-Enhanced Digital Pen |
| title_fullStr | Surface-Free Multi-Stroke Trajectory Reconstruction and Word Recognition Using an IMU-Enhanced Digital Pen |
| title_full_unstemmed | Surface-Free Multi-Stroke Trajectory Reconstruction and Word Recognition Using an IMU-Enhanced Digital Pen |
| title_short | Surface-Free Multi-Stroke Trajectory Reconstruction and Word Recognition Using an IMU-Enhanced Digital Pen |
| title_sort | surface free multi stroke trajectory reconstruction and word recognition using an imu enhanced digital pen |
| topic | trajectory reconstruction inertial measurement unit sensor-based deep learning convolutional neural network handwriting recognition digital pen |
| url | https://www.mdpi.com/1424-8220/22/14/5347 |
| work_keys_str_mv | AT mohamadwehbi surfacefreemultistroketrajectoryreconstructionandwordrecognitionusinganimuenhanceddigitalpen AT danielluge surfacefreemultistroketrajectoryreconstructionandwordrecognitionusinganimuenhanceddigitalpen AT timhamann surfacefreemultistroketrajectoryreconstructionandwordrecognitionusinganimuenhanceddigitalpen AT jensbarth surfacefreemultistroketrajectoryreconstructionandwordrecognitionusinganimuenhanceddigitalpen AT peterkaempf surfacefreemultistroketrajectoryreconstructionandwordrecognitionusinganimuenhanceddigitalpen AT dariozanca surfacefreemultistroketrajectoryreconstructionandwordrecognitionusinganimuenhanceddigitalpen AT bjoernmeskofier surfacefreemultistroketrajectoryreconstructionandwordrecognitionusinganimuenhanceddigitalpen |