Staff, Symbol and Melody Detection of Medieval Manuscripts Written in Square Notation Using Deep Fully Convolutional Networks
Even today, the automatic digitisation of scanned documents in general, but especially the automatic optical music recognition (OMR) of historical manuscripts, still remains an enormous challenge, since both handwritten musical symbols and text have to be identified. This paper focuses on the Mediev...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2019-06-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/9/13/2646 |
| _version_ | 1818862819739697152 |
|---|---|
| author | Christoph Wick Alexander Hartelt Frank Puppe |
| author_facet | Christoph Wick Alexander Hartelt Frank Puppe |
| author_sort | Christoph Wick |
| collection | DOAJ |
| description | Even today, the automatic digitisation of scanned documents in general, but especially the automatic optical music recognition (OMR) of historical manuscripts, still remains an enormous challenge, since both handwritten musical symbols and text have to be identified. This paper focuses on the Medieval so-called square notation developed in the 11th−12th century, which is already composed of staff lines, staves, clefs, accidentals, and neumes that are roughly spoken connected single notes. The aim is to develop an algorithm that captures both the neumes, and in particular its melody, which can be used to reconstruct the original writing. Our pipeline is similar to the standard OMR approach and comprises a novel staff line and symbol detection algorithm based on deep Fully Convolutional Networks (FCN), which perform pixel-based predictions for either staff lines or symbols and their respective types. Then, the staff line detection combines the extracted lines to staves and yields an <inline-formula> <math display="inline"> <semantics> <msub> <mi>F</mi> <mn>1</mn> </msub> </semantics> </math> </inline-formula>-score of over 99% for both detecting lines and complete staves. For the music symbol detection, we choose a novel approach that skips the step to identify neumes and instead directly predicts note components (NCs) and their respective affiliation to a neume. Furthermore, the algorithm detects clefs and accidentals. Our algorithm predicts the symbol sequence of a staff with a diplomatic symbol accuracy rate (dSAR) of about 87%, which includes symbol type and location. If only the NCs without their respective connection to a neume, all clefs and accidentals are of interest, the algorithm reaches an harmonic symbol accuracy rate (hSAR) of approximately 90%. In general, the algorithm recognises a symbol in the manuscript with an <inline-formula> <math display="inline"> <semantics> <msub> <mi>F</mi> <mn>1</mn> </msub> </semantics> </math> </inline-formula>-score of over 96%. |
| first_indexed | 2024-12-19T10:05:56Z |
| format | Article |
| id | doaj.art-169e7abcb89944edbc83c40437377192 |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-12-19T10:05:56Z |
| publishDate | 2019-06-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-169e7abcb89944edbc83c404373771922022-12-21T20:26:30ZengMDPI AGApplied Sciences2076-34172019-06-01913264610.3390/app9132646app9132646Staff, Symbol and Melody Detection of Medieval Manuscripts Written in Square Notation Using Deep Fully Convolutional NetworksChristoph Wick0Alexander Hartelt1Frank Puppe2Chair for Artificial Intelligence and Applied Computer Science, University of Würzburg, 97074 Würzburg, GermanyChair for Artificial Intelligence and Applied Computer Science, University of Würzburg, 97074 Würzburg, GermanyChair for Artificial Intelligence and Applied Computer Science, University of Würzburg, 97074 Würzburg, GermanyEven today, the automatic digitisation of scanned documents in general, but especially the automatic optical music recognition (OMR) of historical manuscripts, still remains an enormous challenge, since both handwritten musical symbols and text have to be identified. This paper focuses on the Medieval so-called square notation developed in the 11th−12th century, which is already composed of staff lines, staves, clefs, accidentals, and neumes that are roughly spoken connected single notes. The aim is to develop an algorithm that captures both the neumes, and in particular its melody, which can be used to reconstruct the original writing. Our pipeline is similar to the standard OMR approach and comprises a novel staff line and symbol detection algorithm based on deep Fully Convolutional Networks (FCN), which perform pixel-based predictions for either staff lines or symbols and their respective types. Then, the staff line detection combines the extracted lines to staves and yields an <inline-formula> <math display="inline"> <semantics> <msub> <mi>F</mi> <mn>1</mn> </msub> </semantics> </math> </inline-formula>-score of over 99% for both detecting lines and complete staves. For the music symbol detection, we choose a novel approach that skips the step to identify neumes and instead directly predicts note components (NCs) and their respective affiliation to a neume. Furthermore, the algorithm detects clefs and accidentals. Our algorithm predicts the symbol sequence of a staff with a diplomatic symbol accuracy rate (dSAR) of about 87%, which includes symbol type and location. If only the NCs without their respective connection to a neume, all clefs and accidentals are of interest, the algorithm reaches an harmonic symbol accuracy rate (hSAR) of approximately 90%. In general, the algorithm recognises a symbol in the manuscript with an <inline-formula> <math display="inline"> <semantics> <msub> <mi>F</mi> <mn>1</mn> </msub> </semantics> </math> </inline-formula>-score of over 96%.https://www.mdpi.com/2076-3417/9/13/2646optical music recognitionhistorical document analysismedieval manuscriptsneume notationfully convolutional neural networks |
| spellingShingle | Christoph Wick Alexander Hartelt Frank Puppe Staff, Symbol and Melody Detection of Medieval Manuscripts Written in Square Notation Using Deep Fully Convolutional Networks Applied Sciences optical music recognition historical document analysis medieval manuscripts neume notation fully convolutional neural networks |
| title | Staff, Symbol and Melody Detection of Medieval Manuscripts Written in Square Notation Using Deep Fully Convolutional Networks |
| title_full | Staff, Symbol and Melody Detection of Medieval Manuscripts Written in Square Notation Using Deep Fully Convolutional Networks |
| title_fullStr | Staff, Symbol and Melody Detection of Medieval Manuscripts Written in Square Notation Using Deep Fully Convolutional Networks |
| title_full_unstemmed | Staff, Symbol and Melody Detection of Medieval Manuscripts Written in Square Notation Using Deep Fully Convolutional Networks |
| title_short | Staff, Symbol and Melody Detection of Medieval Manuscripts Written in Square Notation Using Deep Fully Convolutional Networks |
| title_sort | staff symbol and melody detection of medieval manuscripts written in square notation using deep fully convolutional networks |
| topic | optical music recognition historical document analysis medieval manuscripts neume notation fully convolutional neural networks |
| url | https://www.mdpi.com/2076-3417/9/13/2646 |
| work_keys_str_mv | AT christophwick staffsymbolandmelodydetectionofmedievalmanuscriptswritteninsquarenotationusingdeepfullyconvolutionalnetworks AT alexanderhartelt staffsymbolandmelodydetectionofmedievalmanuscriptswritteninsquarenotationusingdeepfullyconvolutionalnetworks AT frankpuppe staffsymbolandmelodydetectionofmedievalmanuscriptswritteninsquarenotationusingdeepfullyconvolutionalnetworks |