Punctuation Restoration for Speech Transcripts using seq2seq Transformers
DOI:
https://doi.org/10.47611/jsrhs.v10i4.2249Keywords:
Speech to text, Natural Language Processing, Artificial Intelligence, Machine LearningAbstract
When creating text transcripts from spoken audio, Automatic Speech Recognition (ASR) systems need to infer appropriate punctuation in order to make the transcription more readable. This task, known as punctuation restoration, is challenging since punctuation is not explicitly stated in speech. Most recent works framed punctuation restoration as a classification task and used pre-trained encoder-based transformers, such as BERT, to perform it. In this work, we present an alternative approach, framing punctuation restoration as a sequence-to-sequence task and using T5, a pretrained encoder-decoder transformer model, as the basis of our implementation. Training our model on IWSLT 2012, a common punctuation restoration benchmark, we find its performance is comparable to state of the art classification-based systems with an F1 score of 80.7 on the test set. Furthermore, we argue that our approach might be more flexible in its ability to adapt to more complex types of outputs, such as predicting more than one punctuation mark in a row.
Downloads
References or Bibliography
Alexei Baevski and Henry Zhou and Abdelrahman Mohamed and Michael Auli (2020). wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. CoRR, abs/2006.11477. https://doi.org/10.21437/interspeech.2021-717
Garofolo, J. S., Lamel, L. F., Fisher, W. M., Fiscus, J. G., Pallett, D. S., & Dahlgren, N. L. (1993). DARPA TIMIT: https://doi.org/10.6028/nist.ir.4930
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015). Librispeech: An ASR corpus based on public domain audio books. 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
https://doi.org/10.1109/icassp.2015.7178964
Tündik, M. Á., Szaszák, G., Gosztolya, G., & Beke, A. (2018). User-centric Evaluation of Automatic Punctuation in ASR Closed Captioning. Interspeech 2018. https://doi.org/10.21437/interspeech.2018-1352
Tilk, O., & Alumäe, T. (2016). Bidirectional Recurrent Neural Network with Attention Mechanism for Punctuation Restoration. Interspeech 2016. https://doi.org/10.21437/interspeech.2016-1517
Tilk, O., & Alumäe, T. (2015). LSTM for punctuation restoration in speech transcripts. Interspeech 2015.
https://doi.org/10.21437/interspeech.2015-240
Attila Nagy and Bence Bial and Judit Ács (2021). Automatic punctuation restoration with BERT models. CoRR, abs/2101.07343.
Jacob Devlin and Ming-Wei Chang and Kenton Lee and Kristina Toutanova (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR, abs/1810.04805. https://doi.org/10.18653/v1/N19-1423
Courtland, M., Faulkner, A., & McElvain, G. (2020). Efficient Automatic Punctuation Restoration Using Bidirectional Transformers with Robust Inference. In Proceedings of the 17th International Conference on Spoken Language Translation (pp. 272–279). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.iwslt-1.33
Kolár, J., Svec, J., & Psutka, J. (2004). Automatic punctuation annotation in czech broadcast news speech.
Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. CoRR, abs/1910.10683.
Noam Shazeer and Mitchell Stern (2018). Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. CoRR, abs/1804.04235.
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, & Blake Hechtman. (2018). Mesh-TensorFlow: Deep Learning for Supercomputers.
Thomas Wolf and Lysandre Debut and Victor Sanh and Julien Chaumond and Clement Delangue and Anthony Moi and Pierric Cistac and Tim Rault and Rémi Louf and Morgan Funtowicz and Jamie Brew (2019). HuggingFace's Transformers: State-of-the-art Natural Language Processing. CoRR, abs/1910.03771.
Gravano, A., Jansche, M., & Bacchiani, M. (2009). Restoring punctuation and capitalization in transcribed speech. In 2009 IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 4741-4744). https://doi.org/10.1109/ICASSP.2009.4960690
Yinhan Liu and Myle Ott and Naman Goyal and Jingfei Du and Mandar Joshi and Danqi Chen and Omer Levy and Mike Lewis and Luke Zettlemoyer and Veselin Stoyanov (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR, abs/1907.11692.
Alam, F. (2020). Punctuation Restoration using Transformer Models for High-and Low-Resource Languages. In Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020) (pp. 132–142). Association for Computational Linguistics.
https://doi.org/10.18653/v1/2020.wnut-1.18
Stephan Peitz, Markus Freitag, Arne Mauser, & H. Ney (2011). Modeling punctuation prediction as machine translation. IWSLT.
Vāravs, Andris & Salimbajevs, Askars. (2018). Restoring Punctuation and Capitalization Using Transformer Models: 6th International Conference, SLSP 2018, Mons, Belgium, October 15–16, 2018, Proceedings. https://doi.org/10.1007/978-3-030-00810-9_9
Published
How to Cite
Issue
Section
Copyright (c) 2021 Aviv Melamud; Alina Duran
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Copyright holder(s) granted JSR a perpetual, non-exclusive license to distriute & display this article.
