Files in this item



application/pdfContribution_393_final.pdf (1MB)
(no description provided)PDF


Title:Optimizing the neural network training for OCR error correction of historical Hebrew texts
Author(s):Suissa, Omri; Elmalech, Avshalom; Zhitomirsky-Geffet, Maayan
Subject(s):Optical character recognition (OCR)
OCR post-correction
Neural networks
Hebrew historical newspapers
Digital humanities
Abstract:Over the past few decades, large archives of paper-based documents such as books and newspapers have been digitized using Optical Character Recognition. This technology is error-prone, especially for historical documents. To correct OCR errors, post-processing algorithms have been proposed based on natural language analysis and machine learning techniques such as neural networks. Neural network's disadvantage is the vast amount of manually labeled data required for training, which is often unavailable. This paper proposes an innovative method for training a light-weight neural network for Hebrew OCR post-correction using significantly less manually created data. The main research goal is to develop a method for automatically generating language and task-specific training data to improve the neural network results for OCR post-correction, and to investigate which type of dataset is the most effective for OCR post-correction of historical documents. To this end, a series of experiments using several datasets was conducted. The evaluation corpus was based on Hebrew newspapers from the JPress project. An analysis of historical OCRed newspapers was done to learn common language and corpus-specific OCR errors. We found that training the network using the proposed method is more effective than using randomly generated errors. The results also show that the performance of the neural net-work for OCR post-correction strongly depends on the genre and area of the training data. Moreover, neural networks that were trained with the proposed method outperform other state-of-the-art neural networks for OCR post-correction and complex spellcheckers. These results may have practical implications for many digital humanities projects.
Issue Date:2020-03-23
Series/Report:iConference 2020 Proceedings
Genre:Conference Poster
Rights Information:Copyright 2020 Omri Suissa, Avshalom Elmalech, and Maayan Zhitomirsky-Geffet
Date Available in IDEALS:2020-03-17

This item appears in the following Collection(s)

Item Statistics