Evaluation of OCR can be done in a number of different ways. From the point of view of reproducibility and objectivity, the preferred way is to develop ground truth transcriptions for an evaluation dataset, and evaluate by measuring character error rate and/or word error rate. An essential prerequisite for this is a tool which compares the OCR with the ground truth transcription, and collects statistics about the amount of errors, both in terms of characters (Character Error Rate, CER) and words (Word Error Rate, WER). Most evaluation tools proceed by alignment of ground truth and OCR, allowing to precisely locate the discrepancies between GT and OCR. An alternative evaluation measure which gives a rough indication of word error rate without considering the location of the words is the bag-of-words error rate. This may be used when complex layouts render alignment problematic.
OCR evaluation, and indeed development of innovative approaches to OCR itself, benefits enormously from publication of well-structured evaluation sets.
We provide training materials for the following tools: