The OCR-algorithm (optical character recognition) described in this paper is a module in the assistive device SubPal, which should be able to read subtitles from television and camera aloud. The SubPal system is described in detail in (Nielsen & Bothe, 2007). By sampling the television signal (PAL) a binary image is created. This binary image is analysed using the OCR-algorithm for generating text-strings that can be passed on to a speech synthesis box. The requirements and the implementation of the OCR are discussed and some initial results are presented. The algorithm is developed with the purpose of later being implemented in hardware (FPGA).
Interviews with both visually impaired people and dyslexic have revealed, that a large group is cut off
from full understanding of the visual media (television, DVD, VHS, cinemas), when this is given in a
language that they are not comfortable with. Even if a visual impaired person is able to grasp the
overall content of the screen they are unable to read the subtitles. A solution to this is presented in the
paper
Before going into detail with the modules of the OCR, we asses the overall requirements that should
be considered with respect to the application (described in
Adaptive – the font differ from channel to channel depending on the subtitling company. The algorithm should be versatile enough to encounter for this. Noisy backgrounds (e.g. white Tshirt behind subtitles), should not reduce the detection rate significantly.
Orientation – the spatial orientation of the image should be transparent to the algorithm.
Commercial OCR-solution have been surveyed, but the majority of the available OCR-solutions
targets the software market and depends on a specific operating system and hardware-architecture,
which imposes additional overhead with respect to performance. The OCR discussed in this paper is
intentioned for the portable device described in
In the following sections we will look into the design of such an OCR.
The task of recognising characters in television as well as
cinema can be divided into the modules illustrated in figure
1. This division is consistent with the standard approach
used in OCR-systems
First of all the raw signal from the composite video signal is sampled to create a binary image, from which the subtitles can be extracted. When using the signal from the camera it is also necessary with some spatial adjustment to ensure that the lines of text are horizontal in the image, which is a precondition for our OCR-algorithm. Next the binary image is prefiltered to remove noise and enhance characteristic features. After the optimal filtering the image can be divided into separate lines, words and letters. Each of these characteristic features, statistical or semantic, are detected and compared with an already existing database (based upon training set). After choosing the most likely letters in a given word, the word can be compared with a dictionary lookup, to verify if the letter combination is likely.
Each processing step will be explained in more detail in the following sections. 2 Field Programmable Gate Array
The images used for the character recognition are created by sampling the composite signal from the
television/video camera using a Tektronix TDS1002 oscilloscope and applying a threshold. This is
described in more detail in
In the case where the images come from the CCD video camera, it will often be necessary to carry out
some minor spatial adjustments, since our later described feature extraction is not rotational invariant.
The subtitles consist of one or two lines of densely packed letters with the same orientation. This
a’priori knowledge can be used for finding the rotation of the image, which maximizes the horizontal
sum in the frame (corresponding to a horizontal orientation of the subtitles). Another approach would
be to use a rotation invariant feature extraction, such as Transformation Ring Projection
The success of the OCR-algorithm depends on the initial filtering. The aim of this filtering and
segmentation is to separate the text into separate letters and at the same time make sure that each
letter is as characteristic as possible. To begin with we lowpass-filter the image to remove high
frequency noise which was not removed in the sampleprocess, further more we use dilation3 to avoid
that letters are being divided into several regions (see
With the optimal preconditions given the lines can then be separated by simply detecting minimas in the horizontal projections. By looking at the vertical projections instead each line can be separated into words and letters (see figure 6). After the segmentation each letter is mapped into fixed height and width, thereby making it comparable with the letters in the database, when extracting the features.
To compare each region with our database we need to extract relevant features. We use a
combination of simple statistical and semantic features, which minimize the amount of calculations.
The statistical features are relations between area, width/height, background/foreground and the 1st
order moment. Furthermore the horizontal and vertical projections (see figure 9) are used. A way to
reduce the data from the horizontal and vertical projection is to do a Fourier transform of the rowsums
as described in
Our semantic method detects holes, feet, heads and arms in the letters as illustrated in figure 8. An
improvement could be to further more use feature point extraction for detection of intersections and
corners (see
The horizontal and vertical projections are compared with the database using crosscorrelation. When comparing the statistical features such as height/width-ratio, foreground/background, area and first order moment, a normal distribution is assumed. For each letter the mean of this distribution is given in the database, while the variance is chosen for optimal detection. An extracted feature such as the pixelarea can now be checked towards the distribution of each letter and a probability can be calculated (see figure 7). The results from the projections and the other statistics is finally combined and the most likely letter is chosen. Afterwards the semantical features are used for correction of the most likely misclassifications.
Even with a good detection rate of each letter, errors will occur from time to time. To cater for these,
each word is checked with a dictionary, containing the most common words. If the word is not present,
the most likely match is chosen, e.g. by finding the word with most letters in the correct position. It is
then evaluated which of the 2 words is the most likely, based on the probabilities from the
classification. The dictionary lookup is done using the large English database WordNet® (see
reference
The aim of these considerations is to obtain an implementation in hardware for optimal response times and optimal power consumption. Further work is necessary to make the OCR more versatile, fulfilling the requirements laid out in section 2. A possibility for doing this could be to implement the classification by using an associative neural network.