Word segmentation is an important task in natural language processing, especially for languages without word boundaries, such as Thai language. Many Thai word segmentation programs have been developed. Researchers and developers in Thai documents usually spend a tremendous amount of time in studying and trying di erent Thai word segmentation programs. This paper presents the performance of six Thai word segmentation programs which include Libthai, Swath, Wordcut, CRF++, Thaisemantics, and Tlexs. Based on experimental results, we compare these programs in terms of usage, response time, time outs, and relevance.
Natural Language Processing (NLP) enables computers to understand human languages. It consists of many processes such as word segmentation, part-ofspeech tagging, automatic summarization, and speech synthesis. Most NLP applications require input text to be segmented into words before being processed ?? Corresponding author further. For example, in sentences similarity application, text must rst be tokenized into a series of terms before being analyzed grammatically and semantically. Word segmentation is an essential part for Asian languages such as Thai, Chinese, Japanese, and Korean. This is because these languages are written without delimiter spaces for words in the same sentence. 2
Many researches and several programs have been developed for word
segmentation. We chose six Thai word segmentation programs which were Libthai, Swath,
Wordcut, CRF++, Thaisemantics, and Tlexs. They were selected because they
were actively maintained and widely used. Libthai [
Usage Table 1 summarizes the features about usage, o ine support, and whether installation is needed. Response Time We ran the experiments and measured response times on the computer with Intel Dual 1.73GHz and 3 GB of RAM using Ubuntu 64bit as an operating system. The program execution times are shown in Table 2. Libthai performed the best with the response time 0.022 seconds while Swath and CRF++ had response times as 0.024 seconds. On the other hand, the responses times of Thaisemantics, Wordcut, and Tlexs were large. This is because Thaisemantics and Tlexs are programs that are used over the internet thus their response times depend on the internet bandwidth and tra c time. Wordcut is implemented using JavaScript language which may increase the response time of the program. Number of Time Outs Tlexs and Thaisemantics had several time outs because they are called over the internet. Tlexs had average 2,070 time outs while Thaisemantics had 5 time outs. Tlexs caused a large number of occurrences of time outs might be due to server settings or errors.
Relevance Messages from the BEST were sent to each program and the word segmentation output was kept in a list. After that, we compared this log list with a correct list from BEST by using the percentage of precision, recall and F-measure. We ran this test by using 5-fold cross-validation, and then computed the average value as shown in Table 3. Both Tlexs and CRF++ have the best F-measure because they use CRF. 4
This paper presents the comparison of six Thai word segment programs in terms of usage, response time, time outs, and relevance. Swath, Libthai, and CRF++ programs provide the smallest response times because they are native programs. Thaisemantics yields the largest response time because Thaisemantics is called over the internet and uses a dictionary. Although Tlexs is also called over the internet, it has better response time because it uses CRF. Both Tlexs and CRF++ give the highest F-measure because they employ CRF.