论文信息 - n-Grams and their implication to natural language understanding

n-Grams and their implication to natural language understanding

Abstract This paper presents the results of a comparative information-theoretic study that was carried out between Greek and English texts. The rank frequency correlation ( p ) between the two appears to be very high; the correlation is between 0.915 and 0.989. The results also include positional letter analyses, n -gram analyses, word analyses, empirical semantic correlations between the Greek and English n -grams, and entropy calculations. The findings presented here are of interest to researchers in the fields of natural language understanding, text processing and compression, speech synthesis and recognition as well as error detection and correction. The results are interesting because they encompass the complete range of hierarchic text patterns (i.e. letters, n -grams (or sub-word patterns) and words).

Emmanuel J. Yannakoudakis | P. J. Hutton | I. Tsomokos

[1] Ching Y. Suen,et al. n-Gram Statistics for Natural Language Understanding and Text Processing , 1979, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2] Emmanuel J. Yannakoudakis,et al. An intelligent spelling error corrector , 1983, Inf. Process. Manag..

[3] Emmanuel J. Yannakoudakis,et al. An Insight into the Entropy and Redundancy of the English Dictionary , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[4] Claude E. Shannon,et al. Prediction and Entropy of Printed English , 1951 .

[5] Emmanuel J. Yannakoudakis,et al. Quasi-Equifrequent Group Generation an Evaluation , 1982, Comput. J..

[6] Michael F. Lynch,et al. Variety generation - A reinterpretation of Shannon's mathematical theory of communication, and its implications for information science , 1977, J. Am. Soc. Inf. Sci..