A Rational Design for a Weighted Finite-State Transducer Library

1 Overview We describe the design principles and main algorithms for an object-oriented library for weighted nite-state transducers, which are nite automata in which each transition has an output and a weight as well as the more familiar input. The main goal of the library is to provide algorithms and representations for all the symbolic processing components (language models, dictionaries, acoustic realization rules, word lattices) in large-vocabulary speech recognition systems. This goal leads to several requirements: generality, to support the representation and use of the various information sources in speech recognition; modularity, to allow rapid experimentation with diierent representations of speech recognition tasks; and eeciency, to support competitive large-vocabulary recognition. Rational power series provide the theoretical foundation for the library by giving the semantics for the objects and operations in the library and by creating the opportunity for optimizations (on-demand composition, determinization and minimization) that are not available in more \ad hoc" speech recognition frameworks. The generality of the library has made it also valuable in other language-processing applications, such as word segmentation for Chinese text 25]. 1.1 Design Rationale Current speech-recognition systems rely on a variety of probabilistic nite-state models, for instance n-gram language models 21], multiple-pronunciation dictionaries 11], and context-dependent acoustic models 10]. However, most speech recognizers do not take advantage of the shared properties of the information sources they use. Instead, they rely on special-purpose algorithms for speciic representations. That means that the recognizer has to be rewritten if representations are changed for a new application or for increased accuracy or performance. Experiments with diierent representations are therefore diicult, as they require changing or even completely replacing fairly intricate recognition programs. This situation is not too diierent from that in programming-language parsing before lex and yacc 2]. Furthermore, specialized representations and algorithms preclude certain global optimizations based on the general properties of nite-state models. Again, the situation is similar to the lack of general