Menu
Home
Contact us
Stats
Categories
Calendar
Toggle Wiki
Wiki Home
Last Changes
Rankings
List pages
Orphan pages
Sandbox
Print
Toggle Image Galleries
Galleries
Rankings
Toggle Articles
Articles home
List articles
Rankings
Toggle Blogs
List blogs
Rankings
Toggle Forums
List forums
Rankings
Toggle File Galleries
List galleries
Rankings
Toggle Maps
Mapfiles
Toggle Surveys
List surveys
Stats
ITHEA Classification Structure > I. Computing Methodologies  > I.2 ARTIFICIAL INTELLIGENCE  > I.2.7 Natural Language Processing 
MULTILINGUAL REDUCED N-GRAM MODELS
By: Tran Thi Thu Van and Le Quan Ha (2468 reads)
Rating: (1.00/10)

Abstract: Statistical language models should improve as the size of the n-grams increases from 3 to 5 or higher. However, the number of parameters and calculations, and the storage requirement increase very rapidly if we attempt to store all possible combinations of n-grams. To avoid these problems, the reduced n-grams’ approach previously developed by O’Boyle? 1993 can be applied. A reduced n-gram language model can store an entire corpus’s phrase-history length within feasible storage limits. Another theoretical advantage of reduced n-grams is that they are closer to being semantically complete than traditional models, which include all n-grams. In our experiments, the reduced n-gram Zipf curves are first presented, and compared with conventional n-grams for all Irish, Chinese and English. The reduced n-gram model is then applied for large Irish, Chinese and English corpora. For Irish, we can reduce the model size, compared to the 7-gram traditional model size, with a factor of 15.1 for a 7-million-word Irish corpus while obtaining 41.63% improvement in perplexities; for English, we reduce the model sizes with factors of 14.6 for a 40-million-word corpus and 11.0 for a 500-million-word corpus while obtaining 5.8% and 4.2% perplexity improvements; and for Chinese, we gain a 16.9% perplexity reductions and we reduce the model size by a factor larger than 11.2. This paper is a step towards the modeling of Irish, Chinese and English using semantically complete phrases in an n-gram model.

Keywords: Reduced n-grams, Overlapping n-grams, Weighted average (WA) model, Katz back-off, Zipf’s law.

ACM Classification Keywords: I. Computing Methodologies - I.2 ARTIFICIAL INTELLIGENCE - I.2.7 Natural Language Processing - Speech recognition and synthesis

Link:

MULTILINGUAL REDUCED N-GRAM MODELS

Tran Thi Thu Van and Le Quan Ha

http://www.foibg.com/ijitk/ijitk-vol04/ijitk04-2-p07.pdf

Print
I.2.7 Natural Language Processing
article: SYNTACTIC OPERATIONS – MODELING LANGUAGE FACULTY · ON MENTAL REPRESENTATIONS: LANGUAGE STRUCTURE AND MEANING REVISED · IMPROVING AUTOMATIC SPEECH RECOGNITION ACCURACY BY MEANS OF PRONUNCIATION VARIAT · УНИВЕРСАЛЬНАЯ СИСТЕМА ПРОГРАММ МОРФОЛОГИЧЕСКОГО АНАЛИЗА НАУЧНО-ТЕХНИЧЕСКИХ ... · SPAM AND PHISHING DETECTION IN VARIOUS LANGUAGES · GRAMMATICAL PRIMING DOES FACILITATE VISUAL WORD NAMING, AT LEAST IN SERBIAN · MULTILINGUAL REDUCED N-GRAM MODELS · COGNITIVE MODEL OF TIME AND ANALYSIS OF NATURAL LANGUAGE TEXTS · IMPLEMENTATION OF DICTIONARY LOOKUP AUTOMATA FOR UNL ANALYSIS AND GENERATION · О МОДЕЛИРОВАНИИ ПОНИМАНИЯ · ФОРМАЛЬНОЕ ОПРЕДЕЛЕНИЕ СИТУАЦИИ ДЛЯ СЕМАНТ · THE EDUCATIONAL TECHNOLOGY FOR LEARNING FOREIGN WORDS · PARAMETERIZATION OF COMMENTS FROM PERUVIAN FACEBOOK AND TWITTER... · THE STUDY OF FACTORS RELADED WITH SINGLE-DOCUMENT KEYWORD EXTRACTION · AUTOMATED TAG EXTRACTION & CLUSTERING IN DOCUMENTS CONTAINING COMPOSITIONAL ... · STUDYING SPECIAL TEXT RUSSIAN CORPORA BY THE LEXICO-SYNTACTIC MODELS · STUDYING SPECIAL TEXT RUSSIAN CORPORA BY THE LEXICO-SYNTACTIC MODELS · CLASSIFICATION OF PRIMARY MEDICAL RECORDS WITH RUBRYX-2: FIRST EXPERIENCE · MACHINE TRANSLATION IN THE COURSE “COMPUTER TECHNOLOGIES IN LINGUISTICS” .. · CLASSIFICATION OF FREE TEXT CLINICAL NARRATIVES (SHORT REVIEW) · METHODS AND TOOLS OF COMPUTATIONAL LINGUISTICS FOR THE CLASSIFICATION ... · LEXISTERM – THE PROGRAM FOR TERM SELECTION BY THE CRITERION OF SPECIFICITY · ELECTION DATA VISUALIZATION · COMPUTER SUPPORT OF SEMANTIC TEXT ANALYSIS OF A TECHNICAL SPECIFICATION ON ... · MOBILE ELECTION · MOBILE SEARCH AND ADVERTISING · ALGEBRA LOGIC APPROACH TO PERSON’S THINKING MECHANISMS FORMALIZATION · COMPUTER SUPPORT OF SEMANTIC TEXT ANALYSIS OF A TECHNICAL SPECIFICATION ON DESIG · LSPL-PATTERNS AS A TOOL FOR INFORMATION EXTRACTION FROM NATURAL LANGUAGE TEXTS · NUMERIC-LINGUAL DISTINGUISHING FEATURES OF SCIENTIFIC DOCUMENTS · HIERARCHICAL THREE-LEVEL ONTOLOGY FOR TEXT PROCESSING · HIERARCHICAL THREE-LEVEL ONTOLOGY FOR TEXT PROCESSING · COMPUTER-AIDED SYSTEM OF SEMANTIC TEXT ANALYSIS ... · METHODOLOGY FOR LANGUAGE ANALYSIS AND GENERATION ... · ANALYSIS AND COORDINATION OF EXPERT STATEMENTS IN THE PROBLEMS ... · SEMANTIC SEARCH OF INTERNET INFORMATION RESOURCES ON BASE OF ONTOLOGIES ... · INTELLIGENT SEARCH AND AUTOMATIC DOCUMENT CLASSIFICATION AND CATALOGING ... · VERBAL DIALOGUE VERSUS WRITTEN DIALOGUE · INFORMATION PROCESSING IN A COGNITIVE MODEL OF NLP · EXPERIMENTS IN DETECTION AND CORRECTION OF RUSSIAN MALAPROPISMS BY MEANS ... · COMMON SCIENTIFIC LEXICON FOR AUTOMATIC DISCOURSE ANALYSIS OF SCIENTIFIC ... ·
Login
[ register | I forgot my password ]
World Clock
Powered by Tikiwiki Powered by PHP Powered by Smarty Powered by ADOdb Made with CSS Powered by RDF powered by The PHP Layers Menu System
RSS Wiki RSS Blogs rss Articles RSS Image Galleries RSS File Galleries RSS Forums RSS Maps rss Calendars
[ Execution time: 0.08 secs ]   [ Memory usage: 7.60MB ]   [ GZIP Disabled ]   [ Server load: 0.14 ]
Powered by Tikiwiki CMS/Groupware