Word prediction and word probability exemplified in searches over a pharmaceutical database

Table of Contents

1 Introduction to CREAM

1.1 The Corpus Research for Exploitation of Annotated Metadata project (CREAM)

1.1.1 The main topic

1.1.2 The Corpus - eNova Database

1.1.2.1 eNova Application and Sample Walk Through

1.1.3 The main goal

2 State of the Art

2.1 Contributions to Word Prediction and Word Probability

2.1.1 Current Contributions to n-Gram Analysis

2.2 Applied n-Gram Analysis

2.2.1 National Security

2.2.2 Spelling Correction

2.2.2.1 Other Areas Related to Spelling Correction

3 n-Grams - Word Count and n-Gram Modeling

3.1 Introduction to Word Count in Corpora

3.2 Tokenization - Word Segmentation in Corpora

3.2.1 Word Types vs. Word Tokens

3.2.2 Stemming and Lemmatization

3.2.3 Non-word Characters

3.3 Parameters of Tokenization

3.3.1 Compounding and Words Separated by Whitespace

3.3.2 Hyphens

3.3.3 Case-(In-)Sensitivity

3.3.4 Other Cases in English

3.3.5 Stop words

3.3.6 Other Languages

3.4 Introduction to Word Probability and Word Prediction

3.4.1 Markov Assumption and n-Gram Modeling

3.4.2 n-Grams - n-Token Sequences of Words

3.4.3 Simple n-Gram Analysis - Maximum Likelihood Estimation

3.5 n-Gram Analysis over Sample Corpus

4 Waterfall Model

4.1 Introduction

4.2 Introduction to the Waterfall Model of Software Development

4.2.1 Requirement Analysis

4.2.2 Specification Phase

4.2.3 Design

4.2.4 Implementation and Testing - Monogram Code

4.2.5 Implementation and Testing - Bigram Code

4.2.6 Integration

5 Interpretation of n-Gram Retrieval

5.1 Introduction

5.2 Presentation of Monograms

5.2.1 Interpretation of Monograms

5.2.2 Spelling Correction

5.3 Bigrams

5.3.1 Relative Frequency of Sample Bigrams

5.4 Trigrams

5.5 Main interpretation and forecast

Objectives and Research Themes

The primary goal of this thesis is to improve the user experience and guided search functionality of the eNova pharmaceutical database through the application of n-gram modeling. By analyzing historical search queries, the work aims to infer user search strategies, predict intended keyword combinations, and optimize database performance to support both scientific and expert users.

• Application of n-gram modeling for word probability analysis in a specialized pharmaceutical database.
• Methodological preparation of data corpora, including tokenization, stop word filtering, and handling of domain-specific constraints.
• Implementation of Ruby-based computational tools to extract monograms, bigrams, and trigrams from search query logs.
• Evaluation of search patterns and performance improvements for the eNova database interface.

Excerpt from the Book

Summary of Chapters

1 Introduction to CREAM: Introduces the CREAM project and the eNova pharmaceutical database, setting the stage for the research objective of utilizing n-gram modeling to enhance guided search capabilities.

2 State of the Art: Provides a theoretical overview of word prediction, n-gram modeling, and related computational linguistics concepts, including historical context like the Shannon Game.

3 n-Grams - Word Count and n-Gram Modeling: Discusses the methodology of corpus preparation, focusing on tokenization, lemmatization, and the mathematical foundations of word probability and n-gram sequences.

4 Waterfall Model: Details the software development process used to create the Ruby-based n-gram analysis tools, covering requirement analysis, design, implementation, and integration of the code.

5 Interpretation of n-Gram Retrieval: Presents the empirical results of the n-gram analysis performed on the eNova database, including interpretations of monograms, bigrams, and trigrams, and concludes with a forecast for future work.

Keywords

n-gram modeling, corpus analysis, pharmaceutical database, eNova, Novartis, tokenization, word probability, Markov assumption, information retrieval, Ruby, software development, waterfall model, spelling correction, search strategy, lexicography

Frequently Asked Questions

What is the core focus of this research?

The work focuses on analyzing search query data from the eNova pharmaceutical database to understand user search behavior and improve the efficiency of guided search interfaces through n-gram modeling.

Which fields are covered in this study?

The study integrates computational linguistics and software engineering, applying natural language processing (NLP) techniques to optimize professional, domain-specific search engines.

What is the primary objective of the thesis?

The main goal is to implement a method for predicting keyword combinations that users are likely to look for, thereby enabling more intuitive and powerful search interfaces using picklists and statistical distribution data.

What methodology is employed?

The research uses n-gram analysis—specifically monograms, bigrams, and trigrams—combined with the Maximum Likelihood Estimation (MLE) method to calculate word probabilities within the provided search corpora.

How is the main body structured?

The main body follows a software development lifecycle approach (Waterfall Model) to conceptualize, design, and implement customized Ruby code that parses search queries and generates statistical models.

Which keywords characterize this study?

Key terms include n-gram modeling, corpus analysis, eNova, Novartis, tokenization, and Markov assumption.

How does the author handle spelling errors in the data?

The author notes that while spelling errors exist (approx. 0.7-1.1%), they are deemed minor enough not to significantly skew the statistical results, though future improvements might incorporate automated spelling correction.

How is the Ruby code utilized?

Three distinct Ruby scripts are used to handle different levels of n-gram analysis (monograms, bigrams, and trigrams), specifically filtered for pharma-industry search syntax (e.g., ignoring Boolean operators).

Why are there two different corpora in the analysis?

The study contrasts "internal expert" queries with "external expert" queries to demonstrate differences in search strategies and keyword usage between different user groups.

What is the conclusion regarding future developments?

The author suggests that future work should focus on optimizing code for syntactic case-insensitivity and shifting the focus towards analyzing suffixes to better categorize generic search intents.