RealBias Detection in Media: An NLP-Based Approach Using Corpus Statistics and Sentence Embeddings
Under the mentorship of Dr. Clayton Greenberg at the University of Pennsylvania, RealMedia's Neeraj Gummalam conducted research using natural language processing techniques — corpus statistics, sentence encodings, and other vectorization methods — to detect bias in written text.
The work developed a model combining Pointwise Mutual Information (PMI) with a dual TF-IDF strategy to classify biased and unbiased sentences more effectively, highlighting its potential as a tool to help readers identify media bias. Along the way, Neeraj and his mentor also identified a possible flaw in the training data of Google's Universal Sentence Encoder — contributing to broader discussions about data quality in machine learning.
Abstract
We implement a Natural Language Processing solution for binary classification to categorize a sentence as biased or unbiased. Detecting bias is a challenge in the media today, but can help readers identify which sources portray it. Our final model relied on probabilistic data about the connection between words, sentences, and each class, using Pointwise Mutual Information (PMI) and Term Frequency–Inverse Document Frequency (TF-IDF) as heuristics, and leveraged Google's Universal Sentence Encodings (USE) to capture sentence meaning. Our results revealed a possible limitation in USE's training data for bias detection. Through topic analysis, we uncovered which topics are characterized by minimal bias, and used those discoveries to contextualize the model's performance.