Twitter Sentiment Analysis using Python

Twitter sentiment analysis uses natural language processing and machine learning to determine the sentiment expressed in tweets. A tweet can be classified as positive or negative based on the words and patterns present in its text.

Input: "I love this!"
Output: Positive
Explanation: The tweet expresses a positive opinion, so the trained model classifies it as positive.

How is Twitter Sentiment Analysis Useful?

• Twitter Sentiment Analysis is important because it helps people and businesses understand what the public thinks in real time.
• Millions of tweets are posted every day, sharing opinions about brands, products, events or social issues. By analyzing this huge stream of data, companies can measure customer satisfaction, spot trends early, handle negative feedback quickly and make better decisions based on how people actually feel.
• It’s also useful for researchers and governments to monitor public mood during elections, crises or big events as it turns raw tweets into valuable insights.

Step by Step Implementation

Step 1: Install Necessary Libraries

The following libraries are used to load the dataset, convert text into numerical features, train classification models, and evaluate their performance.

pip install pandas scikit-learn
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import BernoulliNB
from sklearn.linear_model import LogisticRegression
from sklearn.svm import LinearSVC
from sklearn.metrics import accuracy_score, classification_report

Step 2: Load Dataset

The Sentiment140 dataset contains 1.6 million tweets with sentiment labels. The polarity is stored in column 0, while the tweet text is stored in column 5.

df = pd.read_csv('training.1600000.processed.noemoticon.csv.zip', encoding='latin-1', header=None)
df = df[[0, 5]]
df.columns = ['polarity', 'text']
print(df.head())

Output:

Explanation:

• pd.read_csv() loads the compressed CSV file.
• encoding="latin-1" allows the dataset's character encoding to be read correctly.
• Columns 0 and 5 are selected because they contain the sentiment label and tweet text.
• The columns are renamed to polarity and text.

Step 3: Keep Only Positive and Negative Sentiments

Sentiment140 uses 0 for negative tweets and 4 for positive tweets. Since this project uses binary sentiment classification, the labels are converted to 0 and 1.

df = df[df.polarity != 2]

df['polarity'] = df['polarity'].map({0: 0, 4: 1})

print(df['polarity'].value_counts())

Output:

Explanation:

• Tweets with polarity 0 and 4 are retained.
• Negative sentiment is represented by 0.
• Positive sentiment is represented by 1.
• The dataset contains an equal number of positive and negative tweets.

Step 4: Clean the Tweets

Text is converted to lowercase so that words with different capitalization are treated consistently.

def clean_text(text):
    return text.lower()

df['clean_text'] = df['text'].apply(clean_text)

print(df[['text', 'clean_text']].head())

Output:

Explanation:

• clean_text() converts each tweet to lowercase.
• apply() runs the function on every tweet.
• The cleaned text is stored in a separate clean_text column.

Step 5: Train Test Split

The data is divided into training and testing sets. The training set is used to build the models, while the testing set is used to evaluate their performance.

X_train, X_test, y_train, y_test = train_test_split(
    df['clean_text'],
    df['polarity'],
    test_size=0.2,
    random_state=42
)

print("Train size:", len(X_train))
print("Test size:", len(X_test))

Output:

Train size: 1280000
Test size: 320000

Explanation:

• X_train and X_test contain tweet text.
• y_train and y_test contain sentiment labels.
• test_size=0.2 reserves 20% of the data for testing.
• random_state=42 keeps the split reproducible.

Step 6: Perform Vectorization

Machine learning models require numerical input, so the tweets are converted into numerical features using TF-IDF (Term Frequency-Inverse Document Frequency).

vectorizer = TfidfVectorizer(max_features=5000, ngram_range=(1,2))

X_train_tfidf = vectorizer.fit_transform(X_train)
X_test_tfidf = vectorizer.transform(X_test)

print("TF-IDF shape (train):", X_train_tfidf.shape)
print("TF-IDF shape (test):", X_test_tfidf.shape)

Output:

TF-IDF shape (train): (1280000, 5000)
TF-IDF shape (test): (320000, 5000)

Explanation:

• TfidfVectorizer converts text into numerical feature vectors.
• max_features=5000 limits the vocabulary to 5,000 features.
• ngram_range=(1, 2) includes both individual words and two-word combinations.
• fit_transform() learns the vocabulary from the training data and transforms it.
• transform() applies the same learned vocabulary to the test data.

Step 7: Train Bernoulli Naive Bayes model

The Bernoulli Naive Bayes classifier is trained using the TF-IDF features from the training set.

bnb = BernoulliNB()
bnb.fit(X_train_tfidf, y_train)

bnb_pred = bnb.predict(X_test_tfidf)

print("Bernoulli Naive Bayes Accuracy:", accuracy_score(y_test, bnb_pred))
print("\nBernoulliNB Classification Report:\n", classification_report(y_test, bnb_pred))

Output:

Explanation:

• BernoulliNB() creates the classifier.
• fit() trains the model using the training features and labels.
• predict() generates sentiment predictions for the test tweets.
• accuracy_score() calculates the proportion of correct predictions.
• classification_report() displays precision, recall, and F1-score.

Step 8: Train Support Vector Machine (SVM) model

A Linear Support Vector Machine is trained on the same TF-IDF features.

svm = LinearSVC(max_iter=1000)
svm.fit(X_train_tfidf, y_train)

svm_pred = svm.predict(X_test_tfidf)

print("SVM Accuracy:", accuracy_score(y_test, svm_pred))
print("\nSVM Classification Report:\n", classification_report(y_test, svm_pred))

Output:

Explanation:

• LinearSVC() creates a linear SVM classifier.
• fit() learns the relationship between TF-IDF features and sentiment labels.
• predict() classifies the test tweets.
• The accuracy and classification report show the model's performance.

Step 9: Train Logistic Regression model

Logistic Regression can also be used for binary sentiment classification.

logreg = LogisticRegression(max_iter=100)
logreg.fit(X_train_tfidf, y_train)

logreg_pred = logreg.predict(X_test_tfidf)

print("Logistic Regression Accuracy:", accuracy_score(y_test, logreg_pred))
print("\nLogistic Regression Classification Report:\n", classification_report(y_test, logreg_pred))

Output:

Explanation:

• LogisticRegression() creates the classification model.
• max_iter=100 sets the maximum number of training iterations.
• The model learns from the TF-IDF representation of the training tweets.
• Predictions are evaluated against the actual test labels.

Step 10: Compare Model Performance

The three models can be compared using their accuracy scores.

print("Bernoulli Naive Bayes:",
      accuracy_score(y_test, bnb_pred))

print("SVM:",
      accuracy_score(y_test, svm_pred))

print("Logistic Regression:",
      accuracy_score(y_test, logreg_pred))

Output:

Bernoulli Naive Bayes: <accuracy>
SVM: <accuracy>
Logistic Regression: <accuracy>

The exact accuracy can vary depending on the dataset version, preprocessing, library version, and model configuration.

Step 11: Make Predictions on sample Tweets

After training, the models can classify new tweets. The same TF-IDF vectorizer used during training must be used to transform the new text.

sample_tweets = ["I love this!", "I hate that!", "It was okay, not great."]
sample_vec = vectorizer.transform(sample_tweets)

print("\nSample Predictions:")
print("BernoulliNB:", bnb.predict(sample_vec))
print("SVM:", svm.predict(sample_vec))
print("Logistic Regression:", logreg.predict(sample_vec))

Output:

Explanation:

• The sample tweets are converted into TF-IDF features using the existing vectorizer.
• Each trained model predicts 1 for positive sentiment and 0 for negative sentiment.
• The third tweet may be classified as negative because the model was trained only on positive and negative classes; it does not have a separate neutral class.

We can see that our models are working fine and giving same predictions even with different approaches.

You can download the Source code from here- Twitter Sentiment Analysis using Python