README.md

SVM Model with TF-IDF

This repository provides a pre-trained Support Vector Machine (SVM) model for text classification using Term Frequency-Inverse Document Frequency (TF-IDF). The repository also includes utilities for data preprocessing and feature extraction.:

Installation

Before running the code, ensure you have all the required libraries installed:

pip install nltk beautifulsoup4 scikit-learn pandas

Download necessary NTLK resources for preprocessing.

import nltk
nltk.download('stopwords')
nltk.download('wordnet')

How to Use:

1. Data Cleaning
   The data_cleaning.py file contains a clean() function to preprocess the input dataset:

• Removes HTML tags.
• Removes non-alphanumeric characters and extra spaces.
• Converts text to lowercase.
• Removes stopwords.
• Lemmatizes words.

from data_cleaning import clean
import pandas as pd

# Load your data
df = pd.read_csv('test_data_random_subset.csv')

# Clean the data
cleaned_df = clean(df)

2. TF-IDF Feature Extraction
   The tfidf.py file contains the TF-IDF vectorization logic. It converts cleaned text data into numerical features suitable for training and testing the SVM model.

from tfidf import tfidf

# Apply TF-IDF vectorization
X_train_tfidf = tfidf.fit_transform(X_train['title'])
X_test_tfidf = tfidf.transform(X_test['title'])

3. Training and Testing the SVM Model
   The svm.py file contains the logic for training and testing the SVM model. It uses the TF-IDF-transformed features to classify text data.

from sklearn.svm import SVC
from sklearn.metrics import accuracy_score, classification_report

# Train the SVM model
svm_model = SVC(kernel='linear', random_state=42)
svm_model.fit(X_train_tfidf, y_train)

# Predict and evaluate
y_pred = svm_model.predict(X_test_tfidf)
accuracy = accuracy_score(y_test, y_pred)
print(f"SVM Accuracy: {accuracy:.4f}")
print(classification_report(y_test, y_pred))

4. Training a new dataset with pre-trained model
   To test a new dataset, combine the steps above：

-