ai_project.py

# -*- coding: utf-8 -*-
"""AI Project.ipynb

Automatically generated by Colaboratory.

Original file is located at
    https://colab.research.google.com/drive/1oiortdPk8zoqOU3EP6fa4sMkvN7CuExk

## **Topic - Classification of conversational threads on Twitter as Rumourand Non-Rumour**

### **Data Collection** - 
Collected tweets from the streaming API relating to newsworthy events that could potentially prompt the initiation and propagation of rumours. Selected rumours were then captured in the form of conversation threads.

*   Candidate rumourous stories from Twitter were collected. 
*   Second, journalists in the research team selected from these candidate rumours those that met the rumour criteria and identified the tweets that actually introduced them. 
*   Finally, associated conversations for these rumour-introducing tweets were collected and annotated using an annotation scheme specifically crafted for this purpose.

### **Attributes** -
* The datasets consist of rumour stories, represented by squares, which can be one of true (green), false (red), or unverified (orange). 
* Each of the rumour stories has a number of rumour threads associated with it, which is represented as black lines that form a timeline where threads are sorted by time. 
* When a story is true or false, the journalists also pick, within the story’s timeline, one tweet as the resolving tweet.

### **Data Instances** - 
For the purposes of this study, the focus was on the **19991 tweets** annotated as rumourous. These rumourous tweets were collected and annotated in three different languages: English, German and French. 

All the replies to the 584 rumourous source tweets for the nine events in the dataset were also looked at. Collection of tweet IDs of all the tweets replying to the source tweet, i.e. the conversational thread, from which we can form the whole tree was done. Their, metadata was also collected.

### **Data Annotation** - 
The annotation work performed by the journalists was twofold
* annotation of each source tweet as being a rumour or not
* grouping of source tweets into stories

**The outcome of the annotation work was summarized in a table containing** -
* 61 attributes
* It has five instances for the five events under consideration

### **Target Label** -
The target label is Rumour for each Thread id and Tweet id

### **ML task type** - Classification

### **Visualizations** - 
Some of the visualizations needed would include- 
* Charts to visualise, in all, 103 rumours which remain unverified, 159 which were later proven, and 68 which were found to be false
* Plotting distribution of delays in resolving false and true rumours to look at the delay between a rumour being posted for the first time on Twitter and the rumour being resolved as being either true or false
* Plotting to shows networks of interactions between users, the connections being coloured according to their accuracy.
* Table for showing percentages of retweets for unverified, accurate, and inaccurate tweets.
* We further examine these retweet networks by looking at the time in which the retweets occur. We want to know the extent to which each type of rumour is retweeted, as well as whether there is a time-effect on rumour diffusion patterns. Hence, plotting average distribution of retweets for different rumourous stories (true, false) over time using histogram.
* Plotting a graph to compares the number of retweets that each type of tweet receives overall, compared with the average number of retweets that all rumours get.
* Graph to show distribution of support ratios before and after resolving tweets for true and false rumours, as well as for rumours that remain unverified.
* Support heatmap representing the number of rumours that spark more support,  more denials, the same number of each, or neither.
* Plot to compares the amount of discussion that different types of rumourous tweets spark before and after resolving tweets. 
* Plot to show shows the support ratio that different types of tweets (true, false, pre and post resolution) receive.
* Plotting the distribution of certainty ratios before and after resolving tweets for true and false rumours, as well as for rumours that remain unverified.
* Visualization of distribution of evidentiality ratios before and after resolving tweets for true and false rumours, as well as for rumours that remain unverified.
* Plot to show the statistics for the average follow ratio of users who express support, certainty and evidentiality for rumours at different veracity status levels.

**Dataset loading & Pre-processing**
"""

from google.colab import drive
drive.mount('/content/drive')

#Importing libraries

import pandas as pd 
import matplotlib.pyplot as plt
import numpy as np
import sys
import statsmodels.api as sm
from pandas.plotting import scatter_matrix
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
import seaborn as sns
import statsmodels.formula.api as smf
from sklearn import metrics

"""**Import dataset and its pre-processing**"""

missing_value_formats = ["n.a.","?","NA","n/a", "na", "--"]
dt1 = pd.read_csv (r'/content/drive/MyDrive/aifinal/charliehebdof.csv', na_values = missing_value_formats, engine='python')
dt1

missing_value_formats = ["n.a.","?","NA","n/a", "na", "--"]
dt2 = pd.read_csv (r'/content/drive/MyDrive/aifinal/fergusonf.csv', na_values = missing_value_formats, engine='python')
dt2

missing_value_formats = ["n.a.","?","NA","n/a", "na", "--"]
dt3 = pd.read_csv (r'/content/drive/MyDrive/aifinal/germanwings-crashf.csv', na_values = missing_value_formats, engine='python')
dt3

missing_value_formats = ["n.a.","?","NA","n/a", "na", "--"]
dt4 = pd.read_csv (r'/content/drive/MyDrive/aifinal/ottawashootingf.csv', na_values = missing_value_formats, engine='python')
dt4

missing_value_formats = ["n.a.","?","NA","n/a", "na", "--"]
dt5 = pd.read_csv (r'/content/drive/MyDrive/aifinal/sydneysiegef.csv', na_values = missing_value_formats, engine='python')
dt5

#Merging datasets into one 

df=dt1.append(dt2)
df=df.append(dt3)
df=df.append(dt4)
df=df.append(dt5)
df

"""**Dataset scheme:**

*   "is_rumor": 1 if rumor else 0
*   "is_source_tweet" : 1 if source id there else 0
*   "is_truncated": 1 if truncated else 0
*   "has_smile_emoji": 1 if "😊" else 0
*   "sensitive": 1 if sensitive else 0
*   "has_place": 1 if place is there else 0
*   "has_coords": 1 if it has coordinates else 0
*   "has_quest": 1 if it has question else 0
*   "has_exclaim": 1 if it has exclaim else 0
*   "has_quest_or_exclaim": 1 if it has a question/exclaim else 0
*   "user.verified": 1 if user is verified else 0
*   "user.notifications": 1 if user notifications are there else 0
*   "user.desc_length": user description length if it is there else 0
*   "user.has_bg_img": 1 if user bg image is there else 0
*   "user.default_pic": 1 if user profile pic is there else 0
*   "user.location": 1 if user location is there else 0





"""

print(type(df))
print(len(df))

#Checking null values
bool_n = pd.isnull(df)
print(bool_n)

# handling null values for colums containing text
df["text"].fillna("No text", inplace = True) 
df["user.handle"].fillna("No handle", inplace = True) 
df["event"].fillna("Event not specified", inplace = True) 
df["user.time_zone"].fillna("Time zone not specified", inplace = True)
df 

# replacing the numeric null values with 0
df.fillna(0)

print('Total Rumorous threads:')
print(len(df.loc[df['is_rumor']== 1]))
print('Total Non-Rumorous threads:')
print(len(df.loc[df['is_rumor']== 0]))

#Ratio of number of rumours to the number of non-rumours

r=len(df.loc[df['is_rumor']== 1])
nr=len(df.loc[df['is_rumor']== 0])
print('Ratio:')
ratio=r/nr
print(ratio)

df.info()

df.head()

"""**Visualisation**"""

# Import dependencies for this notebook

import pandas as pd
import numpy as np
import networkx as nx
import seaborn as sns  
import matplotlib.pyplot as plt
from matplotlib.lines import Line2D
from datetime import datetime

#Heatmap

f, ax = plt.subplots(figsize=(18,18))
sns.heatmap(df.corr(), annot=True, linewidth=0.5, fmt='.1f', ax=ax)

#Distribution of Feature Values (All News Events) Violin plot

df[["is_source_tweet","is_rumor"]] = df[["is_source_tweet", "is_rumor"]].astype(bool)
y = df['is_rumor']
X = df[["sentimentscore", "FirstPersonPronoun", "ThirdPersonPronoun", "hashtags_count", "tweet_length", "user_mentions", "urls_count", "media_count"]]
X = (X - X.mean()) / X.std()  # normalize
dt = pd.concat([X, y], axis=1)
dt = pd.melt(dt, id_vars="is_rumor", var_name="features", value_name="value")
plt.figure(figsize=(10,10))
ax = sns.violinplot(x="features", y='value',hue="is_rumor", data=dt, split=True, inner="quart")
plt.xticks(rotation=90)
ax.set_title("Distribution of Feature Values (All News Events)")

#Proportion of Rumor to Non-Rumor tweets by events

twts_event_rumor = df[["event", "tweet_id", "is_rumor"]].groupby(["event", "is_rumor"]).agg(len) \
    .rename(columns={"tweet_id": "total"}).reset_index()
plt.figure(figsize=(10,10))
plt.title("Proportion of Rumor to Non-rumor Tweets by News Event")
ax = sns.barplot(x="event", y="total", hue="is_rumor", data=twts_event_rumor)

#Proportion of Source and Response Tweets by News Event

plt.figure(figsize=(13,7))
plt.title("Proportion of Source and Response Tweets by News Event")
ax = sns.countplot(x="event", hue="is_source_tweet", data=df, palette="husl")

thread_lengths = df.groupby(["event", "thread"]) \
    .agg({ "tweet_id": len, "is_rumor": max }) \
    .reset_index() \
    .rename(columns={"tweet_id": "thread_length"})[["event", "thread", "is_rumor", "thread_length"]]
thread_lengths.head()

thread_lengths.groupby("event").describe()

#Thread Length by Event and Rumor

fig, axes = plt.subplots(figsize=(12,7))
ax = sns.boxplot(y="event", 
            x="thread_length", 
            hue="is_rumor", 
            data = thread_lengths)
ax.set_title("Thread Length by Event and Rumor")

"""**ML Task type: Classification**

For each of the nine given stories, we need to classify to identify the number of tweets associated with it. Also, annote each tweet as rumourous or non-rumourous.

Since the data is labelled, this is an instance of SUPERVISED MACHINE LEARNING.This would help us to find all kinds of hidden patterns in the data and find features which are useful.

We need to group the tweets as RUMOURS or NON-RUMOURS. Classification best suits our project.
"""

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
from sklearn.metrics import f1_score
from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import classification_report,confusion_matrix
from sklearn.metrics import confusion_matrix, accuracy_score
from sklearn.linear_model import LogisticRegression
from sklearn import metrics
from sklearn import preprocessing
from sklearn import svm
import matplotlib.pyplot as plt
from sklearn.metrics import accuracy_score
import matplotlib.mlab as mlab

df.fillna(-99999, inplace=True)
df = df.rename(columns={'user.time_zone':'U'})

X =df.drop(["is_rumor","U","text","event","user.handle"],axis=1)
y=df["is_rumor"]
#y =df.iloc[:,2]
#y=y.astype('int')
X_train,X_test,y_train, y_test = train_test_split(X, y, test_size=0.5,random_state=0, stratify=y)

X_train.describe()

X_test.describe()

y_test.describe()

y_train.describe()

import math
math.sqrt(len(y_test))

# display numerical variables

numerical = [col for col in X_train.columns if X_train[col].dtypes != 'O']

numerical

# check missing values in X_train

X_train.isnull().sum()

# check missing values in X_test

X_test.isnull().sum()

X_train.shape

X_test.head()

X_test.shape

"""**Naive Bayesian Classifier**"""

#Naive Bayesian Classifier
from sklearn.metrics import plot_confusion_matrix
# instantiate the model
gnb = GaussianNB()

# fit the model
gnb.fit(X_train, y_train)

y_pred = gnb.predict(X_test)
y_pred

print('Model accuracy score: {0:0.4f}'. format(accuracy_score(y_test, y_pred)))

y_pred_train = gnb.predict(X_train)

y_pred_train

print('Training-set accuracy score: {0:0.4f}'. format(accuracy_score(y_train, y_pred_train)))

# print the scores on training and test set
print('Training set score: {:.4f}'.format(gnb.score(X_train, y_train)))
print('Test set score: {:.4f}'.format(gnb.score(X_test, y_test)))
print('f1 score:')
print(f1_score(y_test,y_pred))

cm = confusion_matrix(y_test, y_pred)

print('Confusion matrix\n\n', cm)
print('\nTrue Positives(TP) = ', cm[0,0])
print('\nTrue Negatives(TN) = ', cm[1,1])
print('\nFalse Positives(FP) = ', cm[0,1])
print('\nFalse Negatives(FN) = ', cm[1,0])

#Observations:
#Naive Bayesian Classifier yields lowest accuracy of prediction.

# visualize confusion matrix with seaborn heatmap

cm_matrix = pd.DataFrame(data=cm, columns=['Actual Positive:1', 'Actual Negative:0'], 
                                 index=['Predict Positive:1', 'Predict Negative:0'])

sns.heatmap(cm_matrix, annot=True, fmt='d', cmap='YlGnBu')

print(classification_report(y_test, y_pred))

# print the scores on training and test set

print('Training set score: {:.4f}'.format(gnb.score(X_train, y_train)))

print('Test set score: {:.4f}'.format(gnb.score(X_test, y_test)))

import matplotlib.pyplot as plt
plt.figure(figsize=(5, 7))
ax = sns.distplot(df['is_rumor'], hist=False, color="r", label="Actual Value")
sns.distplot(y_pred, hist=False, color="b", label="Predicted Values", ax=ax)
plt.title('Actual vs Precited value for outcome')
plt.show()
plt.close()

"""**k-nearest neighbors(KNN) Classifier** """

#k-nearest neighbors (KNN) classifier 

classifier = KNeighborsClassifier(n_neighbors=13,p=2,metric='euclidean')
classifier.fit(X_train,y_train)
y_pred =  classifier.predict(X_test)
y_pred
class_names=np.array(['0','1'])

# Function to plot the confusion Matrix
def plot_confusion_matrix(cm, classes,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    fmt = 'd' 
    thresh = cm.max() / 2.
    #for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
    for i in range (cm.shape[0]):
      for j in range (cm.shape[1]):
        plt.text(j, i, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')

cm= confusion_matrix(y_test,y_pred)
cm
plot_confusion_matrix(cm,class_names)
print(classification_report(y_test,y_pred))

print('Our criterion give a result of ' 
      + str( ( (cm[0][0]+cm[1][1]) / (sum(cm[0]) + sum(cm[1])) + 4 * cm[1][1]/(cm[1][0]+cm[1][1])) / 5))
print('f1 score:')
print(f1_score(y_test,y_pred))
print('Accuracy score:')
print(accuracy_score(y_test,y_pred))

print('We have detected ' + str(cm[1][1]) + ' rumours / ' + str(cm[1][1]+cm[1][0]) + ' total rumours.')
print('\nSo, the probability to detect a rumour is ' + str(cm[1][1]/(cm[1][1]+cm[1][0])))
print("the accuracy is : "+str((cm[0][0]+cm[1][1]) / (sum(cm[0]) + sum(cm[1]))))

cm = confusion_matrix(y_test, y_pred)

print('Confusion matrix\n\n', cm)

print('\nTrue Positives(TP) = ', cm[0,0])

print('\nTrue Negatives(TN) = ', cm[1,1])

print('\nFalse Positives(FP) = ', cm[0,1])

print('\nFalse Negatives(FN) = ', cm[1,0])

#Actual vs Precited value niplot

import matplotlib.pyplot as plt
plt.figure(figsize=(5, 7))
ax = sns.distplot(df['is_rumor'], hist=False, color="r", label="Actual Value")
sns.distplot(y_pred, hist=False, color="b", label="Predicted Values", ax=ax)
plt.title('Actual vs Precited value for outcome')
plt.show()
plt.close()

"""**Decision Tree Classifier**"""

#Decision Tree classifier


#Training model


from sklearn.tree import DecisionTreeClassifier
dtree = DecisionTreeClassifier()
dtree.fit(X_train,y_train)
predictions = dtree.predict(X_test)
from sklearn.metrics import classification_report,confusion_matrix
print(classification_report(y_test,predictions))
print(confusion_matrix(y_test,predictions))
from sklearn.metrics import accuracy_score
print('Accuracy score:')
accuracy_score(y_test,predictions)

cm = confusion_matrix(y_test, predictions)

print('Confusion matrix\n\n', cm)
print('\nTrue Positives(TP) = ', cm[0,0])
print('\nTrue Negatives(TN) = ', cm[1,1])
print('\nFalse Positives(FP) = ', cm[0,1])
print('\nFalse Negatives(FN) = ', cm[1,0])

cm_matrix = pd.DataFrame(data=cm, columns=['Actual Positive:1', 'Actual Negative:0'], 
                                 index=['Predict Positive:1', 'Predict Negative:0'])

sns.heatmap(cm_matrix, annot=True, fmt='d', cmap='YlGnBu')

import matplotlib.pyplot as plt
plt.figure(figsize=(5, 7))
ax = sns.distplot(df['is_rumor'], hist=False, color="r", label="Actual Value")
sns.distplot(predictions, hist=False, color="b", label="Predicted Values", ax=ax)
plt.title('Actual vs Precited value for outcome')
plt.show()
plt.close()

"""**Logistic Regression Classifier**"""

#Logistic Regression classifier

from sklearn.linear_model import LogisticRegression
logmodel = LogisticRegression()
logmodel.fit(X_train,y_train)
predictions = logmodel.predict(X_test)

from sklearn.metrics import classification_report
print(classification_report(y_test,predictions))

y_pred = logmodel.predict(X_test)
print('f1 score:')
print(f1_score(y_test,y_pred))
print('Accuracy:')
print(accuracy_score(y_test, y_pred))
cm = confusion_matrix(y_test, y_pred)

print('Confusion matrix\n\n', cm)
print('\nTrue Positives(TP) = ', cm[0,0])
print('\nTrue Negatives(TN) = ', cm[1,1])
print('\nFalse Positives(FP) = ', cm[0,1])
print('\nFalse Negatives(FN) = ', cm[1,0])

cm_matrix = pd.DataFrame(data=cm, columns=['Actual Positive:1', 'Actual Negative:0'], 
                                 index=['Predict Positive:1', 'Predict Negative:0'])

sns.heatmap(cm_matrix, annot=True, fmt='d', cmap='YlGnBu')

"""**Selecting the best Classifier**

According to the models implemented above, k-nearest neighbors (KNN) Classifier yields the best results. It has an accuracy of 0.83 and f1 score of 0.55. So, this model best suits our project.
"""