A3_ML_31005.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "A3_ML_31005.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/upatel2/UTS_ML2019_ID13253366/blob/master/A3_ML_31005.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fh5MKXWQI6Ij",
        "colab_type": "text"
      },
      "source": [
        "# **Machine Learning 31005 Assignment Task 3 (A3)**\n",
        "\n",
        "> > > > ### **ML_31005_A3_13253366_Utsav_Patel**\n",
        "\n",
        "\n",
        "The link to my Github file is: https://github.com/upatel2/UTS_ML2019_ID13253366/blob/master/A3_ML_31005.ipynb\n",
        "\n",
        "\n",
        "## Question 3: \n",
        "\n",
        "### *Marketing or advertising companies would be very interested in being able to predict whether a Twitter message will spread as a meme or not, and even better, construct it so that it will spread. Why is this a hard problem to solve? Describe two approaches using data analytics to predict whether a tweet will go viral or not. How would you validate these approaches? Discuss the ethical and social consequences of this study.* \n",
        "\n",
        "---\n",
        "\n",
        "In this age of materialism and social media, the expansion of social platforms has been dramatically increasing. Social networking sites like Twitter have now become an imperative source of information for everyone and hence can be vital for advertising and marketing purposes. All people now use these on a regular basis, thus making it essential for companies to try to understand users’ behaviour. Big companies have started to increase their understanding of their user's preferences. Amongst the various social media platforms, Twitter is an example that has a very large following due to many celebrities, athletes and other famous people. This media has a significantly huge user base with approximately 1.3 billion accounts and several millions of users this month.\n",
        "\n",
        "It becomes essential for marketing companies to understand the factors that contribute towards the “viral” tweets as it can be crucial for the analysis of public opinions and further help gather insights to implement prediction tools. The expansion of social media has brought the emerging concept of “memes”. Memes can be developed by anyone and can be based on someone’s photo, tweet or even inappropriate content. Although memes have been mainly developed for entertainment purposes on social media, they are nowadays misused to a high degree and thus businesses must consider ways to ensure that they are not mistreated. \n",
        "\n",
        "It is important for this problem to be solved since it has raised several questions regarding the privacy and security of the individual in terms of social media. This issue is growing today as companies have taken this issue lightly and have not gained much insight into it deeply. \n",
        "\n",
        "Companies now have started to predict the possibility of a tweet being converted to a meme and going viral. This will help companies understand how it can be stopped and understand its positive and negative impacts on the people and their environment. The prediction problem of the conversion into memes can be solved using Data Analytics. \n",
        "\n",
        "In order to predict the probability of a meme spreading, one Data Analytics approaches we can use is Classification algorithms. Classification is a supervised Machine Learning approach in which the computer learns from the data which is inputted. We will explore Decision Trees and SVM classifiers. \n",
        "\n",
        "Decision Tree classifier breaks down the form into a tree structure. It condenses a dataset into smaller sets and develops a tree correspondingly. This classifier can handle both numerical and categorical data hence making it very valuable. Another major advantage of Decision Trees is that it can easily solve complex problems. Once the classifier is implemented, it analyses the data in the dataset and helps to find the probability of the virality of tweets. One the classifier was implemented, and it was found that the tweet size contributes most for predicting how viral a tweet is. \n",
        "\n",
        "Another classifier called Support-Vector Machines (SVM) is also a supervised learning algorithm that can be implemented to predict whether the tweet gets viral or not. It is mainly used by plotting each data as a point in n-dimensional space that helps distinguish the data points. SVM is excellent at solving complex non-linear problems and can be used to solve both classification and regression problems as well. In this research, SVM can been implemented due to its advantage of working with huge amounts of data. In this classifier, we can use different types of kernels like RBF (Radial Basis Function) which helps to find the space between two features. In this way, the classifier can be used to predict the virality of tweets. \n",
        "\n",
        "The next approach that can be used is Artificial Neural Networks (ANN). ANNs are a set of neurons (units) arranged in layers that convert an input into an output. ANNs have only recently come to the spotlight due to their improved performed in real-world applications. It has several advantages such as, high tolerance to noisy data and working with incomplete information to produce an output. ANNs have a hidden network which helps give it a predicted result. This network tries to minimize the difference between the predicted and actual values using optimizers. Since this question is based on how often a tweet is made into a meme, ANN was chosen over other classification approaches. This approach can be used effectively in this prediction. \n",
        "\n",
        "All the approaches described above play a significant role in determining the likelihood of a tweet becoming a meme. \n",
        "\n",
        "Once all the approaches have been implemented, these approaches must be validated to ensure that the approaches are reliable and accurate. Validation is very important in Data Analytics since it ensures the avoidance of ‘overfitting’ (captures noisy data) of data. Overfitting occurs if the algorithm represents low bias and higher variance. Therefore, we validate the two classifiers using ROC curves and Correlation Matrix, and validate the ANN by RMSE, Partitioning and Cross-validation methods. \n",
        "\n",
        "ROC (Receiving Operating Characteristic) curves are another aspect of validating the problem associated with Twitter. The ROC curve is a tool used for test evaluation through graphs. For our problem, it can be calculated that the higher the probability of a tweet being retweeted, the more chances for it being converted to a meme. The ROC curve is plotted with different cut-off points for all parameters. Each point on the curve represents a decision threshold. The paper [4] evaluates the ROC curve for ANN, Decision Tree and SVM approaches discussed above and tries to predict whether tweet becomes a meme. \n",
        "\n",
        "Correlation Matrix is also a validation part which shows the correlation coefficients between variables. Each cell in the matrix shows the correlation between two variables. It is used to display the attributes and their relationship with other attributes in the dataset. The correlation matrix is a great way to understand the variables in relation to other variables and validate the approaches of SVM and Decision trees. \n",
        "\n",
        "Artificial Neural Networks can be validated through the RMSE (Root-mean-square-error) method of evaluation. However, since that method does not explain the behaviour of our approach, the data is split through the validation process of “Partitioning”. Partitioning is performed to split the training and testing data. After that, the RMSE method is used again to increase accuracy of the approach used. Finally, cross-validation is executed to validate the entire approach. In particular, ‘k-fold cross validation’ must then be performed to the ANN approach. This method ensures that the evaluated results are free from sample bias which makes it very useful. \n",
        "\n",
        "Along with the approaches to solve the problem and validating these methods, it becomes essential to consider the ethical and social aspects of this problem. \n",
        "\n",
        "Ethical and social issues must also be addressed due to vast amounts of security threats that Data Analytics offers. Memes today have been expanding rapidly once they are created. \n",
        "\n",
        "The ethical concerns are highlighted by the evolving social media. Ethics are moral principles that govern the behaviour of an individual or business. For this emerging question about tweets being transformed into memes, the ethical issues would be to misinterpret a tweet and spread it as a joke for entertainment purposes. Additionally, the natural language in a tweet may involve sarcasm, which is usually not understood by most, which in turn leads to them misunderstanding the tweet. This issue is causing several issues like cyberbullying for younger teenagers and affecting them psychologically if the meme constructed is based on them. It is not illegal to do this, however, comes down to individual perception and is considered ethically unacceptable. On the flip side, whilst joining Twitter, each user accepts the terms and condition that anyone can use your tweet unless it is private. So, it is important for all marketing and advertising companies to consider these factors when predicting the probability of a tweet becoming viral into a meme. \n",
        "\n",
        "Social consequences are also equally important to be considered for marketing and advertising companies to address. However today, the social is missing in the social media platforms. Social consequences of predicting in Twitter can be both, positive in one way, and negative in the other. There have been several positive impacts like good interaction between friends living overseas, ability to message celebrities and follow them and to interact with many others. However, it also has negative issues like spreading of rumours, tweets going unnoticed, tweets becoming memes and many more. For the context or the question, the prediction will be able to help reduce the negative impacts memes have on people’s mind and companies will therefore try to limit the virality of tweet into a meme. Thus, having a positive social influence on this topic. \n",
        "\n",
        "The approaches used for the model, can be assessed for each tweet and can be validated through different evaluation. Potential firms based on marketing and advertising will benefit from these approaches to understand the prediction of tweets and hence learn about users’ behaviour. The memes will then be restricted to whom it is spread to and controlled to what age group can see it. This research will help in minimising the negative impacts social media has, and ethically make use of Data Analytics methods.\n",
        "  \n",
        "\n",
        "\n",
        "## Bibliography\n",
        "\n",
        "[1] P. Andr´e, M. S. Bernstein, and K. Luther. Who Gives A Tweet? Evaluating Microblog Content Value. In Proceedings of the 16th ACM Conference on Computer Supported Cooperative Work and Social Computing, CSCW, pages 471–474, 2012. [Accessed 1st October 2019] \n",
        "\n",
        "[2] Weng, L., Menczer, F. and Ahn, Y. (2013). Virality Prediction and Community Structure in Social Networks. Scientific Reports, 3(1) [Accessed 1st October 2019] \n",
        "\n",
        "[3] Jenders, M., Naumann, F. and Kasneci, G. (2019). Analyzing and predicting viral tweets. Conference: Proceedings of the 22nd international conference on World Wide Web companion.\n",
        "\n",
        "[4] Nesi, P., Pantaleo, G., Paoli, I. and Zaza, I. (2018). Assessing the reTweet proneness of tweets: predictive models for retweeting. Multimedia Tools and Applications, 77(20), pp.26371-26396. [Accessed 2nd October 2019] \n",
        "\n",
        "[5] Jain, A., Kant Lavania, M., Prasad, A. and Jindal, J. (2019). Advanced Data Mining Project on Will a tweet go viral?. [online] Available at: https://www.researchgate.net/publication/333614650 [Accessed 3 Oct. 2019].\n",
        "\n",
        "[6] Zubiaga, A., Spina, D., Martínez, R. and Fresno, V. (2014). Real-time classification of Twitter trends. Journal of the Association for Information Science and Technology, 66(3), pp.462-473. \n",
        "\n",
        "[7] Sidana, M. (2017). Types of classification algorithms in Machine Learning. [online] Medium. Available at: https://medium.com/@Mandysidana/machine-learning-types-of-classification-9497bd4f2e14 [Accessed 4 Oct. 2019].\n",
        "\n",
        "[8] Kamath, R. (2017). What are the advantages/disadvantages of Artificial Neural networks? [online] Quora. Available at: https://www.quora.com/What-are-the-advantages-disadvantages-of-Artificial-Neural-networks [Accessed 4 Oct. 2019].\n",
        "\n",
        "[9] Schoonjans, F. (2019). ROC curve analysis with MedCalc. [online] MedCalc. Available at: https://www.medcalc.org/manual/roc-curves.php [Accessed 5 Oct. 2019].\n",
        "\n",
        "[10] Digital Vidya. (2018). The Importance of Cross Validation In Machine Learning. [online] Available at: https://www.digitalvidya.com/blog/cross-validation-in-machine-learning/ [Accessed 5 Oct. 2019].\n",
        "\n",
        "[11] Bock, T. (n.d.). What is a Correlation Matrix? | Displayr. [online] Displayr. Available at: https://www.displayr.com/what-is-a-correlation-matrix/ [Accessed 6 Oct. 2019].\n",
        "\n",
        "[12] Lewis, G. (2016). We Asked an Expert if Memes Can Predict the Next President. [online] Vice. Available at: https://www.vice.com/en_au/article/kwxdqa/we-asked-an-expert-if-memes-could-determine-the-outcome-of-the-presidential-election [Accessed 6 Oct. 2019].\n",
        "\n",
        "[13] Eric S. Swirsky, Jinger G. Hoop, and Labott, S. (2015). Using Social Media in Research: New Ethics for a New Meme? [online] Available at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4169004/ [Accessed 7th October 2019]. \n"
      ]
    }
  ]
}