Sentiment Analysis of Social Survey Data for Local City Councils

Big Data analytics can be used by smart cities to improve their citizens’ liveability, health, and wellbeing. Social surveys and also social media can be employed to engage with their communities and these can require sophisticated data analysis techniques. This research was focused on sentiment analysis of social surveys both from traditional sources and social media. 

Initially, standard data analysis techniques using RStudio and Python were applied to several open-source datasets (the 2018 Social Indicators Survey dataset published by the City of Melbourne (CoM) and the Casey Next short survey 2016 dataset published by the City of Casey (CoC)). The qualitative nature of the CoC dataset responses could produce rich insights unlike the quantitative CoM dataset. Word cloud visualizations and bar charts were created for sentiment (as shown for the survey question “Which issue was of most concern?”).

The Casey Next survey analysis provided an initial understanding of issues of interest to the CoC through identification of keywords common among survey respondents. These keywords formed the basis for a Twitter investigation where a Twitter API was created and embedded within RStudio. Sentiment for a specific issue was determined by using a lexicon-based approach where the words in each tweet were compared to those in a lexicon that contains positive and negative words and their associated intensities. The bar chart shows overall sentiment as a percentage ranging from very positive to very negative.

The R codes were converted to web apps and embedded within a website to provide a customisable solution to estimate sentiment on key issues both from the historic 2016 dataset and more immediate Twitter data. Further refinement of the methodology is required to improve the social media twitter app. Future projects may involve analysis of different social media such as Facebook and Instagram that include other media types such as imagery and video.