What is Tony talking about?

by Stubborn Mule on 17 September 2012 · 8 comments

I first experimented with word clouds several years ago and used them to visualise the speeches of Kevin Rudd and Malcolm Turnbull. I have now learned from the Fell Stats blog (via R-Bloggers) that there is an R package for generating word clouds.  The package makes use of tm, a text mining package for R, which I have been meaning to look into for some time. So, it seemed only appropriate to explore the speeches of Tony Abbott.

This word cloud shows the 150 most-used words in Tony’s speeches over the last 18 years. Perhaps disappointingly, since my efforts to strip punctuation also stripped apostrophes, “cant” actually only shows the frequency of the word “can’t”.

Pretty though the word cloud is, a little more can be gleaned from the word usage patterns through time. The correlation in recent years between “carbon” and “tax”, is clearly due to Abbott’s attacks on Labor’s imposition of a price on carbon. His stint as health minister is also evident. I did expect to see more of an impact from his “stop the boats” campaign (here the count for “boat” includes “boats”).

Abbott word count through time

Admittedly, there are no particularly deep insights here, but it was a fun way to learn about the tm and wordcloud packages.

UPDATE: In response to the comment from Dan, I have added a chart showing word frequency rather than count. This accounts for distortions arising from the larger number of Abbott speeches in recent years.

Abbot Word (freq)

 Abbott word frequency through time

For those who are interested, I have uploaded the (python) code for downloading the speeches and the (R) code for generating the charts to github.

Possibly Related Posts (automatically generated):

{ 8 comments… read them below or add one }

1 dan September 18, 2012 at 10:32 am

Sean, haven’t looked at the underlying data, but to what extent are the spikes to the right of the graphs (increases in mentions) simply a function of Abbot saying more since becoming leader?

I think a more interesting graph would be one which shows the changing frequency of a word as a proportion of all words (say for a year – because then you won’t have to seasonally adjust for the quiet periods over Xmas etc).

You might find no change. Or – I suspect – you might find less change. Obviously things like “Carbon” or this week “Muslim” will peak and trough on the news cycle, which is not really in Tony’s control.

2 Stubborn Mule September 18, 2012 at 12:03 pm

It is a very fair point: speeches in the earlier years are fewer and further between. I will roll up my sleeves and turn it into frequencies.

3 Stubborn Mule September 18, 2012 at 6:22 pm

Dan, I have added a frequency chart. There is a pickup in earlier years, but the broad pattern is similar. I also checked “muslim” but the frequencies were very low.

4 Stubborn Mule September 18, 2012 at 7:22 pm

For some reason, 2008 was a quiet year for Tony with only five speeches.

5 Tyler Rinker September 19, 2012 at 1:56 am

As far as the apostrophe problem…
If you’re willing to download my beta package qdap: https://github.com/trinker/qdap you can use strip to remove punctuation except apostrophes.

library(devtools)
install_github(“qdap”, “trinker”)

strip(scrubber(x), apostrophe.remove=FALSE)

6 Tyler Rinker September 19, 2012 at 12:07 pm

Don’t bother with the package: Here’s 3 ways to do it with regexing:
gsub(“[^[:alnum:][:space:]‘\”]”, “”, x) #METHOD 1
gsub(“.*?($|’|[^[:punct:]]).*?”, “\\1″, x) #METHOD 2
gsub(“(.*?)($|’|[^[:punct:]]+?)(.*?)”, “\\2″, x) #METHOD 3

7 Stubborn Mule September 19, 2012 at 2:00 pm

Thanks Tyler: I will give it a try with this regex magic.

8 dan September 19, 2012 at 2:36 pm

Next Mulepost: “What is Tyler talking about?!?”

Leave a Comment

Previous post:

Next post: