Following on from the last post on the Hottest 100, I received a few tweets from @mjdart demanding a deeper dive into the data. One of his questions was

Of artists charting in at least 5 yrs, are Oz artists higher represented?

I decided to broaden the questions to look at artists with at least five tracks in Hottest 100s (so artists with two tracks in one year and one track in three other years would be in). On this criterion, Australia still comes out on top.

Number of artists with at least 5 “Hot” tracks

In the last post, I complained that 2010 data is currently missing from Wikipedia. It seems that this is because Wikipedia is yet to get permission from the ABC. I have decided to risk the wrath of Auntie and have posted the full chart in the table at the bottom of the post. Having pieced it together, I have updated my original chart to include 2010.

As you can see, 2010 was a good year for Australian artists. It also turns out that the artist with the most Hottest 100 tracks is also Australian: Powderfinger. Here are all the artists with at least 10 Hottest 100 tracks.

UPDATE:

@mjdart asked another question which I thought I should also answer:

@stubbornmule If you assign 100 pts for #1 thru 1 pt for #100 each year, is Oz proportion higher/lower? eg Oz filling out top or bottom 50?

To answer this, I assigned 100 points for the top spot, 99 points for the second and so on down to one point for last place. I summed the score for each country and then scaled it by dividing by 50.5. This odd choice arises because 100 + 99 + 98 + … + 2 + 1 = 5050 and so dividing by 50.5 would give a score of 100 for a country that managed to win every spot in the top 100. This makes the score directly comparable to a simple count of places in the top 100. So, how does this weighted score compare to a simple count? The answer, evident in the chart below is not much! So, each country’s artists must be fairly evenly spread through the top 100 over time.

Finally, here is the complete listing of the 2010 Hottest 100, including country of origin. If you are feeling brave, you may wish to update Wikipedia. Just keep in mind, the list may be deleted again if the ABC does not provide permission for the list to be published!

Another year, another Australia Day. Another Australia Day, another Triple J Hottest 100. And that, of course, means an excellent excuse to  set R to work on the chart data.

For those outside Australia, the Hottest 100 is a chart of the most popular songs of the previous year, as voted by the listeners of the radio station Triple J. The tradition began in 1991, but initially people voted for their favourite song of all time. From 1993 onwards, the poll took its current form* and was restricted to tracks released in the year in question.

Since the Hottest 100 Wikipedia pages include country of origin**, I thought I would see whether there is any pattern in whose music Australians like best. Since it is Australia Day, it is only appropriate that we are partial to Australian artists and they typically make up close to half of the 100 entries. Interestingly, in the early 90s, Australian artists did not do so well. The United Kingdom has put in a good showing over the last two years, pulling ahead of the United States. Beyond the big three, Australia, UK and US, the pickings get slim very quickly, so I have only included Canada and New Zealand in the chart below.

Number of Hottest 100 tracks by Country

If you have excellent eyesight, you may notice that 2010 is missing from the chart. For some reason, this is the only year which does not include the full chart listing on the Wikipedia page. There is a link to a list on the ABC website, but unfortunately it does not include the country of origin. Maybe a keen Wikipedian reading this post will help by updating the page.

I make no great claims for the sophistication or the insight of this analysis: it was really an excuse to learn about using the XML package for R to pull data from tables in web pages.

require(XML)
require(ggplot2)
require(reshape2)

results <- data.frame()
col.names <- c("year", "rank", "title", "artist", "country")

# Skip 2010: full list is missing from Wikipedia page
years <- c(1993:2009, 2011)

for (year in years) {
    base.url <- "http://en.wikipedia.org/wiki/Triple_J_Hottest_100,"
    year.url <- paste(base.url, year, sep="_")
    tables <- readHTMLTable(year.url, stringsAsFactor=FALSE)
    table.len <- sapply(tables, length)
    hot <- cbind(year=year, tables[[which(table.len==4)]])
    names(hot) <- col.names
    results <- rbind(results, hot)
}

# Remap a few countries
results$country[results$country=="Australia [1]"] <- "Australia"
results$country[results$country=="England"] <- "United Kingdom"
results$country[results$country=="Scotland"] <- "United Kingdom"
results$country[results$country=="Wales"] <- "United Kingdom"
results$country[results$country=="England, Wales"] <-"United Kingdom"

# Countries to plot
top5 <- c("Australia", "United States", "United Kingdom",
  "Canada", "New Zealand")

# Create a colourful ggplot chart
plt <- ggplot(subset(results, country %in% top5),
    aes(factor(year), fill=factor(country)))
plt <- plt + geom_bar() + facet_grid(country ~ .)
plt <- plt + labs(x="", y="") + opts(legend.position = "none")

Created by Pretty R at inside-R.org

UPDATE: there is a little bit more analysis in this follow-up post.

* Since the shift to single year charts, there have been two all-time Hottest 100s: 1998 and 2009.

** There are some country combinations, such as “Australia/England”, but the numbers are so small I have simply excluded them from the analysis.

We have never been spoiled for choice when it comes to internet music providers in Australia, and things seem to be getting worse not better.

Five or six years ago, I first came across the intriguing internet radio service Pandora which drew upon the painstakingly assembled Music Genome Project to generate customised radio stations. Entering a track or artist on the web site would produce a playlist of “genetically” similar music and the results were impressive. Back then I was able to stream Pandora via my Squeezebox network music player. But it wasn’t long until the music copyright police got onto Pandora and Australians visiting the website would simply see a page explaining why the service was not available. I was lucky enough to still be able to play Pandora stations over the Squeezebox for another year until they discovered that loophole and shut it down.

The on-demand music streaming service Rhapsody has never been available in Australia. Rhapsody’s newer challenger Spotify is also unavailable here and I must admit to a little Schadenfreude when I learned that Spotify is yet to become available in the US. Although I am sure it will be available there before we get it.

With all of these services denied to music-lovers down-under, I had to make do with Last.fm which generated custom stations based on listening habits of other users whose tastes overlapped with your own. Founded almost 10 years ago in London, Last.fm was bought by CBS four years ago, which made me nervous for a while, but the service seemed to continue as usual. Until this February when Australian listeners, and listeners in many other countries around the world, found their Last.fm service abruptly discontinued.

Options for online music should be expanding, yet here in Australia we have fewer services available than we did five years ago.

Last month Rolling Stone published a revised list of the 500 Greatest Songs of All Time. The last version of the list was published in 2004 and, while the update brings the count of 21st century songs from 3 to 28, there have not been too many significant changes. The top ten songs remain the same.

Rolling Stone Top 10 Songs

The Beatles still have more tracks in the list than any other band.

Rolling Stone Top 5 Artists

But what interests me most is what this list has to say about Rolling Stone, its readers and the compilers of the list. A while ago I wrote a post about Triple J’s Hottest 100 of All Time where I noted that the Triple J’s list was heavily skewed to the 1990s. This chart shows the distribution by decade.

So how does the Rolling Stone list compare? Here is its distribution.

The difference between the two should be clear, but just to labour the point, here are the two distributions side by side (and converted to percentages since the Rolling Stone list has five times as many songs in it).

Rolling Stone vs Triple J by Decade

I suppose it should come as no surprise that the baby-boomers love their 60s and 70s music and the Gen-Ys love their 90s music. But, having spent my formative music-listening years in the 80s, I cannot help but feel that decade is under-represented by both charts. Or is that an accurate reflection of the quality of music in the 80s?

And another question: how likely is it that this post will end up in the headlines of Bubblepedia? Fortunately, not very.

Data: the list was obtained from here, a reference obtained from the Wikipedia entry. I fixed some typos, added years and loaded the data into a Google docs spreadsheet. Let me know if you see any remaining errors.

Following on from my post on Visualizing the Hottest 100, I noticed that the UK’s Guardian newspaper has published a list of 1000 songs to hear before you die*. The list was assembled from nominations posted by readers. Even before looking at the list, I suspected that the demographic profile of the Guardian’s readers may be a little different to that of Triple J’s listeners. A look at the distribution of year of release in the two lists bears that out.

Year of Release “Five Number” Statistics

In fact, fully 14% of the tracks in the Guardian’s list were released before the earliest track in the Hottest 100. Interestingly, that track was Bob Dylan’s “Like A Rolling Stone”, which also features in the Guardian’s list.

While the 1000 songs are not presented in any particular rank order, they are grouped by “theme”. The themes are heartbreak, life and death, love, party sonds, people and places, politics and protest and, of course, sex. This allows us to investigate the evolution over time of these different themes.

The chart below is a “box and whisker plot”, also known more prosaically as a “box plot”. It provides a graphical representation of the distribution over songs in each theme by year of release. The box shows the “interquartile range”, from the 1st quartile to the 3rd quartile. This means that half the songs fall inside the box, while a quarter were released in earlier years and a quarter in later years. The solid band shows the median year, which is the year right in the middle of the distribution. The light grey line shows the average year of release. Since most of the distributions are skewed to the left (early years) right (later years) in the interquartile range [see UPDATE below], the mean is a bit higher than the median. The “whiskers” on the plot extend no more than 1.5 times the width of the box. Any outliers beyond the whiskers are shown as points.

Distribution of Year of Release

So what can be made of these distributions? It looks as though love songs are not as popular as they once were and people and places have fared worse still. But while love may be old-fashioned, sex and party songs have become more prevalent and there is still plenty of heartbreak.

And what of the most popular artists? The three most successful artists in Triple J’s Hottest 100 were Nirvana, Jeff Buckley and Radiohead. Nirvana and Radiohead managed one song each in the Guardian’s list: “Lithium” and “Paranoid Android” respectively (both in the life and death theme). Jeff did not make the list, although his father Tim did, with the song “On Top”. The artist with the most entries in the Guardian’s list was Bob Dylan, and the top 12 features a few who did not make it into the Hottest 100 at all, including Randy Newman, Frank Sinatra and The Kinks.

It’s hard to read much more than that into these numbers, but importantly it gave me the opportunity to use a box and whisker plot which this blog has been sorely lacking.

UPDATE: As Mark has commented, this is a bit of a dodgy explanation. There is only so much that can be deduced about a distribution from a box and whisker plot (appealing though they may be). This histogram shows the distribution of the year of release for life and death songs.

Life and Death Theme Histogram

Mark also pointed out that the box and whisker plot does not really show the relative popularity of the different themes over time. I haven’t used pie charts yet, but I am not a fan, so I have come up with a mosaic plot instead.

This confirms the decline in popularity of the love theme, but suggests that, while sex boomed in the 1990s, it has lost ground again in the 21st century. Heartbreak and party songs are the most popular themes of the current decade. The chart also shows that there are more songs in the list from the 60s and 70s than from the 90s, again a departure from the Hottest 100.

I have added this chart to the Guardian Datastore photo pool on flickr.

* To be precise, there are only 988 different songs in the list (and six are duplicated, each appearing in two different categories).

Today radio station Triple J finished broadcasting their Hottest 100 tracks of all time, the first all-time vote since 1998. For those outside Australia and not familiar with the tradition of the Hottest 100, it began back in 1989 and results are determined by listener votes. After two more years the format changed and votes were restricted to tracks released over the previous year, presumably because the top 10 became a list of the usual suspects. Since then 1998 and this year have been the only all-time hottest votes. A traditional favourite, Love Will Tear Us Apart by Joy Division, which was #1 in two of the first three all-time charts only made it to #4 this year, but Nirvana’s Smells Like Teen Spirit was #1 in the third and again in 1989 1998 and this year it made it to #1 for a third time.

Thanks to the wonderful collaborative spirit of Web 2.0, this year’s full list is already up on Wikipedia, complete with the year of release of each track. This allows me to indulge in my data mining hobby, which is why I am posting here rather than over on the The Music Blogs. So, inspired by a suggestion from Mark Lauer, a regular Mule reader (and careful sub-editor), here is a look at the distribution of the hottest 100 tracks by year of release.

Hottest 100 Track Ranking by Year of Release

While the density certainly increases after about 1995, reflecting a lot of new entrants since the early charts, there is no clear trend along the 45 degree line (and, for the technically-minded, the R² is about 0.1%). So, while there are not as many oldies in the chart, those oldies that do make it in are just as likely to rank well as the newer entrants. To make the most of the R code I wrote to produce this chart, here is the same thing showing artist name rather than track name.

Hottest 100 Artist Ranking by Year of Release

To get a better sense of the distribution of rank and year, here is a chart that just shows the location of the tracks by year and rank.

Hottest 100 Rank versus Year of Release

Seeing the data just as points like this shows a concentration of tracks released around the mid-90s. A histogram of the year of release confirms this.

Of course, I’m sure this says more about the demographics of voters than the preponderance of true classics in the 90s.

UPDATE: In this tweet, @nicwalmsley suggested an artist scoring system: 100 points for ranking 1st, 1 point for ranking 100th. As he notes, this system puts Radiohead, Jeff Buckley and Nirvana in 1st, 2nd and 3rd place respectively. Here are the top 10 artists by this measure.

FURTHER UPDATE: @Warlach has laboured hard to assemble the full Hottest 100 as a blip.fm playlist.

YET ANOTHER UPDATE: In case you are wondering about the geographic mix, as expected the list is dominated by the US and the UK.

The pedants should note that I’ve counted System of a Down in the USA (rather than USA/Armenia) and Crowded House as Australia (rather than Australia/New Zealand). I hope that doesn’t offend our Kiwi cousins!

Over the last few years, I have posted a number of times on the subject of music. These posts have ranged from the subject of Krautrock to a critique of the RIAA. From now on, I have decided to post pieces which are directly about music (concert reviews, genres, etc) over on The Music Blogs, where I am now a guest contributor. Anything about the economics of music or music and web 2.0 may continue to appear here on the Mule.

I have begun this shift with a review of the Lee “Scratch” Perry concert at the Sydney Opera House. Perry, one of the pioneers of dub music, is now 73 so it was a historic occasion, not to be missed. It was a great concert and featured the legendary Adrian Sherwood on the mixing desk, but that’s as much as I’ll say here. To find out more, you’ll have to read the review!

Last year I wrote about the the music/social network combination blip.fm. That post was followed up with one on the demise of muxtape and mixwit in which I said “I hope that blip.fm does not become the next victim of the RIAA”. While blip.fm has survived to date, it may only last by significantly changing its laissez-faire approach to streaming music.

A post on their blog last week opens

In the past few weeks we’ve had to make a few difficult decisions that will change the way some things work on Blip.fm.  For the majority of you the changes will be for the better, for others they might be less than ideal for the time being.

It goes on to note that music will “primarily” be sourced from the music service imeem rather than broad-based searches of the internet. Users will no longer be able to submit urls pointing to mp3s. Instead, a set of “approved” urls will be used.

Blip say they cannot give the reasons for the changes, but it is no surprise that blip.fm is under enormous pressure from the music industry to curtail their service and these steps will certainly help ease that pressure. However, as noted in many of the comments on their blog, the risk is that these changes will undermine the very features of blip.fm that has made it so popular. For example, they are likely to fall foul of the mess that is music “territory” rights. Most “legitimate” music services like imeem restrict music streaming to certain countries, filtering based on IP address (in the case of imeem it only plays 30 second clips, while other services like Pandora are completely blocked). If blip.fm ends up relying exclusively on these services for their content, they will lose much of the international audience that has been a major contributer to their growth in recent months.

So, while blip.fm may survive in a compromised form, it provides yet another example of the extent of mess that is music distribution in the digital age.

UPDATE: In further news, the concert-streaming site fabchannel has been forced to close. Their post explaining the closure makes for very interesting reading.

Back in August, muxtape, a popular music playlist site, was forced to close by the Recording Industry Association of America (RIAA). Now mixwit have announced that it is closing too. The only explanation offered was as follows:

We’ve put a year of work into Mixwit so this choice wasn’t taken lightly. I won’t go into the details of our situation but state simply that we boldly marched into in [sic] a position best described as “between a rock and a hard place.”

Reading between the lines, it looks as though they too have fallen at the hands of the RIAA. Under the cover of claims to be protecting artists, claims that do not bear close scrutiny, the RIAA is building an impressive track-record of stifling innovation. While it is possible to take comfort from the fact that attempts to stem the tide of progress always fail in the end, it is nevertheless frustrating to see the suffering of victims of this pernicious organisation in the meantime, whether those victims are single mothers sued for file-sharing or the creators of sites like muxtape and mixwit.

While these sites appeared to be on shaky legal ground from the start, that seems to be a reason to change the rules to fit the new technology, not the other way around. Larry Lessig eloquently summed up the issues years ago in his presentation <free culture>, and the challenges he laid down then are yet to be addressed. I can only hope things begin to change soon and, in the meantime, I hope that blip.fm does not become the next victim of the RIAA.

In my recent post on the future of Microblogging, I expressed concerns about the viability of twitter given that they are yet to find a business model. But perhaps I just wasn’t thinking laterally enough: earlier this week I stumbled across a novel approach to monetising microblogging. The new site Blip.fm brings music to microblogging in a way that initially had me scratching my head, but it is gradually starting to make more sense. Based on a recent post on the Microblogger’s blog, 140char, others are responding in much the same way.

Blip.fm works much like any other microblog, allowing you to post short messages and follow the stream of messages posted by others (if this doesn’t make any sense, have a look at this post on microblogging 101). The difference is that you have to choose a song before every post and you can listen to a stream of posts as well as reading it. Of course, blip.fm also has all the usual microblogging features, with some of the usual tweaking of terminology: you can follow (favorite) other users (DJs), post replies, etc. They also throw in the idea of giving a DJ “props”. Initially this can come across as something of a gimmick, much like “karma” on plurk, but it actually turns out to be an effective way of speeding up the network building process. On other microblogs you decide whether someone is interesting based on what they say. With blip.fm that decision can be based on what they are listening to. I suspect that this is actually a faster track to finding kindred spirits.

Music is not just  blip.fm’s point of differentiation from other microblogs; it is also the way blip hopes to make money. Next to every track there is a “Buy This MP3″ link, which takes you to the Amazon music store. Blip.fm would earn affiliate revenue for any sales generated through these links, although how effective this is remains to be seen. While I suspect that very few people will click though to buy tracks, the plan would be to get the network of users (sorry, DJs) to such a size that even a small percentage of click-throughs would yield a decent income.

Online music can, however, be fraught with danger and getting clicks on the “Buy” links may not be the only challenge blip.fm faces. Back in April, the cnet blog Webware looked at two new music sites for creating online “mix-tapes” and questioned whether they were on solid legal ground. They were right to ask as muxtape has now been shut down, presumably at the hand of the RIAA. Of the two sites, muxtape always seemed to be sailing closest to the wind as they allowed users to upload tracks to share. Becoming popular very quickly would not have helped either.

The other mix-tape site, mixwit, took a different approach and, so far at least, is still up and running. You cannot upload music to mixwit; instead the site provides a user-friendly front end to the music search-engines seeqpod and SkreemR. This creates a protective buffer for mixwit in defending piracy complaints: they would argue that they can not be held any more accountable for infringing material than Google. Of course, the weakness in this argument is that vitually all of the content found on mixwit would be breaching copyright, while most material found on Google would not be.

Judging from the occasionally dodgy track names, I suspect that blip.fm uses the same meta-search approach as mixwit (although it does also allow uploads), so the fates of the two sites are almost certainly linked. I hope that they do both survive so that I can continue to embed my mixes in blog posts and keep blipping.

P.S. I’m a little short on props at the moment, do drop by and click on one of the little stars!

UPDATE: unfortunately mixwit has now been forced to close. I hope that blip.fm is not doomed too.