Following on from the last post on the Hottest 100, I received a few tweets from @mjdart demanding a deeper dive into the data. One of his questions was

Of artists charting in at least 5 yrs, are Oz artists higher represented?

I decided to broaden the questions to look at artists with at least five tracks in Hottest 100s (so artists with two tracks in one year and one track in three other years would be in). On this criterion, Australia still comes out on top.

Number of artists with at least 5 “Hot” tracks

In the last post, I complained that 2010 data is currently missing from Wikipedia. It seems that this is because Wikipedia is yet to get permission from the ABC. I have decided to risk the wrath of Auntie and have posted the full chart in the table at the bottom of the post. Having pieced it together, I have updated my original chart to include 2010.

As you can see, 2010 was a good year for Australian artists. It also turns out that the artist with the most Hottest 100 tracks is also Australian: Powderfinger. Here are all the artists with at least 10 Hottest 100 tracks.

UPDATE:

@mjdart asked another question which I thought I should also answer:

@stubbornmule If you assign 100 pts for #1 thru 1 pt for #100 each year, is Oz proportion higher/lower? eg Oz filling out top or bottom 50?

To answer this, I assigned 100 points for the top spot, 99 points for the second and so on down to one point for last place. I summed the score for each country and then scaled it by dividing by 50.5. This odd choice arises because 100 + 99 + 98 + … + 2 + 1 = 5050 and so dividing by 50.5 would give a score of 100 for a country that managed to win every spot in the top 100. This makes the score directly comparable to a simple count of places in the top 100. So, how does this weighted score compare to a simple count? The answer, evident in the chart below is not much! So, each country’s artists must be fairly evenly spread through the top 100 over time.

Finally, here is the complete listing of the 2010 Hottest 100, including country of origin. If you are feeling brave, you may wish to update Wikipedia. Just keep in mind, the list may be deleted again if the ABC does not provide permission for the list to be published!

Another year, another Australia Day. Another Australia Day, another Triple J Hottest 100. And that, of course, means an excellent excuse to  set R to work on the chart data.

For those outside Australia, the Hottest 100 is a chart of the most popular songs of the previous year, as voted by the listeners of the radio station Triple J. The tradition began in 1991, but initially people voted for their favourite song of all time. From 1993 onwards, the poll took its current form* and was restricted to tracks released in the year in question.

Since the Hottest 100 Wikipedia pages include country of origin**, I thought I would see whether there is any pattern in whose music Australians like best. Since it is Australia Day, it is only appropriate that we are partial to Australian artists and they typically make up close to half of the 100 entries. Interestingly, in the early 90s, Australian artists did not do so well. The United Kingdom has put in a good showing over the last two years, pulling ahead of the United States. Beyond the big three, Australia, UK and US, the pickings get slim very quickly, so I have only included Canada and New Zealand in the chart below.

Number of Hottest 100 tracks by Country

If you have excellent eyesight, you may notice that 2010 is missing from the chart. For some reason, this is the only year which does not include the full chart listing on the Wikipedia page. There is a link to a list on the ABC website, but unfortunately it does not include the country of origin. Maybe a keen Wikipedian reading this post will help by updating the page.

I make no great claims for the sophistication or the insight of this analysis: it was really an excuse to learn about using the XML package for R to pull data from tables in web pages.

require(XML)
require(ggplot2)
require(reshape2)

results <- data.frame()
col.names <- c("year", "rank", "title", "artist", "country")

# Skip 2010: full list is missing from Wikipedia page
years <- c(1993:2009, 2011)

for (year in years) {
    base.url <- "http://en.wikipedia.org/wiki/Triple_J_Hottest_100,"
    year.url <- paste(base.url, year, sep="_")
    tables <- readHTMLTable(year.url, stringsAsFactor=FALSE)
    table.len <- sapply(tables, length)
    hot <- cbind(year=year, tables[[which(table.len==4)]])
    names(hot) <- col.names
    results <- rbind(results, hot)
}

# Remap a few countries
results$country[results$country=="Australia [1]"] <- "Australia"
results$country[results$country=="England"] <- "United Kingdom"
results$country[results$country=="Scotland"] <- "United Kingdom"
results$country[results$country=="Wales"] <- "United Kingdom"
results$country[results$country=="England, Wales"] <-"United Kingdom"

# Countries to plot
top5 <- c("Australia", "United States", "United Kingdom",
  "Canada", "New Zealand")

# Create a colourful ggplot chart
plt <- ggplot(subset(results, country %in% top5),
    aes(factor(year), fill=factor(country)))
plt <- plt + geom_bar() + facet_grid(country ~ .)
plt <- plt + labs(x="", y="") + opts(legend.position = "none")

Created by Pretty R at inside-R.org

UPDATE: there is a little bit more analysis in this follow-up post.

* Since the shift to single year charts, there have been two all-time Hottest 100s: 1998 and 2009.

** There are some country combinations, such as “Australia/England”, but the numbers are so small I have simply excluded them from the analysis.

Spending couple of weeks down the south coast of New South Wales, spotting dolphins and echidnas, has slowed down my blogging. Fortunately, regular contributor James Glover has once more come to the rescue with a guest post. This time his topic is wheelchairs and air-travel.

Perhaps you’ve heard of a recent court case in which a wheelchair user, Sheila King, took Jetstar to court (and lost) on the basis of the Disabilities Discrimination Act? If you are a wheelchair user and you book a flight on one of our airline carriers then a fairly obvious thing won’t happen. Unlike say a bus you won’t be able to board the aircraft in your chair and be strapped in for the journey. What actually happens is that when making the booking you tick a box (or tell the booker on the phone) that you are in a wheelchair. If there are seats available for wheelies when you get to the airport you will give up your chair and be made to use a specially designed “wheelchair” (its a chair, it has wheels) that is designed to be fit the narrow corridor of most planes which I am sure you are aware of – their narrowness, for you, only apparent when the person ahead of you is blocking the aisle loading 3 pieces of carry on luggage into the overhead lockers while chatting to their new friends in the seat they are meant to occupy. We all suffer this situation. These “wheelchairs” are not designed to be used without help, they are more like children’s toy carts and cannot be operated by the user as the wheels are very small and low down. For a wheelchair user to be taken out of their wheelchair in a public place can be quite discombobulating. Many wheelchair users develop a personal relationship with their chair – it is after all a place you spend many of your waking hours.

Digression. The very first time I was in a wheelchair outside the confines of a hospital ward (it was a hospital wheelchair but is the exact same model I now own, like I said it is personal) I was being pushed by none other than the proprietor of this very website! Without going into the details let’s just say it was a pretty dramatic event and we both learned a valuable lesson in wheelchair use and the wheelchair repair workshop at the hospital was kept busy. But I digress.

So here is the thing. According to Google about 1% of the population uses wheelchairs. And a Jetstar plane has about 200 seats so they expect to get about 2 wheelchair users on average per flight. So what is the problem with only allowing this same number on each flight, as some airlines do? Well the problem is that statistically wheelchair users don’t travel in pairs and sometimes there will be less than 2 users and sometimes there will be more. Just as if you toss 10 coins sometimes there will be fewer than 5 heads (the average or expected number) and sometimes there will be more. Only on average will there be 5. In fact it is a simple problem to work out the probability of there being, say, n wheelchair users, given the average of 1% on a 200 seat plane. This is called the Binomial Distribution. If you have access to Excel then the function Binomdist(n,200,1%) will tell you this probability. Before I give you some numbers I admit that the overall population average may not be the same as the average flying on planes. It may be less than 1% due to wheelchair users being put off flying. But maybe on some routes it is higher: but I am guessing the annual “snowbird” migration of retired people from the northern United States to Florida at the start of Winter would track above the 1% rate.

So here are the Binomial probability figures.

For example, assuming a 1% chance of any given passenger a 200 hundred seat plane being in a wheelchair, the probability that there will be exactly 4 wheelchair passengers wanting seats is 9%. To work out the probability of a passenger being denied a seat on their preferred flight, we will assume that we’re dealing with an airline where more than two wheelchair passenges book on a flight, then at least on passenger will have to change their travel plans. From the table above, the chance of the flight only having 0, 1 or 2 wheelchair passengers totals 68%, so there’s a 32% chance that there will be at least one wheelchair passenger who cannot fly. For any one wheelchair passenger, there is a (n-1)/(n+1) chance of being bumped if n other wheelchair passengers book on the flight. Weighting that by the probably that there are n passengers and adding it up for all n>1 gives a probability of 27%. As a frequent flyer in a wheelchair, you can expect to miss out on a seat quite regularly! [Note: these calculations have been updated: the editor's "corrections" were undone. Ed.]

I am quite fortunate now that I no longer need to travel in my wheelchair. But as I still use a walking stick I wait for everyone else to get off the plane. You sit there, looking behind you to see if everyone else has left. But there are always these strange people who seem to sit there at the back of the plane and wait for 10 minutes or more, after everyone has disembarked, before even moving. You wonder why the airline staff don’t just hurry them off? I assume they aren’t disabled because they are sitting at the back of the plane. If airlines really had a problem with the extra time that getting wheelies off the plane then they could make this up by just moving these people along.

When I first read about this case my initial response was that being disabled and traveling is a bit of challenge anyway and you just get on with it. But the more I thought about it I wondered if the airlines just took it for granted that wheelchair users would change their plans to fit in with the rules. I am glad Sheila King took the issue up!

For some time now, I have been meaning to have a look at Beautiful Soup, a python library designed to make it easy to scrape data from web-sites. Now that I have finally tried it out, I wish I had got to it sooner. It really is very handy and easy to use.

As my first Soup project, I turned to an old Mule favourite: petrol data. The Australian Institute of Petroleum (AIP) publishes retail petrol price data, which it sources in turn from MotorMouth. The price data is spread across individual pages for each state, like this one for Victoria and there are separate pages for unleaded and diesel. It would be nice to pull together all of the data and, since the pages are all laid out in exactly the same way and there is a straightforward naming convention for the urls of each page, this is very easy to do using Beautiful Soup. You can see the results showing average weekly petrol prices for the week ending 18 December in the table below.

With all the data in hand, the obvious questions to ask are: where is the cheapest petrol and where is the most expensive petrol? As you can see in the chart below, Adelaide came in as the cheapest place to fill your tank in late December at 134.9 cent/L, while Broome was the most expensive at 165.9, an impressive 23% mark up over Adelaide.

Top 10 and bottom 10 unleaded petrol prices
(average for the week ending 18 December 2011)

Of course this only gives a snapshot at a point in time: Adelaide may not always offer such good value for money and Broome residents may not always pay such a premium. Unfortunately, I have not been able to find any historical data by town on the AIP website. So I have set my data-scraping routine up to collect the data each week. Some time late this year I will revisit this data to see if any patterns emerge over time.

Returning for another guest post, James Glover is once again drawn to a beer coaster for some quick, if somewhat morbid, calculations. For those taking to the road over the Christmas period, this post should also serve as a reminder to drive carefully!

Why did the chicken cross the road? To get to The Other Side. This twist on the ending to the iconic joke was based on the observation of a single dead chicken on the road while I was returning by car from Sydney to Melbourne at the end of my recent touring holiday. The holiday in fact started two weeks earlier when I was driving around Tasmania. While there were no chickens, there was a dead sperm whale on the road.

In Tasmania I noted that in addition to its abundance of quality food, beer and wines, it has a remarkable supply of one other thing compared to the mainland: native fauna road kill. If you think that the occasional dead kangaroo (or more likely fox) you see driving in the country is plenty, then you haven’t been to Tassie. We are talking a dead possum or wallaby every kilometre or so, which means if you are driving at 100km/hr means you see one every 36 seconds. While it is unfortunate, particularly for the animals involved, to see so many native animals dead (but no foxes because they have been eradicated from the island), it is actually cause for joy because it indicates a very healthy population of native wildlife.

This got me thinking. Could you actually use the number of road kill to estimate the density of animals living in the bush? The answer is yes, and without any derivation or proof I present it here:

density = road kill ratio/kill zone area

where

road kill ratio = av. distance between cars/
                   av. distance between road kill
kill zone area = 2 x car velocity x time x car width

It’s a surprisingly simple formula, and can also be handy for keeping children occupied on long car trips (at least in Tasmania). The model it is derived from is admittedly fairly simplistic – let’s just call it the “Frogger model of vehicle/animal interaction”. Here the time is important because clearly if we had an infinite amount of time and no method of disposal of road kill then the number would build up without limit. In practice the attendant carrion birds on each road kill and its, shall we say, “freshness” (blood, guts, brains you get the picture) suggests that they were all products of the previous nocturnal period’s collisions. In fact there are road signs indicating to drivers to be particularly careful between dusk and dawn to avoid animals so I take “time” to be 10 hours and assumed all carrion are fresh. Taking the average distance between road kill to be 1km and cars to be 10km gives a road kill ratio of 10. The average car speed was 100 km/hr and my car is about 2.5m wide so putting this all together gives an estimated bush density of 20 animals per square km. That’s about one per 5 hectares. That seems a little on the low side for a densely populated area but as “beer coaster” estimates go, it’s probably not a bad start.

This reasoning got me thinking that there probably true that there are no Tasmanian tigers left, because one would have shown up as road kill by now. Tasmanian rangers patrol the roads every morning looking for Tassie devils to monitor the spread of that awful face tumour disease they get. In fact, I saw two Tassie devils on the road and they were both alive and moving. I also saw three live echidnas: three more than I have seen in my life up to now. As mentioned I also saw a dead sperm whale on the road. Not some replica or whale skeleton either: it had died the day before. It was on the beach near Strahan which I was driving on at the time. So, yes, it was on my road and hence I feel justified classifying it as road kill. And no, I don’t know why the whale crossed the road. Perhaps it mis-heard someone say that in Tasmania there was an abundance of “road krill”.

As a change from the usual fare of economics and finance, I recently read  Jürgen Trimborn’s biography Leni Riefenstahl: A Life about Hitler’s favourite film-maker, Leni Riefenstahl. Riefenstahl was a highly controversial figure. Her films Triumph des Willen, chronicling the 1935 Nazi party rally in Nürnberg and Olympia, documenting the 1936 Olympic Games in Berlin were critically acclaimed around the world, but also served as propaganda for the Third Reich.

After the war, Riefenstahl was acquitted in de-Nazification trials, but for some years struggled to shake the taint of her association with Hitler and his regime. Over time she found much of the world shifting its attitudes towards her. As Trimborn observes:

The older she became, the more pronounced was the phenomenon of Riefenstahl’s “promotion to the status of a cultural monument,” as Susan Sontag described it in 1974. The critical disputes surrounding her receded further and further into the background, replaced by an enthusiastic, or at least respectful, tribute to Riefenstahl’s ceaseless vitality.

This vitality really was extraordinary. At the age of 71, she lied about her age, claiming to be twenty years younger in order to take a scuba-diving course and from there spent decades developing a new career as an underwater photographer. She died in 2003 at the age of 101.

But Trimborn argues that this fascination with Riefenstahl’s vitality was a distraction from the complexities of Riefenstahl’s character. His biography portrays a brilliant, driven woman who was also a narcissist and a liar, who spent most of her life denying her complicity with the Nazis. For instance, she spent the later years of the war working on a fiction feature, Tiefland. The exigencies of war stymied her plan to film in Spain, so instead she made use of gypsies from Nazi labour camps as extras. She later claimed to have seen her cast fit and well after the war, but later eyewitness testimony not only revealed that many had ended up in concentration camps like Dachau, but that Riefenstahl knew only too well what was happening.

Reading about Leni Riefenstahl is a good recipe for cognitive dissonance: the contradictions are hard to reconcile. Her documentaries were stylistically revolutionary, redefining the genre, and yet the content and the context are appalling. She was a gifted artists, but in many ways highly objectionable. But, who said the real world was simple?

To provide a bit more context for the French government bond spreads discussed in the last post, the chart below shows the 5-year spreads to German bonds for a few more European countries.

With spreads over 4300 basis points (43%), the chart is dominated by Greece, so here is the chart again with Greece removed.

As you can see in both charts, while French spreads are certainly heading north, they have a long way to go.

For those who have spotted the break in the line for Ireland, my data source seems to be missing 2010 data. I am looking into that and will update the casts if I plug the gaps.

Data source: Bloomberg.

Changes of leadership in both Greece and Italy were initially well-received by markets, but investors are getting nervous again. Attention is shifting to France, and French government bonds seem to be on the nose. The chart below shows the “spread” between French and German 5-year government bonds. Measured in basis points (1/100th of 1%), the spread is the difference between the yields on the respective bonds and it has now reached 183 basis points.

Given that yields on 5-year government bonds are only 0.95%, that is a big difference. Investors are demanding almost double the rate of interest on a French bond offered by a German bond if they are to take on the risk that France is not able to repay its debt in 5 years’ time.

Unlike France, the United Kingdom is lucky enough to have its own currency and the spread between UK and German government bonds is only 10 basis points. More on that in a future post.

Data source: Bloomberg.

It has been quite a long time since art was the subject of a post here on the Mule, but today we took the kids to see Sculptures by the Sea. Held each year, this exhibition consists of a series of large sculptures arranged along the coast from Bondi beach to beach. As usual, parking was challenging, but as usual, the effort was worthwhile. We did not make it along the full length of the exhibition (small legs got a bit too tired), but there were some excellent pieces. The family favourite was, without a doubt, the magnificent stag. Here are a few photos.

In response to my post about colour wheels, I received a suggested enhancement from Drew. The idea is to first match colours based on the text provided and then add nearby colours. This can be done by ordering colours in terms of hue, saturation, and value. The result is a significant improvement and it will capture all of those colours with more obscure names.

Here is my variant of Drew’s function:

col.wheel <- function(str, nearby=3, cex=0.75) {
	cols <- colors()
	hsvs <- rgb2hsv(col2rgb(cols))
	srt <- order(hsvs[1,], hsvs[2,], hsvs[3,])
	cols <- cols[srt]
	ind <- grep(str, cols)
	if (length(ind) <1) stop("no colour matches found",
		call.=FALSE)
	s.ind <- ind
	if (nearby>1) for (i in 1:nearby) {
		s.ind <- c(s.ind, ind+i, ind-i)
	}
	ind <- sort(unique(s.ind))
	ind <- ind[ind <= length(cols)]
	cols <- cols[ind]
	pie(rep(1, length(cols)), labels=cols, col=cols, cex=cex)
	cols
}

I have included an additional parameter, nearby, which specifies the range of additional colours to include. A setting of 1 will include colours matching the specified string and also one colour on either side of each of these. By default, nearby is set to 3.

The wheel below shows the results for col.wheel(“thistle”, nearby=5). As well as the various shades of “thistle”, this also uncovers “plum” and “orchid”.

This is far more powerful than the original function: thanks Drew.