Category Archives: sections

Meta-category for grouping categories on the nav menu.

Do Daleks use toilet paper?

I have been watching some (very) old Doctor Who episodes, including the first ever serial featuring the archetypal villains, the Daleks. In this story, the Daleks share a planet with their long-time enemies, the Thal. After a war culminating in the denotation of a neutron bomb, both races experience very different mutations. The Daleks have become shrunken beasts that get about in robotic shells, while the more fortunate Thals mutated into peace-loving blondes.

The Thals hope to make peace with the Daleks, but the Daleks have more fiendish plans and plot to lure the Thals into their city with a gift of food and then ambush them. It is a good plan, but it is the choice of gifts that left me bemused. There is plenty of fruit and some large tins whose contents remain undisclosed. These may be reasonable choices, although I do find it hard to picture the Daleks stacking melons with their plunger hands. But the trap also appears to feature stacks of toilet paper. Granted, toilet paper may be an appealing luxury for the Thal, who have been trekking through the jungle for a year, but the real question here is, why do Daleks even have toilet paper?

Dalek ambush

Chinese non-residents…in China

CCTVRecently I travelled to China for the first time. My first glimpse of Beijing took in the Escher-like headquarters of Chinese TV station CCTV. It is an extraordinary building and to get a proper sense of it, you have to see it from a number of different angles.

Driving across the city, impressed by the scale of the place, I asked one of my hosts about the population of Beijing. He told me there were about 40 million, including non-residents. Almost double the entire population of Australia. Maybe it’s an exaggeration, but more than the figure itself it was the reference to “non-residents” that piqued my interest. Where there really so many people moving to China as to have a significant impact on the population of the capital?

Later, I learned that these non-residents were in fact people from other provinces. Under China’s Hukou system, restrictions are imposed on people’s ability to move from one part of the country to another. Many people from rural areas are drawn to cities to find work, but without residency rights for the city in which they work they cannot access public education or health care. So, Beijing is full of married men who have left their families at home in the provinces. Living in tiny apartments, they work all year and then travel back to their families for Chinese New Year, taking their earnings with them.

Being used to freedom of movement in Australia, it’s hard not to see this as a harsh system. But, reflecting on the numbers, China is a country of 1.3 billion people; if there are already 30 to 40 million people in Beijing, how would the city cope with a sudden influx of millions more? Only a few days ago, the central committee of China’s communist party released new targets to increase urbanisation from 53.7% of the population to 60% by 2020. This plan involves granting urban hukou status to an additional 100 million rural migrant workers. Even so, another 200 million migrants will remain non-residents. It is sobering to consider the potential consequences of granting full freedom of migration to the entire population rather than managing the process in this highly controlled fashion.

I’m not about to renounce my belief in democracy (however challenged it may be in many Western countries today), but, much like the CCTV building, it seems that to better understand China, you have to see it from a number of different angles.

Bringing Harmony to the Global Warming Debate

For some time now, our regular contributor James Glover been promising me a post with some statistical analysis of historical global temperatures. To many the science of climate change seems inaccessible and the “debate” about climate change can appear to come down to whether you believe a very large group of scientists or a much smaller group of scientists people. Now, with some help from James and a beer coaster, you can form your own view.

How I wish that the title of this article was literally true and not just a play on words relating to the Harmonic Series. Sadly, the naysayers are unlikely to be swayed, but read this post and you too can disprove global warming denialism on the back of a beer coaster!

It is true, I have been promising the Mule a statistical analysis of Global Warming. Not only did I go back and look at the original temperature data but I even downloaded the data and recreated the original “hockey stick” graph. For most people the maths is quite complicated though no more than an undergraduate in statistics would understand. It all works out. As a sort of professional statistician, who believes in Global Warming and Climate Change, I can only reiterate my personal  mantra: there is no joy in being found to be right on global warming.

But before I get onto the beer coaster let me give a very simple explanation for global warming and why the rise in CO2 causes it. Suppose I take two sealed glass boxes. They are identical apart from the fact that one has a higher concentration of CO2. I place them in my garden (let’s call them “greenhouses”) and measure their temperature, under identical conditions of weather and sunshine, over a year. Then the one with more CO2 will have a higher temperature than the one with less. Every day. Why? Well it’s simple: while CO2 is, to us, an “odourless, colourless gas” this is only true in the visible light spectrum. In the infra-red spectrum, the one with more CO2 will be darker. This means it absorbs more infrared radiation and hence has a higher temperature. CO2 is invisible to visible light but, on it’s own, would appear black to infrared radiation.  The same phenomenon explains why black car will heat up more in the sun than a white one. This is basic physics and thermodynamics that was understood in the 19th century when it was discovered that “heat” and “light” were part of the same phenomenon, i.e. electromagnetic radiation.

So why is global warming controversial? Well, while what I said is undeniably true in a pair of simple glass boxes, the earth is more complicated than these boxes. Radiation does not just pass through, it is absorbed, reflected and re-radiated. Still, if it absorbs more radiation than it receives then the temperature will increase. It is not so much the surface temperature itself which causes a problem, but the additional energy that is retained in the climate system. Average global temperatures are just a simple way of trying to measure the overall energy change in the system.

If I covered the glass box containing more CO2 with enough aluminium foil, much of the sunshine would be reflected and it would have a lower temperature than its lower CO2 twin. Something similar happens in the atmosphere. Increasing temperature leads to more water vapour and more clouds. Clouds reflect sunshine and hence there is less radiation to be absorbed by the lower atmosphere and oceans. It’s called a negative feedback system. Maybe that’s enough to prevent global warming? Maybe, clouds are very difficult to model in climate models, and water vapour is itself a greenhouse gas. Increasing temperature also decreases ice at the poles. Less ice (observed) leads to less radiation reflected and more energy absorbed. A positive feedback. It would require a very fine tuning though for the radiation reflected back by increased clouds to exactly counteract the increased absorption of energy due to higher CO2. Possible, but unlikely. Recent models show that CO2 wins out in the end. As I as said, there is no joy to being found right on global warming.

So enough of all that. Make up your own mind. Almost time for the Harmony. Perusing the comments of a recent article on the alleged (and not actually real) “pause” in global warming I came across a comment to the effect that “if you measure enough temperature and rainfall records then somewhere there is bound to be a new record each year”. I am surprised they didn’t invoke the “Law of Large Numbers” which this sort of argument usually does. Actually The Law of Large Numbers is something entirely different, but whatever. So I asked myself, beer coaster and quill at hand, what is the probability that the latest temperature or rainfall is the highest since 1880, or any other year for that matter?

Firstly, you can’t prove anything using statistics. I can toss a coin 100 times and get 100 heads and it doesn’t prove it isn’t a fair coin. Basically we cannot know all the possible set ups for this experiment. Maybe it is a fair coin but a clever laser device adjusts its trajectory each time so it always lands on heads. Maybe aliens are freezing time and reversing the coin if it shows up tails so I only think it landed heads. Can you assign probabilities to these possibilities? I can’t.

All I can do is start with a hypothesis that the coin is fair (equal chance of heads or tails) and ask what is the probability that, despite this, I observed 100 heads in a row. The answer is not zero! It is actually about 10-30. That’s 1 over a big number: 1 followed by 30 zeros. I am pretty sure, but not certain, that it is not a fair coin. But maybe I don’t need to be certain. I might want to put a bet on the next toss being a head. So I pick a small number, say 1%, and say if I think the chance of 100 head is less than 1% then I will put on the bet on the next toss being heads. After 100 tosses the hypothetical probability (if it was a fair coin) is much less than my go-make-a-bet threshold of 1%. I decide to put on the bet. It may then transpire that the aliens watching me bet and controlling the coin, decide to teach me a lesson in statistical hubris and make the next toss tails and I lose. Unlikely, but possible. Statistics doesn’t prove anything. In statistical parlance the “fair coin” hypothesis is called the “Null Hypothesis” and the go-make-a-bet threshold of 1% is called the “Confidence Level”.

Harmony. Almost. What is the probability that if I had a time series (of say global temperature since 1880) that the latest temperature is a new record. For example the average temperature in Australia in 2013 was a new record. The last average global temperature record was in 1998. I think it is trending upwards over time with some randomness attached. But there are all sort of random process which produce trends, some of which are equally likely to have produced a downward trending temperature graph. All I can really do, statistically speaking, is come up with a Null Hypothesis. In this case my Null Hypothesis is that the temperature doesn’t have a trend but is just the result of random chance. There are various technical measures to analyse this, but I have come up with one you can fit on the back of a beer coaster.

So my question is this: if the temperature readings are just i.i.d. random processes (i.i.d. stands for “independent and identically distributed”) and I have taken 134 of these (global temperature measurements 1880-2014) what is the probability the latest one is the maximum of them all? It turns out to be surprisingly easy to answer. If I have 134 random numbers then one of them must be the maximum. Obviously. Since they are iid I have no reason to believe it will be the first, second, third,…, or 134th. It is equally likely to be any one of those 134. So the probability that the 134th is the maximum is 1/134 = 0.75% (as it is equally likely that, say, the 42nd is the maximum). If I have T measurements then the probability that the latest is the maximum is 1/T. So when you hear that the latest global temperature is a maximum, and you don’t believe in global warming, then be surprised. As a corollary if someone says there hasn’t been a new maximum since 1998 then the probability of this still being true, 14 years later, is 1/14 = 7%.

So how many record years do we expect to have seen since 1880? Easy. Just add up the probability of the maximum (up to that point) having occurred in each year since 1880. So that would be H(T) = 1 + 1/2 + 1/3 + … + 1/T. This is known as the Harmonic Series. It is famous in mathematics because it almost, but doesn’t quite converge. For our purposes it can be well approximated by H(T) =0.5772+ ln(T) where ln is the natural logarithm, and 0.5772 is known as the Euler-Mascharoni constant.

So for T=134 we get from this simple beer-coaster sized formula: H(134) = 0.5772+ln(134)= 5.47. (You can calculate this by typing “0.5772+ln(134)” into your Google search box if you don’t have a scientific calculator to hand). In beer coaster terms 5.47 is approximately 6. So, given the Null Hypothesis (which is that there has been no statistically significant upward trend since 1880) how many record breaking years do we expect to have seen? Answer: less than 6. How many have we seen: 22. 

Temperature peaks

Global temperatures* – labelled with successive peaks

If I was a betting man I would bet on global warming. But there will be no joy in being proven right.

James rightly points out that the figure of 22 peak temperatures is well above the 6 you would expect to see under the Null Hypothesis. But just how unlikely is that high number? And, what would the numbers look like if we took a different Null Hypothesis such as a random walk? That will be the topic of another post, coming soon to the Stubborn Mule!

* The global temperature “anomaly” represents the difference between observed temperatures and the average annual temperature between 1971 and 2000. Source: the National Climate Data Center (NCDC) of the National Oceanic and Atmospheric Administration (NOAA).

I’m with Felix

FT blogger Felix Salmon and venture capitalist Ben Horowitz have very different views of the future of Bitcoin. Salmon is a skeptic, while Horowitz is a believer. A couple of weeks ago on Planet Money they agreed to test their differences with a wager.

Rather than a simple bet on the value of Bitcoin, the bet centres of whether or not Bitcoin will move beyond its current status, as a speculative curiosity, to serve as a genuine basis for online transactions. The test for the bet will be a survey of listeners in five years’ time. If  10% or more of listeners are using Bitcoin for transactions, Horowitz wins. If not, Salmon wins. The winner will receive a nice pair of alpaca socks.

I have been fascinated by Bitcoin for some time now and have a very modest holding of 1.6 Bitcoin. Nevertheless, I believe that Felix is on the right side of the bet. I have no doubt that the technological innovation of Bitcoin will inform the future of digital commerce, but Bitcoin itself will not become a mainstream medium of exchange.

Volatility

Only days after the podcast, the price of Bitcoin tumbled as MtGox, the largest Bitcoin exchange in the world, suspended Bitcoin withdrawals due to software security problems. Sadly, this means my own little Bitcoin investment halved in value. It also highlights how much of a roller-coaster ride the Bitcoin price is on. As long as Bitcoin remains this volatile, it cannot become a serious candidate for ecommerce. It is just too risky for both buyers and sellers. Horowitz acknowledges that the Bitcoin market is currently driven by speculators, but is confident that the price will eventually stabilise. I doubt this. Even during its most stable periods, the volatility of Bitcoin prices is far higher than traditional currencies, and has been throughout its five year history.

Bitcoin drop

The Ledger

One of the key innovations of Bitcoin is its distributed ledger. Everyone installing the Bitcoin wallet software ends up downloading a copy of this ledger, which contains a record of every single Bitcoin transaction. Ever. As a result, there is no need for a central authority keeping tabs on who owns which Bitcoin and who has made a payment to whom. Instead, every Bitcoin user serves as a node in a large peer-to-peer network which collectively maintains the integrity of this master transaction ledger. This ledger solves one of the key problems with digital currencies: it ensures that I cannot create money by creating copies of my own Bitcoin. The power of the ledger does come at a cost. It is big! On my computer, the ledger file is now almost 12 gigabytes. For a new Bitcoin user, this means that getting started will be a slow process, and will make a dent in your monthly data usage. A popular way around this problem is to outsource management of the ledger to an online Bitcoin wallet provider, but that leads to the next problem.

Trust Problems

A big part of the appeal of Bitcoin to the more libertarian-minded is that you no longer have to place trust in banks, government or other institutions to participate in online commerce. In theory, at least. If you decide to use an online Bitcoin wallet service to avoid the problem of the large ledger, you have to trust both the integrity and the security capability of the service provider. The hacking of inputs.io shows that this trust may well be misplaced. Even if you have the patience and bandwidth to maintain your own wallet, trust is required when buying or selling Bitcoin for traditional currency. There are many small Bitcoin brokers who will buy and sell Bitcoin, but invariably you have to pay them money before they give you Bitcoin, or give them Bitcoin before you get your money. Perhaps the big exchanges, like MtGox, should be easier to trust because their scale means they have more invested in their reputation. But they are not household names, the way Visa, Mastercard or the major banks are. Growth of commerce on the internet has been built on trust in the names providing the transactions more than trust in the technology, which most people don’t understand. I would be very surprised to see the same level of trust being established in the Bitcoin ecosystem, unless major financial institutions begin to participate.

The Authorities

But will banks jump onto the Bitcoin train? I doubt it. Not because they are afraid of the threat to their oligopoly—most bankers still only have the vaguest idea of exactly what Bitcoin is, or how it works. What they do know is that virtual currencies are attractive to criminals and money launderers. Last year saw the FBI crackdown on Liberty Reserve, followed by the crackdown on the underground black-market site Silk Road. More recently, the CEO of one of the better-known Bitcoin exchanges was arrested for money-laundering. In the years since September 11, the regulatory obligations on banks to ensure they do not facilitate money laundering have grown enormously. The anonymity of Bitcoin makes it hard for banks to “know their customer” if they deal with Bitcoin and as law-enforcement increases its focus on virtual currencies, providing banking services to Bitcoin brokers becomes less appealing for banks. When I bought my Bitcoin last year, I used the Australian broker BitInnovate. For several months now, their Bitcoin buying and selling services have been suspended and, I’m only guessing, this may be because their bank closed down their accounts. To become a widely-accepted basis for commerce, Bitcoin will necessarily have to interface effectively with the traditional financial system. At the moment, the prospects for this don’t look good.

For these reasons, I think Felix has a safe bet, and can look forward to cosy feet in alpaca socks. But, even if Bitcoin does not become widely accepted, its technological innovations may well revolutionise commerce anyway. Banks around the world can adopt ideas like distributed ledgers and cryptographically secure, irrevocable transactions to make the mainstream global payments system more efficient.

Shark season

Summer in Australia comes with cicadas, sunburn and, in the media at least, sharks. So far, I have learned that aerial shark patrols are inefficient (or perhaps not) and that the Western Australian government plans to keep swimmers safe by shooting big sharks.

Sharks are compelling objects of fear, right up there with spiders and snakes in the package of special terrors for visitors to Australia. As good hosts, we are quick to reassure: sharks may be the stuff of nightmares and 70s horror movies, but attacks are rare.

But, exactly how rare is death by shark? Over a Boxing Day lunch, I heard an excellent ‘statistic’, designed to reassure a visiting American. Apparently, more people are killed each year in the US by falling vending machines than are killed by sharks around the world. I was skeptical, but had no data to hand. Later, with the help of Google, I discovered that this statistic is 10 years old and the source? Los Angeles life guards. The tale has, however, become taller over time. Originally, vending machine deaths in the US were compared to shark attack fatalities in the US, not the entire world.

While data on vending machine related deaths are hard to come by, subsequent attempts to validate the story concluded that it was plausible, on the basis that there were two vending machine deaths in 2005 in the US but no fatal shark attacks.

Fun though the vending machine line may be, it is not relevant to Australia and, if you are on the beach contemplating a quick dip, then the risk of a shark attack is certainly higher in the sea than death by vending machine. Local data is in order.

According to the Taronga Zoo Australian Shark Attack File (ASAF):

 In the last 50 years, there have been 50 recorded unprovoked fatalities due to shark attack, which averages one per year.

Fatalities have been higher than average over the last couple of years. The ASAF recorded two deaths in 2012 and, although validated figures for 2013 are yet to be published, six deaths have been reported over the last two years, suggesting that fatalities rose further to four this year.

To compare shark fatalities to other causes of mortality, a common scale is useful. My unit of choice is the micromort. A one-in-a-million chance of death corresponds to a micromort of 1.0, a one-in-ten-million chance of death to a micromort of 0.1. Taking the recent average death rate of three per year (more conservative than the longer run average of one), and a population of 23 million in Australia leads to a figure of 0.13 micromorts for the annual risk of death for a randomly chosen Australian.

The most recent data on causes of death published by the Australian Bureau of Statistics (ABS) are for 2009. That year, three people were killed by crocodiles. Sharks are not specifically identified, but any fatal shark attacks would be included among the three deaths due to ‘contact with marine animals’. The chart below illustrates the risk of death associated with a number of ‘external causes’. None of these come close to heart disease, cancer or car accidents. Death by shark ranks well below drowning, even drowning in the bath, as well as below a variety of different types of falls, whether from stairs, cliffs or ladders.

Shark barplot

Annual risk of death in Australia (2009 data)*

Of course, you and I are not randomly chosen Australians and our choices change the risks we face. I am far less likely to suffer death by vending machine if I steer clear of the infernal things and I am far less likely to be devoured by a shark if I stay out of the water.

So, care should be taken when interpreting the data in the chart. Drug addicts (or perhaps very serious Hendrix imitators) are far more likely to asphyxiate on their own vomit than summer beach-goers. The fairest point of comparison is drowning in natural waters. At almost 3.5 micromorts, drownings in the sea (or lakes and rivers) is more than 25 times more common than fatal shark attacks. And the risk of both can be reduced by swimming between the flags.

What does that leave us with for conversations with foreign visitors? If you are headed to the beach, the risk of shark attack would be higher than death by vending machine, but it is still very low. The drive there (at 34.3 micromorts) is almost certainly more dangerous.

I will be taking comfort from my own analysis as I am heading to Jervis Bay tomorrow and sharks were sighted there this weekend:

Bendigo Bank Aerial Patrol spotted up to 14 sharks between 50 and 100 metres from shore at various beaches in Jervis Bay. [The] crew estimated the sharks at between 2.5 and 3.5 metres in length at Nelsons, Blenheim, Greenfields, Chinaman’s Beach and Hyams Beaches.

The beaches are un-patrolled, so wish me luck…but I don’t think I’ll need it.

* The figure for ‘Shark attack’ is based on the estimate of three deaths per year rather than the ABS data.

Power to the people

Regular Mule contributor, James Glover, returns to the blog today to share his reflections on solar power.

I have been investigating solar power for years and finally bit the bullet and signed up for a system. A 4.5kW system cost me $8,500, after receiving the Government rebate (about $3,000). I’ve been meaning to write about my adventures in solar for a while now. It started because of a strange fact I discovered about 4 years ago. Even though the cost of solar cells has been dropping dramatically in the last 4 years (it’s gone down about 75%) the payback time has stayed steady at about 5-10 years. The payback time is based on what you save by not paying your power bills plus what you earn by selling electricity back into the grid. The peak time for solar generation is 10am-2pm while the peak time for domestic use is in the morning and evening outside these times.

The answer to my conundrum is that while the cost of solar cells has been steadily dropping so has the feedback tariff. When the feedback tariff was 60c per kWh, the excess power created during the day paid for the disparity in the price of the power consumed in the evening. In Victoria the feed in tariff has dropped to about 8c. In order to have a net zero cost of solar it is necessary to have an even bigger system as peak power cost is about 32c per kWh. A particularly good website I found for all things solar is SolarQuotes.  I thoroughly recommend it as has lots of info on solar power as well as cost benefit analysis. They recommended two solar companies in my area, both of who were very good.

From a financial point of view it makes sense that power companies would buy solar power at a lower rate than they would provide it‑it’s called the bid/offer spread and is how most companies make money. The cost of producing power is about 5c so it is still cheaper for them to produce and sell the power themselves than buy it from solar power generators.

There is a twist to this tale however. Electricity generators are monopolies and so left to their own devices would naturally gouge buyers. When the state governments privatised electricity generation they set up supervisory boards to ensure the companies made reasonable, but not immodest profits. In the absence of a competitive market one way to do this is on a “cost plus” basis: set the profit at say 10% above cost of electricity generation. It seemed reasonable until power companies found a way to game this system. If they increased the cost of providing electricity then they increase their profits.

But surely, you say, the costs of generating electricity are based on market forces for the raw materials plus the cost of running the plant? One way is to spend much more on investment than is actually necessary. And the electricity companies did this beautifully. They convinced the state government oversight bodies that not only was electricity consumption forecast to rise well above GDP growth but that existing infrastructure needed to be “gold plated”: improved to reduce the probability of a widespread failure. A combination of inflated growth predictions (and hence building new plants) and gold plating is the real reason electricity prices have risen 20% year on year over the last few years. Yes, the carbon tax has had a small effect as a one-off increase. The Coalition (now the Government) exploited this in the run up to the election, although I am pretty sure this not was the real reason the Labor government lost office.

If you take solar power growth into consideration then electricity generation from traditional sources such as coal and hydro is expected to fall, not rise. Gold plating (soon to include actual gold power lines…I think I am joking) is now seen for what it is and is being reined in.

One of things I have always wondered is why someone doesn’t set up a virtual power company which buys solar power and sells it to distributors? Turns out they already exist. The thing which swung me to the solar provider I chose (the price was identical to the others) was that they could hook me into just such a company.  Sunpower is a US company which has set up in Australia to do just this. Currently their feed in tariffs are higher (guaranteed 20c for 2 years as opposed to 8c for coal generating providers) though I have no expectation they will remain this high. Australian Diamond Energy is another example of a virtual power company. Diamond Energy buys green power from retail solar producers (i.e. you and me) as well as independent wind farms. They also invest in their own larger scale solar and wind farms. Market forces will dictate the future price and I am happy to offset the environmental cost of running my air conditioning at full bore over summer.

In the US they already have communities which set up solar farms to provide their bulk electricity and sell their excess to the grid. Old style electricity companies have resorted to making claims that there are problems with solar electricity, either because it’s at the wrong time of the day or because old style inverters produce modified sine waves from direct current rather than pure sine waves and some electrical appliances don’t operate as well with these modified sine waves. Increasingly though inverters are of the pure sine wave type anyway. While there is some truth to their arguments, it is worth remembering that power companies would prefer that there was no solar at all. They have an axe to grind, their arguments are designed to limit the onward march of solar, or totally compensate them for lost revenue which will achieve the same aim through higher solar costs or lower feedback tariffs.

Another example of why traditional power companies are increasingly out of touch is smart meters. Solar power companies, monitor power usage through smart meters and solar panel output monitoring.They then provide feedback directly to your table or smart phone, and also work to help users optimise their power usage and minimise costs. Traditional power companies see smart meters as purely a way to save on meter reader costs, they have no interest in reducing users’ power consumption.

It seems that in Australia, the “sunburnt country” we have missed a few tricks. The dinosaur coal-based power companies are fighting a rearguard action, trying to get governments to lower the feed-in tariff further or let them charge solar customers a fixed fee to cover their “costs”. I think they are on the wrong side of history. A consumer group Solar Citizens has already been effective in reminding governments that over 1m households have solar power. I think that 1m is a tipping point.

There are about 8m households in Australia. At a cost of about $5,000 we could make each a net producer of electricity for $40bn.  About the cost of the NBN. A new national Snowy River Scheme!

Power to the people. From the people. For the people.

Qantas and Adobe

In my last post, I complained about the approach Qantas has taken to password security for its new Qantas Cash website. When I called Qantas to express my concerns, my query was referred to the “technical team”. I was assured they would be able to assuage my concerns. Here is the email response I received:

As I’m sure you’ll understand, we cannot discuss in any depth the security protocols and practices of our products.

However, I can assure you that your password is stored and encrypted on our server, is never transmitted and cannot be viewed by anyone.

The reason we use random ordinal characters rather than full password entry is because it is more secure as it makes harvesting passwords using keylogging software a much more challenging task.

Thank you for taking an interest in the product and we are certain you’ll find the site, the card and the product as a whole, a secure and useful addition to your payment options.

I tried to dig a little deeper, asking whether individual password characters were hashed. This did not help:

Thank you for your email. Your previous question has been queried with our technical team. They have advised that we cannot discuss in any depth the security protocols and practices of our products.

I am far from reassured. Security through obscurity is a poor strategy. Knowing how an effective security practice works does not make it weaker. Quite the contrary: the best security practices are well-known and have been tested and retested and have survived unscathed. The ones that do not pass these tests are discarded. If Qantas is keeping their security methods secret, it simply heightens my fear that they have been devised by web developers who are not experts in security and are vulnerable to attack.

Qantas and I are approaching the question of security very differently, with different threat models. Qantas is focused on preventing me from doing something silly that could compromise my account. Whereas I am worried about Qantas being hacked.

Only a few weeks ago, Adobe was hacked and up to 150 million encrypted passwords have been made public. Their encryption methods were weak (no salted hashing!) and password hints for all of the accounts were also leaked. Enthusiastic hackers are quickly reverse-engineering the passwords.

The same thing could happen to Qantas. If it does, and Qantas is moved to offer a heartfelt apology to their customers, I will not be too upset: I will not be one of those customers.

Security can be tricky

Qantas CashQantas has recently launched Qantas cash, a pre-paid Mastercard which you can charge up with cash in multiple currencies. The contemporary equivalent of traveller’s cheques, cards like this can be as convenient as a credit card with the added advantage of reducing the uncertainty associated with exchange rate volatility. If you have a rough idea of how much you will need in euro, you can charge up the card with euro at today’s exchange rate without having to worry about the Australian dollar dropping in value while you are half way through your trip.

As a Qantas frequent flyer account holder, I received a Qantas cash card in the mail and it seemed worth investigating. However after activating the card, my interest in the card itself was quickly displaced by disappointment in the insecure design of the Qantas cash website.

Computer security is not easy. It should be left to the experts. I am no expert myself, but I have listened to enough of the Security Now podcast to recognise poor security when I see it.

The first sign of trouble came with setting my password. The password had to be 6 to 8 characters long. A maximum of only 8 characters? The longer the password length, the more secure it is and 8 characters is far too short for a secure password.

Somewhat disconcerted, I pressed on, creating a password made up of 8 random characters. Random passwords are far more secure than real words (or even transparently modified “w0rd5”). They are also impossible to remember, but there are plenty of secure password storage tools (such as LastPass) that make that unnecessary.

Having set everything up, I was then prompted to log in. Unexpectedly, instead of being prompted to enter my password, I was asked to enter the “3rd, 4th and 5th character of the password”. Alarm bells started ringing. Quite apart from the irritation that this caused as it prevented LastPass from automatically filling in the password, it confirmed my initial fears that the website’s security model was flawed.

What I had realised was that Qantas servers must be storing passwords. For anyone unfamiliar with password security, this may seem blindingly obvious. If the servers don’t store the password, how can the website confirm you have entered the correct password when you log in?

In fact, there is a far more secure approach, which makes use of so-called “one way functions“. A one-way function takes a string of characters (a password, for example) as input produces a different string of characters as its output. The key feature of a one-way function is that it extremely difficult to reverse the process: given the output, working out what the input must have been is computationally highly intensive. Applying a one-way function is also known as (cryptographic) “hashing”.

Armed with a good one-way function, instead of storing passwords, a web server can store a hash of the password*. Then, whenever a user enters a password, the web site applies the one-way function and compares the result to its database. The password itself can be discarded immediately. The webserver’s user database should only ever contain hashes of user passwords and never the “plain text” original version of the password.

While this approach to password storage is well-established practice in the security community, many corporate websites are not designed by security experts. Back in 2011, hackers were able to get hold of more than a million passwords from Sony which had been stored in plain text.

Unfortunately, it would appear that Qantas cash is not following best practice in its website security. If the site was only storing hashed passwords, it would be impossible for the site to verify whether users were correctly entering the 3rd, 4th and 5th character of the password. Taking a password hash and trying to determine individual characters of the original password is just as difficult as reverse engineering the whole password.**

I then called Qantas cash to seek clarification. I was assured that all passwords were “encrypted” using the same security techniques that any other commercial website, such as Amazon, would use. Furthermore, the requirement to enter individual characters of the password was an additional security measure to prevent users from copying and pasting passwords.

This did not reassure me. Even if the passwords are encrypted, the Qantas cash server itself clearly has the capability of decrypting the passwords, which makes it just as vulnerable as Sony. I am also sure that Amazon does not use this approach. And preventing copying and pasting is a furphy. By preventing users from using secure password stores, this approach simply encourages the use of weaker passwords.

The Qantas cash developers may think they have come up with some excellent security features. But these developers are clearly not experts in security and, as a result, have produced a far less secure site. The call centre promised that the technical team would email me more details of the site’s security. My hopes are not high.

Needless to say, I will not be using the Qantas cash card. This is an e-commerce site, not a movie chat forum. When money is involved, security should be paramount.

Keep your eyes open for news about a Qantas cash website hack.

* Strictly speaking, a “salted hash” should be stored to add an additional layer of security and protect against the use of rainbow tables.

** In principle, Qantas could store hashes of three character combinations (56 hashes would have to be stored or 336 if order is significant). In practice I doubt this is being done.

The price of protectionism

An  article in Friday’s Australian began

Ford has blamed Kevin Rudd’s $1.8 billion fringe benefits tax overhaul for halting production, forcing at least 750 workers to be stood down in rolling stoppages that will further imperil Labor’s chances of retaining the nation’s most marginal seat.

and goes on to report that the Federal Chamber of Automotive Industries has called on Labor to reverse its changes to the application of fringe benefits tax (FBT) to cars.

So what exactly has Labor done to put these jobs at risk?

The previous regime provided two mechanisms to determine tax benefits for expenses incurred for cars used for work purposes:

  1. the “log book” method, whereby the driver maintained records to show what proportion of their use of the car was for work rather than personal use, or
  2. an assumed flat rate of 20% work use of the car (regardless of how often the car is actually used for work purposes).

The government has eliminated the second option. So, the estimated $1.8 billion saving is due to the fact that a significant number of drivers using the 20% method could never come close to a 20% proportion of work use if they took the trouble to maintain a log book. Either that or they don’t think it is worth the effort to maintain the log book records.

While the elimination of this tax-payer largesse for drivers may come at a cost to workers in the car industry, does it really make sense to reverse the changes to save 750 jobs? These jobs would be saved at a cost to the tax payer of $2.4 million per job. Now these are just the jobs at Ford and (for now at least) we should acknowledge that some Holden jobs may also be saved, bringing the cost closer to $1 million per job.

The car industry in Australia has long benefited from government support, but surely there are better ways of saving these jobs. A job guarantee springs to mind.

Of course, industry protectionism is far from unique to Australia and this week I had my attention drawn to an extreme example in the small central American nation of Belize.

On 7 August, the parliament of Belize met for the first time since April. With so long between sittings, there were many bills for parliament to pass that day. Included among these was one which increased the already high import tariff on flour from 25% to 100%.

Wheat

Why such a dramatic increase? For some time, local bakers had been buying their flour from Mexico for 69 Belize dollars per sack (approximately A$38). It was hard to justify buying the more expensive local flour at BZ$81 per sack (A$45). The new tariff will push the price of Mexican flour up to around BZ$110 (A$61), which is good news for the domestic flour mill and its employees.

That domestic flour mill is operated by Archer Daniels Midland (ADM), one of the top 10 global commodity firms. This is the same ADM which is in the process of trying to buy GrainCorp, Australia’s largest agricultural business.

But back to Belize. ADM’s website proudly declares that it “employs more than 40 people” in its Belize mill. Presumably, parliament had an eye to saving these jobs from the threat of cheap Mexican flour when it hiked the import tariff. With a population of only 335,000, Belize is 1/70th the size of Australia. You could argue that saving 40 jobs in Belize is the equivalent of saving 2,800 in Australia and that this is a far more effective form of protectionism than reversing FBT reforms.

But protectionism always has consequences and in Belize these are easier to see than is often the case.

Bread in Belize is subject to price control, along with rice, beans and even local beer. By law, bakers must sell “standard loaves” of bread for BZ$1.75. The August sitting of parliament may have increased flour tariffs, but it did not increase the price bakers could charge for bread.

Bakers in Belize will see their profits squeezed, job losses may follow and there are more bakers in Belize than workers at the ADM mill. Needless to say, the Belize Baker’s Association is lobbying for an increase in the controlled price of bread.

Perhaps it is time for the Belize government to consider abandoning the flour tariff and trying a job guarantee instead.

ngramr – an R package for Google Ngrams

The recent post How common are common words? made use of unusually explicit language for the Stubborn Mule. As expected, a number of email subscribers reported that the post fell foul of their email filters. Here I will return to the topic of n-grams, while keeping the language cleaner, and describe the R package I developed to generate n-gram charts.

Rather than an explicit language warning, this post carries a technical language warning: regular readers of the blog who are not familiar with the R statistical computing system may want to stop reading now!

The Google Ngram Viewer is a tool for tracking the frequency of words or phrases across the vast collection of scanned texts in Google Books. As an example, the chart below shows the frequency of the words “Marx” and “Freud”. It appears that Marx peaked in popularity in the late 1970s and has been in decline ever since. Freud persisted for a decade longer but has likewise been in decline.

Freud vs Marx ngram chart

The Ngram Viewer will display an n-gram chart, but does not provide the underlying data for your own analysis. But all is not lost. The chart is produced using JavaScript and so the n-gram data is buried in the source of the web page in the code. It looks something like this:

// Add column headings, with escaping for JS strings.

data.addColumn('number', 'Year');
data.addColumn('number', 'Marx');
data.addColumn('number', 'Freud');

// Add graph data, without autoescaping.

data.addRows(
[[1900, 2.0528437403299904e-06, 1.2246303970897543e-07],
[1901, 1.9467918036752963e-06, 1.1974195999187031e-07],
...
[2008, 1.1858645848406013e-05, 1.3913611155658145e-05]]
)

With the help of the RJSONIO package, it is easy enough to parse this data into an R dataframe. Here is how I did it:

ngram_parse <- function(html){
  if (any(grepl("No valid ngrams to plot!",
                html))) stop("No valid ngrams.") 
    
  cols <- lapply(strsplit(grep("addColumn", html,
                               value=TRUE), ","),
                getElement, 2)
  
  cols <- gsub(".*'(.*)'.*", "\\1", cols)

I realise that is not particularly beautiful, so to make life easier I have bundled everything up neatly into an R package which I have called ngramr, hosted on GitHub.

The core functions are ngram, which queries the Ngram viewer and returns a dataframe of frequencies, ngrami which does the same thing in a somewhat case insensitive manner (by which I mean that, for example, the results for "mouse", "Mouse" and "MOUSE" are all combined) and ggram which retrieves the data and plots the results using ggplot2. All of these functions allow you to specify various options, including the date range and the language corpus (Google can provide results for US English, British English or a number of other languages including German and Chinese).

The package is easy to install from GitHub and I may also post it on CRAN.

I would be very interested in feedback from anyone who tries out this package and will happily consider implementing any suggested enhancements.

UPDATE: ngramr is now available on CRAN, making it much easier to install.