For the first time, the main Prime Ministerial candidates for the 2010 UK General Elections, will take part in three live debates. Since the BBC have kindly made the full transcripts available, I decided to have a go at analyzing the data and creating a visual representation in the form of word clouds. I am currently working on my own visualization software, but in the meantime these have been done using Wordle.

Preparing the data

The BBC only provides the data in PDF form – to analyze it we need it in text form. Although this is easily done with Acrobat Reader’s "Save as Text" function, the output it produces is not really suitable for automatic processing, so some work has to be done by hand. This basically involves making sure each speaker’s comments are headed by their name and some kind of special character to split each record ( here I have used ‘@’ ), which took about 15 minutes or so.

Download the raw text of the first debate.

Having done that, a command line tool such as awk can be used to split the data by speaker. For example, the following command outputs Clegg’s comments into a separate file:

awk 'BEGIN {RS=""; FS="[@]"} $1=="NC" { print $2 }' debate.txt > clegg.txt

Parsing the data

Python‘s Natural Language Toolkit provides all the functions needed to analyze the text data, such as tokenizing the text by word and even categorizing each word by type, such as proper nouns and prepositions. For example, having extracted Nick Clegg’s speech as above and read the file as a string using Python, the following commands parse the input for sentences, and then tokenize each word procucing a complete word list.

from __future__ import division
import nltk, re, pprint

sentences = nltk.sent_tokenize(text)
tokens=[]
for s = sentences:
 tokens.extend(nltk.word_tokenize(s))
words=[t.lower() for t in tokens]

We can then categorize each word with a POS tag and extract a list appropriately, for example, using the word tokens above the following extracts all the nouns

# this operation takes some time to execute
taggedwords=nltk.pos_tag(words)
nouns=[word for (word,tag) in words if t == 'NN']

Such a list is enough to use with Wordle, however it’s straightforward to create a word frequency list for use with other software.

nounfrequencies = nltk.FreqDist(nouns)

Going further

Word frequency analyses are fairly straightforward, however NLTK is a powerful library and allows for much more detailed and informative analysis based on grammar and sentence structure. It would be interesting to see the results of a more sophisticated approach.

Gallery of word clouds from the first debate

[Show as slideshow]

Disclaimer : this is an opinion piece. I trust that people are capable of using Google to research the facts, though if you’re planning on using Wikipedia I should mention that this article has nothing to do with the film starring Kevin Costner or any episodes of Star Trek.

The idea

Tax ‘speculative banking transactions’ at 0.05% to raise money for public services, and to fight global poverty and climate change. I am not economist enough to object to this on financial grounds, so I want to look at it from a purely ideological perspective.

Is global poverty an economic, or political problem?

The underlying assumption behind the campaign is that global poverty is an economic problem, one that can be solved largely by throwing money at it. Is that really the case? It seems to me that the underlying causes of poverty include poor education, sanitation, the prevalence of superstition, oppressive and unstable political regimes and tribal warfare. Some of the richest nations on earth have tried their hand at tackling these issues over the past century, although perhaps not altruistically, and had little in the way of success.

As of 2009, government support for banking institutions cost the UK taxpayer some £850bn, so it seems that the treasury has few problems in finding the cash when it’s required. The Robin Hood Tax proposes putting even more under their control, can our government institutions really be relied upon to manage it efficiently? Even if more money is earmarked for aid, how can we be sure that it will reach its intended destination? The BBC recently reported that a portion of the aid sent to ease the Ethiopian famine in the 1980s only ended up in the hands of  unscrupulous arms dealers.

Avoidance of responsibility

Charities need money, and on those grounds it would seem that the Tax would be a good idea. However, more than money, charities need volunteers willing to give up their time and their skills. Is there not the worry that by collectively handing responsibility for charitable giving over to the treasury, we wash our hands of the issues and carry on oblivious to the problems around us?

The underlying causes of global poverty and suffering in all their complexities will only be resolved by accepting, and not absolving ourselves of our responsibilities. What is needed is not increased taxation, but participation. If you are interested in volunteering opportunities in your area, see http://www.do-it.org.uk/

Scapegoating the financial services industry

The banking industry needs to take its share of the responsibility for the financial crisis, but so do we all. Everyone rode on the crest of the credit wave over the past decade, taking >100% mortgages, interest-free overdrafts, living on credit and accepting a lifestyle beyond our means.

Recently the government proposed a tax on bankers bonuses but to do such a thing constitutes a massive abuse of power. If the government wish to reduce the bonuses of bankers then they should use their influence as majority shareholders of the institutions that they have bailed out, surely to achieve the same thing through tax legislation is a conflict of interest? These individuals would be taxed not on the basis of what they earn but on what they do for a living. Many individuals receive salaries that the general public perceive to be grossly inflated, but I have yet to hear if the Robin Hood Tax would be extended to box office or football tickets.

Celebrity endorsements

This may seem a trivial objection, but I found the decision to promote this tax with a video featuring actors Bill Nighy ( 2 O levels ) and Robert Hardy ( BA in English from Oxford, so some education, at least ) incredibly patronising. David Attenborough, when asked of his opinion on global warming, used to refrain from making comments about it on the basis that he was a journalist, not a climatologist. It was only after seeing the magnitude of the effects for himself that he began to speak out. I wish more celebrities would follow his example and understand that, for better or worse, their fame gives them influence that far outstrips their knowledge of the subject. I dread to think of the number of Americans currently taking vaccination advice from Jim Carrey ( high school graduate ) and his wife Jenny McCarthy ( abandoned nursing college to pose for Playboy ). It’s unfortunate that the promoters of the Robin Hood Tax have decided to perpetuate the idea that actors are an authority on anything other than acting.