For the first time, the main Prime Ministerial candidates for the 2010 UK General Elections, will take part in three live debates. Since the BBC have kindly made the full transcripts available, I decided to have a go at analyzing the data and creating a visual representation in the form of word clouds. I am currently working on my own visualization software, but in the meantime these have been done using Wordle.

Preparing the data

The BBC only provides the data in PDF form – to analyze it we need it in text form. Although this is easily done with Acrobat Reader’s "Save as Text" function, the output it produces is not really suitable for automatic processing, so some work has to be done by hand. This basically involves making sure each speaker’s comments are headed by their name and some kind of special character to split each record ( here I have used ‘@’ ), which took about 15 minutes or so.

Download the raw text of the first debate.

Having done that, a command line tool such as awk can be used to split the data by speaker. For example, the following command outputs Clegg’s comments into a separate file:

awk 'BEGIN {RS=""; FS="[@]"} $1=="NC" { print $2 }' debate.txt > clegg.txt

Parsing the data

Python‘s Natural Language Toolkit provides all the functions needed to analyze the text data, such as tokenizing the text by word and even categorizing each word by type, such as proper nouns and prepositions. For example, having extracted Nick Clegg’s speech as above and read the file as a string using Python, the following commands parse the input for sentences, and then tokenize each word procucing a complete word list.

from __future__ import division
import nltk, re, pprint

sentences = nltk.sent_tokenize(text)
tokens=[]
for s = sentences:
 tokens.extend(nltk.word_tokenize(s))
words=[t.lower() for t in tokens]

We can then categorize each word with a POS tag and extract a list appropriately, for example, using the word tokens above the following extracts all the nouns

# this operation takes some time to execute
taggedwords=nltk.pos_tag(words)
nouns=[word for (word,tag) in words if t == 'NN']

Such a list is enough to use with Wordle, however it’s straightforward to create a word frequency list for use with other software.

nounfrequencies = nltk.FreqDist(nouns)

Going further

Word frequency analyses are fairly straightforward, however NLTK is a powerful library and allows for much more detailed and informative analysis based on grammar and sentence structure. It would be interesting to see the results of a more sophisticated approach.

Gallery of word clouds from the first debate

[Show as slideshow]

As 2010 is the 100th anniversary of the first published infra red photograph, I thought I’d try my hand using my own digital camera and some easily acquired accessories. If you want, you can skip the theory and go straight to the description of the method and a script for Photoshop.

A quick overview of IR photography

The CCD that is responsible for recording the images photographed by most digital cameras, is already sensitive to the near infra red part of the spectrum. That is, the part of the spectrum outside of the range visible to the human eye, but not so far as that used for, for example, thermal imaging. Since most photographers are not interested in light that they can’t see, this light is usually filtered out by an infra red cutoff filter placed inside the camera body, directly in front of the CCD. However, such filters are imperfect, so with some camera models by combining a long exposure with an infra red transmitting filter placed in front of the lens, some of that IR light can be recovered. The figure shows the basic principle, though I should add that the graphs are just a sketch to illustrate the principle and don’t represent an actual CCD response curve.

Pseudocolour

Since IR light is invisible, some method is needed to represent it in terms of the colours that we can see. Typically, this is often done by simply mapping the intensities of the IR image to a greyscale, with the result being a black and white photo with a slightly surreal look. For my photographs, however, I wanted to try an alternative method.

The LAB Colour Model

The most common method of representing a colour image is by a combination of Red, Green and Blue ‘colour primaries’. The reason for this is simply because it then becomes relatively trivial to display an image using a video display, which also constructs an image using a combination of red, green and blue LEDs or phosphors. This is why such a model is known as a ‘device dependent’ model. However, RGB is not the only colour model, and if you are at all familiar with Photoshop you may be aware of the LAB model.

LAB, or CIELAB to give it its full name, is a device independent model, which like RGB is composed of three components ( or channels ) but in this case only the ‘A’ and ‘B’ components contain colour information, whereas the ‘L’ component contains pure luminosity information. Thus, by separating an image into LAB components it becomes possible to manipulate the luminosity and colour components of an image independently.

Method and Implementation in Photoshop

Equipment

1. Canon G10 Digital Compact Camera
2. LA-DC58K lens adapter for same
3. Hoya R72 58mm Infra Red filter
4. Tripod
5. Photoshop ( or GIMP )

The images in this post have been created from two source images, one a long exposure taken using the IR filter and the other a conventional colour image. Both images are converted into the LAB colour space, and a single image is then produced, using the Luminosity channel from the IR image and the ‘A’ and ‘B’ channels from the colour image. The photoshop script below will automate the process when applied to two source images, you can then manipulate the final image using the adjustment tools. You can also do much the same thing in GIMP using the Decompose/Compose features from the Colour menu, but it does not include Photoshop’s image registration or LAB colour adjustment features.

Download ir_composite.jsx.zip

To use the script, open the IR and corresponding colour image, and ensure that the IR image is selected. Then apply the script.

Practicalities

There are a number of limitations to multiple exposure photography methods such as this, especially ones which require long exposures and changes of camera equipment. It is best suited to landscape photography in good conditions where the multiple exposures can be easily aligned. If either image contains rapidly moving subjects these can ( and will ) show up as artefacts in the resulting image.

Gallery

[Show as slideshow]