workin' in a text mine

OK so what was the point of that last post? I suppose one thing that struck me using Leximancer was that these tools are neat but (like their data mining brethren) it'll be a long time before they move beyond specialist uses into the wider world. So much depends on skilled human operation in terms of framing a problem, selecting & screening terms, and interpreting results.

What experiences have you had with these tools?

Some resources:

• A comparison of unstructured text mining tools from Virginia Uni.
• The National Centre for Text Mining in the UK.
  
• Marti Hearst & NY Times talk Text Mining.
  

leximancer duel - izzard vs hicks

Barry Saunders' stuff has inspired me to have a look at some text mining tools. So I got a trial copy of Leximancer and got busy. I will feed Leximancer various texts over the next few days. I began with a rather hefty challenge: The transcript of Eddie Izzard's Dressed to Kill DVD.

The initial sweep on default settings yielded this (click to enlarge):


Massively increasing the terms (or concepts) yielded this (click to enlarge):

Which is a little clearer if you remove the concepts themselves and just leave the themes (click to enlarge):


And here is some ranking (click to enlarge):


So what can we tell?

• Eddie has a bit of a foul mouth.
• Eddie has a "thing" about religion.
• And war.
• And cake.
• Eddie is probably British.

I then unleashed Leximancer on this transcript of Bill Hicks from the early 90s.

• Here was the out-of-the-box pass...
• ...And with more terms...
• ...And with some clarity...
• ...And some ranking.

Some conclusions for Bill:

• Like Eddie, Bill also has a foul mouth and issues with religion & food.
• I really should use this stuff for something more serious.