Text mining on the horizon

During my undergrad days, I took a database class and just touched the surface of data mining. But the techniques I learned made me think there was a way to mine text the same way.

Researchers at UC-Irvine, according to ZDNet, have done it, with "a relatively new method named topic mining."

From the press release:

Performing what a team of dedicated and bleary-eyed newspaper librarians would need months to do, scientists at UC Irvine have used an up-and-coming technology to complete in hours a complex topic analysis of 330,000 stories published primarily by The New York Times.

And here's how it works:

Text mining allows a computer to extract useful information from unstructured text. Until recently, text mining required a great deal of preparation before documents could be analyzed in a meaningful way. A new text-mining technique called "topic modeling" - which UCI scientists used in their New York Times experiment - looks for patterns of words that tend to occur together in documents, then automatically categorizes those words into topics - all with minimal human effort.

There's also a link to software that researchers can experiment with.

U.S. Congress voting

The great researchers at the Washington Post created a database that tracks member voting from both House and Senate. You can browse every vote since 1991, and can search by House or Senate member, by state, vote missers, bills, most voted on, vote margins and more. Here is the list (with links) of North Carolina 109th Congressional members.

Poverty and Opportunity

The Brookings Institution recently released a 80 page report (PDF) titled, "From Poverty, Opportunity: Putting the Market to Work for Lower Income Families. From the executive summary:
"In general, lower income families tend to pay more or the exact same consumer products than families with higher incomes....At least 1.6 million lower income adults pay excessive fees for furniture, appliances, and electronics. And countless more pay high prices for other necessities such as basic financial services, groceries, and insurance....Reducing the costs of living for lower income families by just one percent could add up to over $6.5 billion in new spending power for these families."

Top 10 Medical Web Sites

The Medical Library Association lists their top 10, most useful medical web sites for health consumers.