Extracting text from HTML file using Python

I'd like to extract the text from an HTML file using Python. I want essentially the same output I would get if I copied the text from a browser and pasted it into notepad.

I'd like something more robust than using regular expressions that may fail on poorly formed HTML. I've seen many people recommend Beautiful Soup, but I've had a few problems using it. For one, it picked up unwanted text, such as JavaScript source. Also, it did not interpret HTML entities. For example, I would expect ' in HTML source to be converted to an apostrophe in text, just as if I'd pasted the browser content into notepad.

Update html2text looks promising. It handles HTML entities correctly and ignores JavaScript. However, it does not exactly produce plain text; it produces markdown that would then have to be turned into plain text. It comes with no examples or documentation, but the code looks clean.


Related questions:

Answers


html2text is a Python program that does a pretty good job at this.


NOTE: NTLK no longer supports clean_html function

Original answer below.


Use NLTK

I wasted my 4-5 hours fixing the issues with html2text. Luckily i could encounter NLTK. It works magically.

import nltk   
from urllib import urlopen

url = "http://news.bbc.co.uk/2/hi/health/2284783.stm"    
html = urlopen(url).read()    
raw = nltk.clean_html(html)  
print(raw)

Need Your Help

Samsung galaxy S GT-I9000 how to

galaxy smartphone

I have just bought a second hand Samsung Galaxy S GT-I9000. I have some data from the previous owner, like text message history, contact list and so on. I was wondering if there is any way to retur...

Perl post to php

php json linux perl

Im am new to Perl, I have a script that gets all the data in my linux server, process the data and form it into json string.