I’m trying to develop a simple web scraper. I want to extract text without the HTML code. It works on plain HTML, but not in some pages where JavaScript code adds text.
For example, if some JavaScript code adds some text, I can’t see it, because when I call:
response = urllib2.urlopen(request)
I get the original text without the added one (because JavaScript is executed in the client).
So, I’m looking for some ideas to solve this problem.
