As if I would’t have enough to do, I started playing with Python, and I loved it: It’s kept simple, it’s powerful, and you don’t have to learn it. I started programming seriously in Python after two hours of command line tests. Everyone who has seen a programming language yet knows Python.I really felt the absence of serious functional programming support and a kind of OCaml-like matching but… it’s another story.
I often need to grasp data from html pages, and I often do that with Java and HTMLParser… but this time I tried with Python because one of my friend told me how beautiful the soup is.
When facing the problem of getting a lot of data from html pages, I always dream a sort of pattern matching for HTML: I want to be able to get “this kind of tag sequence” inside a page. So I created a library to do that for me based on BeautifulSoup, and it’s impressively effective and easy to use!
Theory
An HTML page has a tree of nodes, called DOM, where each node is a tag or a string (this is how BeautifulSoup see it). So we have the DOM of a page and the tree of a pattern, and we want to find all the occurrences of that pattern in the page.
Now the question is: how do we find that two nodes matches ? Well, for my purposes I defined that two nodes are similar if they have the same tag name and if the input node (from the page) matches AS STRING, at least all the attributes of the pattern node. Ok, what does mean that two strings matches ? I defined a simple syntax that worked good for all my targets, and you’ll see it in the following example.
Practice
Let’s suppose you have a page with a list of videos (page.html), and you want to get all the videos:
<html> <head><title>Example</title></head> <body> <div class="video"> <a href="watch?v=0001">Title first video</a><img src="preview1.jpg"/></div> <div class="video"> <a href="watch?v=0002">Title second video</a><img src="preview2.jpg"/></div> <div class="video"> <a href="watch?v=0003">Title third video</a><img src="preview3.jpg"/></div> ... </body> </html>
You just have to copy the tags structure of one video and give it to my python script as pattern, substituting the variables you want, like this (pattern.html):
<div class="video"><a href="watch?v=$code$">$title$</a><img src="$preview$"/></div>
So, just put $variable$ where you want (be as much restrictive as you can to avoid ambiguities). Now if you run the script you get:
claudio@laptop:~$ ./htmlmatch.py index.html pattern.html code: 0001 title: The first video preview: preview1.jpg code: 0002 title: The second video preview: preview2.jpg code: 0003 title: The third video preview: preview3.jpg
.
You can easily access all these filed using the library as a function in your python code and iterating the list (of dictionaries) it gives you back. For example:
page = urllib2.urlopen("http://www.your_video_website.com/")
pattern = open("pattern.html", "r")
matches = htmlmatch(page, pattern)
for map in matches:
for k, v in map.iteritems():
print k, v
print
The Source
I suggest you to use BeautifulSoup version 3.0.7a, because it behaves better with real world HTML (newer versions are based upon the new python HTML parser which is not very improved).
Click here to download htmlmatch-0.1.py as file
#!/usr/bin/env python
import sys
import urllib2
from BeautifulSoup import BeautifulSoup, Tag, NavigableString
import HTMLParser
def htmlmatch(page, pattern):
"""Finds all the occurrencies of the pattern tree into the given html page"""
isoup = BeautifulSoup(page)
psoup = BeautifulSoup(pattern)
def untiltag(gen):
node = gen.next()
while True:
if isinstance(node, Tag):
break
elif len(node.lstrip()) == 0:
node = gen.next()
else:
break
return node
pgen = psoup.recursiveChildGenerator()
pnode = untiltag(pgen)
igen = isoup.recursiveChildGenerator()
inode = untiltag(igen)
variables = []
lastvars = {}
while True:
newvars = nodematch(inode, pnode)
if newvars != None:
if len(newvars) > 0:
lastvars.update(newvars)
try:
pnode = untiltag(pgen)
except StopIteration:
pgen = psoup.recursiveChildGenerator()
pnode = untiltag(pgen)
if len(lastvars) > 0:
variables.append(lastvars)
lastvars = {}
else:
pgen = psoup.recursiveChildGenerator()
pnode = untiltag(pgen)
try:
inode = untiltag(igen)
except StopIteration:
if variables != None:
return variables
return None
return variables
def nodematch(input, pattern):
"""Matches two tags: returns True if the tags are of the same kind, and if
the first tag has AT LEAST all the attributes of the second one
(the pattern) and if these attributes match as strings, as defined in
strmatch function."""
if input.__class__ != pattern.__class__:
return None
if isinstance(input, NavigableString):
return strmatch(input, pattern)
if isinstance(input, Tag) and input.name != pattern.name:
return None
variables = {}
for attr, value in pattern._getAttrMap().iteritems():
if input.has_key(attr):
newvars = strmatch(input.get(attr), value)
if newvars != None:
variables.update(newvars)
else:
return None
else:
return None
return variables
def strmatch(input, pattern):
"""Matches the input string with the pattern string. For example:
input: "python and ocaml are great languages"
pattern: "$lang1$ and $lang2$ are great languages"
gives as output the map:
{"lang1": "python", "lang2": "ocaml"}
The function returns None if the strings don't match."""
var, value = None, None
i, j = 0, 0
map = {}
input_len = len(input)
pattern_len = len(pattern)
while i < input_len:
if var == None:
if pattern[j] == '$':
var = ""
value = ""
j += 1
elif input[i] != pattern[j]:
return None
else:
i += 1
j += 1
else:
while pattern[j] != '$':
var += pattern[j]
j += 1;
j += 1
if j == pattern_len:
while i < input_len:
value += input[i]
i += 1
else:
while i < input_len and input[i] != pattern[j]:
value += input[i]
i += 1
i +=1
j +=1
map[var] = value
var = None
return map
def main(argv):
if len(argv) < 2:
print "example: ./htmlmatch.py input.html pattern.html"
return
page = open(argv[0], "r")
pattern = open(argv[1], "r")
l = htmlmatch(page, pattern)
for m in l:
for k, v in m.iteritems():
print k, v
print
if __name__ == "__main__":
main(sys.argv[1:])
