Scraping Part Of A Wikipedia Infobox

I'm using Python 2.7, requests & BeautifulSoup to scrape approximately 50 Wikipedia pages. I've created a column in my dataframe that has partial URL's that relate to the name

Solution 1:

Rather than try and parse the HTML output, try and parse the raw MediaWiki source for the page; the first line that starts with | Length contains the information you are looking for:

url = 'http://en.wikipedia.org/wiki/' + xyz
resp = requests.get(url, params={'action': 'raw'})
page = resp.text
for line in page.splitlines():
    if line.startswith('| Length'):
       length = line.partition('=')[-1].strip()
       break

Demo:

>>>import requests>>>xyz = 'No_One_Knows'>>>url = 'http://en.wikipedia.org/wiki/' + xyz>>>resp = requests.get(url, params={'action': 'raw'})>>>page = resp.text>>>for line in page.splitlines():...if line.startswith('| Length'):...       length = line.partition('=')[-1].strip()...break...>>>print length
4:13 <small>(Radio edit)</small><br />4:38 <small>(Album version)</small>

You can further process this to extract the richer data here (the Radio edit vs. *Album version) as required.