Code runs in shell but not from the module

This code for downloading a json file from a url runs alright when I type commands one by one in the python shell. However, when I try to run the module containing this code I get: ValueError: No JSON object could be decoded. Any ideas why is that? I run python 2.7.

import urllib2
from urllib2 import Request
import json
import re

url1 = "http://www.skyscanner.net/flights/lond/nyca/130514/130525/airfares-from-london-to-new-york-in-may-2013.html"

req = Request(url1)
res = urllib2.urlopen(req)
the_page = res.read()
theText = str(the_page)

myre = re.compile(r'"SessionKey":"((([a-z0-9]+-)+)[a-z0-9]{12})"')
match = re.search(myre, theText)

print match.group(1)

url2 = "http://www.skyscanner.net/dataservices/routedate/v2.0/"+str(match.group(1))
htmltext = urllib2.urlopen(url2)
data = json.load(htmltext)

the whole code now:

import urllib2
from urllib2 import Request, urlopen, URLError, HTTPError
import json
import re


url1 = "http://www.skyscanner.net/flights/lond/nyca/130514/130525/airfares-from-london-to-new-york-in-may-2013.html"

req = Request(url1)
res = urllib2.urlopen(req)
the_page = res.read()
theText = str(the_page)

myre = re.compile(r'"SessionKey":"((([a-z0-9]+-)+)[a-z0-9]{12})"')
match = re.search(myre, theText)

url2 = "http://www.skyscanner.net/dataservices/routedate/v2.0/%s" % str(match.group(1))

req2 = urllib2.Request(url2)

try:
    response = urlopen(req2)
except HTTPError as e:
    print 'The server couldn\'t fulfill the request.'
    print 'Error code: ', e.code
except URLError as e:
    print 'We failed to reach a server.'
    print 'Reason: ', e.reason
else:
    data = json.loads(response.read())

print data["SessionKey"]

Answers


Ha! You were correct, it is a difference between shell use and scripted use. Whatever that url1 is doing, it hasn't completed (remotely) by the time you open url2. Putting a short wait time in makes it all work!

import time

url2 = "http://www.skyscanner.net/dataservices/routedate/v2.0/%s" % str(match.group(1))

time.sleep(5)

req2 = urllib2.Request(url2)

so it wasn't your code at all... ho hum!


Need Your Help

Adobe livecycle form validation email, numbers etc.

javascript adobe live cycle livecycle-designer

I am not familiar with Adobe live cycle but I am creating a form which I need some validation. When a user inputs a email I put this script to validate my email which works in another PDF but Live ...

Can I delete php5's “error_log.txt”?

php iis iis-6

On our IIS6 server (running on a Microsoft Server 2003), the error_log.txt file located in the php5 folder has grown to a huge size of 500MB, and is massively fragmented. It's so huge I can't even...

About UNIX Resources Network

Original, collect and organize Developers related documents, information and materials, contains jQuery, Html, CSS, MySQL, .NET, ASP.NET, SQL, objective-c, iPhone, Ruby on Rails, C, SQL Server, Ruby, Arrays, Regex, ASP.NET MVC, WPF, XML, Ajax, DataBase, and so on.