Appengine, performance degradation with python27

I wanted to test python27 on appengine so I have migrated my app from python25. Performance got more than 2x slower for every request! Then I've returned to python25 and performance is again as it was before. Here is a picture:

(milliseconds/request) (cgi handler python 27, then python25)

My app uses Werkzeug, Jinja2, and memcache is used quite alot. What reasons can cause such a dramatic decrease in performance? Or is it just because python2.7 on appengine is still in beta?

Some details about application:

It is quite simple online shop. There are some deferred tasks with pdf generation however these don't affect overall graph much because the front page gets most hits. Nearly everything is memcached. It takes up to ~0.8 sec with empty cache to load a page with python 2.5. Non-cached pages takes long to load mainly because there are many db queries. Cached pages load in 60~100 ms. Average load time is ~150 ms. With python 2.7 performance is terrible. Non-cached pages takes 2+ secs to load. Cached pages load in 200+ ms.

Unfortunately I don't have any profiling data and I can't tell what exactly slows things down in python 2.7.

My numbers for page-load time are collected from live page which serves ~10 req/sec and 1 resident python25 instance easily deals with this load.

I have also tested python 2.7 with wsgi and threadsafe:yes, but performance improved just a little compared to python 2.7 and cgi.

Answers


Somewhere on Usenet I read a statement like this from Google "he Python 2.7 runtime is slower than the Python 2.5 runtime in some cases and faster in others. We aren't publicizing the reasons why at this point.". Seems nobody has found so far a scenario where 2.7 is faster than 2.5 thus ...

I read into this

  1. Google is aware of a preformace issue.
  2. They not sure how to handle it.

My profiling indicates that python 2.7 multithreaded applications spend ca. 35 % of their time in {method 'acquire' of 'thread.lock' objects} - and seemingly that happens in googles RPC code. There are also indication that importing has serious locking issues.

Basically you can't do anything about it except waiting for the AppEngine to fix it. See also this comprehensive documentation about the slowdown.

Factors which almost certainly don't play a significant role in this are:

  • uploading pyc files (the GAE infrastructure does that for you)
  • the server provisioning (GAE is such a massive scale and even in closed beta 2.7. was slow)
  • WSGI vs. CGI (wouldn't explain such a huge performance hit)

The ugly thing is that Google did a massive price hike in two steps but told us "by using multithreaded python 2.7 you can run so much more effective that the new prices don't look so bad". Unfortunately the Python 2.7. runtime is still labeled "experimental" and doesn't deliver production quality performance.


Need Your Help

Accessing the query string in ASP.Net Web Api?

routing query-string asp.net-web-api

I'm using the default template generated by Asp.net Web Api. I'm working with the Get() Part:

Tomcat home page is not showing when running in Eclipse

eclipse tomcat servlets

I'm running Tomcat in Eclipse. I would like to see the Tomcat home page, however it doesn't show up. When I browse http://localhost:8080, then I get a HTTP 404 Page Not Found error.

About UNIX Resources Network

Original, collect and organize Developers related documents, information and materials, contains jQuery, Html, CSS, MySQL, .NET, ASP.NET, SQL, objective-c, iPhone, Ruby on Rails, C, SQL Server, Ruby, Arrays, Regex, ASP.NET MVC, WPF, XML, Ajax, DataBase, and so on.