Serving real-time data python web application

I am writing a Python web application with the Flask framework, WSGIServer and geventwebsockets.

I have a thread pool of workers doing heavy processing work which then insert completed data into a MongoDB database. I want to be able to show a real-time stream of new data from MongoDB to the user on site.

What I have done at the moment is open a socket to connect with the client and poll MongoDB for new data every 3 seconds as shown here:

from flask import Flask
from flask_sockets import Sockets
import datetime

app = Flask(__name__)
sockets = Sockets(app)

def echo_socket(ws):
    last_tweet_printed = datetime.datetime.utcnow() - datetime.timedelta(seconds=55) #start printing tweets from 1 minute ago until catch up.
    while True:
        from database_functions import DatabaseFunctions
        import time
        databaseFunctions = DatabaseFunctions()
        tweets = databaseFunctions.loadTweets() # pulls latest tweets from database (all tweets from last 1 minute)

        limit = 5 # max to print out at once to browser
        index = 0

        for tweet in tweets:
            if(limit != index ):

                if(last_tweet_printed < tweet[u'created_at']): #if the last tweet is older than the one we just pulled...
                    last_tweet_printed = tweet[u'created_at'] #update the latest tweet from db...
                    tweet_text = tweet[u'text']

                    ws.send("<font color=\"blue\">"+tweet_text + "</font><br> <font color=\"red\">" + str(last_tweet_printed) + "</font><br>")

                    print('no new tweets in database, wait till next poll.\n')


        time.sleep(3) #sleep for 3 seconds before polling mongoDB again.

def hello():
    return  \
        <title>Test Real-Time</title>
        <script type="text/javascript">
            var ws = new WebSocket("ws://" + + "/echo");
            ws.onmessage = function(evt){
                    var received_msg =;
                    document.getElementById('mark_test').innerHTML += "Tweet: "+received_msg+"<br>";


            ws.onopen = function(){
                ws.send("hello Mark!");


        <h1>Real Time Stream:</h1>
        <div id="mark_test">



if __name__ == '__main__':
    from gevent import pywsgi
    from geventwebsocket.handler import WebSocketHandler
    server = pywsgi.WSGIServer(('', 5000), app, handler_class=WebSocketHandler)

Are there any limitations with the way this has been written? Are there any more efficient/best practise alternatives that could produce a more seamless stream to the users? I want the application to be able to handle a lot more requests to the database.


There are two problems with your approach. One, each client connected to this Flask server polls the database separately, so if you have 100 clients connected you're doing 100 queries every 3 seconds. Better to have one background thread poll the database every 3 seconds, and update the other threads. echo_socket could wait on a global Condition variable that's notified by the background thread after each update.

The other problem with your code is that you're short-polling MongoDB, when you could be long-polling. Long-polling would give you lower latency between a message arriving in the database and your broadcasting it to users, and it will reduce load on the server. Consider Rick Copeland's blog post on MongoDB pub/sub for inspiration.

Need Your Help

ADT18, App Engine Connected Android project and noclassdeffounderror

android google-app-engine adt

Disclaimer...I think I've gone through all of the related posts on this topic and so far, none of them have fixed my problem. That being said, if I missed something, feel free to point me in the p...

PrimeFaces dialog lazy loading (dynamic=“true”) does not work?

jsf jsf-2 primefaces jstl lazy-loading

I have recently changed all my beans from RequestScoped to ViewScoped. Suddenly, the lazy loading of dialogs does not work. I am using PrimeFaces JSF library.

About UNIX Resources Network

Original, collect and organize Developers related documents, information and materials, contains jQuery, Html, CSS, MySQL, .NET, ASP.NET, SQL, objective-c, iPhone, Ruby on Rails, C, SQL Server, Ruby, Arrays, Regex, ASP.NET MVC, WPF, XML, Ajax, DataBase, and so on.