Cron Jobs, Task Queue or Deferred Tasks?
In Google App Engine, for a task like scanning some RSS feeds and adding new entries from the feed to the datastore every 10-15 seconds, should I use Cron Jobs, Task Queue or Deferred Tasks? I'm really confused.
Invoke a cron job every 1 minute, which would get the RSS and sleep for 15 seconds four times. You can locks to prevent overlapping (although the database insertion provides some measure of concurrency control).
Python-like pseudo code:
if cant_get_lock: exit else: for i in (1,2,3,4): get RSS sleep 15 seconds
- I think if it happens every 15 seconds(not skipping) than I would cron jobs because that is easiest to implement. But if you need to be able cancel task then you should use task queue.
- BTW you should use PubSubHubbub(hubbub) to receive updates for the feeds in realtime if I understand you correctly.