Celery group only has results that have completed since the last time it was checked
In Celery 2, I had a TaskSet that was set up like the following:
for (item,jobId) in itemsAndJobs: tasks.append(waitForOutput.subtask((jobId,item))) job = TaskSet(tasks) result = job.apply_async()
And then later on, I would check to see if the TaskSet had completed by checking:
job.ready() and job.successful()
This worked fine- the waitForOutput tasks would slowly move along, and then when they were all done the job check would complete. I could check multiple times without any problems.
In celery 3, I tried the quick and dirty way of changing this to a group by just changing
Unless I wait until all of the waitForOutput tasks to complete before I ever check ready and successful, this never works. ready() always returns false. I added some logging and a default retry of 30 seconds, and this is what I see-
- start 5 waitForOutput jobs
- check ready(), no waitForOutput jobs have completed, ready is false
- 2 waitForOutput jobs complete
- check ready(), ready is false, job.check_completed() is 2
- check ready(), ready is false, job.check_completed() is 0
- remaining 3 waitForOutput jobs complete
- check ready(), ready is false, job.check_completed() is 3
I see the same behavior if I take my Celery 3 code and import TaskSet from task and use that instead of group.
I would love to be told I'm just using groups incorrectly!
I solved this by moving to Redis as my result backend. It appears to be a bug with using AMQP as a result backend.