Celery group only has results that have completed since the last time it was checked

In Celery 2, I had a TaskSet that was set up like the following:

for (item,jobId) in itemsAndJobs:
    tasks.append(waitForOutput.subtask((jobId,item)))
job = TaskSet(tasks)
result = job.apply_async()

And then later on, I would check to see if the TaskSet had completed by checking:

job.ready() and job.successful()

This worked fine- the waitForOutput tasks would slowly move along, and then when they were all done the job check would complete. I could check multiple times without any problems.

In celery 3, I tried the quick and dirty way of changing this to a group by just changing

TaskSet(tasks)

to

group(tasks)

Unless I wait until all of the waitForOutput tasks to complete before I ever check ready and successful, this never works. ready() always returns false. I added some logging and a default retry of 30 seconds, and this is what I see-

  • start 5 waitForOutput jobs
  • check ready(), no waitForOutput jobs have completed, ready is false
  • 2 waitForOutput jobs complete
  • check ready(), ready is false, job.check_completed() is 2
  • check ready(), ready is false, job.check_completed() is 0
  • remaining 3 waitForOutput jobs complete
  • check ready(), ready is false, job.check_completed() is 3

I see the same behavior if I take my Celery 3 code and import TaskSet from task and use that instead of group.

I would love to be told I'm just using groups incorrectly!

Answers


I solved this by moving to Redis as my result backend. It appears to be a bug with using AMQP as a result backend.


Need Your Help

Efficiently getting older versions of R packages

r

A recurring question on SO is that package xx is not available for R version 2.xx.xx. For example the gplots package requires the user to have R 3.0 installed in order for it to install. You can get

Check if Windows machine can act as a host in RDP connection

c++ windows winapi remote-desktop rdp

I'm writing a program and I want to be sure that it will run only on Windows machines that can act as a host in RDP connection. For example, some Windows versions can't do such a thing – e.g. win7 ...

About UNIX Resources Network

Original, collect and organize Developers related documents, information and materials, contains jQuery, Html, CSS, MySQL, .NET, ASP.NET, SQL, objective-c, iPhone, Ruby on Rails, C, SQL Server, Ruby, Arrays, Regex, ASP.NET MVC, WPF, XML, Ajax, DataBase, and so on.