what might delay my celery tasks?
i have a lil problem and maybe someone could help me to figure out what exactly is happening here:
i have thousands of the same tasks which should execute in milliseconds and right after i restart the worker they actually do execute in milliseconds, but right after the logging of a bunch of those tasks i get a big blob from celery saying they are done (http://d.pr/n/66h) all together in a much lager time frame .. and after that each of those tasks take around 5-13s to execute. This made me calculate that celery might be done with all of the tasks in about a week of execution time. (AAAAAhhhhh)
After a worker restart about 30-60 Task get executed in normal speed so to say. After that i get one result every few seconds - like mentioned above kinda like 4-13s. Sure in some cases there is a little bit more to do than than in others, but this does NOT justify differences from 0.08s to 13s!
Im currently using redis as the broker (in gonna switch to RabbitMQ soon) and the result backend and the task itself saves some data to a Solr instance but all of those connections should still be done in milliseconds!
Id like to be able to figure out what actually delays all those tasks. Anybody?
we had the same problem here. We solved the problem by playing aroung with the celery-settings in our django-settings. After decreasing concurrency (to 2) and resetting it to the original state (10) everything worked fine. Maybe give it a try.