How to build a web crawler based on Scrapy to run forever?

I want to build a web crawler based on Scrapy to grab news pictures from several news portal website. I want to this crawler to be:

  1. Run forever

    Means it will periodical re-visit some portal pages to get updates.

  2. Schedule priorities.

    Give different priorities to different type of URLs.

  3. Multi thread fetch

I've read the Scrapy document but have not found something related to what I listed (maybe I am not careful enough). Is there anyone here know how to do that ? or just give some idea/example about it. Thanks!


Scrapy is a framework for the spidering of websites, as such, it is intended to support your criteria but it isn't going to dance for you out of the box; you will probably have to get relatively familiar with the module for some tasks.

  1. Running forever is up to your application that calls Scrapy. You tell the spiders where to go and when to go there.
  2. Giving priorities is the job of Scheduler middleware which you'd have to create and plug into Scrapy. The documentation on this appears spotty and I've not looked at the code - in principle the function is there.
  3. Scrapy is inherently, fundamentally asynchronous which may well be what you are desiring: request B can be satisfied while request A is still outstanding. The underlying connection engine does not prevent you from bona fide multi-threading, but Scrapy doesn't provide threading services.

Scrapy is a library, not an application. There is a non-trivial amount of work (code) that a user of the module needs to make.

Need Your Help

Developing an app for publishing various Facebook page tab apps to a user's page

facebook facebook-graph-api

I want to develop an which allows users to publish various apps to their page as tab app. What is the correct API to be used so that user can select a page and publish with few parameter. No coding

Loop through a grid in Sheet 1, return cell reference with formula in a column on Sheet 2

excel vba excel-vba

I have a very large grid in Sheet 1, there are 365 columns, each one being one day of the year, starting with January 1st.

About UNIX Resources Network

Original, collect and organize Developers related documents, information and materials, contains jQuery, Html, CSS, MySQL, .NET, ASP.NET, SQL, objective-c, iPhone, Ruby on Rails, C, SQL Server, Ruby, Arrays, Regex, ASP.NET MVC, WPF, XML, Ajax, DataBase, and so on.