Skipping web-pages with extension pdf, zip from crawling in Anemone

I am developing crawler using anemone gem (Ruby- 1.8.7 and Rails 3.1.1). How should I skip web-pages with extensions pdf, doc, zip, etc. from crawling/downloading.

Answers


ext = %w(flv swf png jpg gif asx zip rar tar 7z gz jar js css dtd xsd ico raw mp3 mp4 wav wmv ape aac ac3 wma aiff mpg mpeg avi mov ogg mkv mka asx asf mp2 m1v m3u f4v pdf doc xls ppt pps bin exe rss xml)

Anemone.crawl(url) do |anemone|

    anemone.skip_links_like /\.#{ext.join('|')}$/

    ...

end

Need Your Help

using lambda to return a concatenated string

c# lambda

I have a list of user, and on that object i have a property email. Is there a way using lambda in c# to return a single string of each users email in a format of "email1;email2;email3" without any

filled textfield value from select dropdown with split()

javascript php jquery split

when i use split() the textfield value is empty, value for textfield from drop down that i need is e.g 15.

About UNIX Resources Network

Original, collect and organize Developers related documents, information and materials, contains jQuery, Html, CSS, MySQL, .NET, ASP.NET, SQL, objective-c, iPhone, Ruby on Rails, C, SQL Server, Ruby, Arrays, Regex, ASP.NET MVC, WPF, XML, Ajax, DataBase, and so on.