iogf/sukhoi: Minimalist and powerful Web Crawler.


22 bookmarks. First posted by cothrun july 2017.


Minimalist and powerful Web Crawler.
python  spider 
july 2017 by Leonardo.Z
Sukhoi is built on top of the concept of miners, it is similar to what happens with scrapy and its spiders. The basic example below is equivalent to scrapy's main example although it not only scrapes the author's name
but its complete description that stays a layer down from the quotes's pages. Miners inherit from python list class, so they can be used to accumulate data from the pages, they can be placed anywhere too(in this way
it is highly flexible to construct json structures for your fetched data.) Miners can receive pool objects that are used to accurately construct the desired data structure. Note: If sukhoi was useful to you and you feel likely supporting it, please, consider opening
an issue about a donation :)
july 2017 by sechilds
sukhoi - Minimalist and powerful Web Crawler.
python  scraping 
july 2017 by geetarista
Sukhoi

Minimalist and powerful Web Crawler.

Sukhoi is built on top of the concept of miners, it is similar to what happens with scrapy and its spiders. However, in sukhoi the miners can be placed in structures like lists or dictionaries in order to construct json-like structures for the data thats extracted from the pages.
python  web 
july 2017 by cothrun