base Module

tinyCrawl.base.base module

Created on 2021/1/7 13:36

@author: Irvinfaith

@email: Irvinfaith@hotmail.com

class tinyCrawl.base.base.BaseCrawl(iter_url, iter_num_range, thread_num)

基类:object

crawl(*args, **kwargs)

Crawling task function, need to be implemented.

main()

Main function to excute crawl and sink function.

run(obj)

Adding a crawling function into the class

参数

obj – An crawling function object

sink()

Output the crawling result, need to be implemented.

tinyCrawl.base.base_container module

Created on 2021/1/10 14:44

@author: Irvinfaith

@email: Irvinfaith@hotmail.com

class tinyCrawl.base.base_container.Container(*args, **kwargs)

基类:list

classmethod clear()

Clear the data stored in the container.

container = {}
class tinyCrawl.base.base_container.RowContainer(*args, **kwargs)

基类:tinyCrawl.base.base_container.Container