Get a list of URLs from a site
I didn’t mean to answer my own question but I just thought about running a sitemap generator. First one I found http://www.xml-sitemaps.com has a nice text output. Perfect for my needs.
web-crawler
Syntax error, insert “… VariableDeclaratorId” to complete FormalParameterList
I am facing some issues with this code: I am getting the following error: “Syntax error, insert “… VariableDeclaratorId” to complete FormalParameterList” on config.setCrawlStrorageFolder(crawlStorageFolder)
How do I make a simple crawler in PHP?
Meh. Don’t parse HTML with regexes. Here’s a DOM version inspired by Tatu’s: Edit: I fixed some bugs from Tatu’s version (works with relative URLs now). Edit: I added a new bit of functionality that prevents it from following the same URL twice. Edit: echoing output to STDOUT now so you can redirect it to whatever file you want … Read more
Difference between BeautifulSoup and Scrapy crawler?
Scrapy is a Web-spider or web scraper framework, You give Scrapy a root URL to start crawling, then you can specify constraints on how many (number of) URLs you want to crawl and fetch,etc. It is a complete framework for web-scraping or crawling. While BeautifulSoup is a parsing library which also does a pretty good job of fetching contents from URL … Read more
Python: maximum recursion depth exceeded while calling a Python object
this turns the recursion in to a loop:
TypeError: can’t use a string pattern on a bytes-like object in re.findall()
You want to convert html (a byte-like object) into a string using .decode, e.g. html = response.read().decode(‘utf-8’). See Convert bytes to a Python String
Python: maximum recursion depth exceeded while calling a Python object
this turns the recursion in to a loop:
Python: maximum recursion depth exceeded while calling a Python object
this turns the recursion in to a loop: