A few other french SEOs and I have been talking for quite a long time about creating an SEO crawler together. Each of us was working on his own project, doing almost the same thing over and over, without the others.
We wanted to build a community-based SEO crawler, and here it is: Crowl, the opensource SEO crawler!
Crowl is developped on Python and based on Scrapy. It uses MySQL for data storage.
Python is one of the easiest and most used languages, and used by quite a lot of SEOs, while Scrapy is a well-known and easy-to-use scrapping library, but demands quite a lot of tuning to be adapted to SEO needs. That’s one of the issues we want to tackle with Crowl, which we want to be a simple, robust, easy-to-use and innovative tool for any SEO professional.
If you want to try it, you should be able to get it running on any operating system, as long as you have Python3 and MySQL installed. If it doesn’t work, please get in touch with us!
As is, Crowl is a very basic crawler: it scraps pages, gets some information, can store links and content, but that’s about all for now.
However, we’ve already got a lot of ideas: more data to collect, more output options, post-crawl treatments, user interface, …
Feel free to submit your own ideas and contribute! Any help will be appreciated.