Now, I am really proud of introducing you my first tool called Parsero. I hope you enjoy it...
IntroductionsOne of the things you need to do when you are auditing a website is to look at the Robots.txt file, for example: http://www.behindthefirewalls.com/robots.txt. The web administrators write this file to tell the crawlers like Google, Bing, Yahoo... what content they are allowed to index or what directories mustn't be indexed.
- 200 OK The request has succeeded.
- 403 Forbidden The server understood the request, but is refusing to fulfill it.
- 404 Not Found The server hasn't found anything matching the Request-URI.
- 302 Found The requested resource resides temporarily under a different URI
InstallationParsero needs at least Python3 and can be executed in all Operating Systems which support this language development. Also it needs Urllib3.
sudo apt-get install python3 sudo apt-get install python3-pip sudo pip-3.3 install urllib3When you have installed these software, just download the project from:
In Linux you can use the command bellow.
git clone https://github.com/behindthefirewalls/Parsero.git
When you download Parsero, you will see a folder with three files.
You can do this task automatically using Parsero with the command:
python parsero.py -u www.example.com
Example 2In the picture below you can see another robots.txt. The picture has been cut because this server has a lot of Disallow. Can you imagine checking all of them manually?
... and discover for example, the portal login for this site.