Parsero v0.75 has been included in the Kali Linux repository

Some days ago a friend told me, "Ey! Why you didn't write a post talking about how Parsero has been included in the Kali Linux repository?" "Seriously? I forgot it..." So here it is...

As you already know, Kali Linux is one of the most advanced and versatile penetration testing distribution ever made. Kali Linux originally started with earlier version of live Linux distribution named BackTrack. It is a GPL-compliant Linux distribution built by penetration testers for penetration tester. With millions of downloads, it has become the most widely adopted penetration testing framework in existence and is used by the security community all over the world.

That is the reason why I am really proud of announcing that my tool Parsero has been included in the Kali Linux repositories: http://tools.kali.org/information-gathering/parsero

Parsero is a free script written in Python which reads the Robots.txt file of a web server and looks at the Disallow entries. The Disallow entries are the URL path of directories or files hosted on a web server which the administrators don't want to be indexed by crawlers. For example, "Disallow: /portal/login" don't allow to search engines like Google, Bing, Yahoo to index  www.example.com/portal/login  so nobody can locate it by searching on them.

Sometimes these paths typed in the Disallows entries are directly accessible by the users (without using a search engine) just visiting the URL and the Path. Sometimes they are not available to be visited by anybody... Because it is really common that the administrators write a lot of Disallows and some of them are available and some of them are not, you can use Parsero in order to check the HTTP status code of each Disallow entry in order to check automatically if these directories are available or not.

Also, the fact that the administrator write a Robots.txt doesn't mean that the files or directories typed in this file will not be indexed by Bing, Google, Yahoo... For this reason, Parsero is capable of performing searches in Bing to locate content indexed without the web administrator authorization.

Now, you can run Parsero v0.75 directly from this awesome distribution. So, what do you need to use Parsero in Kali Linux?

Installing Parsero in Kali Linux

First of all, you need to execute:

root@kali:~# apt-get update

Then, you can search directly Parsero in the Kali Linux repositories by using the command bellow:

root@kali:~# apt-cache search parsero

Finally run the following command to install it.

root@kali:~# apt-get install parsero

Now you can have fun by checking the directories or files which could have sensitive information and should be "anonymous" to the search engines...

Currently, I'm working on developing the new release which will have another feature. It will be available here: https://github.com/behindthefirewalls/Parsero

Read More

Parsero 0.75 is out!!!!

At the beginning of this month, Parsero v0.71 was presented by ToolsWatch Hacker Arsenal in their blog. That is something that I really appreciate...

Today, I would like to introduce Parsero v0.75. Before writing about that, let me make a brief summary.

As has been written in OWASP Testing Guide v4: Testing: Review Webserver Metafiles for Information Leakage (OTG-INFO-003), robots.txt file could be used "for information leakage of the web application's directory or folder path(s)".

In order to get sensitive information thanks this file, I've developed Parsero which is able to perform this task automatically.

What is new?

Some problems have been fixed in the current version which have three new features that I would like to talk about.

• In the last version, Parsero was able to detect if the content in the Disallow entries had been indexed by Bing by doing searches in this crawler. Now, we are able to check if these links indexed are actually available or not. Notice that Parsero only checks the links of the first Bing results page. It means the first 10 results are analyzed.

• Now Parsero is able to detect if there are Disallows entries repeated in the robots.txt file in order to check each one once to save time. The picture bellow shows you a robots.txt file with the same links repeated.

And how Parsero is able to detect it and check each Dissallow entrie only once.

• In the last version, Parsero downloaded the robots.txt to the machine in order to parse it. Now, Parsero performs this task by doing the same task on the fly.

You can download Parsero here: https://github.com/behindthefirewalls/Parsero

More info here: http://www.behindthefirewalls.com/search/label/Parsero

Read More

Parsero v0.6 is OUT!!!!!

As you already know, Parsero is a free script written in Python which helps you to automatically audit the Robots.txt file of a web server. In just a few seconds, you are able to get a lot of valuable information which is needed  when you are auditing a website.

This tool is available for download here:

https://github.com/behindthefirewalls/Parsero

And here you can learn what Parsero already did.

http://www.behindthefirewalls.com/2013/12/parsero-tool-to-audit-robotstxt.html

How to install Parsero v0.6

Parsero is really easy to install. You can install it  for example, in Kali Linux. You only need to run the commands below.

apt-get install python3
apt-get install python3-pip
pip-3.2 install urllib3
pip-3.2 install beautifulsoup4
git clone https://github.com/behindthefirewalls/Parsero.git

What's new?

If you look at the Parsero help, you will see two new features:

• "-o" :   To only show the available Disallow entries.
• "-sb" :  To search in Bing indexed Dissallows.

Showing only the available Disallows

In the picture below you will see the difference between using the "-o" option and not using it.

If the robots.txt file has a few entries, I recommend you don't use the "-o" option because you will be able to figure out what type of content the administrator wanted to hide looking if you get all the results. But if the file is bigger, you have a lot of information to analyze and it is easer perform the audit getting only the links which are allowed to be visited.

Searching the Disallows entries in Bing

The fact that the administrator wrote a robots.txt to try to hide the crawlers part of his content doesn't mean that the search engines don't index these Disallow entries.

For example, in the picture below, Parsero will find content indexed by Bing which it mustn't have been indexed. Parsero will show you the first 10 Bing results for the indexed Disallows.

By doing CTRL+ click on the links, your browser will be redirected to:

• White links: the search page in Bing.
• Green links: directly to the result found in Bing (the content is not always available and sometimes you will get a 404 HTTP code error).

Read More

Parsero: The tool to audit the Robots.txt automatically

When I was writing Using robots.txt to locate your targets, I felt the necessity of developing a tool to make automatic the task of auditing the Robots.txt file of the web servers.

Now, I am really proud of introducing you my first tool called Parsero. I hope you enjoy it...

Introductions

One of the things you need to do when you are auditing a website is to look at the Robots.txt file, for example: http://www.behindthefirewalls.com/robots.txt. The web administrators write this file to tell the crawlers like Google, Bing, Yahoo... what content they are allowed to index or what directories mustn't be indexed.

But... Why the administrators want to hide some web directories to the crawlers?

Sometimes they want to hide the web portal login, management directories, private info, sensitive data, page with vulnerabilities, documents, etc... If they hide these directories from the crawlers, then they can't be found making Google Hacking or just searching in the search engines...

Why do you need Parsero?

We've said that the administrators tell the crawlers what directories or files hosted on the web server are not allowed to be indexed. They achieve this purpose by writing so much "Disallow: /URL_Path" as they want in the Robots.txt file pointing to these directories. Sometimes these paths typed in the Disallows entries  are directly accessible by the users (without using a search engine) just visiting the URL and the Path even sometimes they are not available to be visited by anybody... Because it is really common that the administrators write a lot of Disallows and some of them are available and some of them are not, you can use Parsero in order to check the HTTP status code of each Disallow entry in order to check automatically if these directories are available or not. 

When we execute Parsero, we can see the HTTP status codes. For example, the codes bellow:

• 200 OK                  The request has succeeded.
• 403 Forbidden     The server understood the request, but is refusing to fulfill it.
• 404 Not Found    The server hasn't found anything matching the Request-URI.
• 302 Found             The requested resource resides temporarily under a different URI
• ... 

Installation

Parsero needs at least Python3 and can be executed in all Operating Systems which support this language development. Also it needs Urllib3.

sudo apt-get install python3
sudo apt-get install python3-pip
sudo pip-3.3 install urllib3

When you have installed these software, just download the project from:

https://github.com/behindthefirewalls/Parsero

https://github.com/behindthefirewalls/Parsero/archive/master.zip

In Linux you can use the command bellow.

git clone https://github.com/behindthefirewalls/Parsero.git

When you download Parsero, you will see a folder with three files.

Before start, you need to check that your default Python version is 3 or later. If you have already installed Python3 but is not your default version,  you can run the script using the command "python3 parsero.py" instead of "python parsero.py".

 

If you don't type any argument, you will see the help bellow.

Example 1

In the picture below you can see the Robots.txt file of a web server in one of my environments. If you are a web security auditor, you should check all the Disallows in order to try to get some valuable information. The security auditor should want to know what directories or files are hosted in the web servers which the administrators don't want to be published on the search engines.

You can do this task automatically using Parsero with the command:

python parsero.py -u www.example.com 

Notice in the picture below that the green links are the links which are available in the web server. You don't need to waste your time checking the other links, just clicking on the green links.

If we visit the www.example.com/server-status/ we can see the Apache logs which are public but hidden for the crawlers...

Example 2

In the picture below you can see another robots.txt. The picture has been cut because this server has a lot of Disallow. Can you imagine checking all of them manually?

 

If you use Parsero, you will audit all the Robots.txt file in just a seconds...

... and discover for example, the portal login for this site.

The future of Parsero

I am working on developing new features of this tool which will be delivered in the next months... I would be really grateful if you decide to give me your feedback about this tool.

I want to give the thanks to cor3dump3d for his support and help!!! He has saved me a lot of time thanks to sharing his knwoledge of Python with me!!

Read More