Sunday, December 08, 2013

When I was writing Using robots.txt to locate your targets, I felt the necessity of developing a tool to make automatic the task of auditing the Robots.txt file of the web servers.

Now, I am really proud of introducing you my first tool called Parsero. I hope you enjoy it...

Introductions

One of the things you need to do when you are auditing a website is to look at the Robots.txt file, for example: http://www.behindthefirewalls.com/robots.txt. The web administrators write this file to tell the crawlers like Google, Bing, Yahoo... what content they are allowed to index or what directories mustn't be indexed.

But... Why the administrators want to hide some web directories to the crawlers?

Sometimes they want to hide the web portal login, management directories, private info, sensitive data, page with vulnerabilities, documents, etc... If they hide these directories from the crawlers, then they can't be found making Google Hacking or just searching in the search engines...

Why do you need Parsero?

We've said that the administrators tell the crawlers what directories or files hosted on the web server are not allowed to be indexed. They achieve this purpose by writing so much "Disallow: /URL_Path" as they want in the Robots.txt file pointing to these directories. Sometimes these paths typed in the Disallows entries  are directly accessible by the users (without using a search engine) just visiting the URL and the Path even sometimes they are not available to be visited by anybody... Because it is really common that the administrators write a lot of Disallows and some of them are available and some of them are not, you can use Parsero in order to check the HTTP status code of each Disallow entry in order to check automatically if these directories are available or not. 

When we execute Parsero, we can see the HTTP status codes. For example, the codes bellow:

  • 200 OK                  The request has succeeded.
  • 403 Forbidden     The server understood the request, but is refusing to fulfill it.
  • 404 Not Found    The server hasn't found anything matching the Request-URI.
  • 302 Found             The requested resource resides temporarily under a different URI
  • ... 

Installation

Parsero needs at least Python3 and can be executed in all Operating Systems which support this language development. Also it needs Urllib3.
sudo apt-get install python3
sudo apt-get install python3-pip
sudo pip-3.3 install urllib3
When you have installed these software, just download the project from:

https://github.com/behindthefirewalls/Parsero

https://github.com/behindthefirewalls/Parsero/archive/master.zip

In Linux you can use the command bellow.

git clone https://github.com/behindthefirewalls/Parsero.git


When you download Parsero, you will see a folder with three files.


Before start, you need to check that your default Python version is 3 or later. If you have already installed Python3 but is not your default version,  you can run the script using the command "python3 parsero.py" instead of "python parsero.py".


If you don't type any argument, you will see the help bellow.


Example 1

In the picture below you can see the Robots.txt file of a web server in one of my environments. If you are a web security auditor, you should check all the Disallows in order to try to get some valuable information. The security auditor should want to know what directories or files are hosted in the web servers which the administrators don't want to be published on the search engines.


You can do this task automatically using Parsero with the command:
python parsero.py -u www.example.com 

Notice in the picture below that the green links are the links which are available in the web server. You don't need to waste your time checking the other links, just clicking on the green links.


If we visit the www.example.com/server-status/ we can see the Apache logs which are public but hidden for the crawlers...

Example 2

In the picture below you can see another robots.txt. The picture has been cut because this server has a lot of Disallow. Can you imagine checking all of them manually?

 
If you use Parsero, you will audit all the Robots.txt file in just a seconds...


... and discover for example, the portal login for this site.

The future of Parsero

I am working on developing new features of this tool which will be delivered in the next months... I would be really grateful if you decide to give me your feedback about this tool.

I want to give the thanks to cor3dump3d for his support and help!!! He has saved me a lot of time thanks to sharing his knwoledge of Python with me!!


Posted on Sunday, December 08, 2013 by Javier Nieto

11 comments

Sunday, December 01, 2013

You already know that the malware developers create packed executables in order to try to thwart the security analyst job and make a lighter file easier to download... If the executable is packed we cannot examine the original program which obstructs us from employing a static analysis to know what the malware does...

In this post we have a malware sample called DEMO.exe which we will work with.

If we open the file with PEiD we can check that it has been compressed with UPX. You already know that it is a common compression...


The majority of times we are able to unpack this type of compression automatically with the UPX program. But this time, we haven't been lucky...


If we want to make a static analysis, first we need to unpack the file. To achieve this purpose, we need to follow the next steps:

  • To unpack the original file into memory.
  • To resolve all of the imports of the original file.
  • To find and transfer execution to the original entry point (OEP).

The key to this technique is to run the packed file until it is decompressed by itself, setting a breakpoint, dumping it and setting the OEP and rebuilding their imports.

To follow these steps we will use OllyDbg v1.10 and the plugin OllyDump.

The first thing we need to do is to load the executable in OllyDbg. We can see that an advertisement appears telling us what we already know: "...reports that its code section is either compressed, encrypted ,or contains large amount of embedded data..."


Ok, let's go. Just loaded the file we can see the program stop at PUSHAD. Press F7 or click on the button to Step into.


Right click at the ESP value and select Follow in Dump...


 ... select the first four characters and set a Breakpoint.


Press the play button and wait until the program stops at the breakpoint. Now we can see the tail jump at 00ACD7BD. The tail jump is the last instruction of the uncompressing action and in this type of compression is usually followed by a lot of 0x00 bytes. These 0x00 bytes are filling to ensure that the section is properly byte-aligned. Notice that function jumps to a site which is very far away, 004090E8. Just in that instruction, the original program will start.


We can set a breakpoint at the tail jump.


We resume the program again and it is stopped at the tail jump. We need to press F7 or click on the Step Into bottom.


Now we are in the OEP. Right-clik on this line and click on Dump debugged process in order to dump the process to a disk.


Notice that the OEP (004090E8) is the same that the last address in the "Modify" field, 90E8 . You can press the "Get EIP as OEP". Untick the "Rebuild import" because the import will be rebuilt using Import REConstructor. It is a good idea to copy the value "90E8" for the future...


Save the dump where you want. I have saved the file as "Dumped.exe". Notice if you want to run the program it will fail because it doesn't have the imports. To fix that, just open Import REConstructor and select the file which is being debugged with OllyDbg. In this case "demo.exe"


The next window will appear.


Just paste the value which was copied before in the OPE section and click on IAT AutoSearch.


Click on the Get Imports first and then click on the Fix Dump buttom.


Just pressed "Fix Dump" select the file which was dumped.


Now, we have two files, the first one which was dumped with OllyDbg and the last one, Dumped_.exe which has the imports rebuilt.


If  you look at the strings in OllyDbg you can see the differences. In the picture below you can see on the left the strings detected in the packed file and on the right, you can see the strings of the unpacked file.



Now, if we open the unpacked file we can detect that is was developed in Visual basic and we can start with the static analysis.


With VB Decompiler we could try to decompiler it...



Posted on Sunday, December 01, 2013 by Javier Nieto

No comments