Web Enumeration Using Gobuster

Hello hackers, today i will be talking about web enumeration basics how we can manually enumerate a web application and also by using an automated tool called gobuster. I will talk about how we can install and operate that tool in Kali Linux machine.

Manual Enumeration

When you start your web enumeration the best tool for you to use is your favorite browser. Mine is Firefox so you can use your own like Chrome, Opera, Brave is pretty nice too. Your browser is as extensive as you are. It keeps record of data it receive and who from. We can use this for a range of activities: finding that exact photo or more usefully — the location of certain files or assets being loaded. This could include things from scripts to page URLs.

Using our Browsers Developer Console

Browsers including Chrome and Firefox have a suite of tools located in the “Developer Tools/Console”. We’re going to be discussing Firefox’s, however, Chrome has a very similar suite. This suite includes a range of tools including:

• Viewing page source code
• Finding assets
• Debugging & executing code such as javascript on the client-side (our Browser)

Using “F12” on our keyboard, this is a shortcut to launch this suite of tools.

In above picture you can see page heading hello world and down there you can see a section “inspector” tool is open. This allows us to see html source code of the web page. This often contains things such as developer comments, and the name to certain aspects of web page features including forms and the likes.

Developers often leave behind comments in the form of the <! — →tags…for example: <! — This is a comment → which are not rendered in the browser as we can see here:

Gobuster

Gobuster is written in Go. Go is an open-source, low-level language (much like C or Rust) developed by a team at Google and other contributors. If you’d like to learn more about Go, visit the website linked above. It is one of the best tool out there to enumerate web and by web i meant directories, DNS, Subdomains and a lot more. You can get more info and source code of it here. When you finish this blog you will be able to do followings things using gobuster:

• How to install Gobuster on Kali Linux
• How to use the “dir” mode to enumerate directories and several of its most useful options
• How to use the “dns” mode to enumerate domains/subdomains and several of its most useful option
• Where to go for help

Let’s start with how to install Gobuster on Kali Linux:

You can install gobuster using following command in your terminal.

sudo apt install gobuster

Useful Global Flag

flag        long flag            description
-t          --threads            No. of concurrent threads
-v          --verbose            Verbose output
-z          --no-progress        Don't display progress
-q          --quiet              Don't print the banner and noise
-o          --output             Output file to write results to

I will typically change the number of threads to 64 to increase the speed of my scans. If you don’t change the number of threads, Gobuster can be a little slow.

Gobuster Modes

Gobuster is very powerful and versatile tool. It have different modes to do different things like dir mode is used when you are searching for web directories in a web app and dns mode is used when your are searching subdomains.

“dir” Mode:
we can pretty much guess what this mode does as it’s name implies it is used to brute force directories in web application. It is a very powerful tool it not only scan wesbite but it also returns it’s status code and this will immediately let you know if you as an outside user can request that directory or not. Additional functionality of Gobuster is that it lets you search for files as well with the addition of a simple flag!

Using dir mode:
To use dir mode type following command

gobuster dir

This isn’t the full command but it tells the gobuster that the user wants to search directories in web. Now after this you will have to add url and the wordlist

gobuster dir -u http://10.10.10.10 -w wordlist.txt

Note: The URL is going to be the base path where Gobuster starts looking from. So the URL above is using the root web directory. For example, in a typical Apache installation on Linux, this is /var/www/html. So if you have a “products” directory and you want to enumerate that directory, you’d set the URL as http://10.10.10.10/products. You can also think of this like http://example.com/path/to/folder. Also notice that I specified the protocol of HTTP. This is important and required.

Other useful flags:

Flag     Long Flag        Description
-c          --cookies     Cookies to use for requests
-x          --extensions  File extension(s) to search for
-H          --headers     Specify HTTP headers, -H 'Header1: val1' 
-k          --no-tls-validation   Skip TLS certificate verification
-n          --no-status   Don't print status codes
-P          --password    Password for Basic Auth
-s          --status-codes Positive status codes
-b          --status-codes-blacklist   Negative status codes
-U          --username    Username for Basic Auth

here -x or — extentsions flag is really useful to search file with specific extentsion.

gobuster dir -u http://10.10.10.10 -w wordlist.txt -x html,php,bak

In above command we specified to search for only those file which have .html, .php, .bak extension.

The -k flag is special because it has an important use during penetration tests and captures the flag events. In a capture the flag room on TryHackMe for example, if HTTPS is enabled, you will most likely encounter an invalid cert error like the one below

if your run gobuster in this site without -k flag gobuster will pop out an long heart breaking error haha so in order to bypass these kinds invalid certifications and give you your results use -k falg in your command.

“dns” mode:
The next mode we'll focus on is the "dns" mode. This allows Gobuster to brute-force subdomains. During a penetration test (or capture the flag), it's important to check sub-domains of your target's top domain. sub-domains increase your scope so larger scope have higher possibilities to find a vulnerability.

using “dns” mode:

You can use dns mode by typing following command in your terminal

gobuster dns

as dir mode this command is incomplete this will tell the gobuster that user wants to do sub-domain brute forcing you have to again specify a domain and a wordlist file.

gobuster dns -d mydomain.com -w subdomains.txt 

This command tells gobuster to do a sub-domain bruteforcing in mydomain.com

Other useful falgs:

Flag     Long Flag    Description
-c      --show-cname  Show CNAME Records(can't use with '-i')
-i      --show-ips    Show IP Addresses
-r      --resolverUse custom DNS server                                 

vhost mode:
The last and final mode we’ll focus on is the “vhost” mode. This allows Gobuster to brute-force virtual hosts. Virtual hosts are different websites on the same machine. In some instances, they can appear to look like sub-domains, but don’t be deceived! Virtual Hosts are IP based and are running on the same server. This is not usually apparent to the end-user. On an engagement, it may be worthwhile to just run Gobuster in this mode to see if it comes up with anything. You never know, it might just find something! While participating in rooms on TryHackMe, virtual hosts would be a good way to hide a completely different website if nothing turned up on your main port 80/443 scan.

using vhost mode:

like dir and dns you can simply type the following command and run vhost mode in gobuster:

gobuster vhost

This notifies the gobuster that the user wants to bruteforce virtual hosts on the target url. After this you have to provide an url and a wordlist file.

gobuster vhost -u http://10.10.10.10 -w wordlist.txt

Other useful flags:

You can see it’s flag and other documentation in the official documentation.

Useful Wordlists:
Here i have used the word wordlist frequently and i didn’t specify exactly which wordlists to use to do different jobs like bruteforcing web directories or dns.
So, in this section i will tell you about which wordlists to use when you are doing specific tasks.

Kali Linux default wordlists:
This are the default wordlists which comes pre-installed in kali linux. There are lots of wordlists i will mention which wordlists i use to do directory bruteforcing in kali linux.

• /usr/share/wordlists/dirbuster/directory-list-2.3-*.txt
• /usr/share/wordlists/dirbuster/directory-list-1.0.txt
• /usr/share/wordlists/dirb/big.txt
• /usr/share/wordlists/dirb/common.txt
• /usr/share/wordlists/dirb/small.txt
• /usr/share/wordlists/dirb/extensions_common.txt — Useful for when fuzzing for files!

/usr/share/wordlists have lots of wordlists for different tasks you can go there and check it all yourself.

Non-Standard Lists:
In addition to the above, Daniel Miessler has created an amazing GitHub repo called SecLists. It compiles many different lists used for many different things. The best part is, it’s in apt! you can simply install all of those wordlists in your kali machin using following command:

sudo apt install seclists

in these two wordlists i think you will get all kinds of worldlists to do your bruteforcing job.

With that i think this will be better to end this blog here cause you know too much reading is boring but before i end i would like to say that gobuster is not the only tool you can use to do web enumeration there are lots of other powerful tools too like nikto, wpscan. I will write about those two in separate blogs.
Thank you so much for reading i hope you learned thing or two from here and yeah keep hacking. ❤