Web security fuzzing with gobuster

The purpose of fuzzing

In penetration testing for web targets, fuzzing refers to a kind of brute force scan that attempts to find accidentally exposed or potentially dangerous files or directories. These can range from improperly secured configuration files with credentials, to entire backups or remnants of previous infections / known backdoors.

On a technical level fuzzing is extremely simple, consisting only of a lightweight http client trying the paths of a wordlist in order against a target base url. This also means that the chance of success is directly tied to the quality / viability of the chosen wordlist. On outdated wordlist may miss many newer security issues, while a wordlist tailored to the apache2 web server would be inappropriate against an nginx reverse proxy setup. Picking the right wordlist is the most important part of fuzzing.

Disclaimer

Even though fuzzing is not technically an exploit itself, it does send a large amount of traffic against a target, effectively mirroring the behavior of a DDOS attack. Make sure you only scan targets you have prior explicit permission to test, and validate that the target system does have spare power / bandwidth capacity to handle the increased traffic spike. Consider running scans outside of office hours or during low-traffic periods to prevent affecting functionality for other users.

Setting up gobuster

The gobuster tool is a directory and domain fuzzing software written in go. it is lightweight and modern, with a focus on simplicity, and can be installed on most systems out of the box:

sudo apt install gobuster

Since fuzzing relies on wordlists to be effective, you will also need

Many quality wordlists are publicly available, the most popular ones being:

• https://github.com/fuzzdb-project/fuzzdb - contains a large set of high-quality wordlists specifically tailored to fuzzing

• https://github.com/v0re/dirb/tree/master/wordlists - the builtin wordlists shipped with dirb (an older directory fuzzing tool)

• https://github.com/danielmiessler/SecLists - a general purpose wordlist repository for password cracking, fuzzing and other security-related techniques

Note that these wordlist repositories can be quite large, so cloning them in their entirety may not be a good idea if you value your remaining disk space. Consider downloading only the wordlists you need selectively, and use compression to reduce their filesize when not in use. Since wordlists mostly contain recurring words and text, they compress extraordinarily well with tools like zip or gzip.

The rest of this guide will reference wordlists from the SecLists repository as if they were installed under /usr/local/share/seclists/, you can use those relative paths ot find the relevant files in the main repository.

Finding directories and files

The primary fuzzing modes used in security assessments is directory fuzzing, used to check the existence of paths (either files or folders). The choice of wordlist matters most here and needs to be adjusted to

gobuster dir -u http://example.com \
  -w /usr/share/seclists/Discovery/Web-Content/common.txt

This rather generic test may already find common points of interest, it's output may look similar to this:

===============================================================
Gobuster v3.5
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Url:                    http://example.com
[+] Method:                 GET
[+] Threads:                10
[+] Wordlist:               common.txt
[+] Negative Status codes:  404
[+] User Agent:             gobuster/3.5
[+] Timeout:                10s
===============================================================
2025/04/27 15:10:39 Starting gobuster in directory enumeration mode
===============================================================
/.gitignore          (Status: 200) [Size: 57]
/.hta                (Status: 403) [Size: 288]
/.htaccess           (Status: 403) [Size: 293]
/.htpasswd           (Status: 403) [Size: 293]
/config              (Status: 301) [Size: 307] [--> http://example.com/config/]
/docs                (Status: 301) [Size: 305] [--> http://example.co/docs/]
/external            (Status: 301) [Size: 309] [--> http://example.co/external/]
/favicon.ico         (Status: 200) [Size: 1406]
/index.php           (Status: 302) [Size: 0] [--> login.php]
/php.ini             (Status: 200) [Size: 148]
/phpinfo.php         (Status: 302) [Size: 0] [--> login.php]
/robots.txt          (Status: 200) [Size: 26]
/server-status       (Status: 403) [Size: 297]
Progress: 4746 / 4747 (99.98%)
===============================================================
2025/04/27 15:10:40 Finished
===============================================================

Although primitive, this scan yielded a fair few interesting results. You can see found files/directories (status code 200), paths that resulted in redirects and where they lead, and even prohibited resources like the /server-status path returning http status code 403 (Forbidden), indicating that the resource exists but the client or webserver user lacks privileges to access it.

You can use the -x flag to specify extensions to check for each wordlist entry, for example to look for not only /config but also /config.ini and /config.php:

gobuster dir -u http://example.com \
  -w /usr/share/seclists/Discovery/Web-Content/common.txt \
  -x ini,php

For more targeted scanning, pick a more specialized wordlist and include common file extensions to increase your detection surface. For example, finding common administration logins / panels can be done using

gobuster dir -u http://example.com \
  -w /usr/share/seclists/Discovery/Web-Content/admin-panels.txt \
  -x php,html,htm,txt,jsp,aspx,js,json

Another common target for fuzzing scans are exposed logs, backups or temporary files:

gobuster dir -u http://example.com \
  -w /usr/share/seclists/Discovery/Web-Content/raft-small-files.txt \
  -x bak,old,zip,tar.gz,log,sql,tmp

It is important to think about what kind of files or directories you are looking for and what kind of extensions (if any) you expect them to have. It can be fruitful to run multiple smaller, more target-specific scans than trying to construct a single merged wordlist to scan for everything at once.

Finding vhosts and subdomains

While most fuzzing refers to finding directories or files, gobuster can also use the same technique to enumerate subdomains. You can either do this by scanning for vhosts:

gobuster vhost -u http://example.com \
  -w /usr/share/seclists/Discovery/DNS/subdomains-top1million-110000.txt

which specifically sends requests to the target web server to try and find subdomains, or use DNS fuzzing instead:

gobuster dns -d example.com \
  -w /usr/share/seclists/Discovery/DNS/subdomains-top1million-5000.txt

which sends requests through the DNS system and find available subdomains that way. The DNS approach can be significantly slower, but is a good fallback option if the web server has some kind of rate-limiting or WAF in place that hinders discovery.

Note how dns fuzzing works with domain only, using the -d flag instead of -u and omitting the http:// protocol prefix and trailing path.

DNS fuzzing can be problematic for hosts using wildcard subdomains or catch-all DNS responses, creating large numbers of false positive results. gobuster will often notice this and warn you, but always manually verify with random subdomain names and tools like dig before running DNS fuzzing tests.

Dealing with faulty 404 responses

Some cms systems respond to "not found" pages with an http 200 response code instead of the appropriate 404 response code. These responses can lead to many false positives, since proper filters of good/bad responses becomes impossible. In such situations, gobuster will quickly detect this problem and warn you accordingly:

Error: the server returns a status code that matches the provided options for non existing urls. http://example.com/cb48330d-5df8-494b-a3dd-691936cc888f => 200 (Length: 80117). To continue please exclude the status code or the length

To fix the problem, you can either ignore the status code 200 entirely (which would blind you to all good requests as well), or ignore responses with this exact content length (80117 for the example above). The adjusted query could then look like:

gobuster -u http://example.com \
  -w /usr/share/seclists/Discovery/Web-Content/common.txt \
  --exclude-length 80117

Using size-based exclusion works for most targets, but is not always reliable. If the target page has varying response sizes for error pages (for example because of a widget displaying dynamic content like the names of currently logged-in admins, then fuzzing may not be possible against it.

Tuning scan parameters

In order to better utilize available network and hardware resources, you can tune the scanning behavior to your needs. By default, gobuster will use 10 threads, each sending requests as fast as possible (aka 10 concurrent requests until wordlist is completed). For fast servers that can deal with higher traffic, you can increase the thread count:

gobuster -u http://example.com \
  -w /usr/share/seclists/Discovery/Web-Content/common.txt \
  -t 50 # use 50 concurrent requests

Older systems may need to decrease thread count, some even requiring additional delays between requests to give backend infrastructure like databases more room to recover from resource-heavy requests:

gobuster -u http://example.com \
  -w /usr/share/seclists/Discovery/Web-Content/common.txt \
  -t 1 --delay 2000ms # use only one thread and wait 2s between requests

Primitive waf or bot detection systems may block requests because of the user agent http header value, requiring testers to supply user-agent values during scans, either random ones with --random-agent or if that doesn't work, a known user agent from a real browser with -a:

gobuster -u http://example.com \
  -w /usr/share/seclists/Discovery/Web-Content/common.txt \
  -a 'Mozilla/5.0 (X11; Linux x86_64; rv:136.0) Gecko/20100101 Firefox/136.0'

There are more options available, for example to supply http basic auth information or handle trailing slashes for path values. See gobuster help dir for a full list.