Content discovery

Everything you can reach by clicking around a site is only the part the site chose to link. Behind that is a second set of pages, admin panels, old endpoints, backup files, API routes, debug pages, that exist and respond, but are linked from nowhere. Content discovery is the craft of finding them, and it is one of the highest-yield things a web tester does.

Why unlinked does not mean unreachable#

A web server answers requests for paths, and it does not care whether a link to that path exists anywhere. If /admin is a real page, then requesting /admin returns it, even though the home page never mentions it. Developers lean on this, "nobody knows the URL, so it's fine", and it is one of the oldest mistakes in the book. This is security through obscurity, and a URL is not a secret: it travels in logs, in browser history, in referer headers, and it can be guessed.

So the plan is to guess, systematically. Take a big list of common path names and ask the server for each one. The ones that come back with a promising response are your discoveries.

Reading responses, not just chasing 200s#

The naive version looks only for 200 OK. The skilled version reads the whole response, because the status code alone is a map:

• 200: the path exists and served something. An obvious find.
• 301 / 302: a redirect. Where to? A redirect from /admin to /login proves /admin exists and is protected, which is still a find.
• 401 / 403: unauthorised or forbidden. The server is saying "this exists, but not for you." That is signal: you have found a protected resource, and now the question is whether the protection actually holds.
• 405: method not allowed. The path exists but wants a different verb (next lesson).
• 404: usually nothing there. But watch for soft 404s: a site that returns 200 with a "not found" page for everything. Then the status lies and you have to compare response sizes or content instead.

The trick that makes discovery reliable is calibration: first request a path you are certain does not exist (/definitely-not-here-9f8a) and note exactly what "nothing" looks like on this server, its status, its length, its wording. Now anything that differs from that baseline is interesting, even if the status code is the same.

Doing it with a tool#

Hand-guessing does not scale past a few paths, so you use a fuzzer. ffuf, feroxbuster, and gobuster all do the same core job: substitute each word from a list into a URL and report the responses.

ffuf -u https://example.com/FUZZ -w /path/to/wordlist.txt \
     -mc 200,301,302,401,403 -ac

FUZZ is where each word goes, -w is the wordlist, -mc matches the status codes you care about, and -ac auto-calibrates the baseline so soft-404 sites do not drown you. Point it at directories first, then, once you find a directory like /api/, fuzz inside it (/api/FUZZ), discovery is recursive, just like the recon loop.

Wordlists are the whole game#

A fuzzer is only as good as its list. The widely used collection is SecLists, which ships lists for exactly this: common web paths, backup file names, API route names, and technology-specific lists. Match the list to the target: use a WordPress list on a WordPress site, an API list on an API. Add extensions the stack uses (-e .php,.bak,.zip,.old) so you catch config.php.bak and database.zip. The most spectacular finds are often not clever paths at all, they are /backup.zip, /db.sql, /config.old, files someone left in the web root.

Discovering parameters too#

Content discovery is not only about paths. A page may accept a URL parameter it never advertises, a hidden debug=1, an admin=true, a file= that reads from disk. Tools like arjun and ffuf (fuzzing the parameter name instead of the path) hunt for these. A page that behaves differently when you add ?debug=true has just told you something.

Be careful, and be in scope#

Content discovery is unmistakably active: you are sending thousands of requests to someone else's server. That has three consequences you must respect.

• Scope. Only ever run a fuzzer against a target you are authorised to test. This is the lesson's hard line. On this Academy's labs you are always fine; on the internet you are not, until a program or client says so.
• Load. Thousands of fast requests can slow or even knock over a fragile server. Throttle with rate limits (ffuf -rate 20) on anything real, and never point a fuzzer at production without permission and care.
• Noise. You will light up every log and alert. On a real engagement that is expected and coordinated; on a bug bounty, read the program's rules about automated scanning first.

Where this goes next#

You have now found hosts, read what they volunteer, named their stacks, and uncovered pages nobody linked. The last recon lesson looks at a different axis entirely: not which URLs exist, but which HTTP methods they answer to, because a path that is boring for GET can be a serious finding for PUT or OPTIONS.