What the site hands out for free

Pick one web app from your map. Before you probe anything, there is a pile of information the site simply gives to every visitor, including you. Files meant for robots, headers on every response, comments left in the page source. Reading these first is free, fast, and often points straight at the good stuff.

robots.txt: the "please ignore" list#

At /robots.txt many sites publish a file telling search-engine crawlers which paths not to index. It looks like this:

User-agent: *
Disallow: /admin/
Disallow: /backup/
Disallow: /internal/api/

Read that as a tester, not as a crawler. The site is handing you a list of the paths it most wants to keep quiet. robots.txt does not stop anyone from visiting those paths, it only asks well-behaved crawlers not to list them, so "Disallow: /admin/" is closer to a signpost than a lock. It is one of the first things to fetch on any target, and one of the most reliably useful:

curl -s https://example.com/robots.txt

Sitemaps: a free list of real routes#

Right next to it, /sitemap.xml (sometimes referenced from robots.txt) lists URLs the site wants indexed. Where robots.txt shows what they hid, the sitemap shows the real structure of the application in one document: every section, sometimes every article or product. It saves you crawling and gives you clean, real routes to start from.

Response headers: the stack, announced#

Every response carries headers, and many of them describe the machinery. Fetch any page with -i to see them:

curl -sI https://example.com/

Look for:

• Server:: the web server and often its version (nginx/1.18.0, Apache/2.4.41).
• X-Powered-By:: the language or framework (PHP/8.1, Express).
• Set-Cookie:: the cookie name frequently names the framework: PHPSESSID (PHP), laravel_session (Laravel), JSESSIONID (Java), connect.sid (Node/Express), csrftoken + sessionid (Django).
• Security headers, or their absence: a missing Content-Security-Policy, a permissive Access-Control-Allow-Origin: *, or a X-Frame-Options that is not set all tell you something about how carefully the app was built.

You are not exploiting anything yet. You are letting the app introduce itself, and it is remarkably talkative.

HTML source and comments: notes the developers left#

View the page source (not the rendered page, the raw HTML). Developers leave things there:

• Comments like <!-- TODO: remove test login before launch --> or <!-- old endpoint: /api/v1/legacy -->. People write themselves notes and forget to delete them.
• Links to scripts and assets whose paths reveal the platform: /wp-content/ and /wp-includes/ mean WordPress; /sites/default/files/ means Drupal; a /_next/ folder means Next.js.
• Hidden form fields and inline configuration: sometimes an entire config object, an API key meant for the client, or a debug flag, sits in a <script> block.
• Referenced JavaScript files. Open them. Front-end JavaScript is where the list of API endpoints usually lives, every URL the app calls is written in there somewhere, including ones you would never reach by clicking.

Other files worth a quick fetch#

A handful of well-known paths are worth requesting on any target, because when they exist they are gold:

• /.well-known/security.txt: contact details for reporting security issues, and sometimes scope hints.
• /.git/: if a site deployed with its Git repository exposed, you can sometimes reconstruct the entire source code. Even just /.git/config returning content is a serious finding.
• /.env: a misplaced environment file can leak database passwords and API keys. Rare, but catastrophic when present.
• /api/, /api/docs, /swagger.json, /openapi.json: API documentation that lists every endpoint and parameter for you.

Fetch these with curl and read the status code. A 200 on any of the sensitive ones is a finding on its own and a map of the rest.

Keep it honest#

Two of these, .git, .env, exposed backups, can hand you secrets or source code. On an authorised target, note them, prove they are readable, and report them; do not go rummaging further than you need to demonstrate the exposure. Off an authorised target, do not fetch them at all. Reading a public robots.txt is browsing; downloading someone's .env full of live credentials is not, wherever it happens to be sitting.

Where this goes next#

You have read what the site volunteers. Some of it, the Server header, the cookie name, the asset paths, already hints at what technology you are up against. The next lesson makes that a deliberate skill: fingerprinting the stack, so you can name the server, the framework, and ideally the versions, and turn that into known weaknesses.