CGE: Visualization of a crawler and hidden connections between subdomains

https://github.com/a11mut3d/CGE

Problems CGE solves

Modern web applications are no longer monolithic, with all functionality contained in a single HTML file — they consist of a host of microservices, APIs, and endpoints overall. Mapping all requests is fairly complex, so you can’t see the full picture of the application surface.

CGE helps solve this problem. It:

— collects all subdomains from SSL certificates (just like crt.sh); — crawls every endpoint, and parses HTML, JS, forms, uploads, and files; — tracks where requests are sent in real time via form interactions; — builds a real-time endpoint interaction graph.

What it looks like

Currently, CGE has 2 usage modes: a web UI and CLI.

There is not much to elaborate on for the CLI (it simply outputs all found endpoints to the console, or saves them to a file on request), so let’s dive deeper into the web UI.

I tried to style the web interface in the vein of Obsidian (spoiler: it didn’t turn out very well).

— Each node represents a host (subdomain). — An edge between nodes represents an HTTP information exchange. — Clicking a node shows a list of all its endpoints (even those detected in requests from other hosts). — Clicking an edge shows all actual requests exchanged between the hosts.

Technical details

I decided to implement it in Python using BS, requests, and DNS. I chose to use Neo4j as the database.

Why I considered this a solid solution for red teams

— Discovery of undocumented APIs. — Building an escalation chain: if you compromise one host, you can see who it communicates with. — Saves time, as you don’t have to deploy the crawler to one endpoint first, then to another. — Preserves scan results — meaning you don’t need to store files on your PC.

Conclusion

As I mentioned earlier, I don’t expect my crawler to shake up the industry, but I’ll be very glad if it helps anyone.