adamchainz/django-crawl

django-crawl is an in-process crawler built on top of Django’s test client, so it can traverse sites without running a separate HTTP crawler. It focuses on testing-style request handling to fetch pages and follow links within the Django environment.