Crawly McCrawlface
Websites crawl, extract content, export translations
Crawl entire websites with a single click. Crawly automatically scans sitemaps and links, extracts clean text, and delivers export-ready content – perfect for SEO analysis, content audits, and professional translation projects.
What Crawly can do:
- Automatic page discovery via sitemap and link tracking
- Clean text extraction – navigation, cookie banners and scripts are automatically removed
- SEO metadata at a glance: Meta descriptions, Open Graph, Twitter Cards, hreflang tags
- XLIFF export for professional translation workflows
- Text export as a single file or one file per page
- Detailed crawl log with HTTP status, response times and word count
I work as a freelance translator and regularly need to extract entire website texts for localisation projects. Up until now, this has always been a tedious copy-and-paste marathon.
Full control over every crawl:
- Three speed levels: Polite, Normal and Fast
- Language filter – crawl only pages in certain languages
- Subdomain control and depth limit
- Up to 5,000 pages per crawl
- robots.txt is respected
- HTTP Basic Auth for protected websites
Who is Crawly designed for?
- SEO professionals – Website audits and metadata analysis in the blink of an eye
- Translators & localisation teams – XLIFF export directly from the website
- Content managers – Complete text content of a website at the touch of a button
- Developers – Quick inventory of web projects
The fact that the app runs completely offline and doesn’t send any data to any cloud is a real plus for sensitive customer projects. One small wish: a CSV export would be the cherry on top.
Mac