Navigating the Web: Unraveling the Challenges Faced by Web Crawlers