Also called: Crawl Engineer, Web Data Engineer
Builds the bots that visit billions of web pages to collect data for search engines, price trackers and AI training, politely and at huge scale.
Your crawlers visit 500 million pages a day, and one website complains you're hitting it too hard. You fix the speed limits and make sure the bot obeys each site's robots.txt rules. Then you debug why pages built with JavaScript come back blank.
Estimated from search and AI company postings.
Approximate: anchored on Bureau of Labor Statistics data for a broader job group.
Bachelor's degree (4-year), about 4 years after high school. Bachelor's in computer science.
Ways in: 4-year college. Training cost: $15k – $60k.
Majors to look at: Computer Science, Software Engineering.
Growing faster than average. AI companies and search engines are hiring crawl teams.
Medium risk for 10 years. AI helps read messy pages, but running huge, polite, legal crawling systems still needs engineers.
Classes: AP Computer Science, Math, Civics (internet law basics).
Official data: O*NET occupation 15-1252.
Career Universe is an early version. Pay, training and AI-risk figures are estimates to start your own research, not verified facts. The research was written with AI research assistants, checked by validation scripts and spot-checked by hand. Check anything important with schools, employers and official sources. How the numbers work.