Technical Search engine optimization: Technological SEO refers to Site and server optimization that helps the crawler in crawling, indexing and ranking operations to rank your website far better. txt file is then parsed and can instruct the robot concerning which webpages are not being crawled. Like a search engine crawler https://www.youtube.com/watch?v=2az-JmMnPV4