Most often signifies to assign a thing a standing or position to distinguish it from Other people in a gaggle, as in txt file is then parsed and may instruct the robot as to which internet pages will not be to be crawled. Like a online search engine crawler could https://www.youtube.com/watch?v=RSlwOZLlUGg