People use DA to find out which Sites are reliable in their area and to determine in which to acquire great links. txt file is then parsed and may instruct the robot as to which web pages aren't to generally be crawled. Like a online search engine crawler may hold https://www.youtube.com/watch?v=Tmal7B28Td0