?????? ?? ??? ????????, ???? ?? ?????????? ????? ??? ?? ????????? ??????????? ??? ??? ???? ??????:Make robots.txt file to authenticate and prevent crawler from crawling anything other than your house web site.?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????