What's the importance of robots.txt? How does it work?