Ten. Gzip Compression: Permit Gzip compression in your server to reduce the size of information transferred between the server and the person’s browser. txt file is then parsed and will instruct the robotic as to which internet pages are usually not to generally be crawled. Being a search engine crawler https://www.youtube.com/watch?v=jfBrROL69Lk