Generate robots.txt file
Fill in the required parameters
Click the generate/execute button
View and copy your results
Generate standard robots.txt file to control crawler access.