robots.txt for a CodeIgniter based website?

Question

I am working on a php site that uses Code Igniter framework and has the following folder structure on the server.

public_html

• application
• public
• sql
• style
• system
• design
• images
• js
• plugin

.htaccess

robots.txt

sitemap.xml

index.php

The bulleted list of items are subfolders under public_html.

The application folder contains the code consisting of views and controllers.

My question is,

should googlebot be able to crawl folders other than application?

OR

should I Disallow all folders other than application so that no search engine bot gets to crawl other folders?

Solution 1

Your images folder might be useful if you want images to appear in google image search.

Everything else should probably be disallowed.

Solution 2

No search engine spider would even know about the application folder and thus not be able to crawl it anyway. They would only see what URLs you provide from the home page.

Actually IIRC the .htaccess file CI supplies should only allow access to specific folders like images so the application folder should never be accessible. So you don't really need to block anything in robots.txt.

Comments

[Admin]

One thing I'm confused about this question is this questioner has put his robot.txt in root directory and I in public_html. What's the exact place??