Updating robots.txt when using vBSEO

My robots.txt file blocks some pages, such as /profile.php. But I'm using vBSEO, so that isn't the actual URL to a profile page. Does this mean I need to change my robots.txt to reflect the re-written URLs?

I noticed that many user profiles are being indexed in Google (and two of them were actually added as sitelinks), but in my robots.txt I have /profile.php which apparently isn't effective. Should I change it to /members/ ?

 

I would like to know this too, I keep thinking profiles are on member.php for some reason too!

 

you are already blocking the php files, they redirect to the vbseo url's
so if you block the member.php file, but you have a CRR that makes those pages /member/ since you have the member.php blocked, it will block the /member/ one as well.

Hope that clears things up, let me know if you have anymore questions on this subject

and ps.. I had blocked the profile pages on my vb site, and blocking them on my ipb site. I want the bots access to the best content and that's it. There is no reason (on my site) for the bots to scan the profile pages, or several other pages like the calendar, or FAQ

 

Thanks, Brandon. That makes sense and saves me some work.

What I don't understand is why profile pages are being added as sitelinks, let alone being indexed - even though it's blocked in robots.txt.

 

Can't you uncheck that box in the sitemap settings, I know they have a few options to click on or off.

If you just added the line in robots.txt file, it will take some time for google to remove those pages.

 

Member profiles already are excluded from the sitemap, and no I didn't just make the change to robots.txt. It's been like that since our launch in May.