how do i set this up so that my members get emails when a thread they started or replied to has a new reply?
You need to setup your vBulletin mail server to use your server as an outgoing mail server. It should be in your vBulletin manual.
AdminCP -> vBulletin Options -> User Registration Options -> Default Registration Options -> Automatic Thread Subscription Mode: Instant email notification
vBulletin Options -> 'User Registration Options' Scroll down to "Default Registration Options", and in the "Automatic Thread Subscription Mode" dropdown menu, choose "Instant Email Notification". Then, to enable it for people who are already registered, go to "Execute SQL Query" in your admin panel, and from the "Automatic Query" menu, choose "Instant Email Notification", and click "Continue".
In your includes/config.php file you need to edit this part; $config['SpecialUsers']['canrunqueries'] = 'x'; Just replace x with your user ID.
UPDATE user SET autosubscribe = 1 restart? Confirm Query Execution This query may modify data in your database. If this change is done in error, it is possible that you will not be able to recover from this change. Are you sure you wish to continue? or continue?