Upgrading software, how do you test it?

It is a general question that applies to all changes in software.

Currently I'm upgrading from 3.7 to 3.8 I have:

• created a testvb copy
• put a recent copy of the database for use there
• the templates needed minor tweaks
• it appears to be working
• invited a few select individuals to test it, but because it is a test copy of the database they quickly got bored with it and did little testing.

Do you have a systematic method of checking all or at least most of the features?
Do you invite a larger group to test software?

I realize that 3.7 to 3.8 is minor, but this would apply to 3.8 to 4.0, or even changing to another forum software such as IPB.

 

I've always done minimal testing of the vb software itself. It's when I do major changes to mods that I've written that I worry about the testing. Basically though, I will update all mods I've downloaded and if there isn't an updated version of a mod, then I do test that particular area of the site to make sure the 'old' mod is really compatible with the new version of vbulletin.

I do, however, always make a database backup as soon as I shut down my site for the upgrade. I copy all files and the database. That way, I know if there are major problems, I can go back to exactly how it was before I did the upgrade.

 

I run a test site (fully licensed on its own, so it acts as a backup site as well).

I always install the new vb version on their first and test/upgrade all the mods as required (including all the mods i release on vb.org).

It will be moving to vb4 as soon as I can.

 

You brought up some good points.

One thing though. It is my understanding that there is no going back to the old version of vB.

If a forum is upgraded then later a problem is found such as compatibility issues with templates or add on products. Then going back to the old version of software will only work with the last saved database under the old vB, all posts under the new version of vB will be lost.

I'm not worried about vB itself, but rather compatibility with other things.

 

Thats not true at all.

In most cases you could re-upload the old files and vb will continue to run, since about 90%+ of database changes only add fields/tables, which will simply have no effect (be ignored) by the older version.

Occasionally, changes will be made that would cause errors in an older version.

 

You are right, there is no going back for a major database upgrade which is why I make sure my backup is made of the database before I do an upgrade. You are probably fine with updates within versions (like within 3.7 or 3.8), but you may have database problems if you go backwards a version (like from 3.8 to 3.7).

I don't know if it was clear, but I always do the upgrade on my test site first. I have too many custom mods I need to make sure work before I would ever upgrade the main site. I do test my tracker mod a whole lot prior to going live with any modified version of it. If that mod doesn't work, then we've got major problems.

 

I do very little testing. I just backup the database, then do what I want. If something breaks, I disable the modification or remove it.

In 2 years, I have only had 1 or 2 problems when upgrading.

 

I upgrade on my own forum, I don't really do a test site...but as said... I should.

 

Well ok, it might be a problem going back to the older version of software. For that sake of this discussion let's assume it makes a difference and it will be a major software upgrade.

It just occurred to me that you are both key people at vb.org :doh:


There must be some development & testing guidelines for people that develop plugins, products, and templates from vB.org along with some generic test databases (3.7.3 small, 3.7.3 large, 3.8.3 small, etc).

Maybe something like:

There has to be some kind of testing procedure but I cannot find it at vb.org, could someone post a link?

 

What you are looking for is commonly referred to as Unit Testing. Instead of creating programmatic tests for the software, you would create operational tests. Basically you would make a list of tasks that you want to test in something like a spreadsheet and then manually go through and test each one.

Forum Home - Display (in all browser), Linking, Permissions, Status Icons
Forum Display - Display, Inline Moderation, Linking, Permissions, Status Icons, AJAX Functionality
Showthread - Display, Inline Moderation, Linking, Permissions, Status Icons, BBCode, AJAX
Etc..

You would then go through each page and check to see if it works properly. If you have specific sections turned off on your forums, you don't need to test them. You would keep notes of anything that doesn't work properly.

 

That's it! And because the vB software works so well there must be some standardized testing like that occurring on some generic databases.

Where are those tests documents & databases located?
I realize every application is a little different and further testing needed, but some basic testing is a good starting point.



Example of generic data base
Users with password "password"

• Jane
• Dick
• Joe
• Lucy
• SuperAdmin
• Admin
• SuperMod
• Mod
• Contributing member1
• Contributing mamner2

Forums:

1. main
2. moderator
3. admin
4. contributing member

With the various members having various privileges such as PM box sizes, social groups enabled, contributing member see contributing members forum but not regular members...

 

I have a heavily-tested(?) test board that I use for a lot of things, including these types of things. If anything happens to it, I'm glad it happened there and not the live forum. When everything seems okay, I can proceed to bring the change live. Like Lynne, there will always be a backup of the live forum right before I do so, just as an extra precaution. I don't usually run into any non quick-fix problems. At least I haven't, yet.

 

Well for our internal purposes, we have a private development server where tests are being made on vBulletin 4. Its the same installation that was created in the vBulletin 3 pre-release period. Its not open to the public because its also an active development site and isn't guaranteed to be accessible. Its a real mess right now during the conversion of templates to the vBulletin 4 style.

I don't know of any testing procedures that have ever been released to the public. I do know that some have been written by the Internet Brands Quality Assurance team and are being used for vBulletin 4 testing. I don't have a copy of them myself though.

 

Thanks Wayne!

Internal company sites certainly should not be made public, and it is good that software QA is being done on the code, and it helps to know the testing procedures are not made public.


But I really don't know how the software can be tested without some guidelines. Maybe people are coding things up then testing the parts they think they should test, on whatever database they have, and calling that 'good enough' until someone else finds the shortcomings.

I just took a peek at vb.org in the 3.8 add-ons. It appears some of those have some really long discussions with fewer installs, and some have many installs with much shorter discussions, so it may be possible some of those are tested better than others. If that is true then Paul M does some fantastic work :thumbup:

Really wish I knew more about these types of things.