From May of 2007 until March 2008 we have been engaged in evaluating 3 course management systems: Angel 7, the rSmart Sakai distro, and our local Blackboard Vista 4.x
Our evaluation included Instructors using them in real course settings (distance and hybrid), including one f2f MIS course where identical tasks were assigned in each system and the Management of Information Students were invited to provide ease of use feedback, also including one f2f Biology course where different sections of the same class were conducted using one of the three CMSs. Our pool of Instructors participating was small, so we also wanted more objective data which we gleaned through the use of Morae by TechSmith .
In a controlled environment we recorded Instructors’ mouse clicks (StarBucks cards were offered as small thank yous for their time), and time on task, while video and audio captured their remarks during the tasks. Again, we had a smaller pool of participants than we would’ve liked, but despite that, we were able to gain more objective data than we would’ve otherwise.
Hopefully we’ll get a chance to share our methodology at an upcoming Educause.
In the meantime, if this mug is familiar to you, you may recognize the Regional Sales Manager of the company representing ‘the winner by a nose’:
You may be surprised to learn the usability data confirmed that among our candidates there were usability trade-offs all around. There was no ‘silver bullet’ out there.
Maybe the next generation.
We’re counting on it.