We indeed cover only a small part of a ‘real’ A/B test and guide users with predefined task. What we measured in this test was the task performance of users on the ‘goal’ task of this page: download Firefox. Comparing this performance can help you to optimize a page, if you don’t have the ability to do real A/B tests on a live website. Quick and dirty.

We could possibly use different types of tasks to compare pages in a “A/B” test to cover other aspects of the AIDA model. Like: “What draws your attention on this page?”, “Which elements are interesting?” etc.

This is fundamentally different to what most people would consider to be “real” A/B testing (e.g. Google Website Optimizer). Why? In real A/B testing (for want of a better term), the users’ goals are not predefined. They are real users, doing their own thing on your site. They need to be enticed to engage in an action. Your test completely skips this important aspect. Consider the old “AIDA” conceptual model (Attention / Interest / Decision / Action). Your test only looks at very last step – Action.

The test you describe above looks useful, your system looks very neat, but this isn’t an alternative to real A/B testing. It’s a different type of user research entirely.