A new research paper by NETSI faculty member Chris Riedl and colleagues has been published in Information Systems Research.

Application/Impact: Information technology has enabled organizations to source ideas for new products and services in an open fashion through idea competitions, crowdsourcing communities, or the lead user method. These approaches frequently lead to a magnitude of diverse ideas from which organizations need to identify the most promising ones. In this paper, we compare two leading mechanisms for open idea evaluation: a multi-dimensional rating scale and a preference market. In an experiment with 120 participants, we find that the rating scale is easier to use than the preference market and that this difference in usability enables users to evaluate the quality of the ideas more accurately. This effect of usability becomes stronger when users perceive the evaluation task as being highly variability and when users evaluate ideas that are more difficult to read. Overall, we find that rating scale users evaluated the ideas about 15% more accurately than preference market users. Our results indicate that organizations should use a multi-dimensional rating scale for open idea evaluation and that the usability of those IT-based system is critical given the difficult nature of the idea evaluation task.