Crowd-sourced challenge-based benchmarking is an emerging paradigm for the development and unbiased assessment of tools and algorithms that aim to translate biological and biomedical big data into useful knowledge. By distributing problems to large communities of expert volunteers, complex biological, health care, and life sciences questions can be addressed efficiently and quickly, while incentivizing adoption of new standards. Challenges provide a successful resolution to the “self-assessmen