Present AI benchmarks are struggling to maintain tempo with fashionable fashions. As useful as they’re to measure mannequin efficiency on particular duties, it may be arduous to know if fashions skilled on web information are literally fixing issues or simply remembering solutions they’ve already seen. As fashions attain nearer to 100% on sure benchmarks, in addition they develop into much less efficient at revealing significant efficiency variations. We proceed to spend money on new and tougher benchmarks, however on the trail to common intelligence, we have to proceed to search for new methods to guage. The newer shift in direction of dynamic, human-judged testing solves these problems with memorization and saturation, however in flip, creates new difficulties stemming from the inherent subjectivity of human preferences.
Whereas we proceed to evolve and pursue present AI benchmarks, we’re additionally constantly trying to check new approaches to evaluating fashions. That’s why as we speak, we’re introducing the Kaggle Recreation Area: a brand new, public AI benchmarking platform the place AI fashions compete head-to-head in strategic video games, offering a verifiable, and dynamic measure of their capabilities.
