All 11 fashions additionally struggled with agentic benchmarks designed to evaluate actual world problem-solving skills round normal data, security, and coding. Claude 3.5 Sonnet and o1 ranked the best on this space, significantly when it got here to extra structured duties with express goals. Still, all fashions had a tough time with software program engineering and different duties requiring open-ended reasoning and planning.
Multimodality is turning into more and more necessary for AI programs, because it permits fashions to course of completely different inputs. To measure this, Vector developed the Multimodal Massive Multitask Understanding (MMMU) benchmark, which evaluates a mannequin’s potential to purpose about photos and textual content throughout each multiple-choice and open-ended codecs. Questions cowl math, finance, music and historical past and are designated as “easy,” “medium,” and “hard.”
In its analysis, Vector discovered that o1 exhibited “superior” multimodal understanding throughout completely different codecs and issue ranges. Claude 3.5 Sonnet additionally did nicely, however not at o1’s stage. Again, right here, researchers discovered that almost all fashions dropped in efficiency when given tougher, open-ended duties.
