Desire alignment has grow to be a vital part in enhancing the efficiency of Massive Language Fashions (LLMs), but its impression in Multimodal Massive Language Fashions (MLLMs) stays comparatively underexplored. Just like language fashions, MLLMs for picture understanding duties encounter challenges like hallucination. In MLLMs, hallucination can happen not solely by stating incorrect details but in addition by producing responses which might be inconsistent with the picture content material. A major goal of alignment for MLLMs is to encourage these fashions to align responses extra carefully with picture data. Just lately, a number of works have launched choice datasets for MLLMs and examined completely different alignment strategies, together with Direct Desire Optimization (DPO) and Proximal Coverage Optimization (PPO). Nonetheless, as a consequence of variations in datasets, base mannequin sorts, and alignment strategies, it stays unclear which particular components contribute most importantly to the reported enhancements in these works. On this paper, we independently analyze every facet of choice alignment in MLLMs. We begin by categorizing the alignment algorithms into two teams, offline (similar to DPO), and on-line (similar to online-DPO), and present that combining offline and on-line strategies can enhance the efficiency of the mannequin in sure situations. We evaluate quite a lot of revealed multimodal choice datasets and focus on how the small print of their development impression mannequin efficiency. Based mostly on these insights, we introduce a novel manner of making multimodal choice knowledge referred to as Bias-Pushed Hallucination Sampling (BDHS) that wants neither further annotation nor exterior fashions, and present that it may obtain aggressive efficiency to beforehand revealed alignment work for multimodal fashions throughout a spread of benchmarks.
- * Authors contributed equally as first authors.
- †Authors contributed equally.
