NeurIPS 2024 Past Generative modelsFairness & ethicsMultimodal

Workshop on Responsibly Building the Next Generation of Multimodal Foundational Models

NeurIPS 2024 Workshop RBFM

Submission deadline
Sep 21, 2024, 23:59 UTC
imported from OpenReview — check the website for extensions
Submission portal
OpenReview
Notes
Auto-imported from the OpenReview venue record on 2026-06-10 — please verify and enrich (topics are keyword-guessed).

Accepted papers (34)

Fetched from OpenReview (v2) on 2026-06-10.

  1. Adversarial Robust Deep Reinforcement Learning is Neither Robust Nor Safe

    Ezgi Korkmaz · PDF
  2. Aligning to What? Limits to RLHF Based Alignment

    Logan Barnhart, Reza Akbarian Bafghi, Maziar Raissi, Stephen Becker · PDF
  3. Attention Shift: Steering AI Away from Unsafe Content

    Shivank Garg, Manyana Tiwari · PDF
  4. BigDocs: An Open and Permissively-Licensed Dataset for Training Multimodal Models on Document and Code Tasks

    Juan A. Rodriguez, Xiangru Jian, Siba Smarak Panigrahi, Tianyu Zhang, Aarash Feizi, Abhay Puri, Akshay Kalkunte Suresh, François Savard, Ahmed Masry, Shravan Nayak, Rabiul Awal, Mahsa Massoud, Amirhossein Abaskohi, Zichao Li, Suyuchen Wang, Pierre-Andre Noel, Mats Leon Richter, Saverio Vadacchino, Shubham Agarwal, Sanket Biswas, Sara Shanian, Ying Zhang, Kurt MacDonald, Sathwik Tejaswi Madhusudhan, Joao Monteiro, Krishnamurthy Dj Dvijotham, Torsten Scholak, Nicolas Chapados, Sepideh Kharaghani, Sean Hughes, M. Özsu, Siva Reddy, Marco Pedersoli, Yoshua Bengio, Christopher Pal, Issam H. Laradji, Spandana Gella, Perouz Taslakian, David Vazquez, Sai Rajeswar · PDF
  5. Building and better understanding vision-language models: insights and future directions

    Hugo Laurençon, Andrés Marafioti, Victor Sanh, Leo Tronchon · PDF
  6. Comparison Visual Instruction Tuning

    Wei Lin, Muhammad Jehanzeb Mirza, Sivan Doveh, Rogerio Feris, Raja Giryes, Sepp Hochreiter, Leonid Karlinsky · PDF
  7. Consistency-diversity-realism Pareto fronts of conditional image generative models

    Pietro Astolfi, Melissa Hall, Jakob Verbeek, Marlene Careil, Oscar Mañas, Matthew J. Muckley, Adriana Romero-Soriano, Michal Drozdzal · PDF
  8. Coordinated Robustness Evaluation Framework for Vision Language Models

    Ashwin Ramesh Babu, Sajad Mousavi, Desik Rengarajan, Vineet Gundecha, Sahand Ghorbanpour, Avisek Naug, Antonio Guillen, Ricardo Luna Gutierrez, Soumyendu Sarkar · PDF
  9. CrossCheckGPT: Universal Hallucination Ranking for Multimodal Foundation Models

    Guangzhi Sun, Potsawee Manakul, Adian Liusie, Kunat Pipatanakul, Chao Zhang, Phil Woodland, Mark Gales · PDF
  10. Decompose, Recompose, and Conquer: Multi-modal LLMs are Vulnerable to Compositional Adversarial Attacks in Multi-Image Queries

    Julius Broomfield, George Ingebretsen, Reihaneh Iranmanesh, Sara Pieri, Ethan Kosak-Hine, Tom Gibbs, Reihaneh Rabbany, Kellin Pelrine · PDF
  11. Exploring Intrinsic Fairness in Stable Diffusion

    Eunji Kim, Siwon Kim, Robin Rombach, Rahim Entezari, Sungroh Yoon · PDF
  12. GUIDE: A Responsible Multimodal Approach for Enhanced Glaucoma Risk Modeling and Patient Trajectory Analysis

    Heman Shakeri, Behnaz Moradijamei · PDF
  13. How to Determine the Preferred Image Distribution of a Black-Box Vision-Language Model?

    Saeid Asgari, Joseph George Lambourne, Alana Mongkhounsavath · PDF
  14. Incorporating Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models

    Ce Zhang, Zifu Wan, Zhehan Kan, Martin Q. Ma, Simon Stepputtis, Deva Ramanan, Russ Salakhutdinov, Louis-Philippe Morency, Katia P. Sycara, Yaqi Xie · PDF
  15. Just rephrase it! Uncertainty estimation in closed-source language models via multiple rephrased queries

    Adam X. Yang, Chen Chen, Konstantinos Pitas · PDF
  16. LEMoN: Label Error Detection using Multimodal Neighbors

    Haoran Zhang, Aparna Balagopalan, Nassim Oufattole, Hyewon Jeong, Yan Wu, Jiacheng Zhu, Marzyeh Ghassemi · PDF
  17. LLAVAGUARD: VLM-based Safeguards for Vision Dataset Curation and Safety Assessment

    Lukas Helff, Felix Friedrich, Manuel Brack, Kristian Kersting, Patrick Schramowski · PDF
  18. MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models

    Mohammad Shahab Sepehri, Zalan Fabian, Maryam Soltanolkotabi, Mahdi Soltanolkotabi · PDF
  19. MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs

    Wenqian Ye, Guangtao Zheng, Yunsheng Ma, Xu Cao, Bolin Lai, James Matthew Rehg, Aidong Zhang · PDF
  20. MMLU-Pro+: Evaluating Higher-Order Reasoning and Shortcut Learning in LLMs

    Saeid Asgari, Aliasghar Khani, Amir Hosein Khasahmadi · PDF
  21. Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Model

    Wenqi Zhang, Zhenglin Cheng, Yuanyu He, Mengna Wang, Yongliang Shen, Zeqi Tan, Guiyang Hou, Mingqian He, Yanna Ma, Weiming Lu, Yueting Zhuang · PDF
  22. Multimodal Situational Safety

    Kaiwen Zhou, Chengzhi Liu, Xuandong Zhao, Anderson Compalas, Xin Eric Wang · PDF
  23. PopAlign: Population-Level Alignment for Fair Text-to-Image Generation

    Shufan Li, Aditya Grover, Harkanwar Singh · PDF
  24. Position Paper: Protocol Learning, Decentralized Frontier Risk and the No-Off Problem

    Alexander Long · PDF
  25. Probabilistic Active Few-Shot Learning in Vision-Language Models

    Anton Baumann, Marcus Klasson, Rui Li, Arno Solin, Martin Trapp · PDF
  26. Rethinking Artistic Copyright Infringements in the Era of Text-to-Image Generative Models

    Mazda Moayeri, Samyadeep Basu, Sriram Balasubramanian, Priyatham Kattakinda, Atoosa Chegini, Robert Brauneis, Soheil Feizi · PDF
  27. Seeing Through Their Eyes: Evaluating Visual Perspective Taking in Vision Language Models

    Gracjan Góral, Alicja Ziarko, Michal Nauman, Maciej Wolczyk · PDF
  28. Skipping Computations in Multimodal LLMs

    Mustafa Shukor, Matthieu Cord · PDF
  29. The Multi-faceted Monosemanticity in Multimodal Representations

    Hanqi Yan, Yulan He, Yifei Wang · PDF
  30. Towards Secure and Private AI: A Framework for Decentralized Inference

    Hongyang Zhang, Yue Zhao, Chao Yang, Ahmad Farhan, Fielding Johnston · PDF
  31. Trust but Verify: Reliable VLM evaluation in-the-wild with program synthesis

    Viraj Uday Prabhu, Senthil Purushwalkam, Jieyu Zhang, An Yan, Caiming Xiong, Ran Xu · PDF
  32. When Do Universal Image Jailbreaks Transfer Between Vision-Language Models?

    Rylan Schaeffer, Dan Valentine, Luke Bailey, James Chua, Cristobal Eyzaguirre, Zane Durante, Joe Benton, Brando Miranda, Henry Sleight, Tony Tong Wang, John Hughes, Rajashree Agrawal, Mrinank Sharma, Scott Emmons, Sanmi Koyejo, Ethan Perez · PDF
  33. WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models

    Pavan Kalyan Tankala, Piyush Singh Pasi, Sahil Dharod, Azeem Motiwala, Preethi Jyothi, Aditi Chaudhary, Krishna Srinivasan · PDF
  34. You Never Know: Quantization Induces Inconsistent Biases in Vision-Language Foundation Models

    Eric Slyman, Anirudh Kanneganti, Sanghyun Hong, Stefan Lee · PDF