ICML 2024 Past Large language models

ICML 2024 Workshop on Foundation Models in the Wild

ICML 2024 FM-Wild Workshop

Submission deadline
Jun 8, 2024, 12:29 UTC
imported from OpenReview — check the website for extensions
Submission portal
OpenReview
Notes
Auto-imported from the OpenReview venue record on 2026-06-10 — please verify and enrich (topics are keyword-guessed).

Accepted papers (95)

Fetched from OpenReview (v2) on 2026-06-10.

  1. $\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs

    Vlad Sobal, Mark Ibrahim, Randall Balestriero, Vivien Cabannes, Diane Bouchacourt, Pietro Astolfi, Kyunghyun Cho, Yann LeCun · PDF
  2. A Critical Look At Tokenwise Reward-Guided Text Generation

    Ahmad Rashid, Ruotian Wu, Julia Grosse, Agustinus Kristiadi, Pascal Poupart · PDF
  3. Adapting LLM Agents with Universal Feedback in Communication

    Kuan Wang, Yadong Lu, Michael Santacroce, Yeyun Gong, Chao Zhang, yelong shen · PDF
  4. Adaptive Concept Bottleneck for Foundation Models

    Jihye Choi, Jayaram Raghuram, Yixuan Li, Suman Banerjee, Somesh Jha · PDF
  5. AdaptiveBackdoor: Backdoored Language Model Agents that Detect Human Overseers

    Heng Wang, Ruiqi Zhong, Jiaxin Wen, Jacob Steinhardt · PDF
  6. Adversarially Robust CLIP Models Induce Better (Robust) Perceptual Metrics

    Francesco Croce, Christian Schlarmann, Naman Deep Singh, Matthias Hein · PDF
  7. An Auditing Test to Detect Behavioral Shift in Language Models

    Leo Richter, Nitin Agrawal, Xuanli He, Pasquale Minervini, Matt Kusner · PDF
  8. An Empirical Study into Clustering of Unseen Datasets with Self-Supervised Foundation Models

    Scott C. Lowe, Joakim Bruslund Haurum, Sageev Oore, Thomas B. Moeslund, Graham W. Taylor · PDF
  9. Benchmarking Robust Self-Supervised Learning Across Diverse Downstream Tasks

    Antoni Kowalczuk, Jan Dubiński, Atiyeh Ashari Ghomi, Yi Sui, George Stein, Jiapeng Wu, Jesse C. Cresswell, Franziska Boenisch, Adam Dziedzic · PDF
  10. Bilingual Adaptation of Monolingual Foundation Models

    Gurpreet Gosal, Yishi Xu, Gokulakrishnan Ramakrishnan, Rituraj Joshi, Avraham Sheinin, Zhiming Chen, Biswajit Mishra, Sunil Kumar Sahu, Neha Sengupta, Natalia Vassilieva, Joel Hestness, Samujjwal Ghosh, Bokang Jia, Onkar Arun Pandit, Satheesh Katipomu, Samta Kamboj, Rahul Pal, Parvez Mullah, Soundar Balaji Doraiswamy, Karim Chami, Preslav Nakov · PDF
  11. Black-Box Detection of Language Model Watermarks

    Thibaud Gloaguen, Nikola Jovanović, Robin Staab, Martin Vechev · PDF
  12. BUILD: Buffer-free Incremental Learning with OOD Detection for the Wild

    Srishti Gupta, Daniele Angioni, Lea Schönherr, Ambra Demontis, Battista Biggio · PDF
  13. Calibrated Self-Rewarding Vision Language Models

    Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, Huaxiu Yao · PDF
  14. CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models

    Peng Xia, Ze Chen, Juanxi Tian, Gong Yangrui, Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan, Yiyang Zhou, Kangyu Zhu, Wenhao Zheng, Zhaoyang Wang, Xiao Wang, Xuchao Zhang, Chetan Bansal, Marc Niethammer, Junzhou Huang, Hongtu Zhu, Yun Li, Jimeng Sun, Zongyuan Ge, Gang Li, James Zou, Huaxiu Yao · PDF
  15. CharED: Character-wise Ensemble Decoding for Large Language Models

    Kevin Gu, Eva Tuecke, Dmitriy A Katz, Raya Horesh, David Alvarez-Melis, Mikhail Yurochkin · PDF
  16. Code Agents are State of The Art Software Testers

    Niels Mündler, Mark Niklas Mueller, Jingxuan He, Martin Vechev · PDF
  17. Combining Pre-trained LoRA Modules Improves Few-shot Adaptation of Foundation Models to New Tasks

    Nader Asadi, Mahdi Beitollahi, Yasser H. Khalil, Yinchuan Li, Guojun Zhang, Xi Chen · PDF
  18. ContextCite: Attributing Model Generation to Context

    Benjamin Cohen-Wang, Harshay Shah, Kristian Georgiev, Aleksander Madry · PDF
  19. Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?

    Jonathan Hayase, Alisa Liu, Yejin Choi, Sewoong Oh, Noah A. Smith · PDF
  20. DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning

    Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, Aviral Kumar · PDF
  21. DistilDIRE: A Small, Fast, Cheap and Lightweight Diffusion Synthesized Deepfake Detection

    Yewon Lim, Changyeon Lee, Aerin Kim, Oren Etzioni · PDF
  22. Domain-Aware Fine-Tuning of Foundation Models

    Uğur Ali Kaplan, Yumeng Li, Margret Keuper, Anna Khoreva, Dan Zhang · PDF
  23. Dual Risk Minimization for Robust Fine-tuning of Zero-Shot Models

    Kaican Li, Weiyan Xie, Ricardo Silva, Nevin L. Zhang · PDF
  24. Efficient Evolutionary Search over Chemical Space with Large Language Models

    Haorui Wang, Marta Skreta, Yuanqi Du, Wenhao Gao, Lingkai Kong, Cher Tian Ser, Felix Strieth-Kalthoff, Chenru Duan, Yuchen Zhuang, Yue Yu, Yanqiao Zhu, Alan Aspuru-Guzik, Kirill Neklyudov, Chao Zhang · PDF
  25. End-To-End Causal Effect Estimation from Unstructured Natural Language Data

    Nikita Dhawan, Leonardo Cotta, Karen Ullrich, Rahul Krishnan, Chris J. Maddison · PDF
  26. Estimating Probability Densities of Tabular Data using a Transformer Model combined with Denoising Diffusion

    Henry W. Leung, Jo Bovy, Joshua S. Speagle · PDF
  27. Evaluating Self-Supervised Foundation Models in Holographic Imaging

    Silas Dietler, Yanick Zeder, Elias Graf, Kilian Koch, Andreas Schwendimann, Tommaso Bendinelli · PDF
  28. Evaluation of RAG Metrics for Question Answering in the Telecom Domain

    Sujoy Roychowdhury, Sumit Soman, H. G. Ranjani, Neeraj Gunda, Vansh Chhabra, SAI KRISHNA BALA · PDF
  29. ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts

    Samar Khanna, Medhanie Irgau, David B. Lobell, Stefano Ermon · PDF
  30. Extracting Training Data from Document-Based VQA Models

    Francesco Pinto, Nathalie Rauschmayr, Florian Tramèr, Philip Torr, Federico Tombari · PDF
  31. Extrapolative Protein Design through Triplet-based Preference Learning

    Mostafa Karimi, Sharmi Banerjee, Tommi Jaakkola, Bella Dubrov, Shang Shang, Ron Benson · PDF
  32. Federated Fine-Tuning of Vision Foundation Models via Probabilistic Masking

    Vasileios Tsouvalas, Yuki M Asano, Aaqib Saeed · PDF
  33. Finding NeMo: Localizing Neurons Responsible For Memorization in Diffusion Models

    Lukas Struppek, Dominik Hintersdorf, Kristian Kersting, Adam Dziedzic, Franziska Boenisch · PDF
  34. FoMu-SSL: Foundation Model-Guided Multi-Sensor Self-Supervised Learning for Remote Sensing

    Dabin Seo, Haeji Jung, Jinkyu Kim · PDF
  35. Generalization vs. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data

    Antonis Antoniades, Xinyi Wang, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, William Yang Wang · PDF
  36. Geometric Median Matching for Robust Data Pruning

    Anish Acharya, Inderjit S Dhillon, Sujay Sanghavi · PDF
  37. GROD: Enhancing Generalization of Transformer with Out-of-Distribution Detection

    Yijin Zhou, Yu Guang Wang · PDF
  38. Improving GFlowNets for Text-to-Image Diffusion Alignment

    Dinghuai Zhang, Yizhe Zhang, Jiatao Gu, Ruixiang ZHANG, Joshua M. Susskind, Navdeep Jaitly, Shuangfei Zhai · PDF
  39. Improving Graph-Language Alignment with Hierarchical Graph Tokenization

    Yongqiang Chen, Quanming Yao, Juzheng Zhang, James Cheng, Yatao Bian · PDF
  40. In Search of Forgotten Domain Generalization

    Prasanna Mayilvahanan, Roland S. Zimmermann, Thaddäus Wiedemer, Evgenia Rusak, Attila Juhos, Matthias Bethge, Wieland Brendel · PDF
  41. In-Context Learning Improves Compositional Understanding of Vision-Language Models

    Matteo Nulli, Anesa Ibrahimi, Avik Pal, Hoshe Lee, Ivona Najdenkoska · PDF
  42. Inference Performance Optimization for Large Language Models on CPUs

    Pujiang He, Shan Zhou, Wenhuan Huang, Changqing Li, Duyi Wang, Bin Guo, Chen Meng, Sheng Gui, Weifei Yu, Yi Xie · PDF
  43. InstructBooth: Instruction-following Personalized Text-to-Image Generation

    Daewon Chae, Nokyung Park, Jinkyu Kim, Kimin Lee · PDF
  44. Instruction Tuning With Loss Over Instructions

    Zhengyan Shi, Adam X. Yang, Bin Wu, Laurence Aitchison, Emine Yilmaz, Aldo Lipani · PDF
  45. Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data

    Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Tomasz Korbak, Henry Sleight, Rajashree Agrawal, John Hughes, Dhruv Bhandarkar Pai, Andrey Gromov, Dan Roberts, Diyi Yang, David L. Donoho, Sanmi Koyejo · PDF
  46. It Takes Two: On the Seamlessness between Reward and Policy Model in RLHF

    TaiMing Lu, Lingfeng Shen, Xinyu Yang, Weiting Tan, Beidi Chen, Huaxiu Yao · PDF
  47. Jogging the Memory of Unlearned Models Through Targeted Relearning Attacks

    Shengyuan Hu, Yiwei Fu, Steven Wu, Virginia Smith · PDF
  48. Language Model-In-The-Loop: Data Optimal Approach to Recommend Actions in Text Games

    Arjun V Sudhakar, Prasanna Parthasarathi, Janarthanan Rajendran, Sarath Chandar · PDF
  49. Leveraging Generative Foundation Models for Domain Generalization

    Sobhan Hemati, Mahdi Beitollahi, Amir Hossein Estiri, Bassel Al Omari, Xi Chen, Guojun Zhang · PDF
  50. LIFTED: Multimodal Mixture-of-Experts for Clinical Trial Outcome Prediction

    Wenhao Zheng, Dongshen Peng, Hongxia Xu, Yun Li, Hongtu Zhu, Tianfan Fu, Huaxiu Yao · PDF
  51. LLM Task Interference: Impact of Task-Switch in Conversational History

    Akash Gupta, Ivaxi Sheth, Vyas Raina, Mark Gales, Mario Fritz · PDF
  52. LoRD: Low-Rank Decomposition of Monolingual Code LLMs for One-Shot Compression

    Ayush Kaushal, Tejas Vaidhya, Irina Rish · PDF
  53. Merging Improves Self-Critique Against Jailbreak Attacks

    Victor Gallego · PDF
  54. MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge?

    Zhaorun Chen, Yichao Du, Zichen Wen, Yiyang Zhou, Chenhang Cui, Zhenzhen Weng, Haoqin Tu, Chaoqi Wang, Zhengwei Tong, Leria HUANG, Canyu Chen, Qinghao Ye, Zhihong Zhu, Yuqing Zhang, Jiawei Zhou, Zhuokai Zhao, Rafael Rafailov, Chelsea Finn, Huaxiu Yao · PDF
  55. Model Breadcrumbs: Scalable Upcycling of Finetuned Foundation Models via Sparse Task Vectors Merging

    MohammadReza Davari, Eugene Belilovsky · PDF
  56. MoRe Fine-Tuning with 10x Fewer Parameters

    Wenxuan Tan, Nicholas Roberts, Tzu-Heng Huang, Jitian Zhao, John Cooper, Samuel Guo, Chengyu Duan, Frederic Sala · PDF
  57. Not Just Pretty Pictures: Toward Interventional Data Augmentation Using Text-to-Image Generators

    Jianhao Yuan, Francesco Pinto, Adam Davies, Philip Torr · PDF
  58. On the Discrepancy and Connection between Memorization and Generation in Diffusion Models

    Hanyu Wang, Yujin Han, Difan Zou · PDF
  59. On the Privacy Risks of Post-Hoc Explanations of Foundation Models

    Catherine Huang, Martin Pawelczyk, Himabindu Lakkaraju · PDF
  60. Open LLMs are Necessary for Private Adaptations and Outperform their Closed Alternatives

    Vincent Hanke, Tom Blanchard, Franziska Boenisch, Iyiola Emmanuel Olatunji, Michael Backes, Adam Dziedzic · PDF
  61. OTTER: Effortless Label Distribution Adaptation of Zero-shot Models

    Changho Shin, Jitian Zhao, Sonia Cromp, Harit Vishwakarma, Frederic Sala · PDF
  62. Out-Of-Context Prompting Boosts Fairness and Robustness in Large Language Model Predictions

    Leonardo Cotta, Chris J. Maddison · PDF
  63. PanSAM: Zero-Shot, Prompt-Free Pancreas Segmentation in CT Imaging

    Abolfazl Malekahmadi, Mohammad Taha Teimuri Jervakani, Armin Behnamnia, Zahra Dehghanian, Amir Shamloo, Hamid R. Rabiee · PDF
  64. Parameter-Efficient Quantized Mixture-of-Experts Meets Vision-Language Instruction Tuning for Semiconductor Electron Micrograph Analysis

    Sagar Srinivas Sakhinana, Sannidhi Gowri Naga Krishna Geethan, Chidaksh Ravuru, Venkataramana Runkana · PDF
  65. PLUTO: Pathology-Universal Transformer

    Dinkar Juyal, Harshith Padigela, Chintan Shah, Daniel Shenker, Natalia Harguindeguy, Yi Liu, Blake Martin, Yibo Zhang, Michael Nercessian, Miles Markey, Isaac Finberg, Kelsey Luu, Daniel Borders, Syed Ashar Javed, Emma L Krause, Raymond Biju, Aashish Sood, Allen Ma, Jackson Nyman, John Shamshoian, Guillaume Chhor, Darpan Sanghavi, Marc Thibault, Limin Yu, Fedaa Najdawi, Jennifer A. Hipp, Darren Fahy, Benjamin Glass, Eric E. Walk, John Abel, Harsha Vardhan pokkalla, Andrew H. Beck, Sean Grullon · PDF
  66. POST: A Framework for Privacy of Soft-prompt Transfer

    Xun Wang, Jing Xu, Franziska Boenisch, Michael Backes, Adam Dziedzic · PDF
  67. Pretrained Hybrids with MAD Skills

    Nicholas Roberts, Samuel Guo, Zhiqi Gao, Satya Sai Srinath Namburi GNVV, Sonia Cromp, Chengjun Wu, Chengyu Duan, Frederic Sala · PDF
  68. Privacy Auditing of Large Language Models

    Ashwinee Panda, Xinyu Tang, Milad Nasr, Christopher A. Choquette-Choo, Prateek Mittal · PDF
  69. Private Fine-tuning of Large Language Models with Zeroth-order Optimization

    Xinyu Tang, Ashwinee Panda, Milad Nasr, Saeed Mahloujifar, Prateek Mittal · PDF
  70. Projected Language Models: A Large Model Pre-Segmented Into Smaller Ones

    David Grangier, Angelos Katharopoulos, Pierre Ablin, Awni Hannun · PDF
  71. Quantum 3D Visual Grounding: A Step Towards Quantum-inspired AI-Visualization

    Adib Bazgir, Rama chandra Praneeth Madugula, Yuwen Zhang · PDF
  72. Rapid Switching and Multi-Adapter Fusion via Sparse High Rank Adapters

    Kartikeya Bhardwaj, Nilesh Prasad Pandey, Sweta Priyadarshi, Viswanath Ganapathy, Rafael Esteves, Shreya Kadambi, Shubhankar Borse, Paul Whatmough, Risheek Garrepalli, Mart Van Baalen, Harris Teague, Markus Nagel · PDF
  73. Recursive Introspection: Teaching LLM Agents How to Self-Improve

    Yuxiao Qu, Tianjun Zhang, Naman Garg, Aviral Kumar · PDF
  74. RNR: Teaching Large Language Models to Follow Roles and Rules

    Kuan Wang, Alexander Bukharin, Haoming Jiang, Qingyu Yin, Zhengyang Wang, Tuo Zhao, Jingbo Shang, Chao Zhang, Bing Yin, Xian Li, Jianshu Chen, Shiyang Li · PDF
  75. RouteFinder: Towards Foundation Models for Vehicle Routing Problems

    Federico Berto, Chuanbo Hua, Nayeli Gast Zepeda, André Hottung, Niels Wouda, Leon Lan, Kevin Tierney, Jinkyoo Park · PDF
  76. SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Sound

    Rishit Dagli, Shivesh Prakash, Robert Wu, Houman Khosravani · PDF
  77. Self-Control of LLM Behaviors by Compressing Suffix Gradient into Prefix Controller

    Min Cai, Yuchen Zhang, Shichang Zhang, Fan Yin, Difan Zou, Yisong Yue, Ziniu Hu · PDF
  78. Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs

    Jiatong Han, Jannik Kossen, Muhammed Razzak, Lisa Schut, Shreshth A Malik, Yarin Gal · PDF
  79. Split, Unlearn, Merge: Leveraging Data Attributes for More Effective Unlearning in LLMs

    Swanand Kadhe, Farhan Ahmed, Dennis Wei, Nathalie Baracaldo, Inkit Padhi · PDF
  80. Strong Copyright Protection for Language Models via Adaptive Model Fusion

    Javier Abad, Konstantin Donhauser, Francesco Pinto, Fanny Yang · PDF
  81. Test-Time Prototype Evolution for Generalizable Vision-Language Models

    Ce Zhang, Simon Stepputtis, Katia P. Sycara, Yaqi Xie · PDF
  82. The Effect of Data Corruption on Multimodal Long Form Responses

    Daniel Z Kaplan, Alexis Roger, Mohamed Osman, Irina Rish · PDF
  83. TimeDiT: General-purpose Diffusion Transformers for Time Series Foundation Model

    Defu Cao, Wen Ye, Yan Liu · PDF
  84. Towards Safe Large Language Models for Medicine

    Tessa Han, Aounon Kumar, Chirag Agarwal, Himabindu Lakkaraju · PDF
  85. TriLM vs FloatLM: Ternary LLMs are more Performant than Quantized FP16 LLMs

    Ayush Kaushal, Tejas Vaidhya, Tejas Pandey, Aaryan Bhagat, Irina Rish · PDF
  86. Two-Level Test-Time Adaptation in Multimodal Learning

    Jixiang Lei, Franz Pernkopf · PDF
  87. Understanding the Role of Functional Diversity in Weight-Ensembling with Ingredient Selection and Multidimensional Scaling

    Alex Rojas, David Alvarez-Melis · PDF
  88. Unsupervised Feature Extraction from a Foundation Model Zoo for Cell Similarity Search in Oncological Microscopy Across Devices

    Gabriel Kalweit, Anusha Klett, Mehdi Naouar, Jens Rahnfeld, Yannick Vogt, Diana Laura Infante Ramirez, Rebecca Berger, Jesus Duque Afonso, Tanja Nicole Hartmann, Marie Follo, Michael Luebbert, Roland Mertelsmann, Evelyn Ullrich, Joschka Boedecker, Maria Kalweit · PDF
  89. Unveiling CLIP Dynamics: Linear Mode Connectivity and Generalization

    Alireza Abdollahpourrostam, Amartya Sanyal, Seyed-Mohsen Moosavi-Dezfooli · PDF
  90. USCILab3D: A Large-scale, Long-term, Semantically Annotated Outdoor Dataset

    Kiran Lekkala, Henghui Bao, Peixu Cai, Wei Zer Lim, Chen Liu, Laurent Itti · PDF
  91. VFA: Vision Frequency Analysis of Foundation Models and Human

    Mohammad Javad Darvishi Bayazi, Md Rifat Arefin, Jocelyn Faubert, Irina Rish · PDF
  92. Vision-Language Models Provide Promptable Representations for Reinforcement Learning

    William Chen, Oier Mees, Aviral Kumar, Sergey Levine · PDF
  93. Waterfall: Framework for Robust and Scalable Text Watermarking

    Gregory Kang Ruey Lau, Xinyuan Niu, Hieu Dao, Jiangwei Chen, Chuan-Sheng Foo, Bryan Kian Hsiang Low · PDF
  94. When Do Language Models Need to Be Large?

    Zhixun Chen, Yali Du, David Henry Mguni · PDF
  95. Zero-Shot Generalization of GNNs over Distinct Attribute Domains

    Yangyi Shen, Beatrice Bevilacqua, Joshua Robinson, Charilaos Kanatsoulis, Jure Leskovec, Bruno Ribeiro · PDF