ICML 2024PastLarge language models

First Workshop on Long-Context Foundation Models @ ICML 2024

LCFM 2024

Official website ↗OpenReview venue ↗See all ICML workshops →✎ Edit this entry

Submission deadline: Jun 7, 2024, 12:59 UTC
imported from OpenReview — check the website for extensions
Submission portal: OpenReview
Notes: Topics were auto-suggested and may be imprecise — edits welcome.

Accepted papers (30)

Fetched from OpenReview (v2) on 2026-06-10.

Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling
Yair Schiff, Chia Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, Volodymyr Kuleshov · PDF
CAMELoT: Towards Large Language Models with Training-Free Consolidated Associative Memory
Zexue He, Leonid Karlinsky, Donghyun Kim, Julian McAuley, Dmitry Krotov, Rogerio Feris · PDF
CD-Pos: Long Context Generalization in LLMs Through Continuous and Discrete Position Synthesis
Zhiyuan Hu, Yuliang Liu, Jinman Zhao, Suyuchen Wang, WangYan, Wei Shen, Chao Yin, Bryan Hooi · PDF
Demonstrations in In-context Learning for LLMs with Large Label Space
Zhan Li, Fanghui Liu, Volkan Cevher, Grigorios Chrysos · PDF
ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Models
Thibaut Thonet, Jos Rozen, laurent besacier · PDF
FastDecode: High-Throughput LLM Serving through Disaggregating Attention Computation
Jiaao He, Kezhao Huang, Jidong Zhai · PDF
Fine-Tuning Medical Language Models for Enhanced Long-Contextual Understanding and Domain Expertise
Qimin Yang, rongshengwang, CHEN JIEXIN, Runqi Su, Tao Tan · PDF
From Text to Pixel: Advancing Long-Context Understanding in MLLMs
Yujie Lu, Xiujun Li, Tsu-Jui Fu, Miguel Eckstein, William Yang Wang · PDF
Improved Algorithms for Kernel Matrix-Vector Multiplication
Piotr Indyk, Michael Kapralov, Kshiteej Sheth, Tal Wagner · PDF
In-Context Learning with Long-Context Models: An In-Depth Exploration
Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R. Gormley, Graham Neubig · PDF
InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, Maosong Sun · PDF
Learning to Reduce: Towards Improving Performance of Large Language Models on Structured Data
Younghun Lee, Sungchul Kim, Ryan A. Rossi, Tong Yu, Xiang Chen · PDF
Long Context Understanding using Self-Generated Synthetic Data
Jerry Li, Subhro Das, Aude Oliva, Dmitry Krotov, Leonid Karlinsky, Rogerio Feris · PDF
Long-Context Vision Large Language Models: Empirical Insights and A Baseline
Yongshuo Zong, Ismail Elezi, Yongxin Yang, Jiankang Deng, Timothy Hospedales · PDF
LongAlign: A Recipe for Long Context Alignment of Large Language Models
Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, Juanzi Li · PDF
Many-Shot In-Context Learning
Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie C.Y. Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, John D Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, Hugo Larochelle · PDF
MINI-SEQUENCE TRANSFORMER: Optimizing Intermediate Memory for Long Sequences Training
Cheng Luo, Jiawei Zhao, Zhuoming Chen, Beidi Chen, Anima Anandkumar · PDF
Mitigate Position Bias in Large Language Models via Scaling a Single Dimension
Yijiong Yu, Huiqiang Jiang, Xufang Luo, Qianhui Wu, Chin-Yew Lin, Dongsheng Li, Yuqing Yang, Yongfeng Huang, Lili Qiu · PDF
MSAMamba: Adapting Subquadratic Models To Long-Context DNA MSA Analysis
Vishrut Thoutam, Dina Ellsworth · PDF
PhaseEvo: Towards Unified Long-Context Prompt Optimization for Large Language Models
Wendi Cui, Jiaxin Zhang, Zhuohang Li, Hao Sun, Damien Lopez, Kamalika Das, Bradley A. Malin, Sricharan Kumar · PDF
Position Coupling: Leveraging Task Structure for Improved Length Generalization of Transformers
Hanseul Cho, Jaeyoung Cha, Pranjal Awasthi, Srinadh Bhojanapalli, Anupam Gupta, Chulhee Yun · PDF
Pretrained Hybrids with MAD Skills
Nicholas Roberts, Samuel Guo, Zhiqi Gao, Satya Sai Srinath Namburi GNVV, Sonia Cromp, Chengjun Wu, Chengyu Duan, Frederic Sala · PDF
Probing the Decision Boundaries of In-context Learning in Large Language Models
Siyan Zhao, Tung Nguyen, Aditya Grover · PDF
RepoQA: Evaluating Long Context Code Understanding
Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, LINGMING ZHANG · PDF
Spatio-Spectral Graph Neural Networks
Simon Geisler, Arthur Kosmala, Daniel Herbst, Stephan Günnemann · PDF
Spectral State Space Models
Naman Agarwal, Daniel Suo, Xinyi Chen, Elad Hazan · PDF
Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystack
Xiaoyue Xu, Qinyuan Ye, Xiang Ren · PDF
Vision-LSTM: xLSTM as Generic Vision Backbone
Benedikt Alkin, Maximilian Beck, Korbinian Pöppel, Sepp Hochreiter, Johannes Brandstetter · PDF
xLSTM: Extended Long Short-Term Memory
Korbinian Pöppel, Maximilian Beck, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael K Kopp, Günter Klambauer, Johannes Brandstetter, Sepp Hochreiter · PDF
ZigMa: A DiT-style Zigzag Mamba Diffusion Model
Vincent Tao Hu, Stefan Andreas Baumann, Ming Gui, Olga Grebenkova, Pingchuan Ma, Johannes Schusterbauer, Björn Ommer · PDF

Accepted papers (30)

☆Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling

☆CAMELoT: Towards Large Language Models with Training-Free Consolidated Associative Memory

☆CD-Pos: Long Context Generalization in LLMs Through Continuous and Discrete Position Synthesis

☆Demonstrations in In-context Learning for LLMs with Large Label Space

☆ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Models

☆FastDecode: High-Throughput LLM Serving through Disaggregating Attention Computation

☆Fine-Tuning Medical Language Models for Enhanced Long-Contextual Understanding and Domain Expertise

☆From Text to Pixel: Advancing Long-Context Understanding in MLLMs

☆Improved Algorithms for Kernel Matrix-Vector Multiplication

☆In-Context Learning with Long-Context Models: An In-Depth Exploration

☆InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory

☆Learning to Reduce: Towards Improving Performance of Large Language Models on Structured Data

☆Long Context Understanding using Self-Generated Synthetic Data

☆Long-Context Vision Large Language Models: Empirical Insights and A Baseline

☆LongAlign: A Recipe for Long Context Alignment of Large Language Models

☆Many-Shot In-Context Learning

☆MINI-SEQUENCE TRANSFORMER: Optimizing Intermediate Memory for Long Sequences Training

☆Mitigate Position Bias in Large Language Models via Scaling a Single Dimension

☆MSAMamba: Adapting Subquadratic Models To Long-Context DNA MSA Analysis

☆PhaseEvo: Towards Unified Long-Context Prompt Optimization for Large Language Models

☆Position Coupling: Leveraging Task Structure for Improved Length Generalization of Transformers

☆Pretrained Hybrids with MAD Skills

☆Probing the Decision Boundaries of In-context Learning in Large Language Models

☆RepoQA: Evaluating Long Context Code Understanding

☆Spatio-Spectral Graph Neural Networks

☆Spectral State Space Models

☆Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystack

☆Vision-LSTM: xLSTM as Generic Vision Backbone

☆xLSTM: Extended Long Short-Term Memory

☆ZigMa: A DiT-style Zigzag Mamba Diffusion Model

Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling

CAMELoT: Towards Large Language Models with Training-Free Consolidated Associative Memory

CD-Pos: Long Context Generalization in LLMs Through Continuous and Discrete Position Synthesis

Demonstrations in In-context Learning for LLMs with Large Label Space

ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Models

FastDecode: High-Throughput LLM Serving through Disaggregating Attention Computation

Fine-Tuning Medical Language Models for Enhanced Long-Contextual Understanding and Domain Expertise

From Text to Pixel: Advancing Long-Context Understanding in MLLMs

Improved Algorithms for Kernel Matrix-Vector Multiplication

In-Context Learning with Long-Context Models: An In-Depth Exploration

InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory

Learning to Reduce: Towards Improving Performance of Large Language Models on Structured Data

Long Context Understanding using Self-Generated Synthetic Data

Long-Context Vision Large Language Models: Empirical Insights and A Baseline

LongAlign: A Recipe for Long Context Alignment of Large Language Models

Many-Shot In-Context Learning

MINI-SEQUENCE TRANSFORMER: Optimizing Intermediate Memory for Long Sequences Training

Mitigate Position Bias in Large Language Models via Scaling a Single Dimension

MSAMamba: Adapting Subquadratic Models To Long-Context DNA MSA Analysis

PhaseEvo: Towards Unified Long-Context Prompt Optimization for Large Language Models

Position Coupling: Leveraging Task Structure for Improved Length Generalization of Transformers

Pretrained Hybrids with MAD Skills

Probing the Decision Boundaries of In-context Learning in Large Language Models

RepoQA: Evaluating Long Context Code Understanding

Spatio-Spectral Graph Neural Networks

Spectral State Space Models

Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystack

Vision-LSTM: xLSTM as Generic Vision Backbone

xLSTM: Extended Long Short-Term Memory

ZigMa: A DiT-style Zigzag Mamba Diffusion Model