[PAPER] Lost in the Middle: How Language Models Use Long Contexts

2024. 5. 20. 19:54·Paper Review

Feb. 23, 2024

Nelson F. Liu et al.

 

1. Problem

- With the advent of RAG (Retrieval Augmented Generation) models, LLMs can now process long input contexts.

- However, there are concerns about whether LLMs can effectively utilize the information within these long contexts.

 

2. Key Findings

- The performance of LLMs degrades when the relevant information is located in the middle of the long input context.

- LLMs perform better when the relevant information is positioned at the beginning or end of the input context.

 

3. Analysis of Causes

- LLMs tend to focus more on recent information and struggle to effectively utilize long contexts.

- This limitation is likely related to the encoder architecture of LLMs, which may not be optimized for processing long-range dependencies.

'Paper Review' 카테고리의 다른 글
  • [PAPER] Self-Consistent Reasoning-based Aspect-Sentiment Quad Prediction with Extract-Then-Assign Strategy
  • [PAPER] Can Large Language Models be Good Emotional Supporter? Mitigating Preference Bias on Emotional Support Conversation
  • [PAPER] Evaluating and Inducing Personality in Pre-trained Language Models
  • [PAPER] Machine learning techniques for emotion detection and sentiment analysis: current state, challenges, and future directions
Sungyeon Kim
Sungyeon Kim
goldstaryeon@sookmyung.ac.kr
Sungyeon Kim
Sungyeon Kim
Sungyeon Kim
전체
오늘
어제
  • 분류 전체보기 (621) N
    • Paper Review (30)
    • Research Record (9)
    • Study Record (143)
      • Cybersecurity (79)
      • AI Data Science (28)
      • Computer Science (24)
      • Linear Algebra (6)
      • SQL (5)
      • LaTeX (1)
    • English Transcription (261) N
    • 한글 필사 (100)
    • 날것 그대로의 생각들 (74)

인기 글

최근 댓글

최근 글

hELLO· Designed By정상우.v4.5.3
Sungyeon Kim
[PAPER] Lost in the Middle: How Language Models Use Long Contexts
상단으로

티스토리툴바

단축키

내 블로그

내 블로그 - 관리자 홈 전환
Q
Q
새 글 쓰기
W
W

블로그 게시글

글 수정 (권한 있는 경우)
E
E
댓글 영역으로 이동
C
C

모든 영역

이 페이지의 URL 복사
S
S
맨 위로 이동
T
T
티스토리 홈 이동
H
H
단축키 안내
Shift + /
⇧ + /

* 단축키는 한글/영문 대소문자로 이용 가능하며, 티스토리 기본 도메인에서만 동작합니다.