LLMs in Chemistry

https://arxiv.org/abs/2402.01439

1. Molecule encoding

1) Representations

(1) Fingerprint

- binary string

 

(2) Sequential 

- SMILES (1988): 문제 많아

- SELFIES (2020): 문제 개선함

- InChI (2013): focuses on uniqueness

 

(3) Graph

 

2) Tokenization

(1) char level

- 이상하게 tokenize됨에도 불구하고 성능 좋아

 

(2) atom level

- customized atom-level tokenizers (2020)

 

(3) motif level

a. chemistry-driven

- break molecules into chemically meaningful substructures

- graph에서 주로 쓰이는 듯 -> cost 증가. 어렵고. expert knowledge 필요.

b. data-driven

- BPE랑 비슷: iteratively merging the most frequent pairs of characters into a single token

 

2. Methods

1) Langauge Modelling Objectives

(1) Masked Language Modelling (MLM)

- prevalent pretraining objective for LLMs

- MLM is conducted on molecular sequential representations (e.g. SMILES, SELFIES, etc.)

- SMIELS-BERT, ChemBERTa, MG-BERT, Molformer, Selformer, T5 Chem, Chemformer, BARTSmiles

 

(2) Molecular Property Prediction (MPP)

- predict molecular properties given molecular sequential representations

- generates properties using cheminformatics tools (e.g. RDKit) -> manual 라벨링 필요없음

- ChemBERTa-2, SPT

- due to the design of MPP, MPP-based models are primarily used for molecular representation learning -> cannot generate tokens

 

(3) Autoregressive Token Generation (ATG)