https://arxiv.org/abs/2402.01439 1. Molecule encoding 1) Representations (1) Fingerprint - binary string (2) Sequential - SMILES (1988): 문제 많아 - SELFIES (2020): 문제 개선함 - InChI (2013): focuses on uniqueness (3) Graph 2) Tokenization (1) char level - 이상하게 tokenize됨에도 불구하고 성능 좋아 (2) atom level - customized atom-level tokenizers (2020) (3) motif level a. chemistry-driven - break molecules into chemi..
Paper Review
1. Bloom (2022) 1) 오픈소스 2) 176B parameters 3) decoder-only Transformer 4) trained on the ROOTS corpus 2. OPT (2022) 1) 오픈소스 2) a suite of decoder-only pre-trained transformers 3) 125M ~ 175M parameters 4 OPT-175B is comparable to GPT-3, while requiring only 1/7th the carbon footprint 3. Mistral 7B (2023) 1) 7B parameters 2) Llama 1, 2 이김 3) transformer based 4) grouped-query attention (GQA) 5) s..
Problem: 1. most methods used in detecting fake news only consider data-oriented text features 2. ignores dual emotion features (1.publisher emotions and 2.social emotions) Proposal 1. Deep Normalized Attention-based mechanism for enriched extraction of dual emotion features 2. Adaptive Genetic Weight Update-Random Forest (AGWu-RF) for classification Methods 1. Deep Normalized Attention-based me..
Title: Twitter Sentiment Analysis via Bi-sense Emoji Embedding and Attention-based LSTM Authors: Yuxiao Chen, Jianbo Yuan, Quanzeng You, Jiebo Luo Publication: ACM Multimedia Conference 2018 1. Background Previous work shows that, it is useful to pre-train a DNN on an emoji prediction task with pre-trained emoji embeddings to learn the emotional signals of emojis. 2. Problem The previous emoji e..
Title: Angle-Optimized Text Embeddings Authors: Xianming Li, Jing Li Submitted on 22 Sep 2023 1. Background High-quality text embedding is pivotal in improving STS (Semantic Textual Similarity) tasks which are crucial components in LLM applications. 2. Problem Existing text embedding models have a vanishing gradients problem, primarily due to the reliance on the cosine function in the optimizati..
Title - emoji2vec: Learning Emoji Representations from their Description Authors - Ben Eisner, Time Rocktaschel, Isabelle Augenstein, Matko Bosnjak, Sebastian Riedal Publication - EMNLP 2016 1. Research Background 1) Increased use of emoji and interest in Social media text analysis 2) NLP often relies on pre-trained word embeddings (e.g., word2vec, GloVe) 3) Yet, neither resource contains Unicod..
Natural language processing: state of the art, current trends and challenges Jul. 2022 1. NLP 1) Natural Language Understanding (NLU) = Linguistics (1) Phonology - sound (2) Morphology - the smallest units of meaning e.g., precancellation -> pre (prefix), cancella (root), -tion (suffix) a. Lexical morpheme (e.g., table, chair) b. Grammatical morphemes (e.g., Worked, Consulting) c. Bound morpheme..
DialogueRNN: An Attentive RNN for Emotion Detection in Conversations https://arxiv.org/pdf/1811.00405.pdf Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, Erik Cambria 1. Raising a Problem Current emotion detection systems in conversations, including the SOTA method, CMN (Conversational Memory Networks, Devamanyu Hazarika's previous research), 1) do not dis..
Beyond Sentiment Analysis: A Review of Recent Trends in Text Based Sentiment Analysis and Emotion Detection https://www.researchgate.net/publication/367297321_Beyond_Sentiment_Analysis_A_Review_of_Recent_Trends_in_Text_Based_Sentiment_Analysis_and_Emotion_Detection - Lai Po Hung and Suraya Alias - accepted August 23, 2022 1. Introduction 1) Sentiment analysis - is one of the earliest methods of ..
Stanford University, Google Brain ICLR 2020 conference 0. MLM 1) mask 15% of the input 2) train to recover the original input → It can learn bidirectional representations 1. A Problem of MLM substantial compute cost → because only learns from 15% tokens per sample 2. Replace token detection 1) replace some tokens of the input → solves a mismath problem of BERT 2) train to predicts every token wh..
링크 : https://arxiv.org/pdf/1909.11942.pdf 학회지 : ICLR 2020 발표 연도 : 2019 *self-supervised learning : - 2단계로 구성. 1) Pre-trained 모델 생성 : untagged data 이용해서 전반적인 특징 학습하고 2) Downstream task : 소량의 tagged data 이용해서 Pre-trained model fine tuning 0. Abstract ALBERT is A Lite version of BERT (have fewer parameters compared to BERT-large) 1) Characteristics (1) use 2 parameter reduction techiniques to addre..
link : https://arxiv.org/pdf/1907.11692.pdf by University of Washington, Seattle, Wa & Facebook AI 1. Introduction Language model pretraining has led to significant performance gains. but, training is computaionally expensive training often done on private datasets of different sizes hyperparameter choices have significant impact on the final results. → so, we present a replication study of BERT..
