Qizhi Pei, Wei zhang, Jinhua Zhu, Kehan Wu, Kaiyuan Gao, Lijun Wu, Yingce Xia, Rui Yan
Recent advancements in biological research leverage the integration of molecules, proteins, and natural language to enhance drug discovery. However, current models exhibit several limitations, such as the generation of invalid molecular SMILES, underutilization of contextual information, and equal treatment of structured and unstructured knowledge. To address these issues, we propose $\mathbf{BioT5}$, a comprehensive pre-training framework that enriches cross-modal integration in biology with chemical knowledge and natural language associations. $\mathbf{BioT5}$ utilizes SELFIES for $100%$ robust molecular representations and extracts knowledge from the surrounding context of bio-entities in unstructured biological literature. Furthermore, $\mathbf{BioT5}$ distinguishes between structured and unstructured knowledge, leading to more effective utilization of information. After fine-tuning, BioT5 shows superior performance across a wide range of tasks, demonstrating its strong capability of capturing underlying relations and properties of bio-entities. Our code is available at $\href{https://github.com/QizhiPei/BioT5}{Github}$.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Drug Discovery | ChEBI-20 | BLEU | 86.7 | BioT5 |
| Drug Discovery | ChEBI-20 | Exact Match | 41.3 | BioT5 |
| Drug Discovery | ChEBI-20 | Frechet ChemNet Distance (FCD) | 0.43 | BioT5 |
| Drug Discovery | ChEBI-20 | Levenshtein | 15.097 | BioT5 |
| Drug Discovery | ChEBI-20 | MACCS FTS | 88.6 | BioT5 |
| Drug Discovery | ChEBI-20 | Morgan FTS | 73.4 | BioT5 |
| Drug Discovery | ChEBI-20 | Parameter Count | 252000000 | BioT5 |
| Drug Discovery | ChEBI-20 | RDK FTS | 80.1 | BioT5 |
| Drug Discovery | ChEBI-20 | Text2Mol | 57.6 | BioT5 |
| Drug Discovery | ChEBI-20 | Validity | 100 | BioT5 |
| Molecule Captioning | ChEBI-20 | BLEU-2 | 63.5 | BioT5 |
| Molecule Captioning | ChEBI-20 | BLEU-4 | 55.6 | BioT5 |
| Molecule Captioning | ChEBI-20 | METEOR | 65.6 | BioT5 |
| Molecule Captioning | ChEBI-20 | ROUGE-1 | 69.2 | BioT5 |
| Molecule Captioning | ChEBI-20 | ROUGE-2 | 55.9 | BioT5 |
| Molecule Captioning | ChEBI-20 | ROUGE-L | 63.3 | BioT5 |
| Molecule Captioning | ChEBI-20 | Text2Mol | 60.3 | BioT5 |
| Text-based de novo Molecule Generation | ChEBI-20 | BLEU | 86.7 | BioT5 |
| Text-based de novo Molecule Generation | ChEBI-20 | Exact Match | 41.3 | BioT5 |
| Text-based de novo Molecule Generation | ChEBI-20 | Frechet ChemNet Distance (FCD) | 0.43 | BioT5 |
| Text-based de novo Molecule Generation | ChEBI-20 | Levenshtein | 15.097 | BioT5 |
| Text-based de novo Molecule Generation | ChEBI-20 | MACCS FTS | 88.6 | BioT5 |
| Text-based de novo Molecule Generation | ChEBI-20 | Morgan FTS | 73.4 | BioT5 |
| Text-based de novo Molecule Generation | ChEBI-20 | Parameter Count | 252000000 | BioT5 |
| Text-based de novo Molecule Generation | ChEBI-20 | RDK FTS | 80.1 | BioT5 |
| Text-based de novo Molecule Generation | ChEBI-20 | Text2Mol | 57.6 | BioT5 |
| Text-based de novo Molecule Generation | ChEBI-20 | Validity | 100 | BioT5 |