Aru Maekawa, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura
Recently, decoder-only pre-trained large language models (LLMs), with several tens of billion parameters, have significantly impacted a wide range of natural language processing (NLP) tasks. While encoder-only or encoder-decoder pre-trained language models have already proved to be effective in discourse parsing, the extent to which LLMs can perform this task remains an open research question. Therefore, this paper explores how beneficial such LLMs are for Rhetorical Structure Theory (RST) discourse parsing. Here, the parsing process for both fundamental top-down and bottom-up strategies is converted into prompts, which LLMs can work with. We employ Llama 2 and fine-tune it with QLoRA, which has fewer parameters that can be tuned. Experimental results on three benchmark datasets, RST-DT, Instr-DT, and the GUM corpus, demonstrate that Llama 2 with 70 billion parameters in the bottom-up strategy obtained state-of-the-art (SOTA) results with significant differences. Furthermore, our parsers demonstrated generalizability when evaluated on RST-DT, showing that, in spite of being trained with the GUM corpus, it obtained similar performances to those of existing parsers trained with RST-DT.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Discourse Parsing | RST-DT | Standard Parseval (Full) | 58.1 | Bottom-up Llama 2 (70B) |
| Discourse Parsing | RST-DT | Standard Parseval (Nuclearity) | 70.4 | Bottom-up Llama 2 (70B) |
| Discourse Parsing | RST-DT | Standard Parseval (Relation) | 60 | Bottom-up Llama 2 (70B) |
| Discourse Parsing | RST-DT | Standard Parseval (Span) | 79.8 | Bottom-up Llama 2 (70B) |
| Discourse Parsing | RST-DT | Standard Parseval (Full) | 56 | Top-down Llama 2 (70B) |
| Discourse Parsing | RST-DT | Standard Parseval (Nuclearity) | 68.7 | Top-down Llama 2 (70B) |
| Discourse Parsing | RST-DT | Standard Parseval (Relation) | 57.7 | Top-down Llama 2 (70B) |
| Discourse Parsing | RST-DT | Standard Parseval (Span) | 78.8 | Top-down Llama 2 (70B) |
| Discourse Parsing | RST-DT | Standard Parseval (Full) | 56 | Bottom-up Llama 2 (13B) |
| Discourse Parsing | RST-DT | Standard Parseval (Nuclearity) | 68.1 | Bottom-up Llama 2 (13B) |
| Discourse Parsing | RST-DT | Standard Parseval (Relation) | 57.8 | Bottom-up Llama 2 (13B) |
| Discourse Parsing | RST-DT | Standard Parseval (Span) | 78.3 | Bottom-up Llama 2 (13B) |
| Discourse Parsing | RST-DT | Standard Parseval (Full) | 55.8 | Bottom-up Llama 2 (7B) |
| Discourse Parsing | RST-DT | Standard Parseval (Nuclearity) | 67.5 | Bottom-up Llama 2 (7B) |
| Discourse Parsing | RST-DT | Standard Parseval (Relation) | 57.6 | Bottom-up Llama 2 (7B) |
| Discourse Parsing | RST-DT | Standard Parseval (Span) | 78.2 | Bottom-up Llama 2 (7B) |
| Discourse Parsing | RST-DT | Standard Parseval (Full) | 55.6 | Top-down Llama 2 (13B) |
| Discourse Parsing | RST-DT | Standard Parseval (Nuclearity) | 67.9 | Top-down Llama 2 (13B) |
| Discourse Parsing | RST-DT | Standard Parseval (Relation) | 57.7 | Top-down Llama 2 (13B) |
| Discourse Parsing | RST-DT | Standard Parseval (Span) | 78.6 | Top-down Llama 2 (13B) |
| Discourse Parsing | RST-DT | Standard Parseval (Full) | 53.4 | Top-down Llama 2 (7B) |
| Discourse Parsing | RST-DT | Standard Parseval (Nuclearity) | 65.4 | Top-down Llama 2 (7B) |
| Discourse Parsing | RST-DT | Standard Parseval (Relation) | 55.2 | Top-down Llama 2 (7B) |
| Discourse Parsing | RST-DT | Standard Parseval (Span) | 76.3 | Top-down Llama 2 (7B) |