IndoNLU

SmSA (Sentiment Analysis) is a task with three possible labels: positive, negative, and neutral. This guide demonstrates how to fine-tune a pre-trained IndoBERT model (e.g., indobenchmark/indobert-base-p1) using the DocumentSentimentDataset and DocumentSentimentDataLoader utilities provided by IndoNLU.

Workflow Overview

1. Setup: Initialize random seeds for reproducibility.
2. Model Loading: Load the BertTokenizer and BertConfig, then instantiate BertForSequenceClassification with the appropriate number of labels.
3. Data Preparation: Use DocumentSentimentDataset to load TSV files and DocumentSentimentDataLoader to create PyTorch DataLoaders.
4. Training Loop: Use forward_sequence_classification to compute loss and predictions, then update the model using an optimizer (e.g., Adam).
5. Evaluation: Evaluate performance on validation and test sets using document_sentiment_metrics_fn.

import torch
from transformers import BertForSequenceClassification, BertConfig, BertTokenizer
from utils.data_utils import DocumentSentimentDataset, DocumentSentimentDataLoader
from utils.forward_fn import forward_sequence_classification
from utils.metrics import document_sentiment_metrics_fn

# 1. Load Model & Config
tokenizer = BertTokenizer.from_pretrained('indobenchmark/indobert-base-p1')
config = BertConfig.from_pretrained('indobenchmark/indobert-base-p1')
config.num_labels = DocumentSentimentDataset.NUM_LABELS
model = BertForSequenceClassification.from_pretrained('indobenchmark/indobert-base-p1', config=config)

# 2. Prepare Dataset
train_dataset = DocumentSentimentDataset('./dataset/smsa_doc-sentiment-prosa/train_preprocess.tsv', tokenizer, lowercase=True)
train_loader = DocumentSentimentDataLoader(dataset=train_dataset, max_seq_len=512, batch_size=32, num_workers=16, shuffle=True)

# 3. Training Loop Snippet
optimizer = torch.optim.Adam(model.parameters(), lr=3e-6)
model.cuda()

for epoch in range(5):
    model.train()
    for i, batch_data in enumerate(train_loader):
        loss, batch_hyp, batch_label = forward_sequence_classification(model, batch_data[:-1], i2w=DocumentSentimentDataset.INDEX2LABEL, device='cuda')
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

The fine-tuning loop for CASA involves using forward_sequence_multi_classification to handle the multi-label classification logic.

Steps:

1. Set model to model.train() and enable gradients.
2. Iterate through the DataLoader.
3. Use forward_sequence_multi_classification(model, batch_data[:-1], i2w=i2w, device='cuda') to get the loss, hypotheses, and labels.
4. Perform standard PyTorch optimization: optimizer.zero_grad(), loss.backward(), and optimizer.step().
5. Use absa_metrics_fn to calculate performance metrics (like accuracy/F1) for the epoch.

from utils.forward_fn import forward_sequence_multi_classification
from utils.metrics import absa_metrics_fn

# Inside training loop
model.train()
torch.set_grad_enabled(True)

for i, batch_data in enumerate(train_pbar):
    loss, batch_hyp, batch_label = forward_sequence_multi_classification(model, batch_data[:-1], i2w=i2w, device='cuda')
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

To fine-tune a model like indobenchmark/indobert-base-p1 for emotion recognition, follow these steps:

1. Configure the model: Load the BertConfig and set config.num_labels to match the number of emotion classes (e.g., 5 for Emot: sadness, anger, love, fear, happy).
2. Instantiate: Use BertForSequenceClassification.from_pretrained with the modified config.
3. Training Loop:
   • Use forward_sequence_classification from utils.forward_fn to compute loss and predictions.
   • Use document_sentiment_metrics_fn from utils.metrics to evaluate performance.
   • Standard PyTorch optimization steps: optimizer.zero_grad(), loss.backward(), and optimizer.step().

from transformers import BertForSequenceClassification, BertConfig, BertTokenizer
from utils.forward_fn import forward_sequence_classification
from utils.metrics import document_sentiment_metrics_fn

# 1. Setup
tokenizer = BertTokenizer.from_pretrained('indobenchmark/indobert-base-p1')
config = BertConfig.from_pretrained('indobenchmark/indobert-base-p1')
config.num_labels = 5 # e.g., for Emot
model = BertForSequenceClassification.from_pretrained('indobenchmark/indobert-base-p1', config=config).cuda()

# 2. Training Loop snippet
optimizer = torch.optim.Adam(model.parameters(), lr=5e-6)
model.train()
for i, batch_data in enumerate(train_loader):
    loss, batch_hyp, batch_label = forward_sequence_classification(model, batch_data[:-1], i2w=i2w, device='cuda')
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

This tutorial demonstrates how to fine-tune a pre-trained Transformer model (specifically indobert-base-p1) for the SmSA (Sentiment Analysis) task. SmSA contains three labels: positive, negative, and neutral.

1. Load Model and Tokenizer

Use HuggingFace transformers to load the tokenizer and configure the model for sequence classification. Ensure config.num_labels matches the number of classes in your dataset.

from transformers import BertForSequenceClassification, BertConfig, BertTokenizer
from utils.data_utils import DocumentSentimentDataset

tokenizer = BertTokenizer.from_pretrained('indobenchmark/indobert-base-p1')
config = BertConfig.from_pretrained('indobenchmark/indobert-base-p1')
config.num_labels = DocumentSentimentDataset.NUM_LABELS

model = BertForSequenceClassification.from_pretrained('indobenchmark/indobert-base-p1', config=config)

2. Prepare Dataset and DataLoader

Use DocumentSentimentDataset and DocumentSentimentDataLoader from the utils.data_utils module to handle tokenization and batching.

from utils.data_utils import DocumentSentimentDataset, DocumentSentimentDataLoader

train_dataset = DocumentSentimentDataset(train_dataset_path, tokenizer, lowercase=True)
valid_dataset = DocumentSentimentDataset(valid_dataset_path, tokenizer, lowercase=True)

train_loader = DocumentSentimentDataLoader(dataset=train_dataset, max_seq_len=512, batch_size=32, num_workers=16, shuffle=True)
valid_loader = DocumentSentimentDataLoader(dataset=valid_dataset, max_seq_len=512, batch_size=32, num_workers=16, shuffle=False)

3. Training Loop

Perform fine-tuning using an optimizer (e.g., Adam) and the forward_sequence_classification utility function to compute loss and predictions.

from utils.forward_fn import forward_sequence_classification
from utils.metrics import document_sentiment_metrics_fn
import torch
from torch import optim

optimizer = optim.Adam(model.parameters(), lr=5e-6)
model = model.cuda()

# Inside training loop:
for i, batch_data in enumerate(train_pbar):
    loss, batch_hyp, batch_label = forward_sequence_classification(model, batch_data[:-1], i2w=i2w, device='cuda')
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

4. Inference on Sample Sentences

To test the model on a single string, tokenize the text, convert to a tensor, and extract the label from the logits.

text = 'Bahagia hatiku melihat pernikahan putri sulungku yang cantik jelita'
subwords = tokenizer.encode(text)
subwords = torch.LongTensor(subwords).view(1, -1).to(model.device)

logits = model(subwords)[0]
label = torch.topk(logits, k=1, dim=-1)[1].squeeze().item()
# Use i2w (index to word) mapping to get the string label
print(f'Text: {text} | Label : {i2w[label]}')

To prepare data for an NER model, follow these steps:

1. Initialize a tokenizer using transformers.AutoTokenizer.
2. Instantiate NerDataset with the path to your preprocessed text file and the tokenizer.
3. Wrap the dataset in a NerDataLoader to handle batching and padding.

Note: The input text file must be tab-separated (token\tlabel) with empty lines separating different sentences.

from transformers import AutoTokenizer

# 1. Setup tokenizer
pretrained_model = 'bert-base-uncased'
tokenizer = AutoTokenizer.from_pretrained(pretrained_model)

# 2. Setup dataset
dataset_path = '../data/ner-grit/train_preprocess_0.txt'
dataset = NerDataset(dataset_path, tokenizer)

# 3. Setup loader
loader = NerDataLoader(dataset, batch_size=32, num_workers=32)

# Iterate through batches
for subwords, subword_to_word_indices, seq_label in loader:
    # subwords: (batch_size, max_seq_len)
    # subword_to_word_indices: (batch_size, max_seq_len)
    # seq_label: (batch_size, max_tgt_len) padded with -100
    pass

WReTe is a textual entailment dataset where the goal is to classify a pair of input sentences into one of two labels: Entail_or_Paraphrase or NotEntail. This guide demonstrates how to finetune an IndoBERT model for this task using the indonlu utilities.

Workflow Overview

1. Setup: Initialize random seeds for reproducibility.
2. Model Loading: Load a pre-trained IndoBERT tokenizer and configuration, then instantiate BertForSequenceClassification with the appropriate number of labels.
3. Data Preparation: Use EntailmentDataset to load CSV files and EntailmentDataLoader to create PyTorch DataLoaders.
4. Training: Use a training loop with forward_sequence_classification to compute loss and update model weights.
5. Evaluation: Evaluate the model on validation and test sets using document_sentiment_metrics_fn to calculate performance metrics.

import torch
from transformers import BertForSequenceClassification, BertConfig, BertTokenizer
from utils.data_utils import EntailmentDataset, EntailmentDataLoader
from utils.forward_fn import forward_sequence_classification
from utils.metrics import document_sentiment_metrics_fn

# 1. Load Model and Tokenizer
tokenizer = BertTokenizer.from_pretrained('indobenchmark/indobert-base-p1')
config = BertConfig.from_pretrained('indobenchmark/indobert-base-p1')
config.num_labels = EntailmentDataset.NUM_LABELS
model = BertForSequenceClassification.from_pretrained('indobenchmark/indobert-base-p1', config=config)

# 2. Prepare Dataset
train_dataset = EntailmentDataset('./dataset/wrete_entailment-ui/train_preprocess.csv', tokenizer, lowercase=True)
train_loader = EntailmentDataLoader(dataset=train_dataset, max_seq_len=512, batch_size=32, num_workers=16, shuffle=True)

# 3. Training Loop Snippet
optimizer = torch.optim.Adam(model.parameters(), lr=5e-6)
model.cuda()

for epoch in range(10):
    model.train()
    for i, batch_data in enumerate(train_loader):
        loss, batch_hyp, batch_label = forward_sequence_classification(model, batch_data[:-1], i2w=EntailmentDataset.INDEX2LABEL, device='cuda')
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

This guide demonstrates how to fine-tune an IndoBERT model for the NERGrit dataset, which involves identifying PERSON, PLACE, and ORGANIZATION entities using IOB chunking. The process involves loading a pre-trained IndoBERT model, preparing the NerGritDataset using NerGritDataset and NerDataLoader, and running a training loop with forward_word_classification.

import torch
from transformers import BertConfig, BertTokenizer
from modules.word_classification import BertForWordClassification
from utils.forward_fn import forward_word_classification
from utils.data_utils import NerGritDataset, NerDataLoader

# 1. Load Tokenizer and Config
tokenizer = BertTokenizer.from_pretrained('indobenchmark/indobert-base-p1')
config = BertConfig.from_pretrained('indobenchmark/indobert-base-p1')
config.num_labels = NerGritDataset.NUM_LABELS

# 2. Instantiate model
model = BertForWordClassification.from_pretrained('indobenchmark/indobert-base-p1', config=config)

# 3. Prepare Dataset
train_dataset = NerGritDataset('./dataset/nergrit_ner-grit/train_preprocess.txt', tokenizer, lowercase=True)
train_loader = NerDataLoader(dataset=train_dataset, max_seq_len=512, batch_size=16, num_workers=16, shuffle=True)

# 4. Training Loop snippet
optimizer = torch.optim.Adam(model.parameters(), lr=2e-5)
model.cuda()

for epoch in range(8):
    model.train()
    for i, batch_data in enumerate(train_loader):
        loss, batch_hyp, batch_label = forward_word_classification(model, batch_data[:-1], i2w=NerGritDataset.INDEX2LABEL, device='cuda')
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()