FinBERT

FinBERT can be fine-tuned for specific financial NLP tasks (like sentiment analysis) using the Huggingface transformers library. The process involves loading a custom dataset, tokenizing the text using the FinBERT tokenizer, and using the Trainer API to run the training loop.

Requirements & Environment:

• Recommended versions: transformers==4.18.0, pytorch==1.7.1.
• Use a GPU for efficient training on large datasets.

Workflow Overview:

1. Load Data: Load your dataset (e.g., via pandas) and ensure missing values in text and label columns are dropped.
2. Split Data: Use sklearn.model_selection.train_test_split to create training, validation, and testing sets.
3. Load Model & Tokenizer: Use BertForSequenceClassification.from_pretrained('yiyanghkust/finbert-pretrain', num_labels=N) and BertTokenizer.from_pretrained('yiyanghkust/finbert-pretrain').
4. Preprocess: Convert pandas DataFrames to Huggingface Dataset objects and apply tokenization with truncation=True and padding='max_length'.
5. Train: Define TrainingArguments and use the Trainer class to execute training.
6. Evaluate & Save: Use trainer.predict() on the test set and trainer.save_model() to persist the fine-tuned weights.

from transformers import BertTokenizer, Trainer, BertForSequenceClassification, TrainingArguments
from datasets import Dataset

# 1. Load pretrained model and tokenizer
model = BertForSequenceClassification.from_pretrained('yiyanghkust/finbert-pretrain', num_labels=3)
tokenizer = BertTokenizer.from_pretrained('yiyanghkust/finbert-pretrain')

# 2. Prepare dataset (assuming df_train is a pandas DataFrame)
dataset_train = Dataset.from_pandas(df_train)
dataset_train = dataset_train.map(lambda e: tokenizer(e['sentence'], truncation=True, padding='max_length', max_length=128), batched=True)
dataset_train.set_format(type='torch', columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'])

# 3. Define training arguments
args = TrainingArguments(
    output_dir = 'temp/',
    evaluation_strategy = 'epoch',
    save_strategy = 'epoch',
    learning_rate=2e-5,
    per_device_train_batch_size=32,
    num_train_epochs=5,
    load_best_model_at_end=True,
    metric_for_best_model='accuracy',
)

# 4. Initialize Trainer and train
trainer = Trainer(
    model=model,
    args=args,
    train_dataset=dataset_train,
    eval_dataset=dataset_val,
    compute_metrics=compute_metrics
)
trainer.train()

# 5. Save the model
trainer.save_model('finbert-sentiment/')

You can use the fine-tuned FinBERT-Sentiment model (hosted as yiyanghkust/finbert-tone) via the Huggingface transformers library. This model is optimized for financial tone analysis and classifies text into three labels: neutral, positive, and negative.

To use it, load BertForSequenceClassification and BertTokenizer using the yiyanghkust/finbert-tone identifier.

from transformers import BertTokenizer, BertForSequenceClassification
import numpy as np

finbert = BertForSequenceClassification.from_pretrained('yiyanghkust/finbert-tone',num_labels=3)
tokenizer = BertTokenizer.from_pretrained('yiyanghkust/finbert-tone')

sentences = ["there is a shortage of capital, and we need extra financing", 
             "growth is strong and we have plenty of liquidity", 
             "there are doubts about our finances", 
             "profits are flat"]

inputs = tokenizer(sentences, return_tensors="pt", padding=True)
outputs = finbert(**inputs)[0]

labels = {0:'neutral', 1:'positive',2:'negative'}
for idx, sent in enumerate(sentences):
    print(sent, '----', labels[np.argmax(outputs.detach().numpy()[idx])])

'''
there is a shortage of capital, and we need extra financing ---- negative
growth is strong and we have plenty of liquidity ---- positive
there are doubts about our finances ---- negative
profits are flat ---- neutral
'''

The TrainingArguments object defines the hyperparameters and strategy for the fine-tuning process.

Commonly used keys for FinBERT fine-tuning include:

• output_dir: Directory where model checkpoints and logs are stored.
• evaluation_strategy: Set to 'epoch' to evaluate at the end of every epoch.
• save_strategy: Set to 'epoch' to save checkpoints at the end of every epoch.
• learning_rate: Typically a small value like 2e-5 for fine-tuning.
• per_device_train_batch_size: Number of samples per GPU/CPU per training step.
• num_train_epochs: Total number of training passes through the dataset.
• load_best_model_at_end: If True, the model with the best score on the evaluation metric is reloaded at the end of training.
• metric_for_best_model: The metric used to determine the 'best' model (e.g., 'accuracy').

from transformers import TrainingArguments

args = TrainingArguments(
    output_dir = 'temp/',
    evaluation_strategy = 'epoch',
    save_strategy = 'epoch',
    learning_rate=2e-5,
    per_device_train_batch_size=32,
    per_device_eval_batch_size=32,
    num_train_epochs=5,
    weight_decay=0.01,
    load_best_model_at_end=True,
    metric_for_best_model='accuracy',
)

Use the yiyanghkust/finbert-tone model to classify financial text into one of three categories: Positive, Neutral, or Negative. This model is fine-tuned on analyst reports from S&P 500 firms.

Input: A financial text string. Output: Sentiment label and confidence score.

from transformers import BertTokenizer, BertForSequenceClassification, pipeline

# Load model and tokenizer
finbert = BertForSequenceClassification.from_pretrained('yiyanghkust/finbert-tone', num_labels=3)
tokenizer = BertTokenizer.from_pretrained('yiyanghkust/finbert-tone')

# Create classification pipeline
nlp = pipeline("text-classification", model=finbert, tokenizer=tokenizer)

# Run inference
results = nlp(['growth is strong and we have plenty of liquidity.', 
               'there is a shortage of capital, and we need extra financing.',
              'formulation patents might protect Vasotec to a limited extent.'])
print(results)

Use the yiyanghkust/finbert-esg model to identify Environmental, Social, or Governance themes in financial text. This is useful for assessing long-term sustainability and associated risks.

Input: A financial text string. Output: Environmental, Social, Governance, or None.

from transformers import BertTokenizer, BertForSequenceClassification, pipeline

# Load model and tokenizer
finbert = BertForSequenceClassification.from_pretrained('yiyanghkust/finbert-esg', num_labels=4)
tokenizer = BertTokenizer.from_pretrained('yiyanghkust/finbert-esg')

# Create classification pipeline
nlp = pipeline("text-classification", model=finbert, tokenizer=tokenizer)

# Run inference
results = nlp(['Managing and working to mitigate the impact our operations have on the environment is a core element of our business.',
               'Rhonda has been volunteering for several years for a variety of charitable community programs.',
               'Cabot\'s annual statements are audited annually by an independent registered public accounting firm.',
               'As of December 31, 2012, the 2011 Term Loan had a principal balance of $492.5 million.'])
print(results)

Use the yiyanghkust/finbert-fls model to identify Forward-Looking Statements (FLS) in corporate reports. This helps distinguish between statements about future events/results and general historical facts.

Input: A financial text string. Output: Specific-FLS, Non-specific FLS, or Not-FLS.

from transformers import BertTokenizer, BertForSequenceClassification, pipeline

# Load model and tokenizer
finbert = BertForSequenceClassification.from_pretrained('yiyanghkust/finbert-fls', num_labels=3)
tokenizer = BertTokenizer.from_pretrained('yiyanghkust/finbert-fls')

# Create classification pipeline
nlp = pipeline("text-classification", model=finbert, tokenizer=tokenizer)

# Run inference
results = nlp(['we expect the age of our fleet to enhance availability and reliability due to reduced downtime for repairs.',
               'on an equivalent unit of production basis, general and administrative expenses declined 24 percent from 1994 to $.67 per boe.',
               'we will continue to assess the need for a valuation allowance against deferred tax assets considering all available evidence obtained in future reporting periods.'])
print(results)

Datasets must be converted from pandas DataFrames to Hugface Dataset objects, tokenized, and formatted for PyTorch.

Tokenization Settings:

• truncation=True: Ensures sequences longer than max_length are truncated.
• padding='max_length': Pads sequences to the specified max_length.
• max_length=128: The standard length used in the example.

Formatting: After mapping the tokenizer, use .set_format(type='torch', columns=[...]) to ensure the dataset returns PyTorch tensors for the required keys: input_ids, token_type_ids, attention_mask, and label.

from datasets import Dataset

# Convert pandas to Dataset
dataset_train = Dataset.from_pandas(df_train)

# Tokenize
dataset_train = dataset_train.map(
    lambda e: tokenizer(e['sentence'], truncation=True, padding='max_length', max_length=128), 
    batched=True
)

# Set format for PyTorch
dataset_train.set_format(type='torch', columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'])