BERT Model and Its Deployment on Google Colab

Natural Language Processing (NLP) has taken a giant leap with the advent of pre-trained language models. One such groundbreaking model is BERT (Bidirectional Encoder Representations from Transformers), introduced by Google. BERT has revolutionized NLP tasks like text classification, question answering, and sentiment analysis due to its ability to understand context in language. Explore the BERT model and its deployment on Google Colab, including step-by-step instructions for leveraging this powerful NLP tool for your projects. In this blog, we will explore the fundamentals of BERT and how to deploy it using Google Colab, with hands-on code examples.

 

1. Understanding the BERT Model

BERT is a transformer-based language model that reads text bi-directionally. It captures contextual relationships between words in a sentence by considering the words before and after a given word. This bidirectional approach allows BERT to understand language nuances better than unidirectional models.

Key Features of BERT:

• Bidirectionality: Unlike traditional language models, BERT reads text in both forward and backward directions, making it better at understanding context.
• Pre-trained on a large corpus: BERT is trained on vast datasets like Wikipedia and BooksCorpus, enabling it to learn deep linguistic patterns.
• Fine-tuning for various NLP tasks: With minimal effort, BERT can be fine-tuned for specific tasks, such as sentiment analysis, named entity recognition, and machine translation.

 

2. Prerequisites

Before we start deploying the BERT model on Google Colab, ensure you have:

• A basic understanding of Python and NLP.
• A Google account (to access Google Colab).
• Some familiarity with Jupyter notebooks (helpful but not mandatory).

 

3. Setting Up the Environment on Google Colab

Google Colab provides free access to GPUs, which is essential for training large models like BERT. Follow these steps to set up your environment:

1. Open Google Colab: Go to Google Colab.
2. Create a new notebook: Click on “File” > “New Notebook.”
3. Enable GPU: Go to “Runtime” > “Change runtime type” > “Hardware accelerator” > “GPU.”

 

4. Installing Necessary Libraries

We need to install the transformer library from Hugging Face, which provides a simple interface to work with BERT and other transformer models.

python

Copy code

!pip install transformers

!pip install torch

 

5. Loading the BERT Model

For this tutorial, we will use the pre-trained bert-base-uncased model for a text classification example.

python

Copy code

from transformers import BertTokenizer, BertForSequenceClassification

import torch

 

# Load the pre-trained BERT tokenizer

tokenizer = BertTokenizer.from_pretrained(‘bert-base-uncased’)

 

# Load the pre-trained BERT model for sequence classification

model = BertForSequenceClassification.from_pretrained(‘bert-base-uncased’, num_labels=2)

 

6. Preprocessing the Text Data

BERT requires input text to be tokenized in a specific way. The text must be converted into tokens that the model understands, with special tokens for padding and attention masks.

python

Copy code

# Sample text

text = “BERT is a powerful model for NLP tasks.”

 

# Tokenize the input text

inputs = tokenizer(text, padding=’max_length’, max_length=32, truncation=True, return_tensors=”pt”)

 

# Display tokenized inputs

print(inputs)

 

7. Performing Inference

We can use the loaded model to make an inference on the tokenized text.

python

Copy code

# Perform inference

outputs = model(**inputs)

logits = outputs.logits

 

# Convert logits to predicted class

predicted_class = torch.argmax(logits, dim=1).item()

print(f”Predicted class: {predicted_class}”)

 

For Free, Demo classes Call: 7507414653

Registration Link: Click Here!

 

8. Fine-Tuning BERT on a Custom Dataset

Fine-tuning BERT on a custom dataset can significantly improve its performance on specific tasks. For this example, we will fine-tune BERT for sentiment analysis using a simple dataset.

Step 1: Preparing the Dataset

Let’s create a small dataset for binary sentiment classification (positive and negative reviews).

python

Copy code

# Sample data

texts = [“I love this movie!”, “This was a terrible experience.”, “Absolutely fantastic!”, “I hated every minute.”]

labels = [1, 0, 1, 0]  # 1 for positive, 0 for negative

 

# Tokenize the data

encoded_inputs = tokenizer(texts, padding=True, truncation=True, max_length=32, return_tensors=”pt”)

 

# Convert labels to tensor

labels = torch.tensor(labels)

 

Step 2: Creating a DataLoader

The DataLoader helps to batch the data and shuffle it for training.

python

Copy code

from torch.utils.data import DataLoader, TensorDataset

 

# Create a dataset and DataLoader

dataset = TensorDataset(encoded_inputs[‘input_ids’], encoded_inputs[‘attention_mask’], labels)

dataloader = DataLoader(dataset, batch_size=2, shuffle=True)

 

Step 3: Training the Model

We will fine-tune the model using the Adam optimizer and a simple training loop.

python

Copy code

from torch.optim import AdamW

 

# Set model to training mode

model.train()

 

# Define optimizer

optimizer = AdamW(model.parameters(), lr=1e-5)

 

# Training loop

epochs = 3

for epoch in range(epochs):

    for batch in dataloader:

        input_ids, attention_mask, labels = batch

 

        # Forward pass

        outputs = model(input_ids, attention_mask=attention_mask, labels=labels)

        loss = outputs.loss

 

        # Backward pass

        optimizer.zero_grad()

        loss.backward()

        optimizer.step()

 

    print(f”Epoch {epoch+1}/{epochs}, Loss: {loss.item()}”)

 

9. Saving and Loading the Model

After fine-tuning, it is essential to save the model for later use.

python

Copy code

# Save the fine-tuned model

model.save_pretrained(‘fine-tuned-bert’)

tokenizer.save_pretrained(‘fine-tuned-bert’)

 

# To load the saved model

model = BertForSequenceClassification.from_pretrained(‘fine-tuned-bert’)

tokenizer = BertTokenizer.from_pretrained(‘fine-tuned-bert’)

 

10. Deploying the Model for Real-Time Inference

For real-time inference, we can deploy the model using web frameworks like Flask or FastAPI. However, for simplicity, here’s how you can perform inference directly on Google Colab.

python

Copy code

# Sample input for inference

new_text = “The product quality is outstanding!”

 

# Tokenize and perform inference

inputs = tokenizer(new_text, padding=’max_length’, max_length=32, truncation=True, return_tensors=”pt”)

outputs = model(**inputs)

predicted_class = torch.argmax(outputs.logits, dim=1).item()

 

print(f”Predicted class for new text: {predicted_class}”)

 

Conclusion

In this guide, we covered the basics of BERT, including loading the model, fine-tuning it on a custom dataset, and deploying it using Google Colab. BERT’s ability to understand language context bidirectionally makes it a powerful tool for various NLP tasks.

With this foundational knowledge, you can start exploring more complex use cases, such as named entity recognition, question answering, or even creating your own chatbot. The flexibility of BERT and the availability of tools like Google Colab make it accessible for beginners to start their journey in NLP.

Do visit our channel to learn more: Click Here

 

Author:-

Aniket Kulkarni

Call the Trainer and Book your free demo Class For Machine Learning Call now!!!
| SevenMentor Pvt Ltd.

© Copyright 2021 | SevenMentor Pvt Ltd.