BERT Model - NLP

BERT (Bidirectional Encoder Representations from Transformers) is a Transformer-based language model developed by Google for understanding the meaning and context of words in text. It uses information from both the left and right context of a token to generate contextual representations.

Working

1. Input Processing

BERT first divides the input text into subword tokens using WordPiece tokenization. It also adds special tokens to indicate the structure of the input.

• [CLS] is added at the beginning of the input. Its final representation can be used for tasks such as text classification.
• [SEP] marks the end of a sentence or separates two sentences.

For example, a single sentence can be represented as:

[CLS] The cat is sleeping. [SEP]

For a pair of sentences:

[CLS] The cat is sleeping. [SEP] It is on the mat. [SEP]

The tokens are then converted into numerical representations and passed through BERT's Transformer encoder.

2. Masked Language Modeling (MLM)

• During pre-training, BERT selects some input tokens as prediction targets and learns to predict their original values using the surrounding context.

For example:

Input: The cat is [MASK] on the mat.
Target: The cat is sleeping on the mat.

• Most selected tokens are replaced with [MASK], while some are replaced with a random token or kept unchanged.
• Because BERT uses information from both directions, it can use the words before and after the selected token to make its prediction.

3. Next Sentence Prediction (NSP)

• BERT is also trained to understand relationships between pairs of sentences.
• It receives two sentences and predicts whether the second sentence follows the first sentence in the training text.

For example:

Sentence A: The student went to the library.
Sentence B: She borrowed a book.

• If the two sentences appeared consecutively in the training text, they form a positive pair.
• For a negative pair, the second sentence is taken from elsewhere in the training text.
• The model predicts whether the pair is IsNext or NotNext.

4. Combined Pre-training

• BERT is pretrained using both MLM and NSP. The model learns to predict selected tokens using their context while also learning whether two sentences form a consecutive pair.
• These objectives allow BERT to learn general language representations before it is adapted to specific NLP tasks.

5. Fine-tuning

• After pre-training, BERT can be adapted to a specific NLP task. These tasks are called downstream tasks and include text classification, Named Entity Recognition (NER) and question answering.
• A task-specific output layer is added to BERT and the model is fine-tuned using labeled data for that task.

For example:

For sentiment analysis, BERT can be fine-tuned to classify a sentence as positive or negative.

• The same pretrained BERT model can therefore be adapted to different downstream tasks without training a language model from scratch.

Architecture

Transformer Encoder

BERT consists of multiple Transformer encoder layers. Each layer contains:

• Multi-Head Self-Attention: Allows each token to attend to other tokens in the input sequence and capture contextual relationships.
• Feed-Forward Network: Applies further transformations to the representations produced by the attention layer.
• Residual Connections: Add the input of a sub-layer to its output to help information flow through the network.
• Layer Normalization: Normalizes the representations to improve training stability.

Multiple encoder layers are stacked to produce increasingly contextual representations.

Input Representation

BERT combines three types of embeddings for each input token:

• Token Embeddings: Represent the individual tokens.
• Segment Embeddings: Indicate which sentence a token belongs to when a pair of sentences is provided.
• Position Embeddings: Provide information about the position of each token in the sequence.

These embeddings are combined and passed through the Transformer encoder.

BERTBASE and BERTLARGE

BERT was introduced in two configurations:

• BERTBASE contains 12 Transformer encoder layers, 12 attention heads and a hidden size of 768, with about 110 million parameters.
• BERTLARGE contains 24 encoder layers, 16 attention heads and a hidden size of 1024, with about 340 million parameters.

Both configurations support input sequences of up to 512 tokens.

Output Representations

BERT produces a contextual representation for each input token. Unlike static word embeddings, the representation of a word can change depending on the words surrounding it.

For example, the representation of bank can differ in:

I deposited money in the bank.

and

We sat near the river bank.

The output corresponding to the [CLS] token can be used for sequence-level classification, while token-level outputs can be used for tasks such as Named Entity Recognition.

BERT Tokenization

BERT uses WordPiece tokenization to divide text into subword tokens. This allows the model to represent words using smaller subword units when the complete word is not present in the vocabulary.

For example:

A word can be divided into multiple subword tokens, where ## indicates a continuation subword in BERT's WordPiece vocabulary.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
text = "BERT understands language context."
tokens = tokenizer.tokenize(text)

print("Tokens:", tokens)

Output:

Tokens: ['B', '##ER', '##T', 'understands', 'language', 'context', '.']

Special tokens such as [CLS] and [SEP] are also added when required by the model.

Implementation

The Hugging Face Transformers library provides pretrained BERT models and tokenizers. The following example uses BERT to generate contextual embeddings for an input sentence.

Step 1: Install Required Libraries

!pip install transformers torch

• transformers provides pretrained BERT models and tokenizers.
• torch provides the tensor operations required to run the model.

Step 2: Import the Required Libraries

from transformers import BertTokenizer, BertModel
import torch

Step 3: Load the Pretrained BERT Model and Tokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
model = BertModel.from_pretrained("bert-base-cased")

The tokenizer converts text into BERT-compatible inputs, while BertModel loads the pretrained Transformer encoder.

Step 4: Tokenize the Input Text

text = "BERT understands the context of words."

inputs = tokenizer(
    text,
    return_tensors="pt"
)

The tokenizer converts the text into token IDs and creates the required input tensors. It also adds special tokens such as [CLS] and [SEP].

Step 5: Generate Contextual Embeddings

with torch.no_grad():
    outputs = model(**inputs)

embeddings = outputs.last_hidden_state

last_hidden_state contains the final contextual representation for each input token.

Step 6: Display the Output Shape

print("Embedding Shape:", embeddings.shape)

Output:

Embedding Shape: torch.Size([1, 11, 768])

The three dimensions represent:

• 1: Number of input sequences
• 9: Number of tokens, including [CLS] and [SEP]
• 768: Hidden size of BERTBASE

Thus, each input token is represented by a 768-dimensional contextual vector.

You can download the complete source code from here.

Variants

BERT inspired several later models that modify its architecture, training process or efficiency.

• RoBERTa: Modifies BERT's pre-training approach and removes the Next Sentence Prediction objective.
• DistilBERT: A smaller and faster model created by distilling knowledge from BERT.
• ALBERT: Reduces the number of parameters through techniques such as parameter sharing and factorized embeddings.
• DeBERTa: Introduces changes to attention and position representations for contextual modeling.

Applications

• Text Classification: Classifies text into categories such as sentiment, spam, topic or intent.
• Named Entity Recognition (NER): Identifies entities such as people, organizations, locations and dates in text.
• Question Answering: Identifies answer spans from a given passage by predicting the start and end positions of the answer.
• Semantic Similarity: Compares the meaning of sentences for tasks such as semantic matching and duplicate detection.
• Information Retrieval: Helps search systems understand the contextual meaning of queries and documents to retrieve relevant information.

Limitations

• Limited Sequence Length: BERT supports input sequences of up to 512 tokens, so longer texts need to be truncated or processed in smaller parts.
• High Computational Requirements: Larger BERT variants require considerable memory and computational resources during training and inference.
• Not Designed for Text Generation: BERT is an encoder-only model designed mainly for language understanding rather than free-form text generation.
• Requires Task-Specific Fine-Tuning: BERT generally needs labeled data and fine-tuning to perform well on a specific downstream task.
• Training Data Bias: BERT can inherit biases and limitations present in the data used to pretrain the model.

BERT vs. GPT