1. 基础介绍

  • BERT(来自Transformers的双向编码器表示)基于Transformer编码器进行预训练(Pre-train),从而对输入词元进行上下文表示。
  • 而在针对具体的自然语言处理任务的训练(Fine-tuning)时,对预训练Transformer编码器的所有参数进行微调,而额外的输出层将从头开始训练。
使用Pre-training的方法與時機| by Jia-Yau Shiau | 軟體之心| Medium | AI Blog TW

2. 输入表示

BERT输入序列的Embedding嵌入,表示为词元嵌入、段嵌入和位置嵌入的矩阵加和。

  • 词元嵌入:词元索引的独热编码的特征表示
  • 片段嵌入:标记来自第一个文本句,还是第二个文本句
    • 每个序列开头加是一个’<cls>‘特殊类别词元。
    • BERT输入序列可以包括1个或者2个文本句子,每个文本句后加一个’<sep>‘特殊分隔词元
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
#@save
def get_tokens_and_segments(tokens_a, tokens_b=None):
    """获取输入序列的词元及其片段索引"""
    tokens = ['<cls>'] + tokens_a + ['<sep>']
    # 0和1分别标记片段A和B
    segments = [0] * (len(tokens_a) + 2)
    if tokens_b is not None:
        tokens += tokens_b + ['<sep>']
        segments += [1] * (len(tokens_b) + 1)
    return tokens, segments

get_tokens_and_segments(["aa"],['b'])
# (['<cls>', 'aa', '<sep>', 'b', '<sep>'], [0, 0, 0, 1, 1])
  • 位置嵌入:与Transformer不同,BERT使用了可学习的位置编码

image-20240815083258457

  • 获取序列的输入表示后,就可以简单地应用到Transformer的编码器中
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
#@save
class BERTEncoder(nn.Module):
    """BERT编码器"""
    def __init__(self, vocab_size, num_hiddens, norm_shape, ffn_num_input,
                 ffn_num_hiddens, num_heads, num_layers, dropout,
                 max_len=1000, key_size=768, query_size=768, value_size=768,
                 **kwargs):
        super(BERTEncoder, self).__init__(**kwargs)
        self.token_embedding = nn.Embedding(vocab_size, num_hiddens) #词元嵌入
        self.segment_embedding = nn.Embedding(2, num_hiddens) #段嵌入
        # 在BERT中,位置嵌入是可学习的,因此我们创建一个足够长的位置嵌入参数(用多少截多少)
        self.pos_embedding = nn.Parameter(torch.randn(1, max_len, num_hiddens))
        # n个Transformer编码层
        self.blks = nn.Sequential()
        for i in range(num_layers):
            self.blks.add_module(f"{i}", d2l.EncoderBlock(
                key_size, query_size, value_size, num_hiddens, norm_shape,
                ffn_num_input, ffn_num_hiddens, num_heads, dropout, True))


    def forward(self, tokens, segments, valid_lens):
        # 在以下代码段中,X的形状保持不变:(批量大小,最大序列长度,num_hiddens)
        X = self.token_embedding(tokens) + self.segment_embedding(segments)
        X = X + self.pos_embedding.data[:, :X.shape[1], :]
        for blk in self.blks:
            X = blk(X, valid_lens)
        return X
  • 示例
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
# 实例化一个encoder
vocab_size, num_hiddens, ffn_num_hiddens, num_heads = 10000, 768, 1024, 4
norm_shape, ffn_num_input, num_layers, dropout = [768], 768, 2, 0.2
encoder = BERTEncoder(vocab_size, num_hiddens, norm_shape, ffn_num_input,
                      ffn_num_hiddens, num_heads, num_layers, dropout)

# demo输入:批量大小为2,序列长度为8
tokens = torch.randint(0, vocab_size, (2, 8))
# 每个序列的段嵌入
segments = torch.tensor([[0, 0, 0, 0, 1, 1, 1, 1], [0, 0, 0, 1, 1, 1, 1, 1]])
encoded_X = encoder(tokens, segments, None)
encoded_X.shape
# torch.Size([2, 8, 768])

3. 预训练任务

  • 上述的encoder前向推断给出了输入文本的每个词元和插入的特殊标记“<cls>”及“<\seq>”的BERT表示。
  • 基于此,来计算预训练BERT的两个损失函数,从而以自监督(无标签)训练方式,学习词元的embedding

3.1 掩蔽语言模型

Masked language modeling

BERT随机掩蔽(masked)词元,并使用来自双向上下文的词元预测掩蔽词元。

  • 这里的双向主要是指在学习特定词元的表示时,可以同时看到该词元前面和后面的序列信息。
  • 这区别于GPT模型从左到右的学习预测方式。
MLM — Sentence Transformers documentation

具体来说,在该预训练中,将随机选择15%的词元作为预测的掩蔽/掩码词元。对于每个被选中的词元,会如下处理:

  • 80%可能变为特殊的“<mask>“词元(例如,“this movie is great”变为“this movie is<\mask>”;
  • 10%时间为随机词元(噪音)(例如,“this movie is great”变为“this movie is drink”);
  • 10%时间内为不变的标签词元(例如,“this movie is great”变为“this movie is great”)。

如下定义一个MaskLM类来预测上述的掩蔽标记

  • 输入:Encoder对词元的编码表示,用于预测的词元位置
  • 输出:这些位置的预测结果
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#@save
class MaskLM(nn.Module):
    """BERT的掩蔽语言模型任务"""
    def __init__(self, vocab_size, num_hiddens, num_inputs=768, **kwargs):
        super(MaskLM, self).__init__(**kwargs)
        # 使用具有单隐藏层的MLP进行预测
        self.mlp = nn.Sequential(nn.Linear(num_inputs, num_hiddens),
                                 nn.ReLU(),
                                 nn.LayerNorm(num_hiddens),
                                 nn.Linear(num_hiddens, vocab_size))

    def forward(self, X, pred_positions):
        num_pred_positions = pred_positions.shape[1] #每个序列的masked词元数
        pred_positions = pred_positions.reshape(-1) #总共的masked词元数
        batch_size = X.shape[0]
        batch_idx = torch.arange(0, batch_size)
        # 假设batch_size=2,num_pred_positions=3
        # 那么batch_idx是np.array([0,0,0,1,1,1])
        batch_idx = torch.repeat_interleave(batch_idx, num_pred_positions)
        masked_X = X[batch_idx, pred_positions]
        masked_X = masked_X.reshape((batch_size, num_pred_positions, -1))
        mlm_Y_hat = self.mlp(masked_X)
        return mlm_Y_hat

示例

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
mlm = MaskLM(vocab_size, num_hiddens)
mlm_positions = torch.tensor([[1, 5, 2], [6, 1, 5]])
mlm_Y_hat = mlm(encoded_X, mlm_positions)
mlm_Y_hat.shape
# torch.Size([2, 3, 10000])
# 10000为词表大小

#计算与真实标签的loss
mlm_Y = torch.tensor([[7, 8, 9], [10, 20, 30]])
loss = nn.CrossEntropyLoss(reduction='none')
mlm_l = loss(mlm_Y_hat.reshape((-1, vocab_size)), mlm_Y.reshape(-1))
mlm_l.shape
# torch.Size([6])

3.2 下一步预测

Next sentence prediction

  • 当BERT的输入序列包含两个句子时,可以预测两个句子是否存在上下文逻辑关系
  • 此时,可以使用序列开头的<cls>词元嵌入表示整个序列,进行建模
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
#@save
class NextSentencePred(nn.Module):
    """BERT的下一句预测任务"""
    def __init__(self, num_inputs, **kwargs):
        super(NextSentencePred, self).__init__(**kwargs)
        self.output = nn.Linear(num_inputs, 2)

    def forward(self, X):
        # X的形状:(batchsize,num_hiddens) 每个序列的<cls>词元表示
        return self.output(X)
  • 示例
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
encoded_X = torch.flatten(encoded_X, start_dim=1)
# torch.Size([2, 6144]) 这里应该只是模拟合适的形状
# NSP的输入形状:(batchsize,num_hiddens)
nsp = NextSentencePred(encoded_X.shape[-1])
nsp_Y_hat = nsp(encoded_X)
nsp_Y_hat.shape
# torch.Size([2, 2])
# 每个样本得到两个概率预测结果

nsp_y = torch.tensor([0, 1]) #真实标签
nsp_l = loss(nsp_Y_hat, nsp_y)
nsp_l.shape
# encoded_X.shape

3.3 综合代码

基于上述定义的BERTEncoder类,以及两个预训练任务MaskLM和NextSentencePred来定义BERTModel

  • 输入的是词元相关信息
  • 输出的是编码后BERT表示encoded_X、以及两个任务的预测结果
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
#@save
class BERTModel(nn.Module):
    """BERT模型"""
    def __init__(self, vocab_size, num_hiddens, norm_shape, ffn_num_input,
                 ffn_num_hiddens, num_heads, num_layers, dropout,
                 max_len=1000, key_size=768, query_size=768, value_size=768,
                 hid_in_features=768, mlm_in_features=768,
                 nsp_in_features=768):
        super(BERTModel, self).__init__()
        self.encoder = BERTEncoder(vocab_size, num_hiddens, norm_shape,
                    ffn_num_input, ffn_num_hiddens, num_heads, num_layers,
                    dropout, max_len=max_len, key_size=key_size,
                    query_size=query_size, value_size=value_size)
        self.mlm = MaskLM(vocab_size, num_hiddens, mlm_in_features)
        self.nsp = NextSentencePred(nsp_in_features)
        # 为nsp任务添加的一层中间隐藏层, 输入为encoder的cls输出,
        # 输出是nsp的输入
        self.hidden = nn.Sequential(nn.Linear(hid_in_features, num_hiddens),
                                    nn.Tanh())
        
    def forward(self, tokens, segments, valid_lens=None,
                pred_positions=None):
        encoded_X = self.encoder(tokens, segments, valid_lens)
        if pred_positions is not None:
            mlm_Y_hat = self.mlm(encoded_X, pred_positions)
        else:
            mlm_Y_hat = None
        # 用于下一句预测的多层感知机分类器的隐藏层,0是“<cls>”标记的索引
        nsp_Y_hat = self.nsp(self.hidden(encoded_X[:, 0, :]))
        return encoded_X, mlm_Y_hat, nsp_Y_hat

4. 训练数据集

  • 示例数据WikiText-2
  • 教材中提供的下载链接失效了,在讨论区有小伙伴及时保存了,目前已下载上传到Github
  • 将该文件手动上传到../data/目录下,再解压
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
import os
import random
import torch
from d2l import torch as d2l

#@save
d2l.DATA_HUB['wikitext-2'] = (
    'https://s3.amazonaws.com/research.metamind.io/wikitext/'
    'wikitext-2-v1.zip', '3c914d17d80b1459be871a5039ac23e752a53cbe')

# 上述链接失效了,收到上传到服务器中
data_dir = '../data/wikitext-2'
ls ../data/wikitext-2
# wiki.test.tokens  wiki.train.tokens  wiki.valid.tokens
  • 预处理
    • 一行代表一个段落
    • 句号为分隔符,将一段拆成多个句子
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#@save
def _read_wiki(data_dir):
    file_name = os.path.join(data_dir, 'wiki.train.tokens')
    with open(file_name, 'r') as f:
        lines = f.readlines()
    # 大写字母转换为小写字母
    paragraphs = [line.strip().lower().split(' . ')
                  for line in lines if len(line.split(' . ')) >= 2]
    random.shuffle(paragraphs)
    return paragraphs

# 示例
# 词元化处理
paragraphs = [d2l.tokenize(
    paragraph, token='word') for paragraph in _read_wiki(data_dir)]

# 3层嵌套的list
# 第一层:段落
# 第二层:句子
# 第三层:词元
paragraphs[0][0][:3]
# ['families', 'of', '45']
# 第一段的第一句的前三个词元

4.1 NSP数据处理

  • 将两句话合并作为一个样本序列
    • 若两句话来自于同一段落的上下文,则为阳性;其余为阴性。
    • 将两种情况序列设置各占50%
  • 如下函数,50%概率返回两个连续的句子(阳性),50%的概率不会
1
2
3
4
5
6
7
8
9
#@save
def _get_next_sentence(sentence, next_sentence, paragraphs):
    if random.random() < 0.5:
        is_next = True
    else:
        # 阴性:随机选择第二个句子
        next_sentence = random.choice(random.choice(paragraphs))
        is_next = False
    return sentence, next_sentence, is_next
  • 如下函数,将对每个段落生成具有两个句子的阳性或阴性序列
    • 返回一个list,包含该段落所有的样本序列,片段嵌入,以及是否连续句子的逻辑符
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
#@save
def _get_nsp_data_from_paragraph(paragraph, paragraphs, vocab, max_len):
    nsp_data_from_paragraph = []
    for i in range(len(paragraph) - 1):
        tokens_a, tokens_b, is_next = _get_next_sentence(
            paragraph[i], paragraph[i + 1], paragraphs)
        # 考虑1个'<cls>'词元和2个'<sep>'词元
        if len(tokens_a) + len(tokens_b) + 3 > max_len:
            continue
        tokens, segments = d2l.get_tokens_and_segments(tokens_a, tokens_b)
        nsp_data_from_paragraph.append((tokens, segments, is_next))
    return nsp_data_from_paragraph

4.2 MLM数据处理

这里所说的序列就是指4.1步骤得到的合并两个句子的词元序列

  • 为每个输入序列做掩码处理,并记录被替换的位置,以及正确的标签
    • 这里的位置就是每个样本序列的词元的位置(不是索引)
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#@save
def _replace_mlm_tokens(tokens, candidate_pred_positions, num_mlm_preds,
                        vocab):
    # 为遮蔽语言模型的输入创建新的词元副本,其中输入可能包含替换的“<mask>”或随机词元
    mlm_input_tokens = [token for token in tokens]
    pred_positions_and_labels = []
    # 打乱后用于在遮蔽语言模型任务中获取15%的随机词元进行预测
    random.shuffle(candidate_pred_positions)
    for mlm_pred_position in candidate_pred_positions:
        if len(pred_positions_and_labels) >= num_mlm_preds:
            break
        masked_token = None
        # 80%的时间:将词替换为“<mask>”词元
        if random.random() < 0.8:
            masked_token = '<mask>'
        else:
            # 10%的时间:保持词不变
            if random.random() < 0.5:
                masked_token = tokens[mlm_pred_position]
            # 10%的时间:用随机词替换该词
            else:
                masked_token = random.choice(vocab.idx_to_token)
        mlm_input_tokens[mlm_pred_position] = masked_token
        pred_positions_and_labels.append(
            (mlm_pred_position, tokens[mlm_pred_position]))
    return mlm_input_tokens, pred_positions_and_labels
  • 输入:词元序列以及相应的词表
  • 输出:序列索引信息,以及待预测的位置信息以及正确的标签
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
#@save
def _get_mlm_data_from_tokens(tokens, vocab):
    candidate_pred_positions = []
    # tokens是一个字符串列表
    for i, token in enumerate(tokens):
        # 在遮蔽语言模型任务中不会预测特殊词元
        if token in ['<cls>', '<sep>']:
            continue
        candidate_pred_positions.append(i)
    # 遮蔽语言模型任务中预测15%的随机词元
    num_mlm_preds = max(1, round(len(tokens) * 0.15))
    mlm_input_tokens, pred_positions_and_labels = _replace_mlm_tokens(
        tokens, candidate_pred_positions, num_mlm_preds, vocab)
    pred_positions_and_labels = sorted(pred_positions_and_labels,
                                       key=lambda x: x[0])
    pred_positions = [v[0] for v in pred_positions_and_labels]
    mlm_pred_labels = [v[1] for v in pred_positions_and_labels]
    return vocab[mlm_input_tokens], pred_positions, vocab[mlm_pred_labels]

4.3 添加填充词元

  • 为每个序列添加填充词元’<pad>’,从而保证输入序列的长度固定,并记录有效长度
  • 同时,序列相关的其它标记信息也需要随之补充到固定长度
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
#@save
def _pad_bert_inputs(examples, max_len, vocab):
    max_num_mlm_preds = round(max_len * 0.15)
    all_token_ids, all_segments, valid_lens,  = [], [], []
    all_pred_positions, all_mlm_weights, all_mlm_labels = [], [], []
    nsp_labels = []
    for (token_ids, pred_positions, mlm_pred_label_ids, segments,
         is_next) in examples:
        all_token_ids.append(torch.tensor(token_ids + [vocab['<pad>']] * (
            max_len - len(token_ids)), dtype=torch.long))
        all_segments.append(torch.tensor(segments + [0] * (
            max_len - len(segments)), dtype=torch.long)) # 片段标记补长
        # valid_lens不包括'<pad>'的计数
        valid_lens.append(torch.tensor(len(token_ids), dtype=torch.float32))
        # 预测词元标记补长: 固定长度的15% - 有效长度的15%
        all_pred_positions.append(torch.tensor(pred_positions + [0] * (
            max_num_mlm_preds - len(pred_positions)), dtype=torch.long)) 
        # 填充词元的预测将通过乘以0权重在损失中过滤掉
        all_mlm_weights.append(
            torch.tensor([1.0] * len(mlm_pred_label_ids) + [0.0] * (
                max_num_mlm_preds - len(pred_positions)),
                dtype=torch.float32)) 
        all_mlm_labels.append(torch.tensor(mlm_pred_label_ids + [0] * (
            max_num_mlm_preds - len(mlm_pred_label_ids)), dtype=torch.long)) # 预测词元真实值,也需要补长
        nsp_labels.append(torch.tensor(is_next, dtype=torch.long))
    return (all_token_ids, all_segments, valid_lens, all_pred_positions,
            all_mlm_weights, all_mlm_labels, nsp_labels)

4.4 综合处理

  • all_token_ids: 输入序列的列表
  • all_segments: 输入序列片段标记的列表
  • valid_lens: 每个序列有效长度的列表
  • all_pred_positions, all_mlm_weights, all_mlm_labels: mlm任务中预测词元的信息
  • nsp_labels: 是否为连续句子对的逻辑符
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
#@save
class _WikiTextDataset(torch.utils.data.Dataset):
    def __init__(self, paragraphs, max_len):
        # 输入paragraphs[i]是代表段落的句子字符串列表;
        # 而输出paragraphs[i]是代表段落的句子列表,其中每个句子都是词元列表
        paragraphs = [d2l.tokenize(
            paragraph, token='word') for paragraph in paragraphs]
        sentences = [sentence for paragraph in paragraphs
                     for sentence in paragraph]
        self.vocab = d2l.Vocab(sentences, min_freq=5, reserved_tokens=[
            '<pad>', '<mask>', '<cls>', '<sep>'])
        # 获取下一句子预测任务的数据
        examples = []
        for paragraph in paragraphs:
            examples.extend(_get_nsp_data_from_paragraph(
                paragraph, paragraphs, self.vocab, max_len))
        # 获取遮蔽语言模型任务的数据
        examples = [(_get_mlm_data_from_tokens(tokens, self.vocab)
                      + (segments, is_next))
                     for tokens, segments, is_next in examples]
        # 填充输入
        (self.all_token_ids, self.all_segments, self.valid_lens,
         self.all_pred_positions, self.all_mlm_weights,
         self.all_mlm_labels, self.nsp_labels) = _pad_bert_inputs(
            examples, max_len, self.vocab)

    def __getitem__(self, idx):
        return (self.all_token_ids[idx], self.all_segments[idx],
                self.valid_lens[idx], self.all_pred_positions[idx],
                self.all_mlm_weights[idx], self.all_mlm_labels[idx],
                self.nsp_labels[idx])

    def __len__(self):
        return len(self.all_token_ids)
  • 制作成数据迭代器
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
#@save
def load_data_wiki(batch_size, max_len):
    """加载WikiText-2数据集"""
    num_workers = d2l.get_dataloader_workers()
    # data_dir = d2l.download_extract('wikitext-2', 'wikitext-2')
    data_dir = '../data/wikitext-2'
    paragraphs = _read_wiki(data_dir)
    train_set = _WikiTextDataset(paragraphs, max_len)
    train_iter = torch.utils.data.DataLoader(train_set, batch_size,
                                        shuffle=True, num_workers=num_workers)
    return train_iter, train_set.vocab

# 词元序列最大长度设置为64
batch_size, max_len = 512, 64
train_iter, vocab = load_data_wiki(batch_size, max_len)

for (tokens_X, segments_X, valid_lens_x, pred_positions_X, mlm_weights_X,
     mlm_Y, nsp_y) in train_iter:
    print(tokens_X.shape, segments_X.shape, valid_lens_x.shape,
          pred_positions_X.shape, mlm_weights_X.shape, mlm_Y.shape,
          nsp_y.shape)
    break
# torch.Size([512, 64]) torch.Size([512, 64]) torch.Size([512]) torch.Size([512, 10]) 
# torch.Size([512, 10]) torch.Size([512, 10]) torch.Size([512])

pred_positions_X[0,:]
# tensor([ 4,  6, 18, 28, 35, 37, 39, 49,  0,  0])
mlm_Y[0,:]
# tensor([3003,  443,  239, 1792,   48,    6,  814,   18,    0,    0])

5. 预训练BERT

5.1 预训练BERT

1
2
3
import torch
from torch import nn
from d2l import torch as d2l
  • 实例化模型
1
2
3
4
5
6
7
8
batch_size, max_len = 512, 64
train_iter, vocab = d2l.load_data_wiki(batch_size, max_len)

net = d2l.BERTModel(len(vocab), num_hiddens=128, norm_shape=[128],
                    ffn_num_input=128, ffn_num_hiddens=256, num_heads=2,
                    num_layers=2, dropout=0.2, key_size=128, query_size=128,
                    value_size=128, hid_in_features=128, mlm_in_features=128,
                    nsp_in_features=128)
  • 损失函数
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
loss = nn.CrossEntropyLoss()

#@save
def _get_batch_loss_bert(net, loss, vocab_size, tokens_X,
                         segments_X, valid_lens_x,
                         pred_positions_X, mlm_weights_X,
                         mlm_Y, nsp_y):
    # 前向传播
    _, mlm_Y_hat, nsp_Y_hat = net(tokens_X, segments_X,
                                  valid_lens_x.reshape(-1),
                                  pred_positions_X)
    # 计算遮蔽语言模型损失
    mlm_l = loss(mlm_Y_hat.reshape(-1, vocab_size), mlm_Y.reshape(-1)) *\
    mlm_weights_X.reshape(-1, 1)
    mlm_l = mlm_l.sum() / (mlm_weights_X.sum() + 1e-8)
    # 计算下一句子预测任务的损失
    nsp_l = loss(nsp_Y_hat, nsp_y)
    l = mlm_l + nsp_l
    return mlm_l, nsp_l, l
  • GPU训练模型
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
devices = d2l.try_all_gpus()

def train_bert(train_iter, net, loss, vocab_size, devices, num_steps):
    net = nn.DataParallel(net, device_ids=devices).to(devices[0])
    trainer = torch.optim.Adam(net.parameters(), lr=0.01)
    step, timer = 0, d2l.Timer()
    animator = d2l.Animator(xlabel='step', ylabel='loss',
                            xlim=[1, num_steps], legend=['mlm', 'nsp'])
    # 遮蔽语言模型损失的和,下一句预测任务损失的和,句子对的数量,计数
    metric = d2l.Accumulator(4)
    num_steps_reached = False
    while step < num_steps and not num_steps_reached:
        for tokens_X, segments_X, valid_lens_x, pred_positions_X,\
            mlm_weights_X, mlm_Y, nsp_y in train_iter:
            tokens_X = tokens_X.to(devices[0])
            segments_X = segments_X.to(devices[0])
            valid_lens_x = valid_lens_x.to(devices[0])
            pred_positions_X = pred_positions_X.to(devices[0])
            mlm_weights_X = mlm_weights_X.to(devices[0])
            mlm_Y, nsp_y = mlm_Y.to(devices[0]), nsp_y.to(devices[0])
            trainer.zero_grad()
            timer.start()
            mlm_l, nsp_l, l = _get_batch_loss_bert(
                net, loss, vocab_size, tokens_X, segments_X, valid_lens_x,
                pred_positions_X, mlm_weights_X, mlm_Y, nsp_y)
            l.backward()
            trainer.step()
            metric.add(mlm_l, nsp_l, tokens_X.shape[0], 1)
            timer.stop()
            animator.add(step + 1,
                         (metric[0] / metric[3], metric[1] / metric[3]))
            step += 1
            if step == num_steps:
                num_steps_reached = True
                break

    print(f'MLM loss {metric[0] / metric[3]:.3f}, '
          f'NSP loss {metric[1] / metric[3]:.3f}')
    print(f'{metric[2] / timer.sum():.1f} sentence pairs/sec on '
          f'{str(devices)}')
    
# 结果
train_bert(train_iter, net, loss, len(vocab), devices, 50)
# MLM loss 5.816, NSP loss 0.757
# 6688.3 sentence pairs/sec on [device(type='cuda', index=0), device(type='cuda', index=1)]

5.2 用BERT表示文本

  • 给定输入句子,经BERT编码后,返回每个词元的嵌入表示
1
2
3
4
5
6
7
def get_bert_encoding(net, tokens_a, tokens_b=None):
    tokens, segments = d2l.get_tokens_and_segments(tokens_a, tokens_b)
    token_ids = torch.tensor(vocab[tokens], device=devices[0]).unsqueeze(0)
    segments = torch.tensor(segments, device=devices[0]).unsqueeze(0)
    valid_len = torch.tensor(len(tokens), device=devices[0]).unsqueeze(0)
    encoded_X, _, _ = net(token_ids, segments, valid_len)
    return encoded_X
  • 句子1:a crane is flying, 加上头的<cls>以及尾的<sep>,共有6个词元
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
tokens_a = ['a', 'crane', 'is', 'flying']
encoded_text = get_bert_encoding(net, tokens_a)
# 词元:'<cls>','a','crane','is','flying','<sep>'
encoded_text.shape
# torch.Size([1, 6, 128])

encoded_text_cls = encoded_text[:, 0, :]
encoded_text_crane = encoded_text[:, 2, :]
encoded_text_cls.shape
# torch.Size([1, 128])
encoded_text_crane[0][:3]
# tensor([-0.3421, -0.6974, -1.9238], device='cuda:0', grad_fn=<SliceBackward0>)
  • 句子2: a crane driver came
1
2
3
4
5
tokens_a2 = ['a', 'crane', 'driver', 'came']
encoded_text2 = get_bert_encoding(net, tokens_a2)
encoded_text_crane2 = encoded_text2[:, 2, :]
encoded_text_crane2[0][:3]
# tensor([-0.3659, -0.6927, -1.8660], device='cuda:0', grad_fn=<SliceBackward0>)

6. 微调BERT

6.1 SNLI数据集

  • 一对文本序列(前提与假设)通常可认为存在3种逻辑关系:
    • (1)蕴含 entailment:前提可以推出假设
    • (2)矛盾contradiction:二者逻辑互相矛盾
    • (3)中性neutral:无法确定
  • 斯坦福自然语言推断(Stanford natural language inferrence, SNLI)数据集包含了50多万个带标签的英语句子组成的集合。
1
2
3
4
5
6
7
8
9
# #@save
# d2l.DATA_HUB['SNLI'] = (
#     'https://nlp.stanford.edu/projects/snli/snli_1.0.zip',
#     '9fcde07509c7e87ec61c640c1b2753d9041758e4')

# data_dir = d2l.download_extract('SNLI')

# 下载snli_1.0.zip到 ../data,解压后,重命名为SNLI
data_dir = '../data/SNLI'
  • 定义如下代码,读取该数据集
    • 分为训练集与测试集,前者约有55万对;后者约有1万对
    • 每个数据集返回a list of 3 list,每个子list分别包括所有的前提序列,假设序列,以及二者的标签。
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
#@save
def read_snli(data_dir, is_train):
    """将SNLI数据集解析为前提、假设和标签"""
    def extract_text(s):
        # 删除我们不会使用的信息
        s = re.sub('\\(', '', s)
        s = re.sub('\\)', '', s)
        # 用一个空格替换两个或多个连续的空格
        s = re.sub('\\s{2,}', ' ', s)
        return s.strip()
    label_set = {'entailment': 0, 'contradiction': 1, 'neutral': 2}
    file_name = os.path.join(data_dir, 'snli_1.0_train.txt'
                             if is_train else 'snli_1.0_test.txt')
    with open(file_name, 'r') as f:
        rows = [row.split('\t') for row in f.readlines()[1:]]
    premises = [extract_text(row[1]) for row in rows if row[0] in label_set]
    hypotheses = [extract_text(row[2]) for row in rows if row[0] \
                in label_set]
    labels = [label_set[row[0]] for row in rows if row[0] in label_set]
    return premises, hypotheses, labels


train_data = read_snli(data_dir, is_train=True)
len(train_data), len(train_data[0])
# (3, 549367)

for x0, x1, y in zip(train_data[0][:3], train_data[1][:3], train_data[2][:3]):
    print('前提:', x0)
    print('假设:', x1)
    print('标签:', y)
# 前提: A person on a horse jumps over a broken down airplane .
# 假设: A person is training his horse for a competition .
# 标签: 2
# 前提: A person on a horse jumps over a broken down airplane .
# 假设: A person is at a diner , ordering an omelette .
# 标签: 1
# 前提: A person on a horse jumps over a broken down airplane .
# 假设: A person is outdoors , on a horse .
# 标签: 0

6.2 加载预训练模型

1
2
3
4
5
6
import json
import multiprocessing
import os
import torch
from torch import nn
from d2l import torch as d2l
  • 教材提供了两个预训练好的bert模型,分别是base和small两个版本
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# d2l.DATA_HUB['bert.base'] = (d2l.DATA_URL + 'bert.base.torch.zip',
#                              '225d66f04cae318b841a13d32af3acc165f253ac')
# d2l.DATA_HUB['bert.small'] = (d2l.DATA_URL + 'bert.small.torch.zip',
#                               'c72329e68a732bef0452e4b96a1c341c8910f81f')

def load_pretrained_model(pretrained_model, num_hiddens, ffn_num_hiddens,
                          num_heads, num_layers, dropout, max_len, devices):
    # data_dir = d2l.download_extract(pretrained_model)
    # http://d2l-data.s3-accelerate.amazonaws.com/bert.small.torch.zip
    data_dir = '../data/bert.small'
    # 定义空词表以加载预定义词表
    vocab = d2l.Vocab()
    vocab.idx_to_token = json.load(open(os.path.join(data_dir,
        'vocab.json')))
    vocab.token_to_idx = {token: idx for idx, token in enumerate(
        vocab.idx_to_token)}
    bert = d2l.BERTModel(len(vocab), num_hiddens, norm_shape=[256],
                         ffn_num_input=256, ffn_num_hiddens=ffn_num_hiddens,
                         num_heads=4, num_layers=2, dropout=0.2,
                         max_len=max_len, key_size=256, query_size=256,
                         value_size=256, hid_in_features=256,
                         mlm_in_features=256, nsp_in_features=256)
    # 加载预训练BERT参数
    bert.load_state_dict(torch.load(os.path.join(data_dir,
                                                 'pretrained.params')))
    return bert, vocab

6.3 数据预处理

  • 根据6.1 ,合并一对文本序列为一个完整的输入序列;以及相应的片段标记,有效长度。
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
class SNLIBERTDataset(torch.utils.data.Dataset):
    def __init__(self, dataset, max_len, vocab=None):
        all_premise_hypothesis_tokens = [[
            p_tokens, h_tokens] for p_tokens, h_tokens in zip(
            *[d2l.tokenize([s.lower() for s in sentences])
              for sentences in dataset[:2]])]

        self.labels = torch.tensor(dataset[2])
        self.vocab = vocab
        self.max_len = max_len
        (self.all_token_ids, self.all_segments,
         self.valid_lens) = self._preprocess(all_premise_hypothesis_tokens)
        print('read ' + str(len(self.all_token_ids)) + ' examples')
        
        
	# 定义_mp_worker,多线程预处理,
    def _preprocess(self, all_premise_hypothesis_tokens):
        pool = multiprocessing.Pool(4)  # 使用4个进程
        out = pool.map(self._mp_worker, all_premise_hypothesis_tokens)
        all_token_ids = [
            token_ids for token_ids, segments, valid_len in out]
        all_segments = [segments for token_ids, segments, valid_len in out]
        valid_lens = [valid_len for token_ids, segments, valid_len in out]
        return (torch.tensor(all_token_ids, dtype=torch.long),
                torch.tensor(all_segments, dtype=torch.long),
                torch.tensor(valid_lens))

        
    # 每个线程的处理细节
    def _mp_worker(self, premise_hypothesis_tokens):
        p_tokens, h_tokens = premise_hypothesis_tokens
        self._truncate_pair_of_tokens(p_tokens, h_tokens)
        tokens, segments = d2l.get_tokens_and_segments(p_tokens, h_tokens)
        token_ids = self.vocab[tokens] + [self.vocab['<pad>']] \
                             * (self.max_len - len(tokens))
        segments = segments + [0] * (self.max_len - len(segments))
        valid_len = len(tokens)
        return token_ids, segments, valid_len

    # 保证原始的p_tokens加上h_tokens的词元数小于最大长度
    def _truncate_pair_of_tokens(self, p_tokens, h_tokens):
        # 为BERT输入中的'<CLS>'、'<SEP>'和'<SEP>'词元保留位置
        while len(p_tokens) + len(h_tokens) > self.max_len - 3:
            if len(p_tokens) > len(h_tokens):
                p_tokens.pop()
            else:
                h_tokens.pop()

    def __getitem__(self, idx):
        return (self.all_token_ids[idx], self.all_segments[idx],
                self.valid_lens[idx]), self.labels[idx]

    def __len__(self):
        return len(self.all_token_ids)
  • 小批量加载数据
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# 如果出现显存不足错误,请减少“batch_size”。在原始的BERT模型中,max_len=512
batch_size, max_len, num_workers = 512, 128, d2l.get_dataloader_workers()
data_dir = d2l.download_extract('SNLI')
train_set = SNLIBERTDataset(d2l.read_snli(data_dir, True), max_len, vocab)
test_set = SNLIBERTDataset(d2l.read_snli(data_dir, False), max_len, vocab)
train_iter = torch.utils.data.DataLoader(train_set, batch_size, shuffle=True,
                                   num_workers=num_workers)
test_iter = torch.utils.data.DataLoader(test_set, batch_size,
                                  num_workers=num_workers)
# read 549367 examples
# read 9824 examples

6.4 微调BERT

  • 在原有的基础上,添加额外的MLP。该MLP由两个全连接层组成(如下的self.hidden与self.output)
  • 基于bert输出中,对于输入序列的<cls>词元的嵌入表示进行微调、预测。

(1)定义前向传播,实例化模型

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
class BERTClassifier(nn.Module):
    def __init__(self, bert):
        super(BERTClassifier, self).__init__()
        self.encoder = bert.encoder
        self.hidden = bert.hidden
        self.output = nn.Linear(256, 3)

    def forward(self, inputs):
        tokens_X, segments_X, valid_lens_x = inputs
        encoded_X = self.encoder(tokens_X, segments_X, valid_lens_x)
        return self.output(self.hidden(encoded_X[:, 0, :]))
    
net = BERTClassifier(bert)

MaskLM类和NextSentencePred类这两个损失函数与微调下游应用无关,因此当BERT微调时,MaskLMNextSentencePred中采用的多层感知机的参数不会更新(陈旧的,staled)。

(2)训练微调模型

1
2
3
4
5
6
lr, num_epochs = 1e-4, 5
trainer = torch.optim.Adam(net.parameters(), lr=lr)
loss = nn.CrossEntropyLoss(reduction='none')
d2l.train_ch13(net, train_iter, test_iter, loss, trainer, num_epochs, devices)
# loss 0.519, train acc 0.791, test acc 0.783
# 7622.9 examples/sec on [device(type='cuda', index=0), device(type='cuda', index=1)]

../_images/output_natural-language-inference-bert_1857e6_102_1.svg