BERT: Pre-training of Deep Bidirectional Transformers

Hello, now lets look at BERT. As we’ve seen earlier, the original transformer was designed as encoder decoder where the encoder takes a text and understands it by doing self attention between the tokens and then pass it through FFN to convert it to a logit. Additionally, the decoder will input the token t-1 and generate a token t recursively. Last time, we’ve seen a decoder only arc when we’re building LLaMA-2, now lets see an encoder only arc. The additional thing is, the decoder only arcs, use only the left context(as we’ve seen masking the right triangular masking) but here since we’re trying to understand what this token is and how it relates with other words in the context, we use both the left and the right one.

Since BERT is designed for tasks requiring language understanding like sentiment analysis classification, QA answering and summarization which none of them require the decoder so we drop the decoder and stack encoders to get the BERT and then stack it with fine-tuning for specific task. We’re stacking 12 encoders if we’re training BASE and 24 if we’re training Large.

We first pretrain on two forms of a task. Masked Language Modeling and Next Sentence Prediction.

During Masked Language Modeling: We take sentences containing a masked token and we’ve to predict what’s that token depending on the context. This masking even though for larger portion of the percentage we use tokens like “MASKED”, some times we’ll also change it with random tokens and sometimes with the original token.

Like lets say we’re predicting “The quick brown fox jumped over the lazy dog”. here we’re masking the quick. 80% of the time we can use the word “Masked” to let our model know this is the one token you need to focus on. 10% of the time we can replace it with random token as “Japan” or whatever and the rest of the time we show it the real answer which is “quick” in our case. While we’re computing the self attention here we’ll notice that the quick will have higher values which give our model a notion that this is a correct token.

During Next Sentence Prediction: We take sentences and we predict whether one follows another, which follows which actions.

Note that these are not sequential and they happen simultaneously. Lets use the arc image from the paper to explain. We’re taking two sentences with masked tokens, they’ve used WordPiece tokenizer in total creating a vocabulary size of 30,000 tokens which also allow sub-word tokens and these will be turned to vector embeddings of size 768 or 1024 depending whether you’re training the Base or Large. Once it passes through our encoder layers it will give us two forms of output. First in the “C” part we get a binary output of 1 if sentence B follows A and 0 if not. And in the Masked LM part we get a embedding vectors containing our prediction too. Note that the embedding dimensions will have same size of the input.

Here the segment embedding is a learned embedding to every token indicating whether it belongs to sentence A or sentence B. Note that, on the input token, you see two special tokens called [cls] and [sep], which state starting of the first sentence by [cls] and [sep] bound the second sentence.

The positional embeddings are absolute embeddings of fixed size of 512 and they’re learned during training, unlike the original transformer which calculate them through sinusoidal and fix them. Note that the size 512 tells you that BERT can only handle sentences having a max size of 512 tokens.

As you might’ve expected in neural world all roads take you to loss optimization, so lets see how we do this for BERT. We take our embedding containing the predicted token values and we pass it through a Feed Forward Layer to change our embedding vector to a probability distribution. Then we compare it with unmasked version and we compute cross entropy loss. Note that we will be calculating loss for the masked tokens only ignoring the unmasked ones. During this process, the embedding vectors will change in a way that better represent the context helping for predicting the masked token.

Now you’ve a language model which understand language, then you can perform supervised learning for your task. Whallla!

References

Original Paper: Devlin, Jacob, et al. “Bert: Pre-training of deep bidirectional transformers for language understanding.” Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 2019.