Rise of the Modern AI Empire – Encoder – Decoder

List of Contents

1. Introduction
2. Where Encoder-Decoder Comes From
3. Encoder-Decoder
4. The Two Papers
5. Conclusion
6. References

I. Introduction

When we talk to people we know, we understand each other. When we say something in English, most students who have learned English understand the meaning. But not the students or other people who don’t understand English. Also, when a Japanese person says, “Good Morning” to us. If we know Japanese, we will understand; if not, we won’t. This is the time we need a translator. A translator is a person who understands both languages, for example, in this case, the Japanese language and the English language.  

For example, we are in Japan. And a woman or a student simply says to us, “I love you”; a translator will say to us, “彼女は「愛してる」と言っています” (Google), meaning, “she says, I love you”. But this is done by a human translator. Imagine if we were to build a system that can translate from one language to another language; this is, in Natural Language Processing (NLP), called Machine Translation (MT, abbreviated for this article).

An MT is somewhat complex in architecture but very simple in understanding. It is a system of translation using Artificial Intelligence (AI) or a computer program without human intervention. Say, a system that translates from the English language to the Japanese language. English is the source language, and Japanese is the target or destination language. But how are we going to build the architecture? The next section is the one that is used in this Machine Translation architecture.

II. Where Encoder-Decoder Comes From

In competitive examinations, like UGC NET, there are some questions about coding and decoding. Say, MENTOR is written as NFMUPS, then how is PROFESSOR written in code? This kind of question we call this as coding-decoding. But this is just an example, and the idea in an encoder-decoder architecture is conceptually similar to this.

We are going to understand the encoder-decoder architecture with the help of two research papers that were published in 2014. The first one is the “Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation” that was published in September 2014 (the month may be different; you can find another blog or source tells it’s June because the very first preprint was in June. But if you search and read the paper in 2026, the month is September as the officially published month). In this paper, the Gated Recurrent Unit and Encoder-Decoder architecture were the most important ideas that were given to the world. And the second one is the “Sequence to Sequence Learning with Neural Networks” that was published in December 2014 (the month may be different; you can find another blog or source tells it’s September because the very first preprint was in September. But if you search and read the paper in 2026, the month is December as the officially published month). The main ideas were Encoder-Decoder and Neural Machine Translation. Both papers introduced encoder-decoder, but nowadays we sometimes take the second paper as the paper of encoder-decoder and fully as the foundation paper of neural machine translation because this paper introduced an end-to-end architecture for machine translation using neural networks. Although we respect both papers as the foundation of encoder-decoder.

NB: In this article, explanations of RNN, GRU, and LSTM are not going to be included. These architectures are left for another article and course.

III. Encoder-Decoder

Before we discuss encoder-decoder, we need to understand one topic in NLP, which is called tokenization. Tokenization is a process of reducing text data into smaller pieces. These pieces are called tokens. Say we have a sentence, “Hi, how are you?” If we tokenize this into words, then the tokens will be: [“Hi”, “,”, “how”, “are”, “you”, “?”]. This is called word tokenization. These tokens are assigned a number (integers) in the vocabulary and then converted into embeddings (or vectors) because any encoder (with RNN, LSTM, and GRU; let’s not include the Transformer here yet) can’t understand the tokens or the integers.

The encoder-decoder is a neural network that was designed for seq2seq learning tasks. It is used in MT, text summarization, speech recognition, etc. Before 2017, the introduction of the Transformer, encoder-decoder was primarily based on Recurrent Neural Networks (RNNs). The encoder processes one token at a time. On the encoder side, the embeddings of the tokens are processed with the hidden state from the previous time step to produce a new hidden state. Meaning, at each time step t, the encoder receives the embedding of the current input token x_t and the hidden state from the previous time step e_t-1 to produce a new hidden state that captures both the information from the current token and the contextual information accumulated from all previously processed tokens. We can represent mathematically how the encoder updates its hidden state at every time step:

e_t = f_enc(x_t, e_t-1)
            where h_t is the hidden state at time step t; the hidden state is a zero vector or learnable parameters,
            x_t is the embedding of the current input token
            e_t-1 is the hidden state from the previous time step
            f_enc denotes the function implemented by the encoder, such as Vanilla RNN, LSTM, GRU.

After the final input has been processed, the encoder gives out a hidden state (from a sequence of hidden states, the last hidden state from the encoder), which represents the entire input tokens (input sequence) as a compact representation or fixed-sized vector that is called the “Context Vector”, c.

The decoder is used to generate the output tokens or sequence one token at a time based on the vector c. The decoder is also implemented using an RNN. Similar to the encoder, the decoder uses previously generated output and hidden state. Here, the decoder is initialized with the previous hidden state, c (context vector from the encoder), and a special initialization token called <SOS> (Start of Sequence). Using these, the decoder generates the next output token and continues until another special token, <EOS> (End of Sequence).

We can represent the hidden state of the decoder as follows:

            d_t = f_dec(y_t-1, d_t-1)

            where d_t = hidden state at time step t

                        y_t-1 = embedding of the previously generated token

                        d_t-1 = hidden state from the previous time step

                        f_dec = representation of decoder computation

The decoder applies a linear transformation to the final hidden state, d_t. This is because d_t is the feature representation that summarizes the contextual information available at the decoding step. But this does not directly correspond to any word in the target vocabulary. To correspond to the words in the target vocabulary, we need scores for the words, and this linear transformation gives them, but they are not normalized. The linear transformation, which is computed by the output layer, can be given as:

o_t = W_od_t + b_o, where o_t is the output score vector, W_o is the output weight matrix, d_t is the decoder final hidden state, and b_o is the bias vector.

Then, the Softmax() function is applied to the scores to get probabilities that are easier to interpret. The general equation can be given as:

P(y_t | y_<t, X) = Softmax(o_t); the Softmax(o_t) is the same as the probability of every possible target word at a time step t, given the input sequence X and all previously generated target tokens y_<t.

After this, the decoder selects the token with the highest probability as

y-hat_t =argmax_{y_in_Vt} P(y_t | y_<t, X), where V_t = target vocabulary.

The output token is fed to the next step as the previous state token to compute the next token. These steps are repeated or run iteratively until the decoder finds the special end-of-sequence token, denoted as <EOS>, indicating that the complete output sequence has been generated.

IV. The Two Papers

The paper “Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation” was never meant to replace Statistical Machine Translation but to improve it. It introduced a little idea about the encoder-decoder architecture we are using nowadays.

When people talk about the paper that introduced the encoder-decoder architecture, we simply say it is the paper “Sequence to Sequence Learning with Neural Networks” (after the Conclusion section and before References, we are going to add some clarification). This paper gave seq2seq learning as a practical end-to-end framework using LSTM encoder-decoder networks for complete sentence translation.

The encoder here in this paper reads one word (or token) at a time and updates its hidden state. After reading the <EOS> token, the last hidden state becomes the context vector, v. The decoder starts from the vector v and predicts the target sentence word by word until it sees <EOS>. The decoder is the same as explained in the above section. But it uses a search-like algorithm during translation generation to find the most probable output sentence based on the probabilities produced by the decoder during translation generation during inference, according to the paper. This search-like algorithm is said to be beam search. However, in the training phase, the decoder learns using ground-truth previous words (training dataset; target), and this approach is called teacher forcing.

The paper also contributed an idea that is a practical technique to improve optimization. They reverse the source sentence; for example, if the source sentence is ABCD, then they reverse it and feed it as DCBA. They also contributed that neural networks can learn meaningful sentence representations while achieving strong translation performance.

V. Conclusion

Machine translation, text summarization, etc. became easier when the encoder-decoder was introduced to the world. Say, there is an input sequence; the input sequence is tokenized, and each token is converted into an embedding. The token is fed to the encoder as the embedding. The encoder processes the input sequence token by token with a hidden state. After steps of the encoder, the final hidden state is taken as the context vector or full representation of the input sequence and fed to the decoder.

In the decoder part, decoding starts with the context vector as the initial hidden state and <SOS> as the initial output token. At every decoding step, a probability is computed, and the token with the highest probability becomes the next output token (also the previous token for another step). Until the decoder encounters <EOS>, it repeats generating the output tokens.

NB: The clarification we want to add here is that. The idea of encoder-decoder was started with “Recurrent Continuous Translation Models” by Kalchbrenner and Blunsom (2013). But we tend to explain the encoder-decoder architecture with the two papers mentioned in Section IV.

References

Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks (arXiv:1409.3215). arXiv. https://doi.org/10.48550/arXiv.1409.3215

Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder–decoder for statistical machine translation (arXiv:1406.1078). arXiv. https://doi.org/10.48550/arXiv.1406.1078

Jurafsky, D., & Martin, J. H. (2024). Speech and language processing (3rd ed. draft). Stanford University. https://web.stanford.edu/~jurafsky/slp3/