(From Seq2Seq processing to GenAI)
(The list of articles to be published under this main article is shown below)
List of Sections.
- Introduction
- Traditional AI
- Machine Learning
- Deep Learning Changes Everything
- Encoder-Decoder
- The Attention Mechanism
- The Transformer: Foundation of Modern NLP and Modern GenAI
- Rise of Large Language Models (LLMs)
- The Current AI Era
- Conclusion
- References
I. Introduction
When people hear the word “Artificial Intelligence” (AI), they think about Elon Musk, robots, self-driving cars, etc. Yes, they are right. Everyone (almost) has a Facebook account, and he/she always finds similar videos to what he/she watched a few minutes ago; that’s Facebook’s recommendation algorithm, and, more importantly, that algorithm is an application of AI. Also, when we search for something like a topic, a story, news, etc. on Google (most people use Google), we find something called ‘AI Overview’. This is an AI system from Google that allows people to access information very quickly. So, in our everyday life, we can say AI is everywhere.
We also hear about ChatGPT, Meta AI, Gemini, Claude, etc.; they are AI systems that can help us in general problem-solving, coding, research, etc. Using Meta AI, we can create a short video; using ChatGPT, we can create an image, etc. There are even voice assistants from Amazon, Apple, etc. to speak to and to speak with.
All of the above examples and considerations, they didn’t become possible overnight. From the 1950s to the 2010s, the events that happened in this era made them possible. ChatGPT, Gemini, etc. weren’t there. However, since 2017, the landscape of AI has changed.
Here is my definition of AI from myself (not from books or any website).
“Artificial Intelligence is the Intelligence created by human beings with the help of computer science, programming, and Mathematics to imitate human intelligence”.
II. Traditional AI
In military boot camp, people wake up, eat, drill, practice, etc. at some specific time. People follow some rules to follow. In boarding school, students wake up early, exercise, study, etc. Students follow some rules made by the school. In prison, people have to follow rules set by the government.
Traditional AI is somewhat similar to this. Say, in military boot camp, the military and the government set the rules, and the people in there have to follow them. In traditional AI, it is more like “if-then” rules. This is because explicit programmed rules, pre-defined knowledge, and logic are there in traditional AI, making complex problems harder to solve. The programmers need to manually code from scratch and set rules. So, we sometimes call this rule-based AI.
In traditional AI, if we don’t mention or discuss briefly a topic called ‘Expert System’, then it is incomplete. So, an expert system is a knowledge-based system that requires human experts to solve problems.
III. Machine Learning
The foundations of Machine Learning (ML) can be traced back to the 1950s. In 1950, Alan Turing (a British mathematician and computer scientist) published Computing Machinery and Intelligence, introducing a test called the Turing test in which he checked if a machine could exhibit human-like intelligent behavior. Although the test is not an ML algorithm, it inspired many researchers to explore ways of creating intelligent machines. ML emerged as a branch of AI that enables computers to learn patterns from data and improve their performance without being explicitly programmed.
From this learning from data, we sometimes call this data-driven learning. Algorithms like kNN, SVM, Bayesian ones, etc. need data to learn the patterns and make predictions. The prediction doesn’t rely on hand-coded or human-specified domain knowledge. The prediction relies on the history and data.
In the Gmail app (most people use Gmail nowadays), we can see the ‘Spam’ folder, and the emails that are not spam will be there in the inbox, but not the Spam ones. Google trained the ML model(s) to perform like this based on data.
V. Deep Learning Changes Everything
Many complex problems cannot be solved with ML algorithms. But they can be solved using Deep Learning (DL) architectures. Here, DL is a subset of ML. Meaning, any problems that can be solved by DL can also be solved by ML. However, problems that involve highly unstructured data, high dimensions, etc. cannot be solved by classical ML algorithms.
In our brains, there is a fundamental unit of the nervous system called a neuron. This unit inspired researchers to introduce artificial neurons in Neural Networks (NNs). NNs are networks of interconnected artificial neurons that are organized in layers. These networks act similarly to human brains to process data, recognize patterns, make decisions, etc., but without explicit programming.
After McCulloch-Pitts introduced a neuron in 1943 that was regarded as the first artificial neuron, in 1957–9158, Frank Rosenblatt introduced the perceptron, the first ever trainable Artificial Neural Network (ANN), which was mainly for binary classification. It had many limitations, but it was and still is one of the foundations of DL.
“Deep Learning is a subset of Machine Learning, but it uses Artificial Neural Networks (ANNs) with multiple hidden layers, layers of neurons, enabling them to learn complex patterns from data”.
DL architectures use a number of layers of artificial neurons to learn. So, more layers, more learning capacity. DL solves many complex problems in one of the easiest ways. It can generate images or videos; it can solve many vision problems in a faster way, and also, even in language, DL can help a lot in solving complex problems.
While DL architectures have proven highly effective, they face significant limitations when applied to Natural Language Processing (NLP). Traditional sequential models — Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) — process text data strictly chronologically. This sequential nature hinders their ability to capture long-range contextual relationships effectively. Furthermore, these architectures suffer from several inherent bottlenecks, including vanishing and exploding gradients.
V. Encoder-Decoder
In cryptography, there is an encryption-decryption mechanism. But the encoder-decoder is not the same as encryption-decryption in cryptography. In encryption, the message is encrypted using some methods to protect it from attackers, and in decryption, the encrypted message is decrypted with some keys (shared keys, not in full discussion). The idea is similar but not the same in encoder-decoder.
Before 2017, in 2014, Sutskever et al. published a paper called “Sequence to Sequence Learning with Neural Networks”. This paper introduced the encoder-decoder idea with a multilayered LSTM.
NB: In this article, I am not going to explain in depth about encoder-decoder, but will explain in Article 1.
The encoder-decoder architecture is based on NLP. Before encoding, we need to convert the raw text data into tokens (the smallest unit), and each token is mapped to numerical forms because DL architectures don’t understand raw text data. The numerical form is read or processed by the encoder one by one sequentially, and with the previous hidden state, it combines the current embedding so as not to lose the context. The final encoded output is the final hidden state, which is called the context vector, a compressed summary of the input sequence that is fixed-size.
The context vector is then passed to the decoder, the context vector as input to the decoder and is used by the decoder as its initial hidden state. The decoder is autoregressive, meaning it generates the output token sequence one by one (one at a time) but uses the previous ones as input to produce the next one. It stops producing output tokens when a special token <EOS> or its predefined length is met. Some important tasks in NLP — machine translation, text summarization — become more effective with the encoder-decoder architecture.
VI. The Attention Mechanism
Before the self-attention in the Transformer, in 2014, a paper, “Neural Machine Translation by Jointly Learning to Align and Translate” by Bahdanau et al., was published, and this was also published at a conference in 2015. In the paper, the limitation of the encoder-decoder architecture was resolved.
The encoder-decoder architecture uses one fixed-length vector to process the task. But it loses information when the sentence is long. The decoder is forced to remember the whole sentence from one fixed-size context vector. But ‘why not let the decoder look back at the encoder whenever it needs information?’ This question gives the “Attention Mechanism”.
Instead of going with a single context vector to the decoder, Bahdanau introduced something to keep every hidden state in the encoder. So that the decoder can use the important information about a different part of the sentence.
In the decoder, the output tokens are generated one at a time. When an output token is generated, the decoder compares its current state with the hidden states of the encoder and assigns an attention score (important scores). These scores are then converted into attention weights and combined with all encoder hidden states using those weights to produce a context vector.
For generating the second or another output token (or simply a word in this case), the decoder uses the same mechanism and produces another new context vector. For every output token, there is a separate context vector that is independent of each other.
Another notable experiment Bahdanau et al. used in this paper is that they used a bidirectional RNN in the encoder because the hidden representation contains information from both directions. This gives the decoder contextual information.
VII. The Transformer: Foundation of Modern NLP and Modern GenAI
In 2014, some complex NLP tasks like machine translation could be solved with the seq2seq architecture, which was introduced as an encoder-decoder architecture. This architecture uses a fixed-size context vector, resulting in loss of information. Again, in this year (as a preprint and later published at a conference in 2015), a paper was published, and this paper solved the problem faced in the previous paper with the “Attention Mechanism,” as briefly explained in Section VI. This gives dynamic context vector(s) and better translation. It also has another problem, meaning (for example) word-2 is computed only after the computation of word-1. In other words, computation here is also sequential, and it forgets information across long distances. This is called the “long-range dependency problem”.
In 2017, Vaswani et al. introduced a paper titled “Attention Is All You Need”. They presented a neural network called “The Transformer”. This architecture laid the foundation of modern NLP and modern Generative AI (GenAI).
NB: The in-depth explanation of the Transformer will be in Article 3.
The Transformer solved the “long-range dependency problem,” and the RNN is completely replaced by the self-attention mechanism. The raw text data is tokenized and converted into numerical forms called embeddings. As the Transformer can’t know the word order, a position encoding is used because it does parallel computing, meaning every token is processed simultaneously.
This original Transformer used N=6, which is the number of encoder layers that are stacked together, and also the number of decoder layers that are stacked together is 6 in number. But the core concept or heart of this is the ‘self-attention’ mechanism.
For example, a sentence like “The chicken, I like it”. Transformer looks at what ‘it’ refers to. It checks every word and learns what ‘it’ refers to: chicken. Self-attention lets every token attend to every other token in the same sequence. (Mathematical calculations will be in Article 3). And they used multi-head attention because the attention here is not performed once. It performs multiple times in parallel.
After the attention, the output is passed through a layer called “Add & LayerNorm,” which performs adding of input data to the output of attention, and the LayerNorm (layer normalization) stabilizes training and accelerates convergence; the final output is the normalized added output.
If x is the input, then we can simply express the operation of layer “Add & LayerNorm” as.
Final Output (the normalized one) = LayerNorm(x + Attention(x)).
After this, the final output is sent through a feed-forward neural network and then again through another “Add & LayerNorm”. These steps are the same for every encoder layer.
The decoder is similar to the ideas or concepts of the encoder, but the only difference is the “Masked Multi-Head Attention”. The decoder cannot see the future tokens or words; the mask hides the future tokens. So the decoder also attends to the encoder outputs, like the attention mechanism by Bahdanau et al. (2014), but by using self-attention, which is used to generate the future tokens. This is called “Cross Attention”.
In the above part of this section, I wrote ‘Transformer laid the foundation of modern NLP and modern GenAI ‘. This is because the architecture is more efficient, more scalable, and better at understanding relationships than the previous ones. The major reasons can also include — parallel processing, solution to long-range dependency, better at understanding contextual relations, and one architecture for many tasks. Since the Transformer, the direction of NLP has changed and led to advancements in both NLP and GenAI.
VIII. Rise of Large Language Models (LLMs)
In 1966, Joseph Weizenbaum published the first-ever Chabot in human history called ‘ELIZA’. This was a rule-based Chabot but wasn’t a language model.
In the 1980s — 1990s, statistical language models were released. The n-gram models were the first language models, and they used probabilities to estimate the next token based on previous context. Meaning, these models were trained on large text corpora, and they used probability to predict the next token by estimating the probability of the next token given the previous context. In the 2000s — 2012, neural language models such as RNN and LSTM were also released.
Before LLMs, let’s see what a language model is.
“Based on probability estimation, a Machine Learning model that can predict upcoming words by creating a probability distribution”.
Large Language Models (LLMs) are language models that are large in number of parameters (trainable weights), in size of dataset, and in quantity of computational resources. LLMs are trained on very large text corpora with a very large number of trainable weights. To train an LLM, an enormous amount of computational resources is needed. With the help of transformers, LLMs emerge very quickly and are very powerful. Even though they are so powerful, they don’t have brains.
When we talk to an LLM like Claude, Gemini, etc., they seem to understand what we are asking or writing. But they don’t, because they don’t have consciousness, and they just understand the context because they are trained on very large text corpora.
IX. The Current AI Era
Artificial intelligence has entered a new era of unprecedented capabilities. For many years, predictive AI dominated the AI landscape. By learning from historical and input data, predictive AI excelled at tasks such as classification, forecasting, recommendation, and anomaly detection across industries.
Meanwhile, GenAI had already existed long before the Transformer architecture. Early generative models were built using techniques such as Markov Chains, RNNs, LSTM, and Generative Adversarial Networks (GANs). Similarly, language models also existed before modern LLMs, but they were generally based on RNNs and LSTMs. These models processed text sequentially, making them slower to train, less scalable, and less effective at capturing long-range context.
The introduction of the Transformer architecture in 2017 fundamentally changed the landscape of Generative AI and LLMs. Its self-attention mechanism enabled models to understand relationships between words regardless of their distance within a sequence while allowing parallel processing during training. As a result, models became significantly more powerful, scalable, and capable of generating coherent, context-aware text, images, code, and other forms of content.
Although multimodal AI research dates back several decades, recent advances have made multimodal models far more capable and practical. Today, models such as ChatGPT and Gemini can process and generate text, images, audio, and video within a single system. These models allow users to communicate naturally through voice, point to objects in images, upload documents, and combine multiple types of information to solve complex tasks.
Another major advancement is the emergence of AI agents and AI copilots. AI agents are intelligent systems capable of planning, reasoning, using external tools, browsing the web, and completing multi-step tasks autonomously. They can break down complex problems into smaller subtasks, execute them, and evaluate their own outputs before producing a final result.
AI copilots, on the other hand, are conversational assistants designed to work alongside humans. They go beyond traditional chatbots by helping automate routine tasks, analyze information, generate content, and improve productivity. Examples include GitHub Copilot and Cursor, which assist developers by writing code, explaining existing code, debugging programs, and suggesting improvements.
Thanks to these advancements, AI has become deeply integrated into everyday life. AI is now available on smartphones, personal computers, enterprise software, and cloud platforms. In healthcare, AI agents and multimodal models can analyze patient records, medical images, and laboratory reports to assist healthcare professionals and identify potential health risks earlier. In software development, AI copilots, AI agents, and LLMs support developers throughout the software development lifecycle — from code generation and debugging to testing and deployment. Despite these remarkable capabilities, AI is not a replacement for human expertise. Human oversight remains essential to verify outputs, make critical decisions, and ensure AI systems are used responsibly and ethically.
X. Conclusion
Artificial Intelligence is not a recent innovation; it has a long history. The field is widely considered to have been formally established in 1956 during the Dartmouth Conference at Dartmouth College, where the term “Artificial Intelligence” was first introduced.
The earliest AI systems were based on traditional, rule-based approaches. These systems relied on predefined rules and expert knowledge, making them effective only within narrowly defined domains. As computing power and data availability increased, ML transformed the field by enabling systems to learn patterns from data instead of relying solely on manually written rules. Later, DL further revolutionized AI by using multi-layer neural networks inspired by the structure of the human brain, enabling significant improvements in computer vision, speech recognition, and natural language processing.
Continued research led to the development of the encoder-decoder architecture, which significantly improved machine translation and other sequence-to-sequence tasks. However, its reliance on a fixed context vector limited its ability to handle long and complex sequences. To address this limitation, researchers introduced the attention mechanism, allowing models to focus on the most relevant parts of the input. Building upon this idea, the Transformer architecture was introduced in 2017. With its self-attention mechanism and parallel processing capability, the Transformer became the foundation of modern Natural Language Processing (NLP) and the driving force behind today’s Large Language Models and Generative AI systems.
Research in AI continues to advance rapidly. Innovations such as AI agents, AI copilots, multimodal models, and reasoning systems are expanding what AI can accomplish and making intelligent systems more capable of assisting people in their daily lives. As AI continues to evolve, the future will likely be defined not by humans or AI working alone, but by effective collaboration between humans and intelligent machines.
References
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need (arXiv:1706.03762). arXiv. https://doi.org/10.48550/arXiv.1706.03762
Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks (arXiv:1409.3215). arXiv. https://doi.org/10.48550/arXiv.1409.3215
Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder–decoder for statistical machine translation (arXiv:1406.1078). arXiv. https://doi.org/10.48550/arXiv.1406.1078
Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate (arXiv:1409.0473). arXiv. https://doi.org/10.48550/arXiv.1409.0473
Jurafsky, D., & Martin, J. H. (2024). Speech and language processing (3rd ed. draft). Stanford University. https://web.stanford.edu/~jurafsky/slp3/