Blog

Sam Altman, CEO of OpenAI toured the world cautioning the governments and public at large about the downside of AI, especially generative AI. Generative AI models that do Natural Language Processing are called Large Language Models (LLMs). Those models such as ChatGPT respond to a prompt or question to generate a string of words ( technically called tokens) sequentially in an autoregressive manner. Here each word contributes to the prompt to generate the next word. Thus we get an answer in an essay-like form, built word by word.

What is there under the hood an LLM? It is a very large ML model with specialised AI architecture called transformers with billions or trillions of computable parameters. This model is trained on large datasets drawn from Internet, books and Wikipedia. It is expensive to train the transformer model since there are trillions of mathematical operations. It involves specialised computer hardware called GPUs. The latest GPT version GPT-4 is larger in size and much more efficient.

Though very beneficial to mankind, there are some downsides. The model can generate large-scale misinformation in the form of fake news. These models do hallucinate despite all the safeguards. The information provided could be factually incorrect.

The models are large-sized. The underlying training process is probabilistic. It is not easy to know how these large models provide a particular output to a given input. There can be outputs which are not aligned with the objectives of its human designers. These are called the black box risks.

In technological evolution, what can be done, will be done. There would be more powerful LLMs. At the same time, research could be directed towards AI safety and alignment. The design should be transparent and fair.

AI has the potential to empower citizens. It is useful in educating people and can assist the general public in medical care. It will be able assist the legal system in India with a backlog of cases. Far from banning it, AI could be used as a change agent to transform the Indian society.

Andrew Ng is known for his online education site Coursera. He is co-founder of Google Brain. He has contributed to deep learning.

Youshua Bengio is co-recipient of Turing Award, 2018 for breakthrough that made deep learning networks a critical component of computing. He has contributed to artificial neural networks and deep learning.

Geoffrey Hinton is also a co-recipient of Turing Award, 2018. He is known for his work on artificial neural networks and deep learning.

Fei-Fei Li is the co-director of Standford Institute of Human-Centered Artificial Intelligence (HAI). She is known for her work on computer vision and ML.

Demis Hassabis is co-founder and CEO of DeepMind. He is known for his work on artificial general intelligence (AGI).

While training language models, (LMs) datasets are drawn from various domains, say publicly accessible dataset (called the Pile) consisting of online data (24%), Wikipedia (9%), and GitHub etc. (4%). The constitution of the data influences how well an LM performs. It should be obvious how much each domain should be included so as to create a model that performs a range of downstream tasks. Either intuition or a series of downstream tasks are used to arrive at domain weights or sample probabilities for each domain. The Pile uses heuristically selected domain weights. Maybe, they are not an ideal choice.

Google and Stanford researchers attempted to identify domain weights so that models perform well on all domains. There is no optimization of domain weights based on a range of downstream tasks. Rather, there is minimization of worst-case loss over domains.

Each domain has an entropy or a unique optimum loss. The DoReMi technique is Domain Reweighting with Minimax Optimisation. It uses distributionally robust optimization (DRO) without being aware of the tasks which will be performed later.

Conventionally, DoReMi begins by training a tiny reference model with 280M parameters. To curtail excess loss, a tiny distributionally resistant LM is introduced (DRO-LM). Domain weights generated by DRO are used in training.

To optimise domain weights on the Pile and the GLaM dataset, they run DoReMi on 280M proxy and reference models.

Intel launches a science-focused generative AI model with a trillion parameters. It is six times the number of ChatGPT parameters. It requires huge computational power to fine tune the model at the hardware level.

This model is expected to cater to the needs of scientific community. It will facilitate research in material science, climate science, cosmology, cancer research, system biology, polymer chemistry.

The project is still a work-in-progress. It will be interesting to see how Intel competes with NVidia, its closest competitor.

OpenAI has witnessed two big miracles — they have an algorithm that can genuinely learn, and second, the algorithm gets better with scale.

There are concerns about AI’s impact on elections and on society. However, as a society, we are going to rise to the occasion.

The software has the capability to generate like mass media. However, at the same time, it has the capability is generate one-on-one interactive persuasion.

Every technological revolution leads to job change. This will be no exception. However, there will be new and better jobs.

The current systems of AI are not dangerous. GPT-4 poses no existential threat, however GPT-10 may be extremely different. We endeavour to align with AGI, and that demands safe systems.

Countries have to integrate AI into other services. LLMs will make government services way better.

AI helps the journalists handle the montonous parts of their jobs better. The time saved can be used for more reporting, thinking of ideas.

The technology can become a tool of oppression in the hands of a dictator.

Just as nuclear materials could be beneficial and dangerous, and are therefore audited by a body like IAEA. So should AI be audited.

OpenAI is building a tool, and not a creature.

Apart from the prevailing payment system such as UPI, NEFT, RTGS, credit-debit-prepaid cards, India needs a separate payment system that is light weight and leaner (no complex network and IT infrastructure). It can work when the existing systems are disrupted due to breakdown of information and communication infrastructure. This could be because of catastrophic events such as acts of God or natural disasters and conflict.

It is wise to have a light and portable system, independent of conventional technologies. It should have minimal hardware and software requirements. A catastrophic event would not be able to impede its performance. It could be run from anywhere with a bare minimum staff.

The RBI wants this system to have near zero-downtime of payment and settlement.

The system will keep liquidity intact in the economy. It will facilitate the essential services, e.g. bulk payments, inter-bank payments and cash availability to institutions.

The system will have simplified system of authentication and verification — there could be a master password for access, and a service authentication password.

Media agencies have indeed expanded their role beyond buying and planning media to partnering with the brands as consultants. To begin with, media planning was fairly simple. Then there was electronic media growth with cable TV and broadcast TV. Later, we have digital media. All this made the role of media agency more significant.

Media agencies play a strategic role today. They offer expertise for traditional and new-age media. Professional consultancies here compete with media agencies. However, media agencies have vast knowledge of brands, and that keeps them in the forefront.

A new terms has been coined – integrated media.

Open source AI models are gaining a foothold. They are a threat to the proprietary models. OpenAI too thinks about releasing an open source model. There is pressure to do so by competitive moves from organisations such as Meta which has released its open source model LLaMa.

OpenAI’s open source model may not have the finesse of its proprietary models. Still it is an important move.

Stability AI has thrown open its LLMs. Databricks Dolly 2.0 AI model is now open source. Together, an AI firm, is developing open source AI models. Big Techs such as Google are concerned about such open source models which are a threat to its own models.

Open source models can be modified and improved upon by anyone. Contributors who come forward to do so are far more than those who work in the organisations. Open source models can be customized. AI is becoming an accessible, inclusive and innovative field.

Youngsters, under 25, are fond of digital games in India. They are not hostile towards watching the ads. Most of the paid gamers in India spend handsomely on in-app purchases.

India is a mobile-first gaming market. Mostly the games are played on hand-held devices.

Consumer goods, automobiles and entertainment are suitable for in-game advertising. There are good conversion rates, depending on the ad format and the targeting tools used.

In-game advertising can become exciting when it is made interactive.

It is to be noted that ads should load quickly. Here user experience matters a lot . Non-intrusive formats should be used which go with the games. Marketers and game developers should co-create the ads. Some ad spots could be native. Some ad spots could be audios. They are less intrusive.

Ads should not appear alongside offensive or inappropriate content.

How to measure the success of in-game placements? Here the approach is more nuanced. Traditional metrics are the number of impressions, click-through rates, engagement rates. Sponsored game modes give additional tracking opportunities. We can measure the time spent on the game and the progress made by a player in that game. It assesses the players engagement.

OpenAI was established as an AI research laboratory. It consisted of OpenAI LP for profit, and its parent company OpenAI Inc. It was founded in 2015. The Board consisted of Elon Musk, Sutskever, Brockman, Blackwell, Chenung, Karpathy, Kingma, Schulman, Vagata, Zaremba and Sam Altman. In 2019, Open AI converted itself from non-profit to ‘capped’ for profit, with profit capped at 100 times any investment. Microsoft provided OpenAI LP with $1 billion investment in 2019, and a further $10 billion investment in 2023.

OpenAI’s notable achievement is the development of GPT series which display an evolution of AI. It is a pathbreaking work, and it continues to revolutionalize AI.

GPT stands for Generative Pre-trained Transformer, a language model which vests systems with intelligence and is used in products such as ChatGPT. GPT models have the capability to generate human-like text and content. ChatGPT is a chatbot released towards the end of November, 2022. Here ‘chat’ refers to chatbot functionality and GPT to Generative Pre-trained Transformer. GPT is a type of large language model (LLM). It is a task-specific GPT that was fine-tuned to target conversational usage. It has been built upon GPT-3 model’s later version GPT-3.5.

ChatGPT works in response to a prompt. It has to understand the prompt. It then produces strings of words which predict best answer to a query asked. Of course, the answer is based on the data it was trained on.

One important thing. ChatGPT’s objective is to be predictive. It predicts the next word in a sentence. (This of course it has learnt on the gigabytes of text data it has been trained on).

A prompt or a query takes through the AI model, and produces a response. It matches the information in the prompt or query and the vast amount of training data.

ChatGPT has varied applications. It, however, excels in natural language processing (NLP) including translation, summarization and question answering. It could be used for generative tasks — text completion, text generation and conversation generation. It can be used for customer service, medical transcription, patient triage.