Yann Lecun is one of the godfathers of AI, along with Bengio and and Geoffrey Hinton. Lecun is the chief scientist at Facebook. At an event in Paris, he declared that generative AI is overhyped, and has reached a dead end. He is into developing new AI models that would show human-like rationality.

Lecun is aware that computers lack common sense. He talked about the image-based Joint Embedding Predictive Architecture (JEPA) which will surpass the capabilities of generative AI. Machines will be able to conceptualise abstract ideas. At present, they just spew out the existing online information.

He feels in a few years generative AI models will be replaced. He feels that AI should align with human understanding of the world. The models should perceive the world and make predictions.

Facebook has adopted generative AI for its platforms discreetly. It has released open source AI models that require less computing power than generative AI.

Facebook is spearheading the development of new AI models which will replicate human rationality.

There was Gold Rush to California of 3 lac people from the rest of the US in 1848 when Marshall found gold here. It was a rapid influx of fortune seekers. It reached its peak in 1852. Of late, Silicon Valley, a region between San Francisco and San Jose, experienced dooms days, when people left for greener pastures elsewhere, say Miami, LA, New York or Puerto Rico. There was pandemic and people left the Bay Area. The headlines were the disaster of crypto currencies, discouraging share prices and the fall of Silicon Valley Bank that funded startups.

There is always a cloud with a silver lining. That came as AI. To be specific, generative AI. It started the new Gold Rush to California. Some 300 enthusiasts and entrepreneurs gathered for Generative AI Meeting of the Minds in May 2023 at Shack15, a swanky social club at the second floor of Ferry Building, San Francisco.

The mood was elevated by the host Peter Leyden, a futurist. He assured the gathering that with the advent of AI, somethings new is cracking open. The whole experience made tech circles buoyant once again.

AI has become the talk of the town. The industry had beat a retreat, but has now come forward to offer AI solutions.

Sam Altman’s OpenAI came up with ChatGPT and Google with Bard. There are image softwares, Dall-E and Midjourney. Altman was assisted financially by Microsoft — $10 billion were poured into OpenAI.

Generative AI brought the gravitas back to Silicon Valley. It all started with a group of 8 researchers at Google who laid the foundation of this with 2017 Vaswani’s and others paper called Attention Is All That You Need. OpenAI was taking cognizance of this development.

Long sequence of data or chunks of text were processed. Each word was weighed in relation to what preceded it. The grammatical structures were considered. It was a breakthrough when computers predicted the next word.

Bay Area at the valley was home to Big Tech, apart from Seattle in the north. Big Tech had hired AI talents for years. Now Google and DeepMind are combining. There are many startups who are mission-driven, and not salary-driven. They expect to be on par with OpenAI in next five years. AI has in fact brought a revival at Silicon Valley. Palo Alto offices of Character AI are creating chatbots.

Employees here process the data in front of high-power computers. These people have practised what OpenAI is doing today.

Of course, there are AI skeptics. AI models lack accuracy. There are issues of bias. There could be an AI hangover or a meltdown. However, AI is not another bubble. Young engineers and entrepreneurs are making it more and more power-packed.

Sam Altman, CEO of OpenAI toured the world cautioning the governments and public at large about the downside of AI, especially generative AI. Generative AI models that do Natural Language Processing are called Large Language Models (LLMs). Those models such as ChatGPT respond to a prompt or question to generate a string of words ( technically called tokens) sequentially in an autoregressive manner. Here each word contributes to the prompt to generate the next word. Thus we get an answer in an essay-like form, built word by word.

What is there under the hood an LLM? It is a very large ML model with specialised AI architecture called transformers with billions or trillions of computable parameters. This model is trained on large datasets drawn from Internet, books and Wikipedia. It is expensive to train the transformer model since there are trillions of mathematical operations. It involves specialised computer hardware called GPUs. The latest GPT version GPT-4 is larger in size and much more efficient.

Though very beneficial to mankind, there are some downsides. The model can generate large-scale misinformation in the form of fake news. These models do hallucinate despite all the safeguards. The information provided could be factually incorrect.

The models are large-sized. The underlying training process is probabilistic. It is not easy to know how these large models provide a particular output to a given input. There can be outputs which are not aligned with the objectives of its human designers. These are called the black box risks.

In technological evolution, what can be done, will be done. There would be more powerful LLMs. At the same time, research could be directed towards AI safety and alignment. The design should be transparent and fair.

AI has the potential to empower citizens. It is useful in educating people and can assist the general public in medical care. It will be able assist the legal system in India with a backlog of cases. Far from banning it, AI could be used as a change agent to transform the Indian society.

Andrew Ng is known for his online education site Coursera. He is co-founder of Google Brain. He has contributed to deep learning.

Youshua Bengio is co-recipient of Turing Award, 2018 for breakthrough that made deep learning networks a critical component of computing. He has contributed to artificial neural networks and deep learning.

Geoffrey Hinton is also a co-recipient of Turing Award, 2018. He is known for his work on artificial neural networks and deep learning.

Fei-Fei Li is the co-director of Standford Institute of Human-Centered Artificial Intelligence (HAI). She is known for her work on computer vision and ML.

Demis Hassabis is co-founder and CEO of DeepMind. He is known for his work on artificial general intelligence (AGI).

While training language models, (LMs) datasets are drawn from various domains, say publicly accessible dataset (called the Pile) consisting of online data (24%), Wikipedia (9%), and GitHub etc. (4%). The constitution of the data influences how well an LM performs. It should be obvious how much each domain should be included so as to create a model that performs a range of downstream tasks. Either intuition or a series of downstream tasks are used to arrive at domain weights or sample probabilities for each domain. The Pile uses heuristically selected domain weights. Maybe, they are not an ideal choice.

Google and Stanford researchers attempted to identify domain weights so that models perform well on all domains. There is no optimization of domain weights based on a range of downstream tasks. Rather, there is minimization of worst-case loss over domains.

Each domain has an entropy or a unique optimum loss. The DoReMi technique is Domain Reweighting with Minimax Optimisation. It uses distributionally robust optimization (DRO) without being aware of the tasks which will be performed later.

Conventionally, DoReMi begins by training a tiny reference model with 280M parameters. To curtail excess loss, a tiny distributionally resistant LM is introduced (DRO-LM). Domain weights generated by DRO are used in training.

To optimise domain weights on the Pile and the GLaM dataset, they run DoReMi on 280M proxy and reference models.

Intel launches a science-focused generative AI model with a trillion parameters. It is six times the number of ChatGPT parameters. It requires huge computational power to fine tune the model at the hardware level.

This model is expected to cater to the needs of scientific community. It will facilitate research in material science, climate science, cosmology, cancer research, system biology, polymer chemistry.

The project is still a work-in-progress. It will be interesting to see how Intel competes with NVidia, its closest competitor.

OpenAI has witnessed two big miracles — they have an algorithm that can genuinely learn, and second, the algorithm gets better with scale.

There are concerns about AI’s impact on elections and on society. However, as a society, we are going to rise to the occasion.

The software has the capability to generate like mass media. However, at the same time, it has the capability is generate one-on-one interactive persuasion.

Every technological revolution leads to job change. This will be no exception. However, there will be new and better jobs.

The current systems of AI are not dangerous. GPT-4 poses no existential threat, however GPT-10 may be extremely different. We endeavour to align with AGI, and that demands safe systems.

Countries have to integrate AI into other services. LLMs will make government services way better.

AI helps the journalists handle the montonous parts of their jobs better. The time saved can be used for more reporting, thinking of ideas.

The technology can become a tool of oppression in the hands of a dictator.

Just as nuclear materials could be beneficial and dangerous, and are therefore audited by a body like IAEA. So should AI be audited.

OpenAI is building a tool, and not a creature.

Apart from the prevailing payment system such as UPI, NEFT, RTGS, credit-debit-prepaid cards, India needs a separate payment system that is light weight and leaner (no complex network and IT infrastructure). It can work when the existing systems are disrupted due to breakdown of information and communication infrastructure. This could be because of catastrophic events such as acts of God or natural disasters and conflict.

It is wise to have a light and portable system, independent of conventional technologies. It should have minimal hardware and software requirements. A catastrophic event would not be able to impede its performance. It could be run from anywhere with a bare minimum staff.

The RBI wants this system to have near zero-downtime of payment and settlement.

The system will keep liquidity intact in the economy. It will facilitate the essential services, e.g. bulk payments, inter-bank payments and cash availability to institutions.

The system will have simplified system of authentication and verification — there could be a master password for access, and a service authentication password.

Media agencies have indeed expanded their role beyond buying and planning media to partnering with the brands as consultants. To begin with, media planning was fairly simple. Then there was electronic media growth with cable TV and broadcast TV. Later, we have digital media. All this made the role of media agency more significant.

Media agencies play a strategic role today. They offer expertise for traditional and new-age media. Professional consultancies here compete with media agencies. However, media agencies have vast knowledge of brands, and that keeps them in the forefront.

A new terms has been coined – integrated media.

Open source AI models are gaining a foothold. They are a threat to the proprietary models. OpenAI too thinks about releasing an open source model. There is pressure to do so by competitive moves from organisations such as Meta which has released its open source model LLaMa.

OpenAI’s open source model may not have the finesse of its proprietary models. Still it is an important move.

Stability AI has thrown open its LLMs. Databricks Dolly 2.0 AI model is now open source. Together, an AI firm, is developing open source AI models. Big Techs such as Google are concerned about such open source models which are a threat to its own models.

Open source models can be modified and improved upon by anyone. Contributors who come forward to do so are far more than those who work in the organisations. Open source models can be customized. AI is becoming an accessible, inclusive and innovative field.