Blog

Generative AI is a blessing for coders. One has to give a prompt in natural language, and the model writes the code. In addition, it has the ability to detect the bugs in the written code.

All this raises concerns about coding as a job. Could there be job losses since coders will use AI programming assistants — Co-pilot and Code Assist?

Co-pilot assists coders to write code, suggest code snippets and provides real-time hints. The coding process is thus streamlined. The whole thing facilitates code writing. There is no replacement of code programmers.

There is a limit to what these AI models do — context length. It is a limit to which the model can comprehend the lines of code. As we know, enterprise-level code consists of millions of lines of code. It is necessary to have a human element in the loop while developing code.

AI coding assistants improve productivity of the programmers. It does not mean they do more work. As there are complex issues while programming, we have to increase the number of coders.

In early days, coding was done in low-level language, say assembly language. In the 1960s and 1970s, Fortran was being taught at educational institutes. Coders did not spend all their time in writing low-level code. They can develop a large scale distributed system in Fortran. However, the new developments bring a new set of complexities — public and private cloud, scaling up and down, dependencies while distributing.

AI coding assistants also learn the code from what a human coder writes. It means that human programmers, and domain experts are still needed.

AI coding assistants facilitate automation and testing. They only automate repetitive tasks. AI cannot reach human-levels of problem solving.

Sam Altman has resigned from OpenAI as CEO and Brockman too has resigned as President. Sam Altman says he was happy working with bright people. Meera Murati of Albanian origin who has been a CTO was appointed as an interim CEO by the Board.

As we know by now, an LLM is a computer algorithm trained on massive datasets to understand and process natural language. This is what we require — AI that generates text, video and audio.

Chinese company 01.AI has released its new LLM, Yi-34B. it is so called since it has been trained on 34 billion parameters. Parameters, as we know, are the weights of inputs a model learns to predict what comes next in a sequence.

Kai-Fu Lee, a Taiwanese computer scientist, has founded this company in March, 2023. Its LLM model is open source. It is available to developers in English and Chinese.

Hugging Face, open source developer community platform, ranked Yi-34B first in a leaderboard of pre-trained LLMs. Though the model is smaller than Falcon-180B and Meta Lla-Ma2 70B, still it beats them. Lee Sees it as a ‘gold standard’ on key metrics.

Lee has worked for American Big Tech and is considered AI pioneer. He has authored two books on the subject.

The words or parts of words ( say plural marker ‘s’ or the prefix ‘un’ ) are stored in the LLM model as tokens. Each token is not represented as a sequence of alphabets or letters but as a vector which is a sequence of numbers.

The vectors assigned to the word tokens in the model place them in a ‘space’. They encode how ‘close together’ they are in that space.

The distances between words are expressed in hundreds of dimensions. These dimensions encode substitutability ( here happy could be close to sad, though on some other dimension, they could be far apart ).

The LLM has to predict what word or sequence of words is or are most likely to come next.

Here two things are used to facilitate this — a transformer and attention.

A transformer is a mathematical process that recalculates vectors for each token. In other words, it assigns new distances between each pair of tokens, depending on what other tokens are. LLM gets first few words by rearranging the question into a response. It has just to find the most probable next word.

LLM weights all the relationships between words it knows (in thousands of dimensions, based on corpus of training data). Then it looks at what words have preceded and reweights those associations. The reweighting step is what the LLM technicians call a transformer. The revaluation of weights based on the salience given to previous bits of the text is called attention.

These steps are applied to every part of the conversation. Attention is a breakthrough development in natural language AI.

Sam Altman is the CEO of OpenAI since 2019. All these years, the company’s mission is to achieve what is called ‘artificial general intelligence’ (AGI). The idea is to make it safe and beneficial for humanity.

It is still not precisely known what AGI is. It is a lofty but vague idea.

In a recent interview with Financial Times, Altman poetically calls AGI ‘magic intelligence in the sky.’ Interpreted in concrete terms, it amounts to some divine or godly entity.

OpenAI itself defines AGI as a system that outperforms humans at most (economically valuable) work. It is so mundane description of something that is superintelligence for Altman.

It is not known to many that Elon Musk is the co-founder of OpenAI who left it in 2018 before Altman joined it as CEO. Musk is worried about AI that could outsmart humans, and could become a digital God. While staying with Larry Page, the cofounder of Google in Palo Alto, Musk would talk about AI safety.

Altman who has made AGI as his mission still sounds ambiguous on the details. Altman expects Microsoft to back OpenAI financially to achieve its mission. He realizes that there is a lot of work, between here and the accomplishment of the mission. Altman also says that OpenAI Board of six people will decide when the company has reached AGI. It leaves a lot of wriggle room for him.

Whether AGI would be just robotized help to high-school students or elevation of tech to divine levels, one thing is certain — AGI requires astronomical investment.

Researchers strive to build better and better AI, but three Google researchers have found out that there is a limit to what has been currently called generative AI. It could be a damp squib on the plans of their superiors to travel to more advanced AI systems.

The paper is authored by a trio of Steve, Lyric Doshi and Nilesh. This paper still has not been peer-reviewed. The paper concludes that AI is not capable of going beyond its training dataset. The paper considers the transformer model — a transformer here converts one type of input into a different type of output.

This model’s architecture was first theorized by a group of researchers in 2017 (Vaswani et al) who wrote a paper called Attention Is All You Need. The researchers thought that the model generates text and other output, and therefore the model can do intuitive thinking on its own. If this is further refined, it can lead to human-level AI, called AGI.

Transformer models did create a lot of buzz — it was felt that they can go beyond their training data.

However, when these are assigned tasks or functions not covered by their training data, they fail, and their generalization is not extended even for simple tasks.

Thus a transformer model is not able to cope with anything if it is not pre-trained on it.

Since the models have been trained on billions of parameters, it was natural to expect that they would be able to cope with tasks on their own. These models have crammed so much knowledge into them, and there is not a whole lot they have not been trained on.

Can a model have some sort of emergent property in AI with enough training data? The research pertains to GPT-2. It is now obsolete. Further models GPT-3, GPT-3.5 and GPT-4 have appeared. Maybe, there could be an emerging AI. Or else, further research could adopt a new approach that overcomes the limitations of the present paradigm.

Of course, this research will sober the sizzle of AI. So far the model depends on the expertise humans already have. We will have to temper the AI expectations.

AGI presumptions require both time and further research. AI still cannot take leaps of thought that separates human beings from machines.

The voice has reached the ears of Sam Altman and Satya Nadela, and they have decided to put in a joint effort towards developing AGI.

Fei-Fei-Li is also considered one of the pioneers of AI. She teaches at Stanford, and her specialisation is computer vision. She is also a founding co-director of Human-centred Artificial Intelligence (HAI).

She is of Chinese origin, emigrated to the US when she was 15. She attended Princeton on scholarship. She became interested in what intelligence is and what it means for a computational machine to be intelligent. She got her Ph.D. studying AI, and specifically computer vision.

Her contribution to contemporary AI is ImageNet in 2009. The model was trained on a huge dataset. The software recognised objects. More than 14 million images were scraped from the web. They were manually labelled into 20,000 noun categories.

The model departed from previous thinking since big data was used to build the model. That defines the deep learning family of algorithms.

ImageNet powered a deep learning neural network algorithm called AlexNet developed by Geoffrey Hinton’s team at Toronto in 2012. The machines got sight for the first time.

LLMs are today built on large data. This they inherit from ImageNet.

She is worried about AI risks which we face here and now, rather than the remote existential threat. There should be a balance between regulation and innovation. It is necessary to debias the algorithms.

She s interested in embodied AI — AI-powered robots that interact with the environment and learns from it.

She is optimistic about ChatGPT like application which can summarize medical reports.

000

Nvidia has become an important supplier of GPU chips to run AI. Microsoft is expected to release its AI chip code-named Athena in collaboration with AMD. OpenAI which uses Nvidia chips loaded supercomputer, built by Microsoft. The computer employs thousands of Nvidia chips. Google has developed its own chips — tensor processing units (TPUs). These TPUs are optimised for use in neural networks. However, these are not good for word processing or executing bank transactions. Google TPUs can be used by other enterprises — say for amounts ranging from $3000 per month to $100,000 per month. Google is also planning to launch chips for its Chromebooks. Baidu, a Chinese search giant has designed chips called Kunlun used in autonomous driving and NLP. Tencent, another Chinese company operating WeChat, is designing chips which can process vast data with a focus on AI, and for image and video processing.

It will take 4-5 years for other companies to catch up with the leader — Nvidia. Nvidia itself has not stopped innovation. Some predict that rival companies may take a decade to catch up with Nvidia.

Nvidia, as we have already observed, partnered with Foxconn to set ‘AI factories’ — a new kind of data centre.

There are many challenges in developing AI chips — complex supply chain issues, dearth of talent, the challenge of getting the chip right, long design and development cycle. The investment is about $1.5 billion to design a single 3nm chip with a complex GPU. Are there buyers to justify the huge investment?

The demand for AI chips will go up. The market will expand. It is going to be a $400 billion market by 2032. It is too big for Big Tech to ignore.

At the conclusion of AI Summit hosted by the UK government, Prime Minister Sunak and Tesla CEO Elon Musk discussed in a conversation a range of issues posed by AI. This conversation was recorded on November 2, 2023 at Lancaster House in London. Sunak plays the chat show host throwing questions at Elon Musk about the riks associated with the fast-developing and ‘transformative’ new generative AI technology.

Musk describes AI as a ‘magic genie’ that could grant limitless wishes and agreed with the need of a referee to monitor the development of the super-computers of the future.

There could be massive job destruction with no job left. If at all one does a job, that could be for personal satisfaction. In other words, AI will do everthing.

Human beings want their life to be meaningful. On account of AI, one of the challenges in the future could be to find meaning in life.

On balance, Musk believed AI would ‘be a force for good.’ He hinted at the possibility of robotic friends in future. Such robots could become part and parcel of life. A software update can make them switch off — they are not friendly any more.

With reference to the safety institutes that would test future AI models before release, Musk called it a good thing. It is like having a referee.

It was an hour long discussion with some members of the UK cabinet and leading entrepreneurs in attendence. The Summit was held at Bletchley Park, Buckinghamshire.

The participating countries signed a Bletchley Declaration. It was a historic agrrement with governments and AI companies working together to test the safety of AI models before and after they are released.

PM Sunak said AI Safety Institute, UK will play a vital role in leading this work in partnership with other countries including AI Safety Institute, USA.

Bletchley Park is the place where Alan Turing took up full-time work in 1939. It could be the stating point in a journey where natural intelligence and artificial intelligence are juxtaposed.

When AI surpasses the intelligence of humans, that is called the singularity. It is a moment that could be happen a few years hence. Bew Goertzel, CEO of SingularityNET believes AGI is some 3-8 years away. AGI means that AI and human intelligence match each other. The result is AI can perform tasks as well as humans.

It is for you to believe him or not. However, it is true that there is no let up in having AI better. All LLMs are pushing hard towards the growth of AI.

More resources and more talents will be deployed to work on AI. As we know, the concept emerged to facilitate the military in the 1950s. Later, the concept has been driven towards a variety of goals. It is useful to the corporates as well as the creative people such as artists or musicians.

It is true that a big leap is necessary to take it from the current position of AI to a singularity. At present, AI is meant for specific tasks. AGI is that push which makes AI understand the world as humans do and hone in its abilities. It is an issue of broadening the understanding of the world. The more AI does so, the closer it goes to AGI and AGI is just one step short of singularity.

Some experts still believe that the technology is far far away. However, the quest is going on. It is just a matter of time.

Sam Altman compares AGI to the ‘magic intelligence in the sky’ for which there is a journey from the earth in a hype-fuelled rocket by companies such as OpenAI. If the rocket reaches the destination, its six members Board will declare that the company has reached singularity. The six members are President Greg Brockman, Chief Scientist Ilya Sutskever, Quora CEO Adam D’Angelo, tech entrepreneur Tasha MaCauley, wife of actor Joseph Gordon-Levitt, professional thinker Helen Toner.

Most of these people believe in altruism — earn money and donate it for good causes.

GPTs have not been invented by OpenAI, but by Google in 2017. OpenAI, however, is consumer friendly. It has released the technology to the public to interface just like a friend. Whether AGI will originate from OpenAI is not certain. Investment flows into the company, but that may not be enough. Between here and AGI, there is a lot of compute, and lot of expenses. All this may not enough to realise the dream of AGI. There could be an upper technical limit on the capabilities of GPTs. The current position is build more capable models, and figure out what they are good for. The business potential of AGI is vast. It can create too much wealth which can be redistributed to the needy.