Understanding the Complexity of LLM Representation
Our current understanding of LLM encoding is not fully unclear. While we can observe the multiplications and node signals within neural networks, the concept of e.g. Charles Darwin’s encoder remains elusive and we do not really know where is it located. However, it is quite certain that in the future, we will gain a deeper understanding of this representation.
Exploring the intricacies of the human brain is much more complex task. Nevertheless, due to the ease of monitoring and x-raying LLMs compared to the human brain, we may have a better chance of comprehending them. Notably, significant efforts have been made in academia to enhance the interpretability of LLMs, which suggests that with time, our understanding will likely improve. Currently, our grasp of LLMs is limited, but even the fragments of knowledge we have acquired have proven immensely beneficial in advancing the field.
Our motivation to understand LLMs goes beyond scientific curiosity; it is driven by the potential to greatly enhance training efficiency and accuracy. Throughout the history of technology, we have witnessed instances where empirical discoveries have been made without a comprehensive understanding of their underlying mechanisms. However, as scientific understanding deepens, these discoveries can be significantly improved.
For instance, the creator of GPT-1 initially developed it without a profound comprehension of its inner workings, yet recognized its impressive capabilities. OpenAI’s efforts to scale and predict advancements in LLMs have led to the creation of demos, which, in turn, have spurred numerous attempts and a progressively better scientific understanding of LLM. Ultimately, this progress stems from initial empirical results.
Looking Ahead
In to the next 2 to 3 years, there are several key milestones and areas of progress that we anticipate:
-
Multimodality: The ability for models like GPT-4 to handle various inputs such as speech, images, and potentially even video. This is a highly sought-after feature, and companies that have already released models capable of processing images and audio have received a much stronger response than anticipated.
-
Reasonability: Currently, GPT-4’s ability to reason is extremely limited. Improving its reasoning capabilities will be a crucial milestone in the coming years.
-
Reliability: Presently, if you pose a question to GPT-4 multiple times, it may provide a good response only once out of those attempts. Enhancing reliability to consistently deliver the best response out of all iterations is a priority.
-
Customizability and Personalization: Recognizing that individuals have unique preferences and requirements, it will be important to allow users to customize their interactions with GPT-4. This includes different styles of responses and the ability to accommodate individual assumptions.
-
Ability to Use Personal Data: Enabling GPT-4 to access and utilize a user’s personal data, such as their email, calendar, and preferences for appointment booking, as well as connecting to other relevant data sources.
-
Adaptability: To optimize efficiency, there is a need for adaptive compute. Currently, the same amount of resources is allocated to process simple tokens and complex mathematical or coded instructions. Implementing adaptive compute will be essential to streamline computational requirements.
These milestones represent significant areas of progress that we aim to achieve within the next few years, pushing the boundaries of AI capabilities and delivering a more versatile and personalized experience for users.
AI Regulations
Currently, there is ongoing discussion and construction of AI regulations (e.g. -> AI Act, the European Commission’s regulatory framework for AI). It is important to strike the right balance when implementing regulations in this space, as history has shown that excessive regulations can hinder progress. However, considering the potential impact of this technology on society and the geopolitical balance of power, especially as we envision future systems with a thousand or a million times greater compute power than GPT-4, there is a need to explore the idea of a global regulatory body.
This body would oversee the regulation of these super-powerful systems, similar to how the International Atomic Energy Agency (IAEA) was established for nuclear energy. The global impact of such systems necessitates a coordinated approach to ensure responsible use. In the shorter term, there will also be issues to address, such as determining what these AI models are allowed or not allowed to say, also in terms of copyright regulations. Different countries may have varying perspectives on these matters.
To address the potential risks associated with exceptionally high compute clusters, it is proposed that any cluster exceeding a certain extreme threshold (which may be limited to few clusters globally) should be subject to scrutiny comparable to international weapons inspectors. This would involve making the model available for safety audits and passing tests both during training and prior to deployment. While this approach cannot guarantee absolute safety, it aims to mitigate the most significant risks associated with these powerful systems.
It is important to note that even with these measures in place, there may still be instances where smaller-scale systems encounter failures, potentially resulting in significant consequences. However, the focus here is primarily on addressing the risks associated with the most powerful tiers of AI systems.
Productivity Improvement from AI
In the current landscape, AI systems have made significant strides in task automation, resulting in increased productivity. While they are not yet capable of replacing entire jobs, we can anticipate a future where AI enables us to accomplish a wider range of tasks. This evolution will undoubtedly lead to the discovery of new and improved job opportunities.
By providing individuals with more powerful tools, we are not simply enhancing their speed but empowering them to engage in qualitatively different endeavors. For instance, if we develop a program that is few times more effective, its impact goes beyond enabling users to complete tasks few times faster. Instead, it elevates them to a higher level of abstraction, tapping into their cognitive capabilities and enabling them to explore entirely new domains.
Analogously, the transition from punched cards to higher-level programming languages didn’t just expedite programming processes but revolutionized the very nature of programming. It allowed us to achieve qualitatively different outcomes and adopt innovative approaches.
In the future we can imagine an AI agent capable of comprehending complex programming tasks, engaging in dynamic dialogue throughout the process. This advancement wouldn’t merely involve writing a few functions at a time, but rather, it would open up entirely new possibilities. In the future, we may even witness AI agents starting companies on our behalf or delving into uncharted territories, such as the discovery of new mathematical concepts.
The area of software development is likely to be at the frontier of significant productivity improvements, however the impact of AI extends to other sectors like education and healthcare.
Conclusion
The ongoing advancements in AI technology hold immense potential to transform industries, create new opportunities, and unleash the full extent of human potential. As we navigate this evolving landscape, it is crucial to recognize the profound impact AI can have on our society and companies while embracing the possibilities it presents. However, keeping up with the pace of this progress is extremely important.
