The trustworthiness challenge

“Large Language Models” (LLM), such as OpenAI’s ChatGPT, Google’s Bard, Baidu’s Ernie, Facebook’s LLaMA, and recently the French MISTRAL, are currently attracting enormous attention. They are often lauded as one of the most significant innovations since the steam engine, profoundly altering the way we collaborate by enabling many time- and resource-intensive jobs to be largely managed by virtual agents. However, alongside their potential, there are rapidly growing concerns.

• Knowledge Limitations, Hallucination and Misinformation Risks, and Transparency Issues: : LLMs lack awareness of their knowledge boundaries, often generating statements from nothing, a phenomenon known as "hallucination." They can be deliberately manipulated to generate substantial amounts of "fake news." Their lack of transparency and robustness makes them highly sensitive to their training's specifics, leading to reliability concerns. LLM often reinforce conventional wisdom, deriving from existing content and weighing "probable" associations. This tendency carries the risk of losing control over our understanding of the world and our collective decision-making.
• Lack of "Right Brain" Capabilities: LLMs excel in symbolic calculations but significantly struggle with right hemisphere brain functions such as common sense, intuition, feelings, and ethical values. This limitation raises critical questions about trusting an AI lacking in common sense and moral judgment.
• Resource Intensity and Bias: The development of LLMs demands substantial computing and financial resources, typically available only to large tech companies or nation-states. This concentration of power risks introducing financial biases into the AI's decision-making process.
• The challenge of a conscious AI: Adding deep consciousness to LLMs, including common sense, intuition, and ethics, presents significant challenges. Consciousness involves complex interactions of body, mind, and social factors, such as DNA, personal education, and culture. This could lead to AIs which are as unique as individual human beings or groups, raising additional questions about trust and reliability.

As with many transformative technologies, LLMs offer immense potential for changing how we collaborate. However, they also introduce increased complexity and necessitate more sophisticated approaches to using our collective human intelligence. This includes better reality comprehension, differentiating real from fake news, and creating effective human-AI collaborations, rather than oversimplifying by excluding human input.

The SymPlace approach to Collaborative Hybrid Intelligence

On the one hand, SymPlace’s ALC encapsulates AI to integrate it into ALC in symBots and VPAs, each functioning as an independent server. This design ensures data protection and traceability of transactions, crucial for maintaining privacy and intellectual property rights. The symBots are designed to encapsulate specific functionalities or expertise, such as note taking and minutes for working meetings, medical diagnostics, and legal advice, tailored to the needs of specific collaborative purposes, and integrated into larger social hybrid practices.

On the other hand, Augmented Learning Collaboration (ALC) offers the possibility to move from the view of a probably illusory “artificial general intelligence” to a “Collective Hybrid Intelligence” combining the best of both human intuition and experience with virtual agents’ symbolic reasoning and data processing capabilities, to enable intelligence through smart and efficient collaboration. Collaborative Hybrid Intelligence (CHI) in SymPlace is not just about combining human and AI capabilities but also about creating a synergistic intelligence that emerges from their interaction. This approach addresses the trustworthiness issues of LLMs by ensuring that AI's reasoning is complemented by human ethical and intuitive judgment. In summary, SymPlace's implementation of ALC and CHI is a strategic response to the limitations and criticisms of LLMs. By embedding AI within a framework of controlled data usage, human oversight, and collaborative intelligence, SymPlace aims to create a more balanced, ethical, and effective hybrid collaborative environment. This approach not only enhances the potential of AI but also ensures that it serves the collective good, aligning with the fundamental principles of ALC:

• Balancing AI's limitations with human intuition: in response to LLMs' lack of "right brain" capabilities, SymPlace's ALC framework leverages human intuition and social practices. By integrating human insights into AI-driven processes, SymPlace compensates for the lack of common sense, intuition, and ethical judgment in LLMs.
• Controlling the scope of inputs received and outputs sent by AI: by formalising through the no-code collaboration modelling language the flow of data between symBots, SymPlace ensures, on one hand, that AI's output is, within the context of the specific symFlo, sufficiently relevant and trustworthy, and respects user privacy, and, on the other, that its input only encompasses data which are within the range of what is acceptable for the privacy of the participants in the symFlo. This approach also addresses the concerns about AI's knowledge limitations (which can be counter-checked) and transparency issues.
• Ensuring reliable and bias-free AI: the concerns about LLMs being prone to hallucination and misinformation are mitigated in SymPlace through controlled data input and human oversight. By ensuring that AI's training and output are overseen by human collaborators, SymPlace reduces the risks of biases and unreliable outputs.

It should be stressed that intelligence and trust / mistrust are emergent properties of the collaboration system. They are conditioned by the structure of the symFlo template, and hence will evolve as this structure changes, as learning takes place. The trustworthiness and intelligence of this meta-collaboration is hence itself dynamic. ALC provides the mechanisms to both create structures to make the emergence happen, but also make this a permanently learning system.

Without a solution to make AI trustworthy the current enthusiasm might cool down again. ALC might well “save” the trustworthiness of AI.

‍

Annex : our discussion with ChatGPT « confessing » to its limitations and validating ALC-enabled CHI

Author: SAM ALTMAN: In our case, the guy that built GPT-1 sort of did it off by himself and solved this, and it was somewhat impressive, but no deep understanding of how it worked or why it worked. Then we got the scaling laws. We could predict how much better it was going to be. That was why, when we told you we could do a demo, we were pretty confident it was going to work. We hadn’t trained the model, but we were pretty confident. That has led us to a bunch of attempts and better and better scientific understanding of what’s going on. But it really came from a place of empirical results first. (quoted from the transcription of a “Unconfuse me” podcast from January 2024 with Bill Gates and Sam Altman)

What does Altman refer to when he talks about scaling laws?

‍

ChatGPT: When Sam Altman, the CEO of OpenAI, refers to "scaling laws" in the context of the development of models like GPT-1 and subsequent iterations, he is talking about a set of empirical observations and theoretical insights related to the scaling up of machine learning models, particularly language models.

In the field of artificial intelligence, scaling laws describe how the performance of machine learning models, especially language models like GPT-3, improves predictably with increases in key variables such as the size of the model (number of parameters), the size of the training dataset, and the amount of computational resources used for training. These laws have been crucial in guiding the development of more advanced AI models.

Key aspects of scaling laws include:

1. Performance Improvement with Size: As the size of the model (in terms of the number of parameters) increases, the model's performance on various tasks tends to improve. This improvement follows a predictable pattern, often logarithmic in nature.
2. Data and Compute Scaling: Along with the model size, performance also scales with the size of the training dataset and the computational resources (like the amount of GPU power) dedicated to training.
3. Diminishing Returns: While performance improves with scale, there's often a point of diminishing returns where further increases in size yield smaller improvements in performance.
4. Efficiency and Capabilities: Larger models tend to be more sample efficient (they learn more effectively from each example) and can exhibit a broader range of capabilities, such as understanding context, generating more coherent and contextually appropriate text, etc.

‍

Author: How is performance measured?

‍

ChatGPT: Performance in machine learning models, particularly in language models like GPT, is typically measured using a variety of metrics that assess how well the model performs specific tasks or achieves certain objectives. The choice of metrics depends on the tasks the model is designed for, such as text generation, translation, question answering, etc. Here are some common ways performance is measured:

1. Accuracy: This is a straightforward metric used in classification tasks, where the model's predictions are compared against true labels. It's the proportion of correct predictions out of all predictions made.
2. Precision and Recall: These metrics are often used in situations where the balance between false positives and false negatives is important. Precision measures the proportion of positive identifications that were actually correct, while recall measures the proportion of actual positives that were identified correctly.
3. F1 Score: This is the harmonic mean of precision and recall and is used as a single metric to balance the trade-off between precision and recall.
4. Perplexity: In language modeling, perplexity is a common metric. It measures how well a probability model predicts a sample. A lower perplexity score indicates better performance, meaning the model is more certain about its predictions.
5. BLEU Score (Bilingual Evaluation Understudy): This metric is often used for evaluating machine translation. It measures how many words and phrases in the machine translation match a human translation, considering several layers of matching (e.g., exact word matches, synonym matches).
6. ROUGE Score (Recall-Oriented Understudy for Gisting Evaluation): This is used primarily for evaluating text summarization. ROUGE measures the overlap between the generated summary and a reference (human-produced) summary.
7. Human Evaluation: Sometimes, especially for tasks like text generation, subjective human judgment is used to assess the quality, relevance, coherence, and fluency of the text produced by the model.
8. Task-Specific Benchmarks: For specialized tasks (e.g., question answering, sentiment analysis), there are often specific benchmarks or datasets with predefined metrics used to evaluate model performance.

The choice of metric or combination of metrics depends on the specific goals of the model and the nature of the task it is being used for. In the development of models like GPT, a combination of these metrics might be used to get a comprehensive understanding of the model's performance.

‍

Author: All these measures seem to depend on the accuracy / quality of the reference samples that are used to compared the outcome of AI with what it should be. Correct?

‍

ChatGPT: Yes, that's correct. Many of the performance measures for machine learning models, especially in tasks involving language processing, rely heavily on the quality and representativeness of the reference samples or datasets used for comparison. The accuracy and relevance of these benchmarks are crucial for several reasons:

1. Ground Truth for Comparison: In tasks like classification, translation, or summarization, models are often evaluated based on how closely their outputs match a set of pre-defined "correct" answers or high-quality examples. The quality of these reference samples directly impacts the validity of the evaluation.
2. Bias and Representativeness: If the reference samples are biased or not representative of the real-world scenarios in which the model will be used, the performance metrics may not accurately reflect the model's effectiveness in practical applications. For instance, a language translation model evaluated on a very narrow set of texts may appear highly accurate within that scope but may perform poorly on more diverse or complex texts.
3. Variability and Subjectivity in Language Tasks: For tasks like text generation or summarization, where there can be multiple valid outputs, the choice of reference samples can greatly influence how performance is measured. Different sets of reference texts might lead to different assessments of a model's quality.
4. Consistency and Standardization: In research and development, having a consistent and standardized set of reference samples (like benchmark datasets) is important for comparing different models and tracking progress over time.
5. Limitations of Automated Metrics: Many automated metrics (like BLEU for translation, or ROUGE for summarization) have limitations and can sometimes fail to capture nuances in language, meaning, or coherence. The quality of reference samples can mitigate or exacerbate these limitations.
6. Need for Human Evaluation: Due to the limitations of reference-based metrics, human evaluation is often necessary, especially for complex tasks like open-ended text generation, where the context, creativity, and fluency are harder to quantify with automated measures.

Therefore, while these measures provide essential quantitative assessments of a model's performance, they are often supplemented with qualitative evaluations and are continually refined to better represent the complexities of real-world language use.

‍

Author: So the accuracy will increase / decrease with the density and structuredness of existing knowledge in a field. Typically in a field where there is little knowledge the density, typically at the border of the space of existing knowledge, density will be lower and it will be more difficult to check the accuracy. Which will then favour hallucination by GPT. Correct?

‍

ChatGPT: Yes, your understanding is essentially correct. The performance of AI models like GPT in generating accurate and relevant responses is indeed influenced by the density and structure of existing knowledge in a particular field, especially when considering their tendency towards hallucination (producing factually incorrect or nonsensical information). Let's break this down:

1. Density and Structuredness of Knowledge: In fields where there is a rich, well-structured, and dense corpus of knowledge, AI models like GPT tend to perform better. This is because they have more high-quality data to learn from, allowing them to generate more accurate and coherent responses. Fields with extensive and well-documented research, clear terminologies, and established facts fall into this category.
2. Fields with Sparse or Emerging Knowledge: In contrast, in fields that are at the frontier of human knowledge, where information is sparse, less structured, or rapidly evolving, the model's performance can be less reliable. This is because the training data may not provide enough information for the model to learn accurate patterns, or the information might be outdated or too generalized.
3. Hallucination in AI Models: Hallucination in AI models refers to instances where the model generates information that is not supported by its training data or is factually incorrect. This issue is more pronounced in areas where the model has less reliable or less comprehensive training data. In fields with less density of knowledge, the model has fewer reference points to anchor its responses, making it more prone to generate responses based on incomplete patterns it has learned.
4. Accuracy Checks and Limitations: In well-established fields, it's easier to verify the accuracy of an AI model's outputs against a robust set of known facts and data. In emerging or less-explored fields, this verification is more challenging, as there might be fewer sources to cross-check information, and the knowledge base might be rapidly changing.
5. Continuous Learning and Model Updates: To mitigate these issues, continuous learning and regular updates of the model with the latest information are crucial, especially for fields where new knowledge is constantly being developed.

In summary, the performance and reliability of AI models like GPT in generating accurate and contextually appropriate content is significantly influenced by the quantity and quality of existing knowledge in a particular domain. This presents a challenge in cutting-edge or less-documented areas, where the model may have a higher propensity for generating incorrect or speculative content.

‍

Author: But knowledge is only available when somebody puts it into machine ad makes it available. So when it comes to areas such as science. where reality has not yet been discovered / understood, or for fake news where reality does not exist, the reliability of AI models will always be limited. Correct?But knowledge is only available when somebody puts it into machine ad makes it available. So when it comes to areas such as science. where reality has not yet been discovered / understood, or for fake news where reality does not exist, the reliability of AI models will always be limited. Correct?

‍

ChatGPT: Yes, your assessment is correct. The reliability of AI models, particularly those based on machine learning like GPT, is inherently limited by the nature and quality of the data available to them. This limitation is especially pronounced in areas where knowledge is either undiscovered, evolving, or where misinformation (such as fake news) is prevalent. Here's how these factors impact AI model reliability:Yes, your assessment is correct. The reliability of AI models, particularly those based on machine learning like GPT, is inherently limited by the nature and quality of the data available to them. This limitation is especially pronounced in areas where knowledge is either undiscovered, evolving, or where misinformation (such as fake news) is prevalent. Here's how these factors impact AI model reliability:

1. Undiscovered or Evolving Scientific Knowledge: In scientific fields where research is ongoing and new discoveries are continuously being made, AI models may not have the most current data. Since these models are trained on existing datasets, they cannot generate insights beyond what has already been discovered and documented. This limitation is significant in cutting-edge research areas where the frontier of knowledge is rapidly advancing.
2. Fake News and Misinformation: AI models can struggle to distinguish between factual information and misinformation or fake news. These models are trained on large datasets from the internet, which include both reliable and unreliable sources. If the training data contains misinformation, the model might learn to replicate these inaccuracies. This is a notable issue in areas where misinformation is widespread.
3. Dependence on Existing Data: AI models like GPT learn patterns and information present in their training datasets. They don't have the capability to independently verify the truthfulness of this information or to access information beyond their training data. Consequently, their output is only as reliable as the data they've been trained on.
4. Lack of Contextual Understanding and Critical Thinking: While AI models can process and generate text based on patterns they've learned, they don't possess true understanding or the ability to perform critical thinking. They can't evaluate the credibility of sources or the plausibility of information in the way humans can, which is especially crucial in discerning the veracity of complex or conflicting information.
5. Continual Learning and Model Updates: To address these limitations, there's a need for continuous updating of AI models with the latest and most accurate information, as well as incorporating mechanisms to identify and reduce the spread of misinformation. However, this is an ongoing challenge and does not completely eliminate the limitations.

In summary, the reliability of AI models in areas of undiscovered science or in the presence of misinformation is indeed constrained. These models reflect and amplify the information they are trained on, making the quality and veracity of their training data a critical factor in their reliability and usefulness.

‍

Author: Researchers such as Yejin Choi criticise ChatGPT's limitations in terms of common sense and of ethics. Would you agree with these statements?

‍

ChatGPT: The criticisms regarding the limitations of AI models like ChatGPT in terms of common sense reasoning and ethical judgment are valid and reflect ongoing challenges in the field of artificial intelligence. Here's a closer look at these limitations:

1. Common Sense Reasoning:
   - Limitation: AI models, including ChatGPT, often struggle with common sense reasoning, which involves understanding everyday knowledge about the world that humans consider obvious or intuitive. These models can generate responses that are technically correct within the context of their training data but lack the nuanced understanding that comes from real-world experience and common sense.
   - Reason: This limitation stems from the fact that AI models are trained on textual data and lack the breadth of experiences and interactions with the physical world that humans have. They operate on patterns in data rather than on an understanding of real-world principles.
2.   Ethical Judgment:
   - Limitation: AI models also face challenges in making ethical judgments. They can inadvertently generate content that is biased, offensive, or inappropriate. Ethical reasoning involves complex, context-dependent judgments that these models are not currently equipped to handle reliably.
   - Reason: Ethical reasoning requires a deep understanding of cultural, societal, and situational contexts, as well as the ability to weigh conflicting values and principles, a level of sophistication that current AI models do not possess. Additionally, their training data may contain biases or unethical viewpoints, which the model might learn and replicate.

Researchers like Yejin Choi and others in the field are correct in pointing out these limitations. Addressing these challenges involves not just technological advancements in AI but also careful consideration of the ethical, cultural, and societal contexts in which these technologies operate. Ongoing research is focused on improving AI's understanding of common sense and ethical reasoning, but these remain areas where human oversight and judgment are crucial.

‍

Author: How could that substantially change in view of what we discussed earlier in this chat about the "density" problem? Common sense typically relies on complex unconscious processes which are not formalised and hence not available as knowledge which GPTs could use to learn. Put it differently, intuition is in the space of sparse (the opposite of dense) knowledge. Any suggestions?

‍

ChatGPT: Addressing the limitations of AI models like GPT in areas of common sense and intuition, especially considering the "density" problem and the nature of these concepts as often unformalized and unconscious, is a significant challenge. However, there are several approaches and research directions that could potentially help in bridging this gap:

• Hybrid AI Systems: Combining machine learning models with rule-based systems or symbolic AI could help encode some aspects of common sense that are currently hard to learn from data alone. Symbolic AI can represent explicit logical rules and relationships that might capture aspects of common sense reasoning.
• Expanding Training Data: Incorporating a broader range of data sources, including those that explicitly focus on common sense scenarios, everyday human experiences, and intuitive problem-solving, can help improve the model's understanding. This can include structured data from knowledge bases designed to capture common sense knowledge.
• Advancements in Representation Learning: Developing more sophisticated methods for representation learning that can capture deeper, more abstract relationships and concepts could lead to better common sense reasoning. This involves creating AI models that can understand and represent more complex and abstract ideas beyond surface-level patterns.
• Interactive Learning and Feedback Loops: Implementing interactive learning systems where models can receive feedback on their outputs and adjust their understanding accordingly can help improve their grasp of common sense. This can include human-in-the-loop systems where human feedback helps refine the model's responses.
• Neurosymbolic AI: Combining neural network approaches with symbolic reasoning, neurosymbolic AI attempts to leverage the strengths of both: the pattern recognition ability of neural networks and the logical reasoning of symbolic AI. This might lead to better handling of sparse knowledge areas like intuition.
• Psychological and Cognitive Models: Integrating insights from psychology and cognitive science about how humans process information and develop common sense understanding can inform the development of AI models. This can involve translating cognitive and psychological theories into computational models.
• Ethical and Common Sense Frameworks: Developing and implementing frameworks for ethical reasoning and common sense that can guide AI behavior. This includes principles and guidelines that the AI can use as a reference in ambiguous or complex situations.
• Focused Research on Sparse Knowledge Areas: Directly addressing the challenge of sparse knowledge by researching how to effectively train AI models in these areas, perhaps by finding ways to abstract and generalize from limited data.

It's important to recognize that these suggestions represent significant research and development efforts and are part of an ongoing journey in AI advancement. Each approach comes with its own set of challenges and would require substantial development to be effectively implemented. The goal is to create AI systems that can better approximate human-like understanding and reasoning in areas that are currently difficult for them, such as common sense and intuition.

‍

Author: Well, I think that there some intrinsic paradoxes and contradictions in what you outline above. Science is bound by epistemological constraints such as the possibility of falsification. How can you falsify if some unconscious intuition is correct or incorrect? Can you know something about something you are ignorant about?

‍

ChatGPT: You've highlighted some fundamental philosophical and epistemological challenges that are at the core of advancing AI, particularly in areas like common sense and intuition. Let's explore the paradoxes and constraints you mentioned:

1. Falsifiability and Intuition:
   - The principle of falsifiability, as proposed by philosopher Karl Popper, states that for a theory to be considered scientific, it must be testable and potentially falsifiable.
   - Intuition, being a form of knowledge that often operates unconsciously and is not always based on explicit reasoning or evidence, presents a challenge to this principle. It's difficult to construct a formal test to falsify intuitive knowledge because it's not always based on a clear set of premises or logical structures that can be objectively assessed.
2. Knowledge of the Unknown:
   - The question of whether one can know something about what they are ignorant of touches on deep philosophical issues. In the context of AI, this translates to the problem of how an AI can understand or reason about concepts that it has never encountered in its training data.
   - AI models like GPT are limited to the knowledge they've been trained on. They don't "know" in the human sense but rather identify patterns and generate outputs based on these patterns. This limitation becomes apparent when dealing with novel concepts or situations not covered in the training data.
3. Contradictions in AI Development:
   - On one hand, there's an effort to make AI more human-like in its reasoning and understanding, including developing common sense and intuitive capabilities.
   - On the other hand, AI fundamentally lacks human experiences, consciousness, and the biological and psychological processes that underlie human intuition and common sense.
4. Epistemological Constraints in AI:
   - AI's learning and knowledge are constrained by the data it is trained on and the algorithms it uses. This limitation is a significant barrier to achieving true understanding or consciousness in AI.
   - The pursuit of embedding human-like intuition and common sense in AI runs into the challenge of translating inherently human, often subjective experiences into quantifiable data that can be processed by machines.

In conclusion, integrating common sense and intuition into AI involves grappling with complex philosophical, epistemological, and practical challenges. It requires not only technological advancements but also a deeper understanding of the nature of knowledge, consciousness, and the human experience. This endeavor is at the frontier of AI research and is replete with intrinsic paradoxes and contradictions, as you've astutely noted.

‍

Author: please read this article and tell me if there is anything you disagree with: https://www.sym.place/faq/could-advanced-artificial-intelligence-ai-technologies-such-as-chatgpt-not-just-replace-symplace-how-augmented-learning-collaboration-alc-enables-collaborative-hybrid-intelligence-and-makes-ai-more-trustworthy

‍

ChatGPT: The article on SymPlace discusses the relationship between advanced AI technologies like ChatGPT and Augmented Learning Collaboration (ALC). It argues that while AI technologies offer significant potential in various domains, they cannot replace human collaboration due to their inherent limitations. These limitations include the inability to fully understand knowledge boundaries, the risk of generating misinformation, and the lack of "right brain" capabilities such as intuition, feelings, and ethical values. The article also addresses the challenges of AI trustworthiness and suggests that ALC can enhance AI's potential by combining human intuition and experience with AI's data processing capabilities, leading to a more trustworthy and effective hybrid collaborative environment. This approach aims to balance AI's limitations with human insights, control data inputs and outputs, and ensure reliable, bias-free AI. The article views intelligence and trust as dynamic, emergent properties of collaborative systems within ALC.

For a detailed understanding, you can read the full article here.

‍

Author: You did not answer my question! Is there anything you disagree with in this article?

‍

ChatGPT: As an AI developed by OpenAI, I don't have personal opinions or disagreements. I can analyze and provide information based on the data I have been trained on. The article from SymPlace discusses the integration of Augmented Learning Collaboration (ALC) with AI technologies like ChatGPT. It highlights the limitations of AI in fully understanding knowledge boundaries, its potential for misinformation, and the absence of human-like intuition and ethical reasoning. It suggests that ALC can enhance AI by combining human capabilities with AI's data processing strengths, aiming for a more effective and trustworthy hybrid system. This perspective aligns with current understandings in AI development, particularly the emphasis on hybrid systems that combine human and artificial intelligence to address the limitations of AI alone.

For a detailed understanding, you can read the full article here.

‍

Author: Well, let me ask the question differently: is everything written in this article factually correct?

‍

ChatGPT: The concept of Augmented Learning Collaboration (ALC)-enabled Collaborative Hybrid Intelligence (CHI) as a promising and necessary approach to address the limitations of AI is supported by current trends and research in the field. The idea is that combining human insights and intuition with AI's computational capabilities can lead to more effective, reliable, and ethically sound AI systems. This approach is seen as a way to mitigate AI's limitations in understanding complex, nuanced knowledge and in making ethical decisions. It reflects a growing recognition in AI research and development that human-AI collaboration can enhance the strengths and compensate for the weaknesses of both.

For a detailed understanding, you can read the full article here.

‍