⚠

Problems Assesment

Problems encountered while progressing the KRE Final Project - Cognitive Bias.

Limitation of LLM (Large Language Model)
An early problem became apparent when we are initially using ChatGPT to generate a definition for each of our biases. The free version of the AI model was outdated in terms of information, and its knowledge became questionable as we detected flaws within the definition that the AI formulated. 
An example of ChatGPT prompt for one of the bias definition
LLM - User Story and CQs
The next problem we encountered was the difficulty in associating the user stories that we have gathered from the AI to the competency questions (CQs) that was required by the eXtreme Design. To solve this, we wrote the CQs ourselves and analyzed whether the user stories could solve the questions or not. 
Evan's Personal Comment
Personally, I found the user stories from the AI to be able to answer the CQs, and I was able to progress the project further. Furthermore, asking the AI to answer the CQs based on the previous user story its generated also proved to be fruitful, as it was capable of answering it as long as the answer is analyzed further. Therefore, I decided to 'feed' the AI our own definition of the biases first to ensure the information to stay updated.
Leonardo's Personal Comment
I personally found the user stories created by the AI to be too broad and vague to be useful in the extraction of CQs. The stories generated by ChatGPT seemed to be too concerned with the general influence that a bias has in the context in which it appears, rather than with the way in which it functions and the elements from which it is composed.  
Nicole's Personal Comment
I found the scenarios proposed by AI generally sufficient for describing cognitive biases. I appreciated their brevity as it allowed me to quickly identify the essential elements that artificial intelligence deemed indispensable for each bias. Of course, at times, the stories were somewhat repetitive, but I believe it was functional. Nevertheless, delving into scientific papers provided a deeper understanding.
Framester Frames' Limited Entities
The next challenge was using Framester Frames to use in the ontologies that the AI had made. We were required to align existing entities with the Framester Frames within our own ontology using the QUOKKA tool. We found our progress to be quite hindered by this predicament, as we found most of the AI-made ontology to be incompatible/unable to be aligned with any of the entities within Framester Frames. 
Upon further research, we found adequate amount of entities from the Framester Frames that we are able to integrate to our own ontologies, although the keyword need to be changed in order to conform with the chosen Framester Frames' entities. Furthermore, additional meeting with the supervisors of the projects explained that we should not limit ourselves to the Framester Frames.
Lack of Framester Frames for the specified keyword - "MemoryPerformance"
Choosing The Appropriate ODPs
Another problem that we found is choosing the appropriate ODPs for each of our cognitive biases ontology. We found there are only few ODPs that correspond to the cognitive bias phenomenon, which corresponds to the field of Humanities, Psychology, and Social Studies. 
At first, we addressed this problem by choosing certain ODPs which fit our understanding of each biases that we are working on. A further meeting with Prof. Gangemi solved the problem further as he told us about the freedom of using multiple ODPs and integrating it in each of our ontologies. Therefore, four of the problems are able to be addressed to enable progress to our project. 
List of potential ODPs to be chosen.
Summary
To summarize, here are the problems we face in the project, with checklists to keep track of them:
​Limitation of LLM - Outdated knowledge for definition of biases. 
Solution: Write own definitions first and give it to the AI before generating user story.
​Limitation of LLM - Associating AI-generated user stories with CQs can be difficult. 
Solution: Ask the AI the CQs based on the generated user stories.
​Framester Frames' Limited Entities - Inadequate choice of keywords.
Solution: Compromise by not rigidly following the keywords, look for alternative entities.
​Choosing The Appropriate ODPs - Few appropriate ODPs can be chosen to create an ontology out of the cognitive biases.
Solution: Use multiple ODPs is allowed, use only the appropriate properties for the ontologies. 
​

Evan's Personal Comment
Personally, I found the user stories from the AI to be able to answer the CQs, and I was able to progress the project further. Furthermore, asking the AI to answer the CQs based on the previous user story its generated also proved to be fruitful, as it was capable of answering it as long as the answer is analyzed further. Therefore, I decided to 'feed' the AI our own definition of the biases first to ensure the information to stay updated.

Leonardo's Personal Comment
I personally found the user stories created by the AI to be too broad and vague to be useful in the extraction of CQs. The stories generated by ChatGPT seemed to be too concerned with the general influence that a bias has in the context in which it appears, rather than with the way in which it functions and the elements from which it is composed.

Nicole's Personal Comment
I found the scenarios proposed by AI generally sufficient for describing cognitive biases. I appreciated their brevity as it allowed me to quickly identify the essential elements that artificial intelligence deemed indispensable for each bias. Of course, at times, the stories were somewhat repetitive, but I believe it was functional. Nevertheless, delving into scientific papers provided a deeper understanding.

Methodology

Ontologies

Last modified 2h ago