Guidelines for doing a cognitive science project, part 7: The power of Learning

In the last post, I talked about the power of explanation, which allows your cognitive computational agent to explain to you how it reached from the inputs to the outputs. In this post, I talk about another power: the power of learning. The idea with this power is for the agent to learn to do the cognitive task the way humans learn.
         In our spreading activation network, this was done by using language processing. You speak sentences to the agent like "Germany won the world cup". It automatically picks up Germany as a team and creates a node in the network called Germany, picks up world cup as an attribute and creates a world cup node in the network and links it to the Germany node and finally, it picks up 'won' and causes the Germany link to world cup to push Germany toward the win side. This is exactly how humans learn. Think about the last time you read a news article about a sports team you did not know much and how certain facts stuck in your brain in the form of concepts linked to that sports team.
        Another very powerful way of learning is learning through mistakes. We make predictions and then compare the predictions to actual results. If the result matches our prediction, the underlying model that led to the prediction is strengthened. On the other hand, if the actual result does not match our prediction, the underlying model is modified or certain links are weakened. In our cognitive agent, we have implemented this ability of learning from mistakes. So, we feed the agent actual results of the matches that it predicts. Using these results, the agent learns which means that certain links in the underlying spreading activation network are strengthened while others may be weakened.          In summary, two ways of learning: 1) learning from language, and 2) learning from mistakes have been implemented in our cognitive agent. While in this post, I talk about these two aspects from the standpoint of our project, these are two general ways of learning that anyone doing a cognitive science project should consider implementing.   

Guidelines for doing a cognitive science project, part 6: The power of Explanation

When creating a cognitive computational agent, you want the agent to have the power of explanation. Meaning, you don't just want it to be a black box that takes in inputs and gives an output without really explaining why. If that were the case, the agent wouldn't be useful to you. Add in the power of explanation, and now the whole process that your agent followed to create the output becomes visible to you. Now, you can pinpoint the exact route followed by your agent to convert the inputs to the outputs. So, when you use the agent the next time to predict your friends performance in that cognitive task, you won't just be able to tell him what his result will be, you'll be able tell him exactly why his result is the way it is. You'll be able to tell him all the factors that he considered, what were the options he weighed and how exactly he reached the end state of the cognitive task. Isn't that cool?

          Continuing with the example of our cognitive task: predicting the winner of a football match, in a previous post, I describe the spreading activation network that we used for the computational agent. Adding in the power of explanation for this agent was tightly coupled with the task of visualization. We had to first visualize the network of teams and their associated attributes as a graph with teams and attributes represented as nodes and the associations between teams and their respective attributes represented as edges. Now, to add in the power of explanation, we highlight the spreading of the activation in the network. For instance, if a game between team A and team B is being predicted, only team A and team B light up. Then, all of the linked attributes to the two teams light up. Next, the activation and inhibition are seen as pushing one team toward the win node and the other team toward the lose node. Thus, the whole process of making a prediction has been visualized and the spreading activation network explains the path it followed toward making the prediction.

        

Guidelines for doing a cognitive science project, part 5: The computational agent

You've defined the problem. You've done the interviews. You've painstakingly parsed the interviews to extract the cognitive process. Now you're all set to create your cognitive computational agent. How do you go about it? The answer is think about what capabilities you want this agent to have. For example, if you want this agent to have the power of explanation, a neural network is not going to work. With all those nodes in hidden layers, its a difficult task to extract what exactly each node is doing. Without knowing exactly what each node is doing, there is no way your agent can explain how it reached from inputs to outputs thereby rendering explanation impossible. Similarly, some other cool things you might want your agent to do might be have the ability to learn, display emotions, show cultural cognition, be capable of distributed cognition. Therefore, it is important to be clear at the outset about the capabilities you want the agent to have.
            In our case, this agent was a spreading activation network. What is a spreading activation network? It basically is a model of working memory, representing how the mind processes related ideas. Think about the word 'dog'. Did the words 'pet', 'bark', 'tail' automatically come to your mind? How did that happen? This is called the 'priming effect'. When you are presented with a concept, the concepts most closely associated to that concept are activated in your brain.
           For our football prediction scenario, we observed from our interviews that people associated attributes with teams and used these attributes to reach their predictions. Thus, the moment someone thinks of say, 'Germany', the thought 'World Cup Winner' is activated and it guides the prediction process. Thus, when a game between two teams is being predicted, the related attributes of both teams are activated and this activation spreads to other related attributes creating a spreading activation network. Now, the activation causes some teams to gravitate toward the 'Win' side and others to gravitate toward the 'Lose' side. Moreover, the spreading activation network explains exactly how it made the prediction. Read the next post to find out how. 

Guidelines for doing a cognitive science project, part 4: From interviews to code (what to do)

The previous post talked about our failed attempt at using a decision tree to represent how people predicted the results of football matches. In this post, I describe how we actually went about solving the problem of finding a computational representation.
       Let me reiterate a point that I've made many times. Interviews are the gold standard. Therefore, you start from the interviews. In our case, we went back to the interview transcripts and tried to see them not just as data with sets of predictions but as a representation of a process. The process in question was: given two teams, how do you reach a prediction about the match between the two given teams? It was fascinating how changing the lens helped us understand the process. While previously, we were looking at the interviews just as training data and test data, now we realized that the interviews basically documented the whole process followed, and they already contained the answer that we were looking for.
      What immediately jumped out from the interviews was that the process did not always start at the same point (thus rejecting the idea of anything like a root node in a decision tree). Instead, based on different teams being considered, people started their thought process with a different feature. Further, thoughts about one feature led to another feature being considered. So, something like a spreading activation of thoughts. Finally, a lot of the thinking was comparative in nature. So, the two teams were being compared on the same feature for most of the time leading to the team with the stronger feature inhibiting the other team's chances of winning.
       These observations pointed toward the following realizations: 1) Features form the core of the decision making process, 2) Features can link to other features, 3) Activation and inhibition are part of the process, and 4) There is no hierarchy or ordering of features to consider. With these realizations, we decided that a spreading activation network would be able to model the prediction process. Details about the spreading activation network in the next post.
        

Guidelines for doing a cognitive science project, part 3: From interviews to code (what not to do)

In this post, I explore the process behind converting interviews to a computational agent that can replicate the cognition represented in the interviews. The most important thing to keep in mind is, as always, avoid bias. The bias in this case is not the same bias that I discussed in a previous post about interviewing people to gather data for a cognitive science project . In that post, I discussed avoiding biasing the interviewee by planting your own ideas while asking the interview questions. In this post, the bias is about biasing your computational agent. Let me explain how.
       Being a team of engineers from AE, CS and ECE, we are rather too familiar and in love with decision trees. For us, decision trees are the one stop solution to all problems. Since the cognitive task at hand, predicting the results of a football match, involved making a decision, we jumped right into programming a decision tree to be our computational agent without really considering the implications or ramifications. More importantly, we never really considered what exactly we were trying to model before jumping into decision trees.
       Sure enough, when the time came for us to actually see if our decision tree could replicate the predictions made by our survey participants, nothing worked and the decision tree did not match the predictions at all. Further, it was really hard for us to get the decision tree to get to any prediction at all. What was going wrong?
        The simple answer is: we were trying to fit an existing answer to a question instead of arriving at an answer that actually solves the question. In cognitive science, the interviews are everything. They are the gold standard against which everything should be measured. Therefore, the solution process should also be guided by the interviews. In our attempt at fitting a decision tree to be the cognitive agent, we never considered the interviews and whether a decision tree really represents how people made football predictions. For instance, a decision tree starts from the top and branches out toward the bottom. This automatically implies that there is a hierarchy in the nodes. Now, as we were trying to represent the features considered in predicting football matches to be nodes of the tree, we were automatically assuming a hierarchical ordering in the features. This absolutely contradicted how people actually made predictions.
       Thus, decision trees did not work. Read the next post to see how we figured out what actually would work.  

Guidelines for doing a cognitive science project, part 2: Interviewing people to gather data

In this post, I try to look back on our interview process with the goal of gathering data for the project on how people make predictions about football match outcomes in the scenario of betting on them.
     The most common response that you would get when interviewing people for a cognitive science project is "My intuition is saying...". The goal is to unravel what goes on behind the scenes of that intuition. It is surprising how infrequently people actually stop and think how they got to that intuition in the first place. Therefore, the goal of the interviewer is to ask questions to get to the bottom of this intuition. However, the interview questions have to be such that: 1) The interviewee does not get biased toward the opinions held by the interviewer and, 2) The interviewer does not plant the seeds of ideas into the interviewee's brain.
      The above points of what the interview questions should not do must be hammered home. If you ask questions in such a way that there are hints within the question toward the answer, you run the risk of planting that idea in the interviewee's brain. Therefore, you basically get answers that do not really reflect the interviewee's thought process but rather your own ideas. From my experience, I have always found it better to ask open ended questions. Let the interviewee ask you questions to get more specific rather than you setting out with a very specific question (possibly which has a hint in its very specificity), which does not let you have that wiggle room necessary to extract the thought process of the interviewee.
      The goal in the data gathering phase was to extract the process people follow to reach a prediction about football matches. Thus, I started the interviews by asking the interviewees to imagine a scenario where they would have to bet on a football game. This way, I tried to scope down the possible responses I could get from them as to the reasons and the process behind their prediction. For instance, reasons like "I lived in France so France will win" were eliminated (although they might be very valid reasons for supporting France but not betting on France). As the eventual goal was to create a computational agent to model and simulate this cognitive process, it was essential to weed out such phenomena that could not really be used for modeling and simulation purposes.
      Once I'd established this betting scenario, I gave the interviewees a set of games to predict and made it very interactive. I had them talk through their thought process including information gathering and inference making. So, if they had to browse the web for a specific statistic or news article, I made a note of which web sites they visited and what bits of information they used from each of these web-sites. Note again that I was playing the role of passive listener/observer thus avoiding any chance of my biasing their thought process. Even in the cases where they asked me for some information, I replied with "Look that up on the internet if you need to".
      Finally, at the end of the process when their predictions were made, I followed up with some clarification questions. For instance, one interviewee said that he thought some player was in-form. My follow up was, "What do you mean by in-form? Can you explain it a bit further to me?" In this way, I was able to get to the bottom of the thought process along with the specific terminology that they were using to denote various considerations. In conclusion, interviews are meant to uncover what goes on actually within intuition and it is extremely important to structure the interviews to enable it.    

Guidelines for doing a cognitive science project, Part 1: Problem definition

In this series of posts, I try and establish a guidelines that now, in hindsight, would have been very useful to know before embarking on doing a cognitive science project. This particular post is about the problem definition process.

          The first step while undertaking any project is problem definition. I consider this to be an art because I see no hard and fast rules to follow for problem definition. Essentially, it is a trade-off between activity and planning. While you don't want to mindlessly set out on a wild goose chase without any clear direction, at the same time, you cannot be stuck in a rut trying to plan out every detail. I think the right balance is to start doing something while at the same time trying to find all the flaws with the direction of what you're doing. Sounds paradoxical, but the only way to keep the creative juices flowing is by actually creating. 

           The project that me and my team were doing was studying how people make predictions about the outcome of football matches (you can call it soccer, but I refuse to call it anything but football). Football is a sport that I love, play and follow, and this was while the Euros were going on so seemed like a great project to do. 

            As I have stressed earlier, it is really important to get started doing something. Had we got stuck trying to dissect the sentence "people predict the outcome of football matches", we would have quite possibly given up right there. Since the project was on cognition, and in this case, human cognition, the best way to get started was introspection. This way, you don't end up bothering anybody else by asking them to predict a ton of football games and at the same time, you don't sit around wondering where to go with the project.   

            The introspection resulted in the realization that emotions were playing a big role and the reasons could be literally anything: "I lived in France so France would win", "I love Italian cuisine so Italy would win". Being an engineer, I love to model things and use the model for predictions. But with reasons so arbitrary, and such a plethora of factors, any kind of analysis let alone prediction seemed a far cry indeed. 

            However, now that I had an idea of the rabbit hole that emotions could lead to, this helped me find things that drive me away from this rabbit hole. We revisited the problem statement and instead of predict, we said bet on. Thus the new problem statement became: "How do people bet on the outcomes of football matches"? This immediately scoped down the problem because now the situation wasn't just trying to see how people support teams, it was trying to analyze who they would put their money on. This made the problem amenable to all forms of model building with its associated analysis and predictive power.
             In conclusion, don't be surprised if cognitive science projects start off being rather broad and rather vague. Start doing something, start trying to build a model and this activity will make you think about the task and better directions would emerge.