In this post, I explore the process behind converting interviews to a computational agent that can replicate the cognition represented in the interviews. The most important thing to keep in mind is, as always, avoid bias. The bias in this case is not the same bias that I discussed in a previous post about interviewing people to gather data for a cognitive science project . In that post, I discussed avoiding biasing the interviewee by planting your own ideas while asking the interview questions. In this post, the bias is about biasing your computational agent. Let me explain how.
Being a team of engineers from AE, CS and ECE, we are rather too familiar and in love with decision trees. For us, decision trees are the one stop solution to all problems. Since the cognitive task at hand, predicting the results of a football match, involved making a decision, we jumped right into programming a decision tree to be our computational agent without really considering the implications or ramifications. More importantly, we never really considered what exactly we were trying to model before jumping into decision trees.
Sure enough, when the time came for us to actually see if our decision tree could replicate the predictions made by our survey participants, nothing worked and the decision tree did not match the predictions at all. Further, it was really hard for us to get the decision tree to get to any prediction at all. What was going wrong?
The simple answer is: we were trying to fit an existing answer to a question instead of arriving at an answer that actually solves the question. In cognitive science, the interviews are everything. They are the gold standard against which everything should be measured. Therefore, the solution process should also be guided by the interviews. In our attempt at fitting a decision tree to be the cognitive agent, we never considered the interviews and whether a decision tree really represents how people made football predictions. For instance, a decision tree starts from the top and branches out toward the bottom. This automatically implies that there is a hierarchy in the nodes. Now, as we were trying to represent the features considered in predicting football matches to be nodes of the tree, we were automatically assuming a hierarchical ordering in the features. This absolutely contradicted how people actually made predictions.
Thus, decision trees did not work. Read the next post to see how we figured out what actually would work.
Being a team of engineers from AE, CS and ECE, we are rather too familiar and in love with decision trees. For us, decision trees are the one stop solution to all problems. Since the cognitive task at hand, predicting the results of a football match, involved making a decision, we jumped right into programming a decision tree to be our computational agent without really considering the implications or ramifications. More importantly, we never really considered what exactly we were trying to model before jumping into decision trees.
Sure enough, when the time came for us to actually see if our decision tree could replicate the predictions made by our survey participants, nothing worked and the decision tree did not match the predictions at all. Further, it was really hard for us to get the decision tree to get to any prediction at all. What was going wrong?
The simple answer is: we were trying to fit an existing answer to a question instead of arriving at an answer that actually solves the question. In cognitive science, the interviews are everything. They are the gold standard against which everything should be measured. Therefore, the solution process should also be guided by the interviews. In our attempt at fitting a decision tree to be the cognitive agent, we never considered the interviews and whether a decision tree really represents how people made football predictions. For instance, a decision tree starts from the top and branches out toward the bottom. This automatically implies that there is a hierarchy in the nodes. Now, as we were trying to represent the features considered in predicting football matches to be nodes of the tree, we were automatically assuming a hierarchical ordering in the features. This absolutely contradicted how people actually made predictions.
Thus, decision trees did not work. Read the next post to see how we figured out what actually would work.
No comments:
Post a Comment