In this post, I try to look back on our interview process with the goal of gathering data for the project on how people make predictions about football match outcomes in the scenario of betting on them.
The most common response that you would get when interviewing people for a cognitive science project is "My intuition is saying...". The goal is to unravel what goes on behind the scenes of that intuition. It is surprising how infrequently people actually stop and think how they got to that intuition in the first place. Therefore, the goal of the interviewer is to ask questions to get to the bottom of this intuition. However, the interview questions have to be such that: 1) The interviewee does not get biased toward the opinions held by the interviewer and, 2) The interviewer does not plant the seeds of ideas into the interviewee's brain.
The above points of what the interview questions should not do must be hammered home. If you ask questions in such a way that there are hints within the question toward the answer, you run the risk of planting that idea in the interviewee's brain. Therefore, you basically get answers that do not really reflect the interviewee's thought process but rather your own ideas. From my experience, I have always found it better to ask open ended questions. Let the interviewee ask you questions to get more specific rather than you setting out with a very specific question (possibly which has a hint in its very specificity), which does not let you have that wiggle room necessary to extract the thought process of the interviewee.
The goal in the data gathering phase was to extract the process people follow to reach a prediction about football matches. Thus, I started the interviews by asking the interviewees to imagine a scenario where they would have to bet on a football game. This way, I tried to scope down the possible responses I could get from them as to the reasons and the process behind their prediction. For instance, reasons like "I lived in France so France will win" were eliminated (although they might be very valid reasons for supporting France but not betting on France). As the eventual goal was to create a computational agent to model and simulate this cognitive process, it was essential to weed out such phenomena that could not really be used for modeling and simulation purposes.
Once I'd established this betting scenario, I gave the interviewees a set of games to predict and made it very interactive. I had them talk through their thought process including information gathering and inference making. So, if they had to browse the web for a specific statistic or news article, I made a note of which web sites they visited and what bits of information they used from each of these web-sites. Note again that I was playing the role of passive listener/observer thus avoiding any chance of my biasing their thought process. Even in the cases where they asked me for some information, I replied with "Look that up on the internet if you need to".
Finally, at the end of the process when their predictions were made, I followed up with some clarification questions. For instance, one interviewee said that he thought some player was in-form. My follow up was, "What do you mean by in-form? Can you explain it a bit further to me?" In this way, I was able to get to the bottom of the thought process along with the specific terminology that they were using to denote various considerations. In conclusion, interviews are meant to uncover what goes on actually within intuition and it is extremely important to structure the interviews to enable it.
The most common response that you would get when interviewing people for a cognitive science project is "My intuition is saying...". The goal is to unravel what goes on behind the scenes of that intuition. It is surprising how infrequently people actually stop and think how they got to that intuition in the first place. Therefore, the goal of the interviewer is to ask questions to get to the bottom of this intuition. However, the interview questions have to be such that: 1) The interviewee does not get biased toward the opinions held by the interviewer and, 2) The interviewer does not plant the seeds of ideas into the interviewee's brain.
The above points of what the interview questions should not do must be hammered home. If you ask questions in such a way that there are hints within the question toward the answer, you run the risk of planting that idea in the interviewee's brain. Therefore, you basically get answers that do not really reflect the interviewee's thought process but rather your own ideas. From my experience, I have always found it better to ask open ended questions. Let the interviewee ask you questions to get more specific rather than you setting out with a very specific question (possibly which has a hint in its very specificity), which does not let you have that wiggle room necessary to extract the thought process of the interviewee.
The goal in the data gathering phase was to extract the process people follow to reach a prediction about football matches. Thus, I started the interviews by asking the interviewees to imagine a scenario where they would have to bet on a football game. This way, I tried to scope down the possible responses I could get from them as to the reasons and the process behind their prediction. For instance, reasons like "I lived in France so France will win" were eliminated (although they might be very valid reasons for supporting France but not betting on France). As the eventual goal was to create a computational agent to model and simulate this cognitive process, it was essential to weed out such phenomena that could not really be used for modeling and simulation purposes.
Once I'd established this betting scenario, I gave the interviewees a set of games to predict and made it very interactive. I had them talk through their thought process including information gathering and inference making. So, if they had to browse the web for a specific statistic or news article, I made a note of which web sites they visited and what bits of information they used from each of these web-sites. Note again that I was playing the role of passive listener/observer thus avoiding any chance of my biasing their thought process. Even in the cases where they asked me for some information, I replied with "Look that up on the internet if you need to".
Finally, at the end of the process when their predictions were made, I followed up with some clarification questions. For instance, one interviewee said that he thought some player was in-form. My follow up was, "What do you mean by in-form? Can you explain it a bit further to me?" In this way, I was able to get to the bottom of the thought process along with the specific terminology that they were using to denote various considerations. In conclusion, interviews are meant to uncover what goes on actually within intuition and it is extremely important to structure the interviews to enable it.
No comments:
Post a Comment