Dissertation Data Collection Lessons

This blog post represents my personal reflections on the dissertation data collection process for people doing social science dissertations. The lessons I have drawn from my experience may or may not help others. This post is primarily a way for me to get my thoughts in order about the experiences I have had. Any other impact is a bonus.

My data collection opportunity was not typical. I collected data for my dissertation through interviews and observations of training events. The research site I found does tough cases year round, but their density (cases per day and number of learners) was rather low most of the year. They only did the number of cases in a time period that suited rapid collection of data (in just 4 weeks as opposed to 4 months or longer) in February. This meant that it was critical for me to produce an acceptable proposal, defend it, and get it submitted to the IRB before Christmas of the year I went ABD (August 2014). I communicated this to my Chair who worked magic with me (my writing organization needed a work), the committee, and the GSEHD senior leadership to make sure my proposal did not sit on anyone's desk. The limiting factor was not how many hours a week I worked, but when participants were available for me. I put a lot of pressure on myself to get a lot of data quickly, because my dissertation completion date affected every major thing I want to do in my life from here on. From what I have read, this is typical of other doctoral students.

I performed quite a few interviews before I had time to review the transcript for a single one. This was a HUGE error. I had taken a lot of time before going to the site for data collection to review my interview guides, create customized notes about protecting participant anonymity, and write notes to make sure that I faithfully executed the participant protections to which I had committed to the university and site IRBs. However, after my first progress update, my Chair very quickly pointed out that I was not honoring the commitment in my proposal to have the first of each kind of interview peer reviewed and checked by the Chair before continuing. Uh-oh. For other doctoral students, if reviews of this nature are not part of your dissertation proposal, perhaps it should be. Read on to understand how they impacted me.

I was making horrific errors in the interviews. I had a feeling this might be the case as I was doing the interviews. It was an uncomfortable, "not sure I am doing this right" kind of feeling that I more sensed than recognized or understood as I did the interviews. Once I had the transcripts in hand, I could clearly see the errors: leading the people I was interviewing, talking too much and liberally interjecting my opinion, interrupting speakers to make a comment or ask a question, summarizing their opinions for them, saying things like "interesting," "that was helpful," and "okay," just like I would in normal conversation, and asking too many questions at once because I had written them in the interview guides that way. These errors may not look so bad, but they could have the effect of contaminating the data with my opinions. This is a major impact for the study’s trustworthiness. My follow up for interesting leads provided by the participants was also insufficient.
I needed to do a better job of balancing providing enough context for the question and my tendency to talk too much, and lead the participants. To do this, I largely followed the interview protocols of the proposal. Despite the specific comments from my Chair, I could not clearly tell the difference between the need to provide a conversational "flow” to the questions and going overboard with "extraneous blather." Besides slowing down to get peer review, the other big lesson for me was not to wait for transcripts to analyze my interviewing performance. I should have listened to the audio recordings a few times after each interview. This probably would have been tedious, but well worth the investment of my time.

Other advice my Chair gave me about doing interviews:

* You have to imagine that you are allotted seven words to ask the participant, and then you have to put tape over your mouth (metaphorically speaking). If it helps, put your hand over your mouth (this helped me).
* Your are an investigative reporter. You are out there to get their opinions. Indicating to them in anyway that you “like", "don’t like", or are surprised (any reaction) by anything they are saying is giving them a cue for anything they say after that.
* If someone asks, “What do you think?” about your study, you have to say “I will be happy to share my thoughts with you at the end of the study. Right now, I am interested in your thoughts.” At the end of the *study*, not the end of the interview. The person might talk to others in the class based on what you say to him.

After getting my first set of interview performance feedback, I felt humbled, embarrassed, frustrated with myself, and that I had let my Chair and participants down, especially since I was going to need to repeat several more interviews, including the fact that I could not be explicit with the participants about why. I just had to hope that they would be understanding and not withdraw their consent to participate because I had wasted their time. Besides not keeping the commitment from my proposal for interview transcript review, the biggest source of errors came from treating the interviews as conversations instead of data collection. A researcher wants a conversational "flow" in the interviews so participants feel comfortable sharing personal views, but these are definitely not conversations. I had to do a better job of talking less and listening more.

The way I dealt with the strong feelings noted above was to carefully review the transcripts, reflect on my errors, produce lists of things to say and not say, and slow down by doing fewer interviews so I could devote more time to improving my performance. Unfortunately, this led to conducting the interviews robotically and producing significant problems with the way I was asking the questions. I was continuing to rush through questions that had a "why?" component built in. This led to asking too many questions at the same time. Why? Because that is the way they were on the interview guides. I should have just asked the question, waited for the answer, and then asked “Why?" only if I needed to. The transcripts showed a lack of conversational flow. It came across as if were using a clipboard and checking off what I wanted to ask, which is a close description of what I was doing because I was so afraid of saying too much that I just stuck to my prepared questions and said little else. This approach gave the interviews an air of interrogation. Of course, because I was so nervous about saying too much, my follow up continued to be poor. My Chair observed from the transcripts from my second round of interviews that they seemed to be producing very little data that answered the research question. Her conclusion was that it was because of the style I was using in the interviews.

Another problem I was having was not getting sufficient details from the learners about things I may have already understood. My Chair suggested that I explain my requests for more detail to learners as follows: "I may or may not understand it, but I need you to explain it for the benefit of the study. So could you please explain it?" You could even explain at the beginning of the interview, "I may ask you to explain things in the interview that you think I should know. The reason I am going to do that is so I can use it in the study because I am not allowed to use what I know."

This feedback and self-assessment led to more feelings of personal inadequacy. Because of my experience as a Navy nuclear operator, I am not easily deterred by failure. I shared my struggle with my wife and close friends, which reinforced the importance of having a strong support network in times like this. This support team convinced me that I could do this and strengthened my resolve to improve. I feel I can do this (get the data) so I just planned to do what a good nuke does: practice a lot and demonstrate results. I resolved to not do any more interviews until I developed an improvement strategy. I needed to show measurable improvement to regain my own confidence and that of my Chair.

As I entered week three of data collection, I stopped collecting data entirely to focus on improving my interviewing and observation skills. I enlisted friends to help me get four hours of interviewing practice. I did a practice interview with a participant. I spent hours carefully studying prior interview transcripts and peer feedback to assess and understand my errors. I updated my "say this, don't say this" quick reference card for interviews, I watched interviews done by Diane Sawyer with Robin Williams and Scott Plausch (who wrote the last The Last Lecture). I listened to many interviews done on podcasts as I was driving to and from the data collection site and critiqued them as I listened. I looked at an interview transcript of Charlie Rose because he is highly regarded as an interviewer. Finally, I scheduled a face to face meeting with my Chair to go over what I had learned and what I planned to do next. One of the most useful things my Chair told me to do was to review the interviews as if I were her and come to the meeting ready to review what I had learned.
The meeting with my Chair to review what I had learned went very well and she removed all restrictions on my data collection speed. I must have convinced her that I was self-reflective enough to keep getting better on my own. I still have many things to improve. Here are a few thoughts based on further reflection.

* As I noted above, I probably did not have the experience to see any of this coming and thus prepare better. There was probably no way to learn and improve my performance without the experience of making all the mistakes.
* My Chair had an "aha" moment when I told her this is all new to me and was not covered in any classes. Even if it had been, it seems doubtful I would have developed the proficiency needed in just one class. I did not convey my lack of interviewing experience to get sympathy or as an excuse. I actually did not bring it up first. My Chair asked about it.
* My deficiencies fell into two general categories:
- cross interview errors like talking to much, leading the interviews, and saying "okay, right" all the time. These were reasonably clear from the transcripts and I can almost see the text in the air in front of me as I do interviews now.
- deficiencies within particular types of interviews, such as poor follow up. I needed guidance from my Chair and peers to spot these and work on them.
* I was only briefly discouraged by the two waves of negative feedback I received. After I created corrective action plans to reflect on my errors, I felt much better. I was told by a dear friend that we all need to realize that we are never too old to be knocked down and that we have to pick ourselves up, dust ourselves off, get back to work, and not let egos get in the way. Taking on the role of Benjamin Franklin's "humble inquirer” is very important for improvement.
* I remained enthusiastic about the whole process of data collection and improving my skill. I looked forward to every day as another opportunity to improve. I was not born knowing how to conduct a good interview for social science research, but the academic apprenticeship represented by the dissertation process is the key to the next chapter of my life. If I was serious about continuing down that path, and I am, I had to focus on getting better.
* The learning I did was recursive on many levels. I could not see any of the challenges coming because it was my first time interviewing for data. Getting lots of interview practice before I started data collection would have been a great idea, but I don't know if my Chair would have had time to provide the feedback before I started collecting data. This is not a criticism of my Chair-she is very busy. Everything is clearer in hindsight, of course, so I might have benefited by reading this in someone else’s blog. I was probably not smart enough at the beginning of this process to look at anyone else's blog post if one even existed. I don't know if this will help anyone else, but it sure helps me to write it down.
* There are many aspects of the dissertation process that are psychologically lonely, so it really helped to share my setbacks and victories with friends and family. In those moments when I feel very alone and vulnerable, I sometimes forget how dear friends and family members I have supporting me can improve my morale. The support I get always grounds me back in that very supportive reality. I am very lucky to have that support, but I also have to remind myself to activate it from time to time by providing updates to them about where I am in the dissertation process. The tenacity, drive, and passion I feel for my dissertation flows from this support.
* Don't get too excited about needing to learn something you were not born knowing and takes practice. My Chair was providing the feedback she did for my own good to make sure I had usable data. Chapter Four of a qualitative dissertation is about rich data provided by study participants, not the researcher. Even if you understand what they are saying, you cannot use it as data because others may not.
* You are not in charge of assessing whether you are improving "fast enough" or if your technique has materially improved. That assessment comes externally. Internally, "fast enough" for me was just about personally assessing my own performance to stay motivated, which I did.
* I put the quick reference card text I created for data collection in a file.
Other Lessons from Data Collection
* I did not “sum up” at the end of interviews as suggested by Lincoln and Guba, Naturalistic Inquiry, 1985, p. 271.
* I did not keep a log during data collection of what I was doing and thinking, but my dissertation journal came close.
I waited until the transcripts were complete for an interview to reflect on it, which was an error.
* Great advice from my Chair: don’t probe what participants are saying about your phenomenon of interest, how are they saying things that are different? After each interview, you need to ask yourself “What did I get out of this interview?” Use this to inform future interviews. I did write analytic memos about these observations that also kept track of how I needed to modify my interview questions. Keep a list of the answers you are getting from participants that are addressing your Reseach Questions.
* Don’t schedule all your interviews in quick succession because it robs you of the reflection time you need between interviews to ask better questions and improve your technique.