Researching comments on news articles

In my master thesis research I am looking at comments on news articles on news websites and on Facebook. During this process I have had to find a way to categorize comments. This has been a long, and sometimes frustrating process. I started out, before beginning the research, with a few categories that I thought would cover all comments, but that I expected would have to be changed. And they were.

I have implemented a heuristic research approach. The qualitative heuristic approach is an exploratory research method where the data affects the categories. This means that the researcher should be “open to new concepts and change his preconceptions if the data are not in agreement with them” (Kleining and Witt 2000). This lead to the creation of 5 broad categories, that I for a long time was quite happy with.

But then I decided to be thorough (stupid me), so I did a reliability test. I gave a person not involved with my master thesis a list of 60 comments to categorize, and then compared them with my own categorization. The results were not good: 60% agreement. After tweaking the definitions of my categories, creating a more procedural method of categorization, and coding the comments again, a second and third reliability test gave me a score of 79 and 75%.

A reliability score of 70% or more is often used as a criterion for exploratory research. But a score of 80-90% would be considered more acceptable in most situations, and above 90% is considered acceptable in all situations (Lombard, Snyder-Duch and Bracken 2002, 593).

Things were looking better, but I was still not happy. So I decided to start completely from the beginning with a new method for finding categories. In stead of coding the comments into one category or another, I wrote any word that came to mind when reading a comment. After doing this with enough comments (about 100) a pattern emerged. Some words were repeated, and these word became the foundation of my new and final categories. And with these categories, another three reliability tests gave me a score of 82-92%. And the categories, which I now am quite confident about, are:

 

Argumentative comments: Arguments contain a proposition that can either be true or false (Blair 2009, 44). These propositions should be testable. They are also formulated for the purposes of persuasion. This means that there needs to be a point of view that the commentator wants someone to adopt, that is backed up by a proposition. Argumentative comments can also contain proofs from classical rhetoric: Ethos, Logos or Pathos.

Opinions: Opinions are comments that are not necessarily meant to persuade, but function as a direct or indirect statement of what the commentator thinks and believes about an issue.

Reactionary comments: Reactionary comments are short expressions of emotions with little or no informative value. They can also be unspecific statements – statements that are not specific enough for the reader to accurately interpret what the commentator is writing about.

Reactionary comments often contain a set of punctuation marks, especially the exclamation mark, or sets of emoticons. Reactionary comments can also be non-verbal. In these cases, the comments contain either only emoticons or written non-verbal expressions, such as “Haha!!”, indicating laughing or joy.

Informative comments: These comments do not directly argue for or against something, although they can be used in discussions to build a case for a point of view. They are meant to provide relevant information – whether or not that information is factual. Informative comments, with the exception of those classified as personal experience, contain testable factual information that can be either true or false.

Derogatory comments: These are comments that uses some form of critique or potentially hurtful discourse, usually directed at another commentator. Put they can also be directed at public figures or the subject of the article.

Humorous comments: These are comments that, with the intention to be funny, brings together two disparate ideas, concepts or situations in a surprising or unexpected manner, or that contains a play of words or self-deprecating, humorous statements.

Tagging comments: Comments containing only a tagged name. They are usually found on Facebook, and seem to be written to direct the attention of the person being tagged to the article.

Suggestions: This is one of those categories where there is not much more to say than the name. Comments containing suggestions.

Questions: Also a category where the name says it all.

Supportive comments: Supportive comments are comments made in defense of someone, including the commentator himself.

Speculative comments: Speculative comments are comments where the commentator is making speculative assumptions, for which there is no real evidence, and making conclusions that cannot reasonably be verified.

Image comments: Sometimes comments contain only an image. Images can contain relevant information, and can even be considered argumentative – a view dating back to classical rhetoric (LaGrandeur 2003, 119).

Links: These are comments that only contain a link to another website. I have not found any of these comments in the data I have analyzed myself, but I’ve made it a category because I did observe these comments while preparing for my research.

Arbitrary comments: Not really a category, just a collection of comments that made me think “WTF??”. Arbitrary comments are comments that are grammatically or contextually difficult to understand.

 

References

Blair, J. Anthony. 2009. “The Rhetoric of Visual Arguments” In Defining Visual Rhetorics, edited by Charles A. Hill and Marguerite Helmers, 41-62. New York: Routledge

Kleining, Gerhard, and Harold Witt. 2000. “The Qualitative Heuristic Approach: A Methodology for Discovery in Psychology and the Social Sciences. Rediscovering the Method of Introspection as an Example” Forum: Qualitative Social Research. 1, no 1, Art 13 – January 2000.

LaGrandeur, Kevin. 2003. “Digital Images and Classical Persusion.” In Eloquent Images: Word and Image in the Age of New Media, edited by Mary E. Hocks and Michelle R. Kendrick, 117-136. Cambridge: MIT Press

Lombard, Matthew, Jennifer Snyder-Duch, and Cheryl Campanella Bracken. 2002. “Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability” Human Communication Research 28, no 4. DOI: 10.1111/j.1468-2958.2002.tb00826.x

Thoughts about Danah Boyd’s Podcast

I was first introduced to Danah Boyd when I read her book “It’s Complicated” about three years ago. I thought it was a great, eye-opening book. Boyd presents a well thought-through argument about the digital lives of teenagers, and introduces perspectives like the fact that kids today (to a higher degree in the US) have fewer opportunities meeting their friends outside of school.

This is one of the topics covered in the podcast, where she also talks about negative aspects of the internet, like bullying and racism. Boyd’s position is that the negative things we hear about on the internet, like cyber-bollying and racist remarks, are not to be blamed on the technology – that they are just a continuation of human behavior in a new medium. While I generally do agree with this, I am interested in how the internet changes the nature of such negative behavior, and if concepts such as anonymity and invisiblity makes it easier for some people to be anti-social online. And this is something that I miss from Boyd’s talk.

But in general, Danah Boyd continues to be an interesting researcher, and the views expressed in her podcast are un-sensational and thought-through.

 

Formulating our collective research inquiry for #ResNetSem

Due to my three day (all-day) VR Unity Hack this past Friday, Saturday, and Sunday, I am now finally blogging for #ResNetSem and our fantastic class last Thursday.  Last week’s #ResNetSem discussion was very dynamic and I think our discussion of networked learning and commercial culture drew to the surface some very important/timely considerations.  I hope to extend these conversations when we meet next Thursday, first by checking in regarding your own specific research vision, and then connecting those interests to the material for our collective consideration this week.

I have asked you all to listen to danah boyd’s recent podcast (link in tweet):

Remember, for this week please write a two-part blog post.  The first part should include a reflection based on your listen to the podcast.  The second part of your blog should include a further clarification of your current research topic to be pursued in this seminar.  In class, I gave each of you a more specific way to think about that clarification.

Remember that this Thursday evening you will have an opportunity to speak with Henry Jenkins & activist digital Esra’a Al-Shafei via google hangout.  Please write me an email to clarify your intention to participate.  I will then send you the link to our forthcoming hangout (to take place online at 23:30 Thursday night).   The conversation will take place from California live at the Digital Media and Learning conference held at the University of California in Irvine.  Our conversation is listed on the Virtually Connecting schedule here: http://virtuallyconnecting.org/blog/2017/09/25/dmlconference/.

Looking forward to seeing you on Thursday!