X-phi - experimenter bias in vignette composition.

**Note: This post is essentially a posting of a half cooked idea - I'm trying to articulate a particular issue I have with one of the suggestions made by the paper under discussion. If anyone reads this and wants to help develop the argument - or explain why it doesn't matter - I'd be deeply appreciative **

Brent Strickland and Aysu Suben have a recent paper out in which they identify (and try to demonstrate experimentally) an important source of potential experimenter bias in the survey style experiments that comprise the bulk of the work in X-phi.

The problem is, simply, that knowledge of the experimental hypothesis that is being tested may affect the design of the experimental stimuli - which, in X-phi, are typically short vignettes. Subjects are exposed to these vignettes and their immediate, intuitive responses recorded for analysis.

 Strickland and Suben attempt to demonstrate how knowledge of the hypothesis under investigation might introduce this kind of bias by doing a replication-with-a-twist of an earlier experimental result of Knobe and Prinz's.

I was going to recap the entire argument and experiment, but in this case a stellar summary is already up over at the experimental philosophy blog. The comments section on this entry is of a characteristically high quality (seriously, read the comments - they really push the debate forward).
Further, the original paper is only 11 pages, and really straightforward - so read that if the X-phi blog summary wasn't enough.

Another problem?

Now that you're back and know all about the experiment ...

 One of the suggestions that Strickland and Suben make in their paper about how to avoid this kind of experimenter bias in X-phi to to have "blind stimulus creation" in which people who have not been exposed to the hypothesis under investigation are used to draw up the vignettes to be used in the eventual experiments.

 I don't think that anyone has yet pointed out just how problematic this may be given that the aim of many of the experiments in X-phi are specifically to test our natural intuitions about broadly philosophical issues.
My (not yet fully developed) problem is that given that if people do in fact have a set of intuitions about the issues that our experimenters are investigating then that would potentially introduce a bias into the stimuli anyway.

Let's take the idea of freedom of will as an example. Let's assume that people are naturally incompatiblist - that is, let's assume that, intuitive, people untainted by years of reading philosophy intuitively hold that freedom of will is possible in a fully deterministic universe.
Suppose, further, that we want to test this hypothesis with vignettes and surveys but are afraid that our exposure to our hypothesis will introduce a bias into our stimuli creation - I fail to see how getting someone who we've hypothesized holds some pre-theoretical intuition about determinism and free-will to draw up the stimuli will help. It seems possible, even likely, that stimuli they create will be influenced by this pre-theoretical (or, if they are philosophers, their pet-theoretical) understanding of the issue at hand.

A step in the right direction to a much more productive - and interesting - approach is suggested in a comment by Strickland:

I'm most interested in comparing the effectiveness of different possible solutions. One simple starting point would be to have three groups of on-line experimenters : (1) receives hypothesis A (2) receives hypothesis B (3) receives no hypothesis. Then you give all m-turkers clear instructions on the types of sentences they need to build (e.g. all sentences must have a group as a grammatical subject and must contain the verb "desire". then the experimenter can choose the tense and any complements)....

This is much more thorough - unlike the Strickland and Suben experiment, which only had groups (1) and (2) in the quote above - the inclusion of the "hypothesis neutral" group would give us the opportunity to not only test peoples's responses to the given vignettes, but further gives us the opportunity to see how the actual vignettes of hypothesis naive stimuli creators behave in comparison to those created by subjects who had been exposed to hypotheses.

Experimental Philosophy - Joshua Alexander

Experimental Philosophy: An Introduction by Joshua Alexander
My rating: 3 of 5 stars

A useful introduction to Experimental Philosophy - a large chunk of the book is dedicated to surveying some of the more important debates in X-phi and so works well as a kind of general review of the literature. Given the pace at which the X-phi community works it's not clear how much of this will remain relevant for long - but Alexander does a good job of using the debates to make broader points about the relevance of Experimental Philosophy as a whole.
I can heartily and unreservedly recommend the book to X-phi neophytes, but anyone with a more than passing familiarity with the literature might want to give it a skip unless you're looking for a refresher.


View all my reviews

Some thoughts on Sam Harris' "The Moral Landscape"

I recently did a first reading of Sam Harris' The Moral Landscape. I'd like to write about it again once I've read another recent book about neuroscience and morality, namely Pat Churchland's Braintrust, but I had two brief notes that I wanted to get down about Harris' book before my first impressions of his book faded completely. 

My first point is that I feel the neuroscience was an unnecessary distraction from the main argument. I'll have more to say about Harris' argument later, but one of the central themes of the book is that moral questions have objectively right and wrong answers, and that these answers are a function (in almost a mathematical sense) of objective facts about the well-being of human animals. What I found jarring about the introduction into the narrative of neuroscientific results was the fact that we already have a wealth of facts about human flourishing. Without drilling down into the neuroscience we already know, for example, that children who are raised without sufficient loving attention from a caregiver are going to have severe developmental issues. We already know, without having to run an fMRI study that repeatedly raping a child - those who have read the book will have encountered a particularly chilling account - will have disasterous consequences for that child's eventual flourishing, or failure to flourish. Please don't understand me as suggesting that neuroscience can't or won't contribute to the eventual science of human wellbeing (which, for Harris feeds into, if not constitutes in itself, a science of morality) - it just felt, to me, that almost every introduction of neuro-talk into his argument was forced.

My second point is that, despite the clunky neuroscience, I enjoyed the book. At first I tried approaching it as I would any work of moral philosophy, but after half an hour or so I  realized that I was reading a different kind of book. If I had tried to read it as serious moral philosophy, I would have enjoyed it far less than I did. To be clear, it doesn't present itself as a work of moral philosophy, at least no more than Dawkins' "The God Delusion" presents itself as a work of theology or philosophy of religion. Both books flirt with serious philosophy, but neither of the books commit themselves to being serious philosophy per se. No - rather than primarily being about knowledge (although both are filled with facts) or philosophical argumentation (and both are primarily argumentative in form) - these books are more about a kind of empowerment. In the case of Sam Harris' book I believe that the central aim, over and above the talk of moral landscapes and shifting notions of well-being, may very well have been to simply empower people who, when faced with clear moral violations, would ordinarily fall back on a naive cultural relativism about values to justify not speaking up, criticising, and condemning. To those who might mute their conscience in the name of, what might be, a justified belief in multiculturalism it serves as both admonition and call to action. 
Of course, I'm not suggesting that others who have read the book have missed this point - not at all - I think Harris makes it blindingly obvious, but it is a lesson worth repeating. 

Our respect for difference does not require that we acquiesce in the face of the truly morally outrageous.  

Spatial ability and socio-economic status

One of the most persistent, and pronounced, findings in gender differences with regards to cognitive performance has to do with spatial skills, in general, and Mental Rotation, in particular (see Linn & Petersen, 1985) - a gap that, except for some notable exceptions (Eastern Canadian Eskimos and Icelanders), seems to hold cross-culturally (Mann, Sasanuma, & Sakuma, 1990).

Fig 1 from Levine et al. 2005

In their 2005 paper -Socioeconomic status modifies the sex difference in spatial skill - Levine, Vasilyeva, Lourenco, Newcombe, and Huttenlocher present the results of a longitudinal study that ran over two years with the intention of investigating what, if any, effect Socioeconomic Status (SES) has on this gender gap in spatial ability. In understanding the differences between male and female spatial skills it's essential to get a handle on whether the observed differences are mainly a function of biology or mainly a function of culture, given that generally any interesting trait is going to be affected by both. This is especially important in the present case because it has been shown that spatial skills tend to correlate with performance in mathematics based subjects - areas where females have traditionally been under-represented. Understanding this gender based performance difference may help enable educators better address the disparity of representation in STEM subjects (see, for example, Cherney, 2008).

SES status was assigned at a school level on the basis of census-track data for Illinois. A total of 547 students were recruited for the experiment, with male and female participants being approximately equally represented across all three SES groups. Testing consisted of administering an aerial-map task (participants are asked to draw correspondences between aerial photographs of an area and a map of the same area), a mental rotation task (based on "the Spatial Relations subtest from the Primary Mental Abilities (PMA) Readiness Level" (Levine et al., 2005: 842)), and a syntax comprehension test. 

Given previous findings, Levine et al. expected to see gender differences in the spatial tasks but not the language task - an expectation that was mostly borne out by their results with one exception. Their findings show (fig 1) that the expected differences in spatial skill held only for middle and high income subjects - low SES male and female subjects failed to show any significant differences in their performance on the aerial-map and mental rotation tasks. That is, in Levine et al's study, the gender gap is virtually non-existent for the low income group. 

The researchers posit two possible explanations for their findings. The first starts with the observation that generally the gender difference manifests itself in the more difficult test items - if both male and female low-SES group subjects failed to succeed in answering the more difficult questions then that difference wouldn't be apparent in the data even if a difference did in fact exist. However, further analysis of their data seems not to support this hypothesis, for example, a difference in spatial ability for the low-SES group did not manifest in the subset of data where performance across all three groups was comparable for spatial tasks while the difference persisted for the higher groups. 

The second possible explanation for the results, and the one the researchers (and their data) seem to favour, is the notion that it is "differentially high level[s] of engagement in the kinds of activities that promote the development of spatial skill[s]" (Levine et al., 2005: 884) that causes the gender gap in spatial ability, and that these kinds of activities (playing with particular toys, freely exploring their neighbourhoods, etc.) might not be readily available to males from low-SES groups, or - at least - as readily available to males as they are to females. 

References:

Cherney, I. D. (2008). Mom, Let Me Play More Computer Games: They Improve My Mental Rotation Skills. Sex Roles, 59(11-12), 776-786. 

Levine, S. C., Vasilyeva, M., Lourenco, S. F., Newcombe, N. S., & Huttenlocher, J. (2005). Socioeconomic status modifies the sex difference in spatial skill. Psychological science, 16(11), 841-5. 

Linn, M. C., & Petersen, A. C. (1985). Emergence and Characterization of Sex Differences in Spatial Ability: A Meta-Analysis. Child Development, 56(6), 1479. 

Mann, V., Sasanuma, S., & Sakuma, N. (1990). Sex differences in cognitive abilities: A cross-cultural perspective. Neuropsychologia, 28(10).

Mental Rotation in Preschoolers

Kids and spinning - a random
unrelated picture

I recently came across an interesting paper by Ping, Ratliff, Hickey, & Levine called Using Manual Rotation and Gesture to Improve Mental Rotation in Preschoolers that might have some implications for my own research. If you're so inclined you can grab a copy of the paper here.

Ping et al's research takes its orientation from studies that show that Mental Rotation is in some respects linked with motor processes, as well as research that shows that propensity to gesture during an MR task is positively correlated with  MR performance (I'll be covering this research soon).

The aim of the study was, then, to investigate the effectiveness of two different training methods on MR performance. The experiment had a standard pre post control design - 63 four year olds were randomly assigned to one of three conditions (two experimental conditions and a control condition) and were administered pre and post-tests assessing their speed and accuracy during a Mental Rotation task.

 The first training condition presented the children with two images of an animal - one of the images was presented in an upright orientation (i.e. the animal was on its feet) while the other was rotated around its centre at one of four different orientations. The children in this condition were then required to bring the rotated image into the same orientation as the upright image by using a joystick that would actually rotate the image clockwise or anti-clockwise in response to the children moving the joystick left and right.
 The second training condition was almost exactly like the first except instead of using a joystick to bring the rotated image into the same orientation as the target image, the children were asked to reach for the image on the screen, pretend that they were grabbing and then rotating the image to bring it in line with the upright image.
Finally, the control condition spent their time performing a task that didn't involve rotation.

The pre and post tests were a more or less standard 2D version of the Shepard-Metzler style MR test - but using images of animals instead of the usual collection of blocks or polygons that are used in these kinds of tests. As well as the Child Mental Transformation Task (CMTT), which was used to measure transfer.

The results were rather interesting (especially point 3 below) - here's a summary:

1. Accuracy and speed improved significantly across all three groups - this is standard stuff, MR tends to have a serious practice-effect.
2. Accuracy improved significantly on the gesture condition compared to the control condition, while the Joystick rotation condition wasn't significantly different from either of the other two conditions.
3. For MR Reaction Time both the Gesture condition and the control condition showed significant gains on the Joystick rotation condition.
4. Further analysis of the data revealed that (3) was explained by the pattern of improvement in females, while males tended to speed up similarly across all three conditions.
5. The same pattern described in 3 and 4 held with transfer to the CMTT 

What's interesting here is that those children that didn't get any training on a task that ostensibly required Mental Rotation improved more than those that were required to physically rotate images.
Ping et al. suggest that what might be happening is that the children physically rotating the object might actually become dependent on this physical rotation while their "pure" MR performance suffers.

Their explanation makes a lot of sense - but I also think that it points to a potential problem in the MR literature. That is, while a lot of thought is given to the design of the experiments, especially the nature of the pre and post-tests, comparatively little thought is given to the nature of the training task. Superficial similarities between Mental Rotation and physically rotating pictures of animals with a joystick are all that seem to be required to justify using the latter as a training task for the former. The fact of the matter is that the computational (broadly construed) task is almost completely different in the two cases. In the one case there is an actual requirement for the subject to mentally rotate an image, in the other there is no mental rotation required at all. Where the two tasks are similar is actually at the point where the two images are compared to see if they are identical. This fact explains why we see improvement in accuracy in the Joystick condition but not in MR Reaction Time performance. The requirements in the Gesture conditions are much closer to the pre and post-tests in this case.

Stats Saturday 1 - MySQL and R

Fig. 1 - QQ-plot of pretest reaction times

I've decided to try and devote my Saturday mornings to working out the technical bits I need for my dissertation - which, at this stage of the project, means stats stats stats.

Although I'm not sure if I'm going to end up using R to do all the stats for my dissertation - but I'm currently using it poke at my dataset a little.

Today I wanted to accomplish two things. I wanted to get R connected to the MySQL database storing all of my raw data, and I wanted to work out how to generate a QQ-plot from that data.

This was all surprisingly easy ...


Step 1 - load the library

MySQL connectivity to R is provided through the CRAN package, rMySQL.
To import the library you merely need to run the command

library(RMySQL)

You shouldn't have any problems unless you haven't actually got the library installed - I had to "apt-get install r-cran-rmysql" under Ubuntu to get it.

Step 2 - get a connection object

Next, we connect to the database and grab a connection object that will be used in subsequent interactions with the DB. We use the command

conn <- dbConnect(MySQL(), user="USER", password="PASSWORD", dbname="DBNAME")

Where "conn" is the connection object, USER is your username, PASSWORD the db password, and DBNAME the name of the database you're trying connecting to (I'm assuming you're connecting to localhost here)

There are a bunch of ways you can set these connections up - check this out (it's a PDF by the way...).

Step 3 - run a query

This part was surprisingly easy - there are a couple of ways you can do this, including stepping through results in stages, but I was just interested in reading the output of a db query into an R dataframe straight.

data <- dbGetQuery(conn,"select ur.user_name,avg(resp_time_per_deg) as resptime \
from tetris4km_mr_test_data as td \
inner join tetris4km_user_roles as ur \
on td.student_number = ur.user_name \
where td.pretest_posttest = 'pretest' and td.reason_excluded is null \
and td.correct_incorrect = 'correct' and td.resp_time_per_deg < 200 \
group by ur.user_name")

The dbGetQuery() function accepts two arguments, the first being our connection we made in step 2, the second being the query itself. In my query here, I'm interested in pulling out my participants' student numbers and their average reaction time on their mental rotation pre-test.
dbGetQuery() then returns a dataframe with our data. We'll use this for our QQ-plot.

Step 4 - do stuff with your data (QQ-plot)

So I wanted to do a QQ-plot of this data to perform a visual test of normality. I simply ran the following commands

qqnorm(data$resptime);qqline(data$resptime)

and got the lovely fig. 1

Mental Rotation 1 - Introduction and Discovery

What is "Mental Rotation"?

Fig.1 - trial pairs from
Shepard & Metzler (1971)

The Wikipedia entry on Mental Rotation tells us that the term designates the ability of (certain) animals, including humans, "to rotate mental representations of two-dimensional and three-dimensional objects".

As we will see in this series of posts, this straightforward, even trivial seeming, definition is not necessarily uncontroversial - so, for the moment, it is best to think of "Mental Rotation" as being whatever it is that allows one to look at the pairs of images in Fig. 1 and tell, without manipulating anything in physical space, whether the "object" in the left circle is the same as that in the right.

In this post we will be taking a quick look at the paper that kick-started the study of Mental Rotation and related phenomena.



Shepard and Metzler's 1971 paper  - uncovering the phenomenon

Roger Shepard and Jacqueline Metzler's seminal paper "Mental Rotation of Three-dimensional Objects" is rightly considered to be one of the classics of cognitive psychology because of its beautifully simple design and the implications of its results.

Fig. 2 - Shepard and Metzler's results

Shepard and Metzler (1971) presented their participants (n=8) with pairs of two-dimensional projections of three-dimensional objects such as those in fig.1 above.There were 1600 trials in total, presented in a randomised order. Half of the trials showed the same object in the left and right positions. These projections of the object were rotated, at multiples of 20 degrees, either in the image plane (around the Z axis) or in depth (around the Y axis). The other half of the trials showed different, enantiomorphic, pairs in the left and right positions, similarly rotated in the image plane or in depth.

A tone signalled the start of each trial to the participants, after which the pair of images were revealed and a timer started. The participants were then required to determine whether the objects in the pair of images were the same or different by pulling one of two levers which recorded their response and stopped the timer.

Fig. 2 shows the results of plotting time as a function of angle of rotation between identical images. The top graph shows the results for rotation in the picture plane, while the bottom shows the results for rotation in depth. What is striking about these graphs is their linearity - they show that as the angle of rotation between the two images increased, so did the time taken to recognise if they were the same image. Further, not only did reaction time increase but it increased at a fixed ratio (this is what gives it it's linear shape).

This is the same kind of result one would get if one took two physical (identical) pictures, laid them down next to each other at different angles, and then rotated one of them at a fixed speed until the two images faced the same direction. As you increased the difference in angle between the two images it would take you longer to rotate them until they were both at the same orientation. Incidentally, Shepard and Metzler fixed the average speed of this (mental) rotation to be around 60 degrees per second for their participants.



Importance of work

The biggest impact that Shepard and Metzler's study has most likely had is in the debate around the nature of mental representation, in particular in the debate about whether the nature of our mental representations (assuming there are such structures) are propositional (that is, sentence-like) or image-like (or that are functionally equivalent (Sternberg, 2011) to images).

It has been supposed that, because the relationship between the angle of rotation and reaction time is so strongly linear, there is a continuous, analogue process underlying mental rotation - something reminiscent of the example I gave above of physically rotating a picture at a fixed speed. There are two important things to note here though. First, this does not mean that there are literally pictures in the head being rotated, it only suggests that there is some process that is, like I said above, functionally equivalent to rotating actual images (i.e. they work the same way, they don't have to look the same - this will be explored in more detail in future posts). Second, it doesn't necessarily mean that the subjects are aware that this process is going on - it could be that this rotation of image-like representations is entirely sub-personal or sub-awareness. However, when interviewed, all Shepard and Metzler's subjects reported experiencing some kind of visual imagery which they "rotated" in mind. Of course, as the researchers point out, introspective data is best "interpreted with caution" (Shepard & Metzler, 1971: 701).


References :

Pitt, David, "Mental Representation", The Stanford Encyclopedia of Philosophy (Fall 2008 Edition), Edward N. Zalta (ed.), URL = . 

Shepard, R. N., & Metzler, J. (1971). Mental rotation of three-dimensional objects. Science, 171(3972), 701-703.

Sternberg, R., & Sternberg, K. (2011). Cognitive Psychology (6th ed.). London: Wadsworth Publishing.