Always Look Beyond the Claims
04/21/09 14:28 Filed in: Research
Recently I had a consultation in Sandwich, Illinois, which meant that I had the opportunity to drive through the countryside on what must certainly have qualified as the loveliest day so far this spring. This type of opportunity is one of the more pleasant side effects of serving a largely rural area.
As I was driving down Route 30 - once referred to by Car and Driver as the Heartland Highway (I swear it was, though I cannot find a valid reference at this point) - I encountered a sign that says “NO WIND FARMS” with a web address at the bottom of it. I was curious enough to stop by the roadside and take a picture of it, and to check out the web address when I got back to the office.
Up front I should note that I have a considerable amount of personal interest and curiosity in alternative energy choices and sources. What I expected to find at the website, frankly, was a bit of propaganda about why having wind farms would be a bad thing - ruins the view, lowers property values, etc.
However, what struck me most - and disheartened me a bit - was a reference to a study, completed by a medical doctor - one Michael A. Nissenbaum - that suggests that people living within a few thousand feet of the turbines are experiencing health effects - headaches, difficulty sleeping, increased need for prescription medications, etc.
Any consumer of scientific information should be aware that the claims made by researchers and others are only as good as their research techniques and level of objectivity. This is extremely important in our field, as parents and professionals supporting and working with people who have developmental disabilities are routinely bombarded with various and sundry claims surrounding treatment for one thing or another, often with little to no good research to support it. I found with a bit of looking about online that this situation - like our review of the 2006 study of the relationship between television viewing and autism (which I will re-post soon to make available for reference) - presented a nice opportunity for evaluation of scientific claims.
To begin with I looked for information on the site about who was responsible for it. There was no immediately easy way to find any information about specific people associated with the site, only the name of an organization called “Citizens for Open Government” with a P.O. Box in Shabbona, Illinois. The website is registered privately as well. It should be noted, in fairness, that a central web address is provided, and I did not use it to gather further information.
The article in question was referencing a study on the Mars Hill wind turbine project in Maine. The article even posts a copy of the slides that Dr. Nissenbaum used for a presentation to the Maine Medical Association Public Health Committee.
Firstly, it’s important for the reader to be aware that Dr. Nissenbaum is a radiologist. These are the folks that read your MRI’s and X-Rays and report that information to the treating physicians. Now, all physicians participate in rotations through multiple areas of practice before settling on a speciality, so it may be that this doctor has some previous experience in treatment of headache and sleep disturbance. Still, it should be made clear to the reader that, as can best be determined by a brief internet search, Dr. Nissenbaum does not currently practice the type of medicine that would be involved with treatment of these types of conditions.
Reviewing the slides from the doctor’s presentation was informative, as what can be determined about the study design suggests significant flaws and considerable opportunities for bias. His slides reveal that he interviewed a very small number of people (15) and that this represented less than half (15 of 35) people who live within 3400 ft of the Mars Hill Wind Turbine Project. What’s more, his study involves use of a questionnaire in which he asks these folks to retrospectively estimate the severity of symptoms before the windmills went up, and to then estimate whether those symptoms are worse now. There is no mention of how the survey participants were selected, suggesting at least the possibility that those interviewed may have been less than objective participants. All of the “symptoms” are based upon self-report, and are items that cannot be independently measured - headaches, dizziness, ringing in the ears, as well as “feelings of ‘stress’, ‘anger’, ‘anxiety’”, etc. The one exception to this is changes in weight but, since the study is retrospective, we must accept the participants report on this matter.
To his credit, Dr. Nissenbaum does provide a review of study weaknesses, and acknowledges that his sample was small, retrospective, and only a case series without a control sample. However, he follows this by saying that there was “no analysis for statistical significance as yet” (emphasis added), and indicates “preliminary findings nonetheless alarming”. I emphasize the “as yet” here because, as a first year research design student could tell you, there’s nothing about the data that he’s collected that would allow for a valid statistical analysis.
But perhaps most telling of all was the fact that a brief internet search (I love the internet - I truly do) revealed that Dr. Nissenbaum is the leader of a group spearheading a moratorium on wind farms in his own home town of Fort Kent, Maine. This fact brings into considerable question his level of objectivity in approaching the study in the first place.
Obviously the purpose of this study was not related to developmental disabilities. However, it is important to understand that, whenever we encounter claims based upon scientific investigation, that we should not simply accept them at face value. Often, as here and as with the television claim, we may find that the conclusions are somewhat less well supported than the claims would suggest.
As I was driving down Route 30 - once referred to by Car and Driver as the Heartland Highway (I swear it was, though I cannot find a valid reference at this point) - I encountered a sign that says “NO WIND FARMS” with a web address at the bottom of it. I was curious enough to stop by the roadside and take a picture of it, and to check out the web address when I got back to the office.
Up front I should note that I have a considerable amount of personal interest and curiosity in alternative energy choices and sources. What I expected to find at the website, frankly, was a bit of propaganda about why having wind farms would be a bad thing - ruins the view, lowers property values, etc.
However, what struck me most - and disheartened me a bit - was a reference to a study, completed by a medical doctor - one Michael A. Nissenbaum - that suggests that people living within a few thousand feet of the turbines are experiencing health effects - headaches, difficulty sleeping, increased need for prescription medications, etc.
Any consumer of scientific information should be aware that the claims made by researchers and others are only as good as their research techniques and level of objectivity. This is extremely important in our field, as parents and professionals supporting and working with people who have developmental disabilities are routinely bombarded with various and sundry claims surrounding treatment for one thing or another, often with little to no good research to support it. I found with a bit of looking about online that this situation - like our review of the 2006 study of the relationship between television viewing and autism (which I will re-post soon to make available for reference) - presented a nice opportunity for evaluation of scientific claims.
To begin with I looked for information on the site about who was responsible for it. There was no immediately easy way to find any information about specific people associated with the site, only the name of an organization called “Citizens for Open Government” with a P.O. Box in Shabbona, Illinois. The website is registered privately as well. It should be noted, in fairness, that a central web address is provided, and I did not use it to gather further information.
The article in question was referencing a study on the Mars Hill wind turbine project in Maine. The article even posts a copy of the slides that Dr. Nissenbaum used for a presentation to the Maine Medical Association Public Health Committee.
Firstly, it’s important for the reader to be aware that Dr. Nissenbaum is a radiologist. These are the folks that read your MRI’s and X-Rays and report that information to the treating physicians. Now, all physicians participate in rotations through multiple areas of practice before settling on a speciality, so it may be that this doctor has some previous experience in treatment of headache and sleep disturbance. Still, it should be made clear to the reader that, as can best be determined by a brief internet search, Dr. Nissenbaum does not currently practice the type of medicine that would be involved with treatment of these types of conditions.
Reviewing the slides from the doctor’s presentation was informative, as what can be determined about the study design suggests significant flaws and considerable opportunities for bias. His slides reveal that he interviewed a very small number of people (15) and that this represented less than half (15 of 35) people who live within 3400 ft of the Mars Hill Wind Turbine Project. What’s more, his study involves use of a questionnaire in which he asks these folks to retrospectively estimate the severity of symptoms before the windmills went up, and to then estimate whether those symptoms are worse now. There is no mention of how the survey participants were selected, suggesting at least the possibility that those interviewed may have been less than objective participants. All of the “symptoms” are based upon self-report, and are items that cannot be independently measured - headaches, dizziness, ringing in the ears, as well as “feelings of ‘stress’, ‘anger’, ‘anxiety’”, etc. The one exception to this is changes in weight but, since the study is retrospective, we must accept the participants report on this matter.
To his credit, Dr. Nissenbaum does provide a review of study weaknesses, and acknowledges that his sample was small, retrospective, and only a case series without a control sample. However, he follows this by saying that there was “no analysis for statistical significance as yet” (emphasis added), and indicates “preliminary findings nonetheless alarming”. I emphasize the “as yet” here because, as a first year research design student could tell you, there’s nothing about the data that he’s collected that would allow for a valid statistical analysis.
But perhaps most telling of all was the fact that a brief internet search (I love the internet - I truly do) revealed that Dr. Nissenbaum is the leader of a group spearheading a moratorium on wind farms in his own home town of Fort Kent, Maine. This fact brings into considerable question his level of objectivity in approaching the study in the first place.
Obviously the purpose of this study was not related to developmental disabilities. However, it is important to understand that, whenever we encounter claims based upon scientific investigation, that we should not simply accept them at face value. Often, as here and as with the television claim, we may find that the conclusions are somewhat less well supported than the claims would suggest.
