The blog survey (was) even worse than I thought
[UPDATE: The original title of this blog post was "The blog survey is even worse than I thought". Since it was posted, the people in charge have gone out of their way to restore credibility in the survey, and have posted the correct slides, with annotations. See their comments below.]
I was discouraged to see that the company that was responsible for reporting the recent SA Bloggers survey (ImagiNATION Alliance) thought it unnecessary to report that 2.9% of respondents earned more than R3,000 per month from blogging. Instead, the company decided to deliberately mislead the public by strongly highlighting that not one respondent made that much. That issue has now been resolved, and the slide has been updated with the new information (although not on the .pdf download).
It would be good if that was that. But it isn’t. A quick look through the slides points to lots of other inconsistencies, such as :
The survey purports that 80% of bloggers have 3 children. That is absurd. It is even made more absurd by the fact that the survey also reports that about 50% of bloggers have no children. So that is already more than 100%, and where do I (with one child) fit in?
It is an embarrassing error, easy to spot, and makes me think that that error, together with their thinking that 2,9% of respondent can be purposefully ignored, invalidates the entire survey.
But there is more.
The survey states that “A vast majority”of bloggers have only one blog. I interpret “vast” to mean about 75% vs 25%. In fact, bloggers who have only one blog are in the minority (48%) versus those that have more than one (52%). Once again, it is obvious that the person analysing the data has little concept of how to read and interpret graphs.
The survey states that 59.6% have their current blog as their first blog, and 40.4% don’t. Yet the analysts conclude from that that “A majority of bloggers have at least one previous blog”.
The thing that ruins the survey completely though, are the pithy little comments on the analysis of the data. Here are some to put a smile (or look of utter puzzlement) on your face:
“There are a substantial number of blogs about parenting, documenting that uniquely personal journey” (but no stats to support this)
“The blogging phenomenon has given people access to a means of expressing themselves not previously available”. Umm, OK?
“This indicates that they are in the main embracing all aspects of the technology available to them online” Huh?
I would also like to know why the sample size varies so greatly from one question to the next. Some questions have well over a 1000 respondents, some just over 600. That is a big variance.
It is very possible that the survey company has since amended the survey. No good looking stupid in public. But the fact is, the study was released, and highlights their incompetency.
My suggestion to you is to ignore all results, and to wait for someone with a bit more knowledge of blogging to survey us, and report on the results properly.
PS I do commend 24.com for initiating the survey. We certainly need statistics. Pity someone dropped the ball.
14 Comments to “The blog survey (was) even worse than I thought”
Post comment
Follow me on Twitter!
Recent Posts
- I need more sleep, thank you
- Geekretreat Stanford Valley 2010
- South Africa’s confusing ADSL industry
- Want to rent a desk in a co-op work space?
- The SA Web Bible: a handy resource for web industry
The survey was conducted well and the questions were fab, but slipping up like this has ruined it somewhat :/
It was certainly conducted well (although I would have preferred if respondents *had* to answer all questions).
Some of the questions were fab, some were irrelevant, and some were clearly missing. (What is your main reason for blogging? Do you expect your blogging activity to increase or decrease in the next year? Have you personally and directly encouraged others to blog as a result of your own blogging? HAve you seen an increase in indirect income as a result of your blogging (say in consultancy requests etc).
Let’s hope it’s an annual exercise, and next time they do a better job.
I think you’re really throwing the baby out with the bathwater here Eve. Telling people to ignore the whole survey because of few quirks is frankly ridiculous. And doing so with such obvious relish is fairly distasteful. I stand by our survey – it has both value and validity. No, it’s not perfect, but it’s still a hell of a lot better than nothing, which is what we all had before.
@Allistair
That’s like saying two wrong make a right.
If I found glaring mathematical/statistical errors in two of your slides I have zero confidence that the rest of the slides are correct.
Sorry. It’s harsh, I know. I would love to see you release the survey results. The data is accurate. The representation of it not so much.
Eve,
There are a few points to remember with the survey:
- It’s the FIRST time it’s been done in SA (Technorati did their first survey of bloggers in 2005 together with Edelman PR and got 821 responses. Incidentally, it wasn’t perfect either: http://www.technorati.com/weblog/2005/09/49.html; http://blog.basturea.com/archives/2006/01/12/edelman-survey-methodology/
- The results are, for the most part, valid, (the independent research house responsible for some anomalies has publicly provided a mea culpa)
- A couple of errors do not, unto themselve disqualify the other results. Effectively, they present opportunities for improvement.
- Harping about what should or should not have been included as questions is a matter of opinion. Everyone is an expert in surveys in retrospect. The fact is the survey is a starting point for further discussion about the degree of development, diversity and dialogue that is happing in SA’s online community. The study also serves as a base to build on for next year’s survey.
- To extrapolate like a rabid conspiracy theorist based on a couple of criticisms that the entire survey is invalid is like saying that NASA should never try to fly another space mission, despite a “couple of errors.” For goodness sake, appreciate what has been accomplished – be critical, yes – but not a snarky ivory tower naysayer.
@P&Qs
First of all, let me repeat that I commend 24.com for initiating the survey. You are absolutely right, it is a long time coming, and the implementation of it was great. Of course, everyone will have an opinion on questions that could have been asked but weren’t, but that is pedantic. The survey was an excellent idea, and I participated in it as soon as I heard of it.
However, whether it is a first survey or the hundredth, the responsibility to crunch the numbers correctly does not go away. Crunching numbers is not that difficult, especially if you employ a professional firm to do so, as News24 did.
News24 had very little reason to doubt the accuracy of the numbers. Presumably, they paid a nice invoice to the firm responsible, and relied on them to do a stellar job. Mistakes were made. Shit happens.
I would have liked those mistakes to have been fixed asap. I am not referring to those around matter of opinion, but hte ones that pertain to cold, hard maths.
A clear explanation and apology, outlining which data was affected, and how the mistakes were fixed would have gone a long way of reassuring others about the accuracy of the survey.
As it is, the slides were changed quietly presentations were swopped for correct ones, and the only notice was a defensive stance by the strategic firm employed to do the crunching.
I stand by my stance that 24com, in the spirit of full disclosure should have issued a statement and an apology, and assured that the survey was in fact accurate – now. It is the least they can do as compensation for all those who filed it out in the first place.
To conclude, your NASA analogy is wrong. By all means should NASA have more missions. But if they see that there is a huge error on the current one, and the mission is compromised, they should abort it asap.
This is an(other) excellent post Eve and it saddens me that your points seem to have been wasted on the surveyors. Comments like “first of it’s kind in SA” are quite frankly offensive. Research is research and I wonder what a company like Markinor would say at the discovery that a market research project has occurred for the first time in South Africa. I am being purposefully obtuse but my point is that the methodology is the same for all research and from your post it appears that the numbers and conclusions just don’t match.
I do not want to defend my business with this comment but rather to take responsibility for what has happened. We are the research house that was tasked with “crunching the numbers” and although we are a professional firm with a solid track record, we made mistakes with this data. This should not have happened and it is us (not 24.com) who owe the bloggers that took the time to complete the survey, an apology and an explanation of what happened.
Here is the apology, imagineNATION Alliance made an error in two slides, one was an omission where we did not include a figure that ought to have been in and the other was an erroneous figure that was entered into the graph datasheet. Although, neither of these figures materially change the findings, these mistakes have understandably shaken the confidence in the study overall and for this we are very sorry.
I want also to clear up what happened with the re-issuing of the data and the transparent way in which we have done so. The two slides in question have been re-issued and given red flags to indicate that they have been altered so that there is no confusion about what has been changed.
What saddens me most about this is that some of the findings that will have a material impact on how the blogosphere morphs and changes over time, are not being debated at all because of the two data errors and that there are some that are questioning the validity of the survey in its entirety.
I have offered to meet interested bloggers to discuss the findings, the data errors and anything else that is needed. I will do this and take the criticism on the chin because I believe passionately that there is value in the study and that it is a good foundation of information on which to build an understanding of bloggers in SA. It seems to me to be the most enormous shame and not in the interests of the blogging community at large for this study to be disregarded.
Amanda Reekie – imagineNATION Alliance
Props to you Amanda – good reply.
UPDATE: We have posted both the updated slides this morning and marked them both clearly as “corrected”. Again, we apologise for the mistakes.
Thanks Amanda and Alistair. Kudos for admitting and fixing the errors.
Brand new to all of this and researching the blogosphere, where can I download the data?
[...] the controversy has focused on the really weak interpretation of the data. But if it had been open, then this interpretation would have been just one weak [...]
Good day.
Did you ever think about creating your own blog? There are many excellent platforms, but by far the best is Wordpress. It is simple to set up, however the themes just never fit my specifications. I searched for a simple solution to this problem and realized that there wasn’t one. I then had a template custom made for my needs and was so happy with the outcome. I then decided to build a website that would show the world how to easily hire an expert in wordpress design.
Wordpress Design