The DIY trend (“Do It Yourself”) has overtaken the world. It’s everywhere, and market research is not any different. Companies increasingly turn to DIY techniques to shorten the time from market research questions to answers, using internal employees and available technological platforms. This hasn’t always been the case. In the past, getting answers required skilled market research agencies, professionals, and resources (i.e., time and money). However, not all is sunshine and rainbows – when DIY-ing in market research, there are some things to watch out for.
In this post, we’ll explore the DIY trend in market research, see how to take advantage of it, discuss what to watch for, and try to understand where this trend will take us.
How it came to be? (the facilitators)
There are numerous contributing factors to DIY in market research, and I’ll name just a few.
Technological advancements are dominant; surveying platforms are user-friendly, easy to use, and flexible. You don’t need to be a programmer to create a good survey. My favorite tool is Alchemer, which offers a premium product for a reasonable price. There are many additional products in this space, such as Qualtrics, SurveyMonkey, and Typeform.
When considering technological advancements in market research, surveying platforms are not the only thing. Some companies provide a market for buying and selling responses (i.e., online panels), and fielding in multiple regions is much easier than in the past. You can reach out to an online panel with global reach and ask for respondents within specific demographic criteria to take your survey.
In addition to technological advancements, many companies hold a rich and up-to-date database of their customers/users. Such a database is truly a treasure: it can be used to field market research within their customer community (without needing external agencies or panels). In addition to reaching out to customers, a lot can be learned from analyzing user transactions and behavior, for example, what products are trending, what should a specific page look like, and what is users’ attention span (UX/UI research, but not only that).
Open-source data and information have opened up the option of using data for market research without actually collecting it – because someone else has gathered it and made it available to the general public. You can have an analyst within your company researching such data without needing a full-service agency.
The obvious benefits
Other than the availability side of things (tech, evolving data collection markets, databases, and open-source data), there is an ever-growing need for obtaining information quickly and efficiently. Extensive field studies, which take months to complete, become irrelevant if the business decision needs to be made within weeks (or even days). In our fast-paced world, it’s common to have a business question that needs an answer as soon as possible. In some sectors, more than others, for example, startups (which work fast and agile as part of their culture), media, and more. DIY tools can help you get answers quickly.
Besides the speed of obtaining information, it also helps control costs.
There’s a twist, tough. DIY might lead to wrong insights and wrong decisions, and there are some things you need to watch out for.
What to watch out for
In contrast to DIY, when you work with market research agencies or consultants, they watch out so that you won’t make stupid mistakes. Here are just a few of the common mistakes that companies do in DIY market research:
- Improper research design: when using statistics to identify a phenomenon, you must properly engineer the experiment. For example, if it’s an A/B test (comparison of two options or products), you must ensure it’s a random allocation to either option. Otherwise, you risk adding bias that might cause misinterpretation of the results. Let’s assume you want to compare two feature sets for a specific product using an A/B experiment. Still, instead of a random allocation, you allocate young respondents to the A option and older respondents to the B option. You might see differences between A and B that are age differences rather than actual differences. A less obvious case would be to run the experiment with option A for one month and then option B for the next month (and then detected differences might be affected by seasonality rather than the actual difference between A and B).
- Improper sampling: collecting the data from the wrong population, e.g., sending a survey to friends (or via social networks) is not a representative sample of anything (other than, maybe, of your friends).
- Biased questions: asking a question incorrectly might lead to a specific answer. This might be a bias in the question itself or the answer scale of the question.
- Ineffective presentation methods (e.g., visualizations): Presenting results in a way that decision-makers need help figuring out the main messages. I have seen a lot of DIY tools that provide misleading or distorted charts, for example. Perhaps even conveying the wrong message.
- Deductions based on small samples (or minor effects, statistically insignificant results, etc.). You have to understand statistics to make sure that your results are valid. For example, what sample size will allow you to make safe and proper deductions? If you use samples that are not big enough, the differences you detect might be statistically insignificant, i.e., a result of probabilistic effects rather than an actual difference. On the contrary, statistically significant differences might represent small effects (which do not justify making major business shifts).
Each of these bullets can be a blog post, and this was a partial list. DIY tools can be beneficial, but experts can help you avoid pitfalls.
The future
Soon, we will see a lot of language-based models (e.g., Chat GPT) getting incorporated into statistical and insight tools. For example, our R&D department has developed an integration to Chat GPT that, given summary statistics, yields a description with insights and recommendations. Microsoft and Google are developing similar integrations to their office products (word, powerpoint, docs, and slides) that can generate content and provide insights. Moving analysis from Excel to R and Python will become more accessible since these language models can support code generation (for example, I generated the code for this post using OpenAI’s Chat GPT and google’s BARD.
These solutions will still need careful guidance by the user since sometimes these language-based models miss or provide unreasonable output.
Conclusions
In this post, I reviewed some of the fundamentals of DIY market research tools: their facilitators of it and motivators.
I also provided some pointers on what you should watch out for when applying DIY tools – that professionals know about, and you should too.
Finally, a short outlook on the future of DIY in market research, specifically from the point of view of large language models (such as Chat GPT).
If you found this post helpful, let me know! Reach out by filling out the contact us form or via twitter @SaridResearch.