What is AB Test? How Are Statistical A / B Tests Done? — 2
The first part of the article was that we discussed conceptually and result-oriented, now in the second part will be technically related to the background tests and the theoretical working principles of these tests.
Note: I usually will use some abbreviated words below:
- Data Science — DS
- Artificial Intelligence — AI
- Machine Learning — ML
- Big Data — BD
Let’s continue…
What path should be followed and what should be considered? We started by saying, under the big heading in the first article. Let’s continue from here.
4. The optimum answer to the question of how long should the measurement period be?
This question is the key point of these studies. There is no single valid approach in this subject that should be considered together with business knowledge, sampling theory, and the law of large numbers. It must be decided together with business knowledge, sampling theory and the law of large numbers. I suggest you take a look at this article on the law of large numbers.
In this step, we need to continue with the assumption (this is an important assumption and one of the key points) that the results of the experiments that occur in each day or period are an observation. So, are we going to do a ratio test for the observations made today or will we do a ratio test based on the observations made within a week?
For our example, let’s assume that testing will be made over the rates that occur over a month. And let’s move on to the next step.
A question before moving on to the next section:
If we wanted to do the same test two months later. Should we approach the problem in the same way in this case? Bayesian statisticians benefit from the power of past knowledge and experience by incorporating the results of these ratio tests and hypotheses that have been created in the past into new tests they will carry out in the future. For example, in the tests carried out with different colors each month for 6 months, it can be shown that the 7th month is considered in the previous 6 months.
5. Hypotheses should be interpreted and relevant action recommendations should be submitted to business units.
Let’s assume that we have, for example, data belonging to two groups that were obtained for 30 days with the steps determined in the previous steps, and let’s assume that the button renewal yielded successful results with the p ratio test run as a result of these tests and evaluate this step.
It is necessary to pay attention to how the H0 hypothesis is interpreted and it should not be forgotten that the academic community is carefully examining the interpretation of the hypothesis test result.
Just as meaninglessness occurs when the letters do not come together correctly in a word, this is also true in statistics (Pearson), which is the alphabet of science. All processes that are intertwined in all tests and interpretations must be done correctly and completely.
H0 hypothesis is interpreted as follows:
“H0 hypothesis is rejected” or “H0 hypothesis cannot be rejected”. Therefore, interpretations such as H0 hypothesis are accepted or H1 hypothesis is accepted cannot be made. Equivalent will come, often used, but can not be done.
We can explain its justification as follows:
We know the probability of making an error when we reject H0 0.05 (alpha value), but when we accept H1, we do not know the possibility of making an error, so no comment can be made on the basis of H1. Therefore, it cannot be interpreted as H0 is acceptable. H0 is rejected or H0 cannot be rejected (there is not enough evidence to reject it) comments can be made.
If the error types in statistics are remembered, the subject will be clarified more.
One of the criticisms brought by Bayesian statisticians to frequencyist statisticians is this situation encountered in hypothesis tests. In response to this situation, the suggestion of Bayesian statisticians will be explained in another article called Bayesian Hypothesis Tests. Of course, the level of interest in these theoretical subjects does not lead to revising the motivation to write.
Consequently, let the H0 hypothesis result in the result of “H0 is rejected” as a result of the test, and it is concluded that changing the button color creates a statistically significant difference. Necessary notifications will be given to the business units.
If the “H0 hypothesis cannot be denied” result, it would mean “the improvements made are nice, but it does not make statistical significance, it is not worth bothering each other for a button color” for the business units.
In the second part, the hypotheses that can be created for the testing of machine learning project outputs and R application for these will take place, except for the “p ratio test”.
This is all I have written about the “What is AB Test? How Are Statistical A / B Tests Done?”. If you want to know more about DS and related others, you can check out my other serial articles. Sample:
Roadmap to Become a “Data Scientist”
You can reach me from my Linkedin account for all your questions and requests.
Hope to meet you in other series articles and articles…🖖🏼
References
1. https://www.invespcro.com/blog/how-to-split-test-without-harming-your-sites-seo/
2. https://visme.co/blog/animated-infographics-interactive-infographics/
3. https://www.statisticsfromatoz.com/blog/new-video-fail-to-reject-the-null-hypothesis
4. https://www.veribilimiokulu.com/blog/istatistiksel-a-b-testleri-nasil-yapilir/
5. https://nl.pinterest.com/pin/114982596716992244/
6. https://web.stanford.edu/~hastie/CASI_files/PDF/casi.pdf
7. https://www.udacity.com/course/ab-testing--ud979
