Recommendation Systems
We watch a movie or listen to music on apps like Netflix and Spotify, and then a movie or music comes in the same round! How do these apps accomplish this or what’s in the background? Let’s try to learn together.
With the development of technology, our habits are also changing. Instead of shopping from stores, we buy our needs through e-commerce, we watch Netflix instead of watching TV or listen to music on Spotify. One of the reasons why these companies are so popular can be shown that their business structures are based on recommendation systems. If you have noticed, Netflix offers you special movie / series recommendations instead of just your own TV show / music preferences. Spotify prepares a list every week according to your music style. There are many product suggestions that may be suitable for you while browsing e-commerce sites or when you come to the product.
Some questions come to mind to understand the underlying logic:
- How does he know me and make these suggestions?
- When I click on the product, similar products come instantly, how does this system work so fast?
- What kind of recommendation algorithm is there in the background?
For the answers to such questions, I will explain with a little more general information.
So what is the Recommendation Systems? and What does it do?
According to the interests of the application or site users, they are the algorithm structures that are organized in order to continue the application use by predicting their next movements. Suggestion engines create models by learning the choices of consumers and produce results related to their needs and interests. The methods to be applied can simply be arranged according to the previous movements of the person or based on the movements of similar people.
Benefits of Recommendation Systems
Personalization:
A personalized experience is provided due to user specific predictions. It gives good results because it directs people to use.
Income:
When used properly, apps can be a good source of traffic and revenue.
Customer satisfaction:
With the right suggestions, people are directed to use and products that the person may never discover are offered.
What are the Types of Recommendation Systems?
There are many types of recommendation systems currently available, we will touch on some of them in this article.
Content-Based Filtering Systems:
It is simply the recommendation of a similar content product according to the content of the product or service. For example, if you bought a blue shirt, suggest a different color of the shirt, or if you are an electronic music lover, the next song suggestion is again an electronic music-based song. So, if we know that A and B artists are making music with similar content, according to the content-based filtering method, song B can be suggested after the user listens to a song. The important point here is to know the information about the content in detail. If we give an example in terms of e-commerce, making a phone case recommendation when you buy a phone can be realized by defining the contents of these products well. The method to be used depends on the current problem.
Collaborative Filtering Systems:
Collaborative systems analyze user and (or) item similarities. It is based on the assumption that similar users will make similar choices. The collaborative filtering technique works by creating a preferences database (user-item matrix) for items by users. It then matches users with relevant interests and preferences by calculating similarities between their profiles to make recommendations. Suggestions such as “Those who bought this also bought this, those who liked this also liked it, those who looked at this also looked at it” can be given. The Cosine Similarity and Pearson Correlation Coefficient techniques are widely used to find statistical or mathematical similarities. On a model basis, techniques such as Dimension Reduction, Association Rules, Decision Trees, Artificial Neural Networks and Clustering can be used.
This system is divided by user and item:
User-to-user filtering:
User-based filtering is a system that takes into account the similarity of user tastes. If two users have purchased joint products, then it is based on the likelihood that the other user also purchased a product that was purchased later. According to the scores of the users, predictions are made as a result of Pearson Correlation or cosine similarity methods.
Item-to-item filtering:
Item-item filtering algorithm analyzes product similarities from user scores. Clicking or viewing instead of scoring can also be considered a variable. The purpose of this system is to ensure that the products purchased or viewed by the user are recommended to other similar users by considering the similarity. In this method, product similarities are found by accepting the products as neighbors, using Pearson Correlation or cosine similarity methods.
Demographic-based recommendation system:
This system aims to make suggestions to users based on demographic classes. It is easy to apply and practical. Estimates can be made such as products to be recommended to women, products to be recommended for certain age groups. Here, the user ratings are not taken into account, the user actions are not taken into consideration, and suggestions are presented according to the similarities after the demographic questions asked to the user.
Popularity Based Recommendation Systems:
It is a system that is not based on the user’s personal preferences. Generally preferred products or services or applications are recommended. It can be an opportunity to broaden the interests of the person, to spread their application preferences.
Hybrid Recommendation Systems:
These are systems where content-based systems and collaboration-based systems are used together. In this way, the lack of one system can be complemented by the other system.
For example, Spotify makes weekly suggestion lists. Here, it combines songs similar to the music I listen to and songs that similar users listen to and offers suggestions.
Recommendation system steps:
- Data collecting:
Implicit and explicit data are used to be used in recommendation systems. For specific data, the member’s order history and the movements on the site can be given as examples. Uncertain data, on the other hand, are data such as the member’s ratings and comments. These two data types can be combined and used. In addition to these, external data support can be provided. Weather, exchange rate information can also be important. - Retention of data:
The process here varies according to the size of the company and the size of the data accordingly. While SQL databases will be sufficient for some operations, it may be essential to use systems such as Cloud for some operations and to use Hadoop and Spark infrastructures while storing and processing data. This way, it may be more efficient to distribute and store a transaction rather than from a single machine. - Analyzing the data:
In this part, it should be decided that the operation to be performed will be analyzed with real-time, batch, near real-time options. Real-time analysis can be defined as processing the data when it arrives, grouping and processing the data periodically, and processing close to real-time every few minutes or seconds. Sometimes real-time transactions can tire systems, or they may not be so critical. In this case, periodic employee, for example, daily operation of the process may be sufficient. - Filtering data (algorithm implementation):
In this part, the methods mentioned above are applied. Content-based, collaboration-based methods are used and the necessary algorithms are applied.
What are the difficulties that can be encountered while setting up the Recommendation System?
Cold start:
This problem is the difficulty of submitting suggestions to a newcomer. This is one of the main problems of the collaborative approach. It can be avoided by showing popular products to a new user or by adding some rule-based filters for a new item.
Scalability:
While making decisions such as the size of the data set and which type of recommendation system to apply, consideration should be given to cost, labor and having sufficient technical knowledge. This problem can be solved in a number of ways. For example, the data set may be very large, in which case a migration to big data systems such as Hadoop is possible. Or, models or rules can be created on the sample. Another solution might be to divide the model into stages and run it.
Interpretability:
While the content of content-based systems is a bit easier to understand, the structure of collaborative systems can be a little more complicated.
Exploitation-Exploration:
Recommendation systems should not exceed past user item preference data. First, training data should be avoided entirely from previous recommendations. Recommendations can be posted on different sites or random suggestions can be made.
How to measure whether a recommendation system is working well?
First of all, it is very important to position the recommendation systems correctly. Considering that impressions are made in invisible or unclicked areas, no matter how successful an algorithm is used, the result may not be successful. For example, recommendation systems will come to the fore as a more personal experience is offered on mobile.
Apart from the model success metrics, it is a method to present it to the user as A / B test. Simply, with this method, users are divided into two groups. Multiple tests can be done instead of two. The two algorithms prepared afterwards are shown to these different audiences. The more voted and used algorithm, the more successful it can be accepted and implemented.
Apart from this, the “Multiarm Bandit” method can also be used. With this method, 10% of the time, incoming traffic is split equally between the two versions. In the remaining 90% time interval, the traffic is directed to the best performing version. In this way, it is planned to increase the conversion rates. The version that performs well in this way is expected to reach statistically more significant results.
Let’s leave it here for now and say to see you in my next article and subject…🖖🏼
You can reach me from my Linkedin account for all your questions and requests.
References
1. https://morioh.com/p/129aab71f5c2
2. https://thedatascientist.com/right-way-recommender-system-startup/
3. https://1.bp.blogspot.com/-JiroTj4WfJE/XyARMYs1WwI/AAAAAAAAAgw/g5rf_SM5vNExdLKtAXsvRIlGF-j4NdsNACLcBGAsYHQ/s572/Presentation2.png
4. https://www.bluepiit.com/blog/classifying-recommender-systems/
5. https://www.sciencedirect.com/science/article/pii/S1110866515000341
6. https://mathchi.medium.com/association-rules-on-business-problem-bd273479f156
7. https://spiyer99.github.io/Recommendation-System-in-Pytorch/
8. https://predictivehacks.com/item-based-collaborative-filtering-in-python/
9. https://www.veribilimiokulu.com/blog/oneri-sistemleri-101/
10. https://qph.fs.quoracdn.net/main-qimg-37f5b6c5f63e8b8b511c06f9deef2c89
11. https://www.klipfolio.com/blog/recommender-system
12. https://ebaytech.berlin/deep-learning-for-recommender-systems-48c786a20e1a
13. https://mathchi.medium.com/what-is-ab-test-how-are-statistical-a-b-tests-done-1-52b741167d72
