How to improve recommendation engines with real-time context and business rules

This post was kindly contributed by SAS Users - go there to comment and to read the full post.

Companies have recognized that the key to maintaining customer loyalty and increasing engagement is to anticipate customer’s needs and desires. To that end, companies have invested heavily in AI technologies to create recommendation engines that present offers, communications, and products to fulfill those needs. Nowadays, recommendation engines can be found just about everywhere from news websites to e-commerce sites to online streaming services. However, customers frequently encounter recommendations that are inappropriate or repetitive. Just because you accidentally clicked on something 6 months ago, doesn’t mean that you should receive 100 variations of that same article/product today.

The challenge is that companies have often focused too much on building advanced AI algorithms to power their recommendation systems, but frequently miss out on rapid changes in the marketplace. Since recommendation algorithms are based heavily on historical behavior, they may fail under rapidly changing environments where new products are introduced, consumer tastes change rapidly, or market conditions deteriorate.

Recommendation engines also frequently fail to account for real-time events and context. For example, during holidays, people’s tastes can be highly seasonal. Using recommendations based on purchases made during other times of the year may have no relevance to what people want today.

Companies are also under pressure to recommend products and content that are most profitable and high value. Unfortunately, the most profitable products may not be the products most preferred by customers. This can result in conflict between marketers who want to push products and data scientists who want to create good recommendation engines. If the marketers override the recommendations, this can result in consumers losing trust in the recommendations.

To address these limitations, companies need to think about developing an approach to recommendation engines that account for the following factors:

Business objectives: Which products are most profitable? How do we optimize the recommendations to generate the highest revenue?
Context: What do we know about the customer before we deliver a recommendation? Are they at home/school/work/vacation? Did a significant life event occur such as getting married, having a baby, or buying a house?
Historical behavior: Analyzing past transactions to generate recommendations. This involves using AI and machine learning techniques such as collaborative filtering, market basket analysis, factorization machines to look at previous purchase history and compare them to others who purchased similar products.
Real-time trends: Using real-time information to address sudden changes in consumer demand. This real-time information can come from social media feeds or by analyzing real-time streaming data.

To address these different aspects of a recommendation engine, I want to walk you through an example of building a recommendation engine that incorporates these different aspects.

Scenario: A cable company would like to develop an app for subscribers that provides real-time recommendations for live TV. To improve the relevancy of the recommendations, they would like to consider several factors. Firstly, they would like to better predict if a family or child is watching at that moment and make age-appropriate recommendations. Secondly, if a show or content is extremely popular right now with other viewers (such as a breaking news event or a sports event in their area), they would like to override the default recommendation with the show that is extremely popular. This can be accomplished in 5 steps:

1. Use factorization machines to analyze historic viewing behavior and come up with personalized recommendations

Factorization machines are one of the most powerful recommendation algorithms currently available. It uses matrix factorization to project ratings on a very sparse matrix of users and products. SAS Viya provides a powerful distributed in-memory engine to train factorization machines on extremely large, sparse datasets consisting of thousands of products (or TV shows) and millions of users.

In the example below, we trained a factorization machine on set-top box viewing data. Our target variable was viewing seconds of the show. The factorization machine attempts to predict how long people will watch a program that they haven’t seen before based on the viewing habits of similar viewers. After training the factorization machine, we can generate a prediction for every program that an individual hasn’t watched before. Using this prediction, we can then rank-order all shows by the predicted viewing time from the factorization machine algorithm.

2. Build predictive models to predict who’s watching

To determine who is watching at any given time, you can use predictive models to best predict whether a child or family is watching. By collecting data on when and where users were historically watching family-friendly content, we can train a model that will predict the likelihood that a family or a child may be watching TV at that time. Using SAS Visual Data Mining and Machine Learning, users can build scalable modeling pipelines that take in historical viewing data, transform data for modeling, and build out a series of candidate models (such as a gradient boosting, neural network, random forest, etc.). After evaluating the modeling performance on a hold-out sample, the champion model can be published in production and leveraged within a decisioning flow.

3. Use SAS Event Stream Processing to capture what is popular right now

To better calculate what’s happening right now, we need a tool that can analyze real-time streaming data and act on it. SAS Event Stream Processing was designed precisely to analyze real-time streaming results before it lands in a data lake or database. Real-time tuning records from set-top boxes, mobile apps, websites, and smart TVs can be aggregated and analyzed in real-time to determine the most popular shows that are playing in real-time for a demographic, region, or genre.

4. Use SAS Intelligent Decisioning to deploy business rules

SAS Intelligent Decisioning is a solution for orchestrating real-time decisions that incorporate business rules and predictive models. It allows non-technical users to design decision flows using an easy GUI interface. After a decision flow is created, it can then be published as a REST API that can be called in real-time from edge devices (such as set-top boxes, mobile apps, Smart TVs, etc.) to receive a real-time recommendation. We can also export these decisions and embed them directly within SAS Event Stream Processing Engine. For more sophisticated users with a strong programming background, these business rules can also be coded directly in SAS Event Stream Processing without using Intelligent Decisioning.

In the example below, we can orchestrate a decision flow that determines what to recommend given certain circumstances. If the predictive model predicts that a child or family is watching, then a family-friendly recommendation will be presented. If event stream processing determines that a certain show is extremely popular right now, then it will override the baseline recommendation with the popular show. Otherwise, it will send the recommendation that was generated by the factorization machine.

5. Orchestrate the entire decisioning process using SAS Event Stream Processing

To bring this all together into a single flow that can work in real-time, we need a tool that can ingest real-time streaming data, enrich the data with all the relevant information we need to make an intelligent recommendation, aggregate real-time data, and execute the decisioning flow. This will result in a final recommendation. Event stream processing can orchestrate all these elements into a single project.

In the example below, SAS Event Stream Processing takes in real-time streaming set-top box records and then enriches it with data from the customer data warehouse. The event stream processing engine then aggregates real-time TV viewing across all devices and determines which shows are most popular right now. Then it scores the data using the predictive model to determine whether a child or family is watching. Finally, it executes the decision flow created in SAS Intelligent Decisioning to determine what the final recommendation will be. Event Stream Processing has a REST API that allows third-party applications or devices to connect to this flow and receive the requested recommendation.