How to Track Changes in Product Opportunities with Online Reviews

Product planners need to know what customers want before they can decide the next direction for improvement. Surveys take time to produce results. If a new product launches in the meantime, customers’ interests change as well. Even results from a one-time review analysis may no longer match the current situation a few months later.

This raises a question. Can we find product attributes that are “mentioned often but have low satisfaction” in customer reviews, and track every month when those opportunities appear and disappear?

Let us read one paper that answers this question.

• Title: Dynamic Monitoring of Product Opportunities with Online Product Review Data
• Journal: Journal of the Korean Institute of Industrial Engineers
• Year: 2019
• DOI: 10.7232/JKIIE.2019.45.5.387

The paper proposes a method for identifying dynamic (time-varying) product opportunities through social media mining. The method extracts the product topics that customers discuss in online reviews. It then calculates three indicators for each topic every month and combines the three indicators into a single priority. The analyst watches how this priority moves over time.

The terms used throughout this article are as follows.

• Product opportunity: an opportunity to develop a new product that customers will be satisfied with, or to advance an existing product
• Topic modeling: a text mining technique that uses a probabilistic model to find the topics hidden in a collection of documents. The paper treats each topic obtained this way as one customer requirement for the product.
• Importance, Satisfaction, Growth (浮上性): three indicators that express the opportunity level of a topic numerically
• TOPSIS (Technique for Order Preference by Similarity to Ideal Solution): a multi-criteria decision-making technique that ranks alternatives by considering several criteria together

Motivation: Customer Needs Change Quickly, but Existing Analysis Stays at a Single Point in Time

Why Product Opportunities Need Continuous Tracking

The paper gives five reasons.

• In a highly competitive environment, finding innovative product opportunities based on customer requirements is important for the sustained growth of product-based companies.
• With survey-based methods, it has become increasingly difficult to grasp requirements quickly and track how they change. The causes are the global business environment, shorter product lifespans, and more complex customer requirements.
• Social media provides large volumes of real-time data containing customers’ opinions and requirements. Because customers exchange product information on their own, the analysis results become an important resource for R&D and marketing strategy.
• As product life cycles shorten and the market environment becomes more complex, customer requirements change quickly. Therefore, requirements must be analyzed dynamically and reflected in the business.
• Once a technology has reached a certain level of maturity, it is more useful to set the direction of improvement according to customer and market demand than to look for new opportunities in the technology.

What Existing Research Has Not Addressed

There have been several prior studies that used social media data to find business opportunities. One study found emerging topics in news data, and another measured satisfaction by applying sentiment analysis and multi-criteria decision-making techniques to mobile service reviews. There are also studies on a text analysis framework for finding product defects, on deriving product opportunities with opportunity discovery theory, and on applying an opportunity analysis algorithm to social media data.

The paper points out three gaps.

• Prior studies quantified opportunities but did not capture opportunities that change over time.
• Most studies did not identify the priority of the quantified opportunities. To set the order of product development and marketing, a company needs the relative importance among opportunities.
• Some studies analyzed from a technology perspective, so they gave too little consideration to the customer and market perspectives.

Method: Group Reviews into Topics and Rank Them Every Month with Three Indicators

The paper’s method has four steps. In Step 1, online product reviews are collected and product-related keywords are extracted. In Step 2, customer requirements are defined through topic modeling, and in Step 3, indicators representing the opportunity level are calculated over time. In Step 4, TOPSIS produces priorities for each time point so that changing product opportunities can be monitored. This article follows the paper’s division into steps as is.

Case and Data

The case product is Apple’s tablet PC, the iPad. The iPad was first released in the United States on April 3, 2010, and held the top share of the tablet PC market for several years. It has diverse functions, new products keep coming out, and customers exchange opinions actively. On this basis, the paper judged it a suitable case for a method that tracks change over time.

The data source is Reddit. Reddit consists of more than 1.2 million subreddits (topic-specific boards). On Twitter or Facebook, an analyst can collect data only with keywords or hashtags defined in advance. On Reddit, by contrast, using a subreddit that matches the topic makes it possible to collect relevant posts without defining search conditions beforehand.

The researchers collected posts written in the iPad subreddit between December 2015 and May 2018.

Source: Section 4.1 of the paper

The documents used in the final analysis are about 25.3% of the collected documents (42,269÷166,882=25.3%).

Step 1: Review Collection and Keyword Extraction

• Input: online reviews of the target product
• Processing: collection, keyword extraction, stopword removal
• Output: document-keyword matrix

1. The analyst collects reviews through web crawling or an open API (Application Programming Interface) provided by social media.
2. For each review, keywords are extracted with a natural language processing tool. The paper gives TextRazor, Alchemy API, and spaCy as examples.
3. Stopwords such as emoticons, onomatopoeia, and greetings unrelated to the product are removed from the list.
4. For each review, keywords and their frequencies are turned into a vector, and all the vectors are combined into a single document-keyword matrix.

Stopwords are removed because keywords that are not needed for the analysis get mixed into the extraction results. Social media data is easier to collect than other public data, is more current, and is more closely related to business.

In the case, the researchers first deleted unnecessary documents such as “[deleted]” and “[removed]”. They then extracted 56,789 keywords with spaCy. This list contained web page links, colloquial expressions, and emoticons. The researchers filtered out stopwords using two conditions.

• Keywords contained in only one document, that is, keywords with a Document Frequency of 1
• Keywords not directly related to the iPad

What remained was 10,401 keywords and the 42,269 documents containing them. Only about 18.3% of the original keyword list remained (10,401÷56,789=18.3%).

Step 2: Extracting Product Topics with LDA

• Input: document-keyword matrix, number of topics
• Processing: LDA (Latent Dirichlet Allocation) topic modeling and topic interpretation
• Output: topic-keyword distribution, document-topic distribution, topic names

1. The analyst decides the number of topics. The paper uses a method that chooses the number of topics at which the cosine similarity between topics or the Perplexity is smallest.
2. LDA is run to obtain the two distributions.
3. Keywords with high contribution are checked in the topic-keyword distribution.
4. Documents with high contribution to that topic are read, and the topic name is decided.

LDA is used because it performs better than other topic modeling techniques and is widely used. The topic-keyword distribution tells what each topic consists of. The document-topic distribution tells which topics each document talks about. The paper uses the former distribution to define requirements and the latter to evaluate them.

A topic is only a bundle of keywords, so a person has to give it meaning. The main keywords of topic 25 were ‘true tone (0.2436)’, ‘final cut (0.0346)’, and ‘true tone display (0.0306)’. The documents with high contribution also discussed the True Tone feature. The researchers therefore named this topic ‘True tone display’.

In topic 49, the contribution of ‘best buy’ was 0.5020. Because this single keyword accounts for more than half of the total contribution, the researchers defined this topic as ‘Best buy’, the topic of the consumer electronics retailer.

In the case, the researchers ran LDA with scikit-learn. They calculated Perplexity while increasing the number of topics from 1, and Perplexity was smallest at 3140.66 when there were 70 topics. The main extracted topics include the following types.

• Usage and functions: ‘Data usage’, ‘Split screen’, ‘Applications’, ‘Photo editing’
• Product models and options: ‘32GB model’, ‘64GB model’, ‘iPad Air’, ‘iPad Mini’
• Accessories: ‘Apple pencil’, ‘Smart keyboard’

Step 3: Calculating Importance, Satisfaction, and Growth

• Input: document-topic distribution, document writing time, document polarity
• Processing: summation by time point and normalization
• Output: three indicators by time point for each topic

1. Importance: At each time point, the contributions in the document-topic distribution are added up to obtain the topic volume. This value is normalized to between 0 and 1.
2. Satisfaction: Sentiment analysis is run on all documents to obtain polarity (the degree of positive or negative sentiment). The document polarity is multiplied by the document-topic distribution to build a sentiment matrix, which is summed at each time point and normalized to between 0 and 1.
3. Growth: Section 3.3, which explains the method, says it is obtained as the ratio of the topic volume at the current time point to that at the previous time point. Because the result is itself a ratio, it is not normalized. However, the conclusion says Growth was obtained as the one-month increase in the contribution in the document-topic distribution. The two descriptions in the paper, ratio and increase, do not match each other, so readers who want to replicate the work should keep this in mind.

The three indicators each point to a different kind of opportunity. A topic mentioned more than other topics is a product element that customers consider important. A topic with low satisfaction still has room for improvement. For a topic whose mentions have surged, a company can respond quickly to a need that did not exist before.

TOT (Topics over Time) and DTM (Dynamic Topic Modeling) are also ways to look at topic trends. The paper chose to add up contributions over time, because the three indicators must be calculated in the same way to maintain consistency.

The Python package VADER was used for sentiment analysis. The paper introduces VADER as a deep learning-based package.

Let us summarize the calculation with symbols. Suppose a document written in a given month has contribution a to topic 25 and sentiment polarity p. This document adds a to the topic volume for that month’s Importance. It adds a×p to the Satisfaction sum. Even with the same contribution, a document with lower polarity adds a smaller value to Satisfaction. Table 1 of the paper lists the contributions and sentiment weights of documents related to topic 25. The actual values can be checked in the original paper (DOI link).

Growth cannot be calculated without a previous time point. In December 2015, the first month of the case, the Growth of every topic is 0.

Step 4: Producing Monthly Priorities with TOPSIS

• Input: three indicators by time point for each topic
• Processing: TOPSIS, moving average
• Output: priority time series by topic

1. Set the positive ideal solution (PIS). It is a hypothetical topic with the highest Importance, the lowest Satisfaction, and the highest Growth.
2. Set the negative ideal solution (NIS). It is a hypothetical topic with the lowest Importance, the highest Satisfaction, and the lowest Growth.
3. For each topic, calculate the Euclidean distance to the two ideal solutions.
4. Calculate the relative closeness. It is the distance to the NIS divided by the sum of the two distances.
5. Repeat at all time points to obtain a closeness time series, reduce fluctuation with a moving average, and then rank.

A closeness of 1 means the best rank, and 0 means the worst. The higher the closeness, the higher the opportunity level. The following is a calculation in which the author plugged values into the paper’s definition. If a topic is at the same position as the PIS, its distance to the PIS is 0. The closeness is then distance to the NIS÷(0+distance to the NIS)=1.

TOPSIS allows trade-offs between indicators. A good value on one indicator can compensate for a bad value on another. The paper also cites ease of understanding and simple computation as advantages.

The moving average period in the case is 6 months. The paper reflected the fact that 5 iPads were released during the 30 months of collection (30÷5=6). The paper describes the iPads released during the analysis period as follows.

• 1st-generation iPad Pro 9.7-inch
• iPad 5th generation
• 2nd-generation iPad Pro 12.9-inch and 10.5-inch (released together on June 13, 2017)
• iPad 6th generation

The count of 5 follows the paper’s description as is. It is a different number from the 5 post-volume surge periods discussed in the results below.

Results: Rankings Move with New Product Releases, and Some Topics Stay Near the Top

Review Volume Rose Right After Announcements and Releases

During the collection period, post volume surged on 5 occasions.

• March 22, 2016, 176 posts: the day after the announcement of the 1st-generation iPad Pro 9.7-inch
• March 22, 2017, 117 posts: the day after the announcement of the 5th-generation iPad 9.7-inch
• June 14, 2017, 221 posts (the most): the day after the release of the 2nd-generation iPad Pro 12.9-inch and 10.5-inch
• September 15, 2017, 136 posts: presumed to be the effect of the iPad price increase and the new product event three days earlier
• March 28, 2018, 153 posts: the day after the release of the 6th-generation iPad 9.7-inch

From these figures, the researchers judged that interest in the iPad Pro was relatively higher than interest in the iPad. These surges were also reflected in Growth. In March 2016 and June 2017, the Growth of most topics surged.

Different Topics Stood Out on Each Indicator

On Importance, two topics showed contrasting trends.

• ‘iPad Air’: Its Importance was higher than other topics but gradually declined after June 2017. No successor to the iPad Air appeared after the 2nd generation was released on October 22, 2014.
• ‘Smart keyboard’: It rose steadily, without sudden surges or drops. The researchers interpreted this as interest in the iPad Pro carrying over to the keyboard used with it.

Two topics also stood out on Satisfaction. ‘Best buy’ dropped sharply in January 2017 and fell after 2018, but it remained relatively very high throughout the analysis period. The Satisfaction of ‘Smart keyboard’ increased similarly to its Importance. The researchers interpreted this as the Smart Keyboard becoming more important while satisfying customers.

A Topic That Stayed Near the Top: Battery Life

Among the 70 topics, topic 35, ‘Battery life’, had an average rank of 8.67 over the whole period. It ranked first in November and December 2017 and in January, February, and April 2018. Its five first-place finishes were the most among all topics.

Reading the documents with high contribution, the detailed themes were as follows.

• The effect of iOS version updates on battery life: this was mentioned most, and the polarity of most documents was negative.
• The battery of accessories such as the Apple Pencil and Smart Keyboard themselves, and the iPad battery drain caused by using accessories
• Comparison of battery life with other products
• The effect of charging methods on the battery

The researchers expected the opportunity level of this topic to remain high after the analysis period.

A Topic Whose Rank Changed with Releases: 64GB and 32GB

Topic 22, ‘64GB model’, rose from 66th to 6th in June 2017. By the author’s calculation, that is a rise of 66−6=60 places. It then stayed in the upper ranks and was 1st in May 2018, the most recent month. ‘32GB model’ did not record high ranks but showed the opposite pattern to ‘64GB model’.

The researchers interpreted this pattern in terms of the lowest-capacity option.

• Models whose lowest capacity is 32GB: 1st-generation iPad Pro 9.7-inch (March 31, 2016), iPad 5th generation (March 24, 2017), iPad 6th generation (March 27, 2018)
• Models whose lowest capacity is 64GB: 2nd-generation iPad Pro 12.9-inch and 10.5-inch (June 13, 2017)

The two groups do not offer each other’s capacity option. The researchers considered that when a new model comes out, mentions of that model’s lowest capacity become more frequent, and the rank of the corresponding topic rises. This interpretation is an inference based on the observation that the release schedule and the rank changes appeared together. It suggests that new product releases can affect the ranking of customer requirements, but it is an interpretation of a single case from one product family (iPad).

A Topic with High Satisfaction and a Low Rank: True Tone Display

Topic 25, ‘True tone display’, had a highest rank of 45th during the analysis period and an average of 60th over the whole period. Most of the documents with high contribution expressed satisfaction with this feature. The polarity of the majority of documents was 0.7 or higher. Among the weights of 10 sample documents, the highest value was 0.9922 and the lowest was 0.7003.

The researchers interpreted that high polarity raised Satisfaction, and that high Satisfaction appears to have had a large influence on lowering the rank. However, Importance and Growth also act on the TOPSIS rank. So what this case confirms is only that a topic with high Satisfaction appeared together with a low rank. The author sees this result as consistent with the intent of a method that does not treat attributes customers are already satisfied with as improvement opportunities.

The Next Opportunities Seen in the Last Six Months of Rankings

The researchers point out that recent rankings should be examined along with ranking trends. This is because past data older than a certain period cannot explain the current market and needs. Table 2 of the paper summarizes the ranks over the last 6 months of the top 5 topics in May 2018. The five topics in the table are as follows.

• ‘64GB model’
• ‘Photo editing’
• ‘YouTube’
• ‘Product options’
• ‘Apple pencil’

The monthly rank figures can be checked in Table 2 of the original paper (DOI link).

The interpretations of two topics contrast. ‘YouTube’ ranked consistently high for 6 months. The researchers expected the need to watch video on the iPad to remain high in the future. For ‘Apple pencil’, the rank rose steadily from January 2018 and climbed 47 places in 5 months. The researchers regarded this topic as a product opportunity with potential.

Significance and Limitations

What Changes

• Product opportunities are monitored continuously over time rather than once.
• Importance, Satisfaction, and Growth are combined with TOPSIS to quantify a priority for each requirement.
• Reviews written directly by customers are used to find opportunities from the customer and market perspectives.
• The effect of new product releases on the ranking of product topics can be confirmed.

Value for Practitioners

Continuous monitoring provides information that snapshot analysis misses. From trends in rank changes, one can find opportunities with potential and confirm the effect that newly released products had on customer requirements. On this basis, one can judge whether past strategy was appropriate and decide where to focus next.

Because it uses large volumes of data collected from the web, it can detect and respond to changes in needs faster than surveys. The paper presents this method as a monitoring tool that supports customer-centered product planning and development.

Value for Researchers

This paper links topic modeling, sentiment analysis, and TOPSIS into a single procedure. The paper chose to add up the contributions in the document-topic distribution, on the grounds that the three indicators must be calculated in the same way to maintain consistency. It also clearly leaves follow-up research topics such as keyword selection criteria, rank prediction, and additional indicators.

Limitations Stated in the Paper

The paper states four limitations of its own.

• The analyst’s subjectivity can enter the process of selecting product-related keywords. This selection affects the topic extraction results.
• Changes in the opportunity level after the analysis period were not predicted. Deep learning-based prediction or time series analysis is needed.
• Besides Importance, Satisfaction, and Growth, more indicators that reflect the characteristics of social media data are needed.
• It was applied to only one product family. Studies applying it to other fields or to individual product models are possible.

Conditions for Application Seen by the Author

What follows is not content from the paper but conditions the author adds with practical application in mind.

• Enough posts must come in each month. Growth is not normalized, so it is strongly affected by the number of documents at each time point. In months with few posts, rankings can fluctuate widely.
• Your own product release dates and event dates should be recorded alongside. As in the 64GB and 32GB case, the cause of a rank change can be explained only if you know the release schedule.
• Topic names are assigned by people. With 70 topics, the person in charge needs to set aside separate time to read the keywords and representative documents.
• A topic with high Satisfaction tends to get a low rank if the other indicators are similar. Because TOPSIS allows trade-offs between indicators, the rank can differ if Importance or Growth is high. To manage strengths that customers like, the Satisfaction indicator should be examined separately.
• The moving average period should be set to match your own release cycle. The paper’s 6 months came from the conditions of the iPad, with 5 products released in 30 months.

What to Try Right Away

1. Collect posts, together with their writing dates, from community boards or review pages that cover your products. After extracting keywords, delete keywords with a document frequency of 1 and keywords unrelated to the product to build a document-keyword matrix. This matrix is the output of the first step.
2. Calculate Perplexity while increasing the number of topics, and run LDA with the number of topics that gives the smallest value. For each topic, read the keywords and documents with high contribution and give it a name.
3. Calculate Importance, Satisfaction, and Growth by month and rank with TOPSIS. After applying a moving average matched to your own release cycle, find the topics whose rank has risen steadily over the last 6 months and make them a review list for the next planning meeting.