Feeds:
Posts
Comments

Archive for the ‘Data Mining’ Category

Theme 3: Integrating third-party data into predictive analysis

[tweetmeme source=”atripathy” only_single=false]This is the third installment of the eight part series on predictive analytics (see part 1, part 2).

Perhaps one of the most significant opportunities for organizations using predictive analytics is incorporating new relevant third-party data into their analysis and decision-making process.  Investment in a targeted and relevant dataset generates far greater returns than spending time in developing sophisticated models without the right dataset.

Suppose a wedding gown retailer wants to pursue a geographical expansion strategy. How would they determine where to open new stores? For that matter how should they evaluate the performance of existing stores?  Should a store in Chicago suburbs produce the volume of business as a store in Austin?

To answer the above questions, you need a lot of data that is not within the organization’s firewalls. One will need to know where people are getting married (demand in a market), how many competitor stores sell wedding gowns in the same area (competitive intensity in a market), how far potential brides are willing to travel to buy a wedding gown (real estate costs in city vs. suburbs will be vastly different), income and spend profile of people in the market (how much are customers willing to spend)

Marriage registration data from NCHS, socio-demographic data from a company like Claritas or US census, business data from Dun & Bradstreet or InfoUSA, cost data for real estate and maybe a custom survey data of potential brides should all be input variables into the store location analysis. Data about existing store sales and customer base are important, but they tell only part of the story and do not provide the entire context to make the right decisions.

Using the above data the retailer will be able to identify favorable markets with higher volumes and growth in marriages and appropriate competitive profiles. It can also use existing store performance data to rank the favorable markets using a regression or cluster analysis and then corroborate the insights using mystery shopping or a survey. Such a data driven methodology represents a quantum improvement over how new store locations are identified and evaluated.  While the datasets are unique to the problem, I find that such opportunities exist in every organization. A clear framing of the problem, thinking creatively about the various internal or external data, and targeted analysis leading to significantly better solutions is what information advantage is all about.

We are in the midst of an open data movement, with massive amounts of data being released by the government under the open government directive. Private data exchanges are being set up by Microsoft, InfoChimps among others. Not to mention all the new types of data now available (e.g., twitter stream data). Companies that build capabilities to identify, acquire, cleanse and incorporate various external datasets into their analysis will be well positioned to gain the information advantage

Read Full Post »

[tweetmeme source=”atripathy” only_single=false]Earlier this week, I suggested a potential business application of IRS’s internal migration data for a moving and relocation company.

Folks at Neilsen Claritas just found a far more interesting correlation which should have driven a lot of business decisions.  They note:

Today’s presence of underwater mortgages, or homes with negative equity, seem to be correlated to two common regional U.S. population trends: 1) domestic immigration from the Northeastern region to the South and Southwestern regions of the U.S., and 2) migration from coastal California inland

While such retrospective analysis is interesting for reports and blogs, it is not particularly useful for businesses. Maybe as means to generate  interesting hypothesis for future. It would have been useful had the chart been available to the strategic planning or risk group of businesses signing up people for these housing loans in 2006 and 2007.

Data is valuable only when it is used to drive decisions. Most companies have a huge opportunity to do a better job in bringing together data, analytics and visualization and delivering them to the points of decision.

Read Full Post »

[tweetmeme source=”atripathy” only_single=false]One subject that has not received a lot of coverage in the analytics blogging circle is the current administration’s data.gov project. While still in its infancy the data.gov is an outcome of the government’s transparency initiative called Open Government Directive. In December, all government agencies were asked to produce and publish three new ‘high value’ public data feeds on data.gov website.

The data.gov site still has to work through some of the kinks but eventually it will become a wonderful resource for the data analytics industry. Probably as critical as the US Census data and its American Factfinder tool, which has spawned multiple companies and supports all kinds of interesting analysis across a wide range of industries.

The Sunlight Foundation tracks the new datasets that are being released. For example one of the Labor departments datasets is the “weekly reports of fatalities, catastrophes and other events.” The data, compiled by the Occupational Safety and Health Administration, briefly describes workplace accidents, identifies the company at which and the date when the accident occurred. I think a lot of  insurance companies with worker compensation insurance products will be interested in analyzing the data to better price their products. Or take for instance the IRS internal migration data by state and county based on tax returns. Can it be used by moving companies to better understand the shift in demand for their services? There are thousands of such datasets available, and a lot of them will potentially be valuable to businesses. The value of a dataset to business like beauty, is in the eyes of the beholder. This makes the categorization challenging but at the same time makes it interesting for businesses as it can be a potential source of competitive advantage. If you can figure out to interpret the IRS migration data to better align your marketing campaigns for your moving and relocation assistance business, you can get better return on investment on your spend than your competition.

It is time for organizations to look outside their firewalls and build a strategy of collecting, incorporating and analyzing external data into their analytics and strategic planning efforts.  Companies like Infochimps, which is a private clearinghouse and market place for third-party data are betting on this trend. They already collect, cleanse and  format the data.gov datasets so that it is analysis ready.

Take out the time to check the datasets that are available. You never know what you may find.

Read Full Post »

Theme 2:  Modeling Strategic vs. Operational Decisions

[tweetmeme source=”atripathy” only_single=false]

In the first post of this eight part series, I wrote about the importance of understanding the cost of a wrong decision prior to making an investment in a predictive modeling project.

Once we determine the need for the investment, we need to focus on type of modeling approach.  The modeling approach depends on the type of decision that we want predictive model to drive. Decisions can be broadly categorized as either operational or strategic .

I define operational decisions as those that have a specific and unambiguous ‘correct answer’, whereas in strategic decisions an unambiguous ‘correct answer’ is not available. Moreover such decisions have a cascading effect on adjacent and related decisions of the system.

Think about a health plan predicting a fraudulent insurance claim versus predicting the impact of lowering reimbursement rates for patient re-admissions to hospitals.

An insurance claim is either fraudulent or not. The problem is specific and there is an unambiguous correct answer for each claim. Most transaction level decisions fall in this category.

Now consider the second problem. Lowering the reimbursement rate of patient readmission will certainly incent the physician and hospitals to focus on good patient education, follow-up outpatient care and to ensure complete episodes of care for the patient during their time in the hospital. This should result in lower cost for the health plan. However, it can also lead to hospitals delaying the discharge of patients during their first admission or physicians treating patients in an out-patient setting when they should be at the hospital and ending up in emergency room visits. This is the cascading effect of our first decision that will increase the cost of care. Strategic decisions have multiple causal and feedback loops which are not apparent and an unambiguous right answer is hard to figure.  Most policy decisions fall in this category.

The former is an operational decision and requires established statistical (regression, decision tree analysis etc.) and artificial intelligence techniques (e.g. neural networks, genetic algorithms). The key focus of the problem is to predict whether a claim is fraudulent or not based on historical data. Understanding the intricacies of causal linkages is desirable but not necessary (e.g.  neural networks).  The latter needs predictive modeling approaches that are more explanatory in nature. It is critical to understand causal relationships and feedback loops of the system as a whole.  The idea is to develop a model which accurately captures the nature and extent of relationships between various entities in the system based on historical data, to facilitate testing of multiple scenarios. In the re-admission policy example, such a model will help determine the cost impact based on the various scenarios of provider adoption and behavior change (percentage of providers and hospitals that will improve care vs. those that will not adapt to the new policy). Simulation techniques like systems dynamics, agent based models, monte carlo and scenario modeling approaches are more appropriate for such problems.

Bottom line, it is important to remember that strategic and operational decisions need different predictive modeling approaches and the two questions you have to ask yourself:

  1. Is the decision you want to drive operational or strategic in nature?
  2. Are you using the appropriate modeling approach and tools?

Cross-posted on TheInformationAdvantage blog

Read Full Post »

« Newer Posts