What is Data Wharehousing and Data Mining

[hafizfarooq]

Hello,

1. What is Data Wharehousing and Data Mining ?

2. Waht are the benefits of both ?

3. How can we use both ?

Farooq

[smUX]

Data mining is using a specific system/filter to extract useful information from a large source of data...i.e. taking a list of the ages of a group of people and adding it together and dividing by the number of people to get their average age. Data warehousing I'm not sure about

[RhinoBull]

Data Warehousing Review
Data Mining

[leinad31]

For an organization requiring data/trend analysis, you don't retain, or rather you don't use for analysis, all data that ever comes your way... its just too much to manage especially when for long durations, large customer base, data becomes obsolete, etc. So you setup a data warehouse that contains the most recent/relevant data you will subject to data mining. Data mining was already explained.

[hafizfarooq]

Thankyou all, specially RHINOBULL

[smUX]

Some more useful links, courtesy of the everpopular WikiPedia:

http://en.wikipedia.org/wiki/Data_warehousing
http://en.wikipedia.org/wiki/Data_Mining

Google is your friend, but so is WikiPedia...and don't let Google know it, because he becomes jealous!

[RhinoBull]

...Google is your friend...

Yea ... but you could've at least check my post...

[RhinoBull]

Thankyou all, specially ...

You're very special welcome!

[smUX]

Yea ... but you could've at least check my post...

I did for the data warehousing one, so nyaa :-P

[RhinoBull]

It keeps me wonder what that means...

[Predictor]

Data mining is using a specific system/filter to extract useful information from a large source of data...i.e. taking a list of the ages of a group of people and adding it together and dividing by the number of people to get their average age.

What you have described sounds more like report writing to me. Data mining (also called knowledge discovery in databases or KDD) is a sophisticated statistical analysis of data, most often predictive modeling. For more information, see, for instance, KDnuggets:

http://www.kdnuggets.com/


-Will Dwinnell
http://matlabdatamining.blogspot.com/

[smUX]

Data mining is mining for data, as the name suggests...filtering out the information you want from a larger cache of data. The work done on this data (in my example, adding the ages together and dividing by the number of people to get their average age) might be called report writing, but the gaining of the data in the first place is data mining :-)

[szlamany]

Data mining is not as described in some of these posts...

It first requires the data warehouse - this is the most important step.

This is where the data in a OLTP system is flattened and leaves 3rd normal form to enter the OLAP system.

The users of the OLAP system are not technical - so as they "mine" the data it must be changed in form so that they can more easily accomplish there goals.

For example, in the OLTP system we have INVOICE header records and INVOICE detail records and lots of linking table for VENDOR and what not.

When it's moved to the OLAP system it's flattened - we might have "sales" totals by state - or zip code - or region - that table is filled from the OLTP INVOICE/DETAIL/VENDOR tables.

Now the accounting dept when they want to mine for some info on how a particular region has done, or a zip code - they have a simple place to look for that data. JOIN's are pretty much out of the question - as not properly done they will skew results.

A data warehouse is created based on the needs of those who will be mining it.

It can always be argued that the OLTP system can produce reports for analysis - but then you always have to go back to programmer-level people to produce those reports.

The purpose of the warehouse is to present data to low-end users that can be easily manipulated to produce meaningful output.

[Predictor]

Data mining is mining for data, as the name suggests...filtering out the information you want from a larger cache of data. The work done on this data (in my example, adding the ages together and dividing by the number of people to get their average age) might be called report writing, but the gaining of the data in the first place is data mining

No, this is absolutely incorrect. The result of a data mining operation is a statistical model: code, essentially, not data. The whole point of performing data mining is to be able to make inferences about future behavior.

As an example, below is a (denatured) model I recently produced for a bank. The goal was to predict whether individual customers will repay loans. In this case, it is a logistic regression run over a single table of data. The resulting code is intended to generate predictions (probability estimates) about future cases, not summarizing old ones:

XModelLinear =
-0.12405053679991 +
-0.00232402562179 * PRIOR_AVAILABLE +
-0.4963737314753 * DOB +
-0.82299458796435 * cnt_4247 +
-0.48519697312454 * cnt_5341 +
0.5056754163603 * cnt_7831 +
2.32062670810524 * cnt_4939 +
1.9624082049473 * cnt_7961 +
0.67703628829194 * cnt_5643 +
0.00213194069755 * amt_5754 +
-0.0006523147294 * Att_amt_off +
0.11162160204319 * UtilCount +
0.0126422117559 * VeCount +
0.06749157992633 * CasCount +
5.71685305090582 * SCF416;

! Apply transfer function
XModel = 1 / (1 + exp(-XModelLinear));


-Will Dwinnell
http://matlabdatamining.blogspot.com/

[Predictor]

Data mining is not as described in some of these posts...

It first requires the data warehouse - this is the most important step.

While using a data warehouse can make data mining easier, it is not necessary. The important thing is to have a relevant source of historical data upon which to perform analysis. Obviously, the better organized and more complete that source is, the easier analysis becomes, so a well-constructed data warehouse can help. On the other hand, I have completed data mining projects using flat files, and others using production relational databases.


-Will Dwinnell
http://matlabdatamining.blogspot.com/

[szlamany]

And I've just watched data warehouse projects fail because the vendors don't seem able to evaluate the users need for mining.

I'm an OLTP person myself - but I've fed warehouses on mainframes for decades. Those were direct and served the purpose well. Some of our OLTP systems have million+ row tables - it's much easier (and faster) to flatten that for OLAP.

In one of our industries (school districts) there is an initiative to make "data driven decisions". Those are way more abstract then sales projections or actuarial projections. Trying to find elementary school predictors for why a senior is doing well in a particular type of class - the users don't know what to look for and almost want the suggestion to jump out at them all on it's own.

[bruce.smith]

Hi,

DATA MINING: Data mining is the process of extracting patterns from data. As more data are gathered, with the amount of data doubling every three years, data mining is becoming an increasingly important tool to transform these data into information. It is commonly used in a wide range of profiling practices, such as marketing, surveillance, fraud detection and scientific discovery.

While data mining can be used to uncover patterns in data samples, it is important to be aware that the use of non-representative samples of data may produce results that are not indicative of the domain. Similarly, data mining will not find patterns that may be present in the domain, if those patterns are not present in the sample being "mined". There is a tendency for insufficiently knowledgeable "consumers" of the results to attribute "magical abilities" to data mining, treating the technique as a sort of all-seeing crystal ball. Like any other tool, it only functions in conjunction with the appropriate raw material: in this case, indicative and representative data that the user must first collect. Further, the discovery of a particular pattern in a particular set of data does not necessarily mean that pattern is representative of the whole population from which that data was drawn. Hence, an important part of the process is the verification and validation of patterns on other samples of data.

BENEFITS:

Retail / Marketing

* Identify buying behavior patterns from customers.

* Find associations among customer demographic characteristics.

* Predict which customers will respond to mailing.

Banking

* Detect patterns of fraudulent credit card usage.

* Identify "loyal" customers.

* Predict customers that are likely to change their credit card affiliation.

* Determine credit card spending by customer groups.

* Find hidden correlations between different financial indicators.

* Identify stocks trading rules from historical market data.

Insurance and Health Care

* Claims analysis - determine which medical procedures are claimed together.

* Predict which customers will buy new policies.

* Identify behavior patterns of risky customers.

* Identify fraudulent behavior. Transportation

* Determine the distribution schedules among outlets.

* Analyze loading patterns.

Medicine

* Characterize patient behavior to predict office visits.

* Identify successful medical therapies for different illnesses.

DATA WAREHOUSING:Data warehouse is a repository of an organization's electronically stored data. Data warehouses are designed to facilitate reporting and analysis.

A data warehouse houses a standardized, consistent, clean and integrated form of data sourced from various operational systems in use in the organization, structured in a way to specifically address the reporting and analytic requirements.

This definition of the data warehouse focuses on data storage. However, the means to retrieve and analyze data, to extract, transform and load data, and to manage the data dictionary are also considered essential components of a data warehousing system. Many references to data warehousing use this broader context. Thus, an expanded definition for data warehousing includes business intelligence tools, tools to extract, transform, and load data into the repository, and tools to manage and retrieve metadata.

BENEFITS: * A data warehouse provides a common data model for all data of interest regardless of the data's source. This makes it easier to report and analyze information than it would be if multiple data models were used to retrieve information such as sales invoices, order receipts, general ledger charges, etc.

* Prior to loading data into the data warehouse, inconsistencies are identified and resolved. This greatly simplifies reporting and analysis.

* Information in the data warehouse is under the control of data warehouse users so that, even if the source system data is purged over time, the information in the warehouse can be stored safely for extended periods of time.

* Because they are separate from operational systems, data warehouses provide retrieval of data without slowing down operational systems.

* Data warehouses can work in conjunction with and, hence, enhance the value of operational business applications, notably customer relationship management (CRM) systems.

* Data warehouses facilitate decision support system applications such as trend reports (e.g., the items with the most sales in a particular area within the last two years), exception reports, and reports that show actual performance versus goals.

Here i would suggest an e book which is available online and can be directly downloaded by visiting " DATA MINING & DATA WARE HOUSING. " You can learn more better from here.

Thanks

http://www.amazon.com/Multidimensional-Data-Modeling-Toolkit-Intelligence/dp/0981775306

[BackWoodsCoder]

ugh all this talk about data mining is bringing back nightmarish memories of learning SAS and Clementine.

[szlamany]

bruce.smith should be reported for the spam that post of his was - this thread is nearly 3 years old!

[BackWoodsCoder]

whoa, necro post lol. Didn't even see that....

[techgnome]

I didn't even notice either.... that explains the moldy smell. *koff* *koff*

-tg