In my years as a consultant implementing data management solutions, my first question to a client would be …

Can I see the data model?

I have long felt that gaining a better understanding of an organization’s data landscape involves to primary artifacts; a data model and a data profile.  This is because, in a lot of cases, these two artifacts represent the two most fundamental states of an organizational data landscape:

what the data should look like and what it actually looks like

With knowledge of these two states, I felt armed with the ability to quickly and easily identify areas of conformity and areas of anomaly.  These two perspectives tended to be the basis of most Information Technology questions I was there to solve.  In this way, I looked at a data model as the starter kit to a data management strategy, or a 101 crash course to an organization’s data management state.

If there were a lot of anomalies, I knew they would require a lot of data quality strategy and remediation, as well as a robust data governance initiative.  If there was a lot of conformity, I knew the organization was mature enough to handle new data management initiatives like Master Data Management or Big Data implementation.

The sad reality is that most organizations either did not have data models for critical applications or felt that the data model was so out-of-date that it was not going to be very helpful to me in my quest for understanding.

Lack of Data Models: Data Management 100

Without a viable data model I was unable to reach these valuable conclusions quickly and was forced to, in a sense, reverse engineer profiling results which was time-consuming and based on some brash assumptions.

Here are some activities which helped me to mitigate the risks of performing educated guessing:

1. Perform Orphan Analysis
1. Analyzing orphans can help you determine the validity of a data model, how users are adding or deleting data and whether referential integrity constraints are even in place in production (which happens more than most will admit during interviews)
2. Analyze Documented versus Actual Data Types
1. Again, this addresses the validity of the design and how users are entering data (very often data is entered in formatted form, entry fields are used for purposes other than the original intention and developers build architectures without really understanding the scenarios that the app is required to support)
3. Analyze most and least commonly occurring values
1. This can help create a profile of how often standards are conformed to, word-of-mouth work-arounds in place and areas that are conforming and do not need attention (as valuable as identifying areas that do need attention)

Data Modeling Profiles: Data Management 102?

Having been through this many times and knowing how much time and effort this requires (often not accounted for in project plans), I feel strongly about developing a tool that can turn data profiles into a data model.  Most of the functionality is there already, someone just needs to make a case for it (just call me somebody).

Such a solution could take profile results, which include actual data types and inferred relationships, and create a data model that supports data management best practices like data governance and data quality.

In addition, a profiling-to-model function could go a long way in reducing the amount of time and error involved in building an MDM hub.  After all, profiling all the contributing sources is one the best practices in defining an MDM hub, why not take it the next step and bake that in?

I completely agree that there are going to be cases where there were design considerations based on performance and that a profile is not always the most accurate source for a model’s design, but there are many cases where a profile-to-model function would increase accuracy and performance and decrease error and time required to model data landscapes.

What do you think?

Take Our Poll

Why data discovery leads to better ETL design

Let’s start with why I feel this way.  Before I’d even heard of data quality I was doing it on a daily basis.  You see I spent several years as an ETL developer on many data warehousing implementation projects.

Typically after a couple of briefing meetings, I’d start developing ETL mappings.  Like any development effort that was followed by some unit testing where I would discover that although my ETL was written to specifications, the load didn’t “look right”.

After some digging I usually found the culprit was the fact that the source data did not match the expectations I had going into the development effort and it was time to, at the very least, add some transforms to the mapping to accommodate for the discrepancies.  In effect, I was performing two critical functions left out of the original development plan, data profiling and enhancement.  I feel strongly that had these two processes not been left out, I would have had a more complete and accurate ETL development experience from the get-go.

Unfortunately this was not an isolated event and, in fact, happened on almost every ETL project.  First hand experience is why I feel so strongly that data discovery leads to a better development process and, ultimately, outcome.

Supporting Research for Data Discovery in ETL Design

In a fairly recent polling exercise the Passioned Group, an analyst and consultancy company, based in The Netherlands, specializing in Business Intelligence, Data Integration and ETL tools, conducted a polling of 2,000 participants where they ask what they thought were the most important requirements when choosing an ETL tool.  The results demonstrated just how important data discovery is to ETL developers.

As you can see aside from performance, data profiling was the most important feature.  My intuition tells me that the people who responded to the poll had similar experiences to mine when developing ETL solutions.

Way back in 2001 William Laurent of Information Management wrote a piece entitled, Best Practices for Data Warehouse Database Developers.  The number one best practice was make sure you are provided with a usable data dictionary before starting heavy-duty development.  Data discovery can help build that data dictionary without relying on assumptions and assertions made by business analysts and database administrators.

In defining what ETL is the Passioned Group mentions data profiling by explaining how it can help build a system that

that is robust and has a clear structure.

The Data Warehouse Information organization , a site “Powered by “DWH Professionals“, “DWH Enthusiasts” and People alike” graphically depicts data profiling  in their recommended ETL design process.

Here is an important statement they make about the benefits of data profiling during ETL design.

Data Profiling is a process that familiarizes you with the data you will be loading into the warehouse/mart

So how do I use data discovery to achieve a better ETL design?

As I mentioned in my previous post, I recommend starting with the following question:

What are the critical data domains we are looking to integrate into the target?

The reason I start with this seemingly basic question is so that you can build true discovery processes into the ETL design.  True discovery finds data unbeknownst to the data consumer that also needs to be included in the target.  To me, this is one of the most value added services that the ETL team can provide to the data consumers.  Here is an example, taken from my previous experience, that demonstrates what I mean.

I had a marketing client that was looking to build a repository from which they could perform campaign management and analytics.  They had done a fair mount of quality due diligence and identified what they felt were the required sources.

When I asked my generic question there was a fair amount of dissent in the room and some even pointed to the source to target matrix (STTM) as my source of information.  However, I pressed on and discovered that some of the more executive users of the analytics were interested in performing analysis on customers were were marketed to but the address of record, for which the source systems was included in the STTM, was not deliverable (or was returned by the USPS).

As it turns out, this information was not stored in a source system but rather kept in a spreadsheet (of course) by one of the marketing administrators.  Of course knowing this allowed me to incorporate  the spreadsheet in the ETL sources but it also help us build in another process which discovered and profiled address data in critical business applications which were then included in an enrichment process so that undeliverable addresses could be updated with the proper addresses (where applicable).

Data discovery is a simple process once you know where to point the discovery tool.  This focus is obtained by asking the general but effective question I mentioned above.  Data domains, like address, help you ask more intelligent and specific questions like …

what critical applications store, collect or consume address data?

Once this is uncovered, data discovery works much the same way that data profiling works.  You define the source, build a connection, define and execute the profile jobs and decipher the results.

Data Discovery for ETL Tips

Here are a few tricks I use when performing data discovery for an ETL design proof of concept.

1. Profile early and often
2. Translate data profiles into a metadata dictionary
3. Identify data anomalies
4. Never develop an ETL map from a specification, do it based on profile results
5. Communicate where metadata and data distributions do not match the businesses expectations and look for the root cause

I know this list seems basic, but you’d be surprised how often it does not happen and how much rework and cost is incurred as a result.

Your thoughts?

the discovery of relationships between data elements, regardless of where the data is stored.

If you expand your mind beyond the conventional relational database meaning of relationships, I agree with this definition.  Relationships in this context, or rather the context I chose to apply, means much more than a primary – foreign key relationship.

In this context relationships is defined as commonality.  This commonality can be of a data type, value pattern, or business use.  If you can profile data and understand the relationships you can set yourself up for more efficient data management practices in the areas of ETL, MDM, and application lifecycle (or application retirement).  Let’s take each of these and examine how a data discovery can increase the quality of the effort.

ETL and Data Discovery

Classical ETL takes data from a source and loads it to a target.  If you perform data discovery profiling on the sources before you build the ETL mapping you can achieve the following:

• a more accurate picture of the required data type of the attribute
  • By examining the profile you can determine if the assigned data type is most appropriate for the data element
• a more accurate specification for the type of transform required
  • If the data and metadata are not 100% coordinated you can build transforms to accommodate for this
• identification of data anomalies and outliers which require further investigation for possible remediation prior to the migration of data
  • this leads to a more robust error handling and exception handling process
• the identification of data, previously unknown, that meets the business requirements and needs to be migrated
  • discovery can lead to uncovering data that was previously undefined or unobtainable for data migrations

As discover tools mature, it may also be possible to generate ETL mappings directly from the tool.  If the target is more richly defined in the discovery tool and the sources are more accurately identified, it makes sense to me that a discovery tool can build a better ETL mapping.

This will require a tight coupling between the discovery and ETL tool, however, there are vendors in the market with this type of coupling available to them.

MDM and Data Discovery

In the same way that data discovery can aid ETL, so too can it aid the efforts of an MDM implementation.  Since MDM implementations are so dependent on ETL, the same leverage is available and can lead to a better MDM hub definition and ETL specification.

Here too can a feature to generate a data mapping be particularly useful.  With so much configuration required for match and merge rules, cutting some development form the scope of the effort would only add benefit.

Another particularly interesting feature would be the ability to generate candidate schemas for the MDM hub based on the data and metadata obtained in the profiles of the sources.

Application Lifecycle and Data Discovery

Finally, during a data discovery investigation it is possible to segment data by data ranges derived from last create / update dates.  This can be leveraged to perform application and/or data lifecycle management which would basically archive data past a certain date line or retire an application which has not be accessed in a predetermined, business driven date.

Here is another use for dynamically generated data mappings which would migrate the retired data to a target or archive destination.

Discovery is only the first step

As you can see from this quick summary, there are many uses for data discovery and as the tools mature there are many more things that can be done to leverage a discovery effort.

Your thoughts?

I will be joining the world class team at Informatica to take the Data Explorer product to the next level.  Data Explorer (DE) is a profiling and discovery engine that focusing on the reporting of data distributions and metadata definition.

There could be no better place for me to land than Informatica.  I have been working with their products for close to 6 years now and find them among the easiest to learn and use in the marketplace.

I intend on using my knowledge gained as a consultant performing countless implementations to enhance the product through increased functionality, usability and transfer of functions with numerous products such as PowerCenter, Lifecycle Management, Metadata Manager, and, of course,  Data Quality.

I’d like to extend my gratitude to the colleagues and clients that have helped me get to this point in my career. 

With this move comes a shift for the blog.  I will primarily focus on those topic areas that influence or affect data discovery.  I’ll post updates about the Data Explorer product and poll the masses for ideas and feedback. 

I hope you enjoy the new and exciting path I am about to embark on as much as I will!

When Bieber tweeted 9 digits of his phone number and asked his twitter followers to guess the 10th digit and call him, he set two unsuspecting victims phones on fire. He also proved how similar phone numbers are and why they are bad candidates for match strategies.

Using phone numbers in match strategies is, in my opinion, a waste of time.  You are only going to increase your chances of generating false positives (unless you do an exact match).  My issue with the exact matches is that it is very easy to make a “fat-finger” error and still identify a false positive.

If you care, here is the link to the Bieber event.  If you are a Bieber fan and found your way to my blog by some search engine failure, my apologies (please don’t terrorize me).

Justin Bieber abuses Twitter with phone gag, may get sued – Technolog on msnbc.com.

Having done this before, I recommend using an address service, Address Doctor, and profiling the data prior to building the cleansing rules.

The latter is a topic I speak often about.  Profiling is an essential part of defining cleansing rules.  Well, meaningful cleansing rules.  Anyone can “stub” in those generic rules, but after profiling the data you can gain much more contextual insight and build customized data cleansing rules.

Check out the whole disucssion here:

I’m a long-time IT professional, but a newcomer to MDM and data cleansing. Any tips on cleansing Address data? | LinkedIn.

I was just thinking about this topic of making data discovery fun last night so this article really spoke to me.  I think Cindy is dead-on when she reminds us to make sure we engage our audience with a storyboard around data analysis that makes the user want to know the ending.

This is often forgotten,  albeit challenging, with regard to presenting data analysis.  I struggle, at times, with how to make presentations interesting.  And let’s face it snazzy icons in a PowerPoint deck does not count as entertaining.

What Cindy emphasizes in the article is to engage users with data stories using these five basic principles when writing your data story:

1. Refresh your data often
2. Build a complete dashboard, but don’t over complicate it (tricky and important!)
3. Encourage further investigation with data discoveries (my favorite!)
4. Analyzing data is fun, not just a job
5. Draw conclusions with your analysis that are accurate and meaningful

To me, points 3 & 5 are the real important concepts here.  I think you can facilitate further investigation with meaningful results.  And the key lies in the term meaningful.  To do this effectively, you need to bear in mind your audience.  Meaningful to a controller is not the same as it is to a DBA.  However, data discovery activities can support both these roles and you need to be sure to deliver something they care about in your story.  I think, if you do, that individual will be compelled to conduct further investigations which is where Cindy’s point of being accurate is important.  Make sure you are on point in your analysis!

Read the referenced article here:

Clues to a Great Business Intelligence Story | Visual Data Group.

Check out Cindy on Twitter: @CindyBHarder

Join the discussion!

I’m curious about how folks here are using data discovery and what tools they are using to do discovery #datadiscovery | LinkedIn.

With regard to the opportunities, one of the most interesting things I see developing is increased access to customers.  From traditional to mobile platforms, there are more avenues to interact with customers, presenting an opportunity for product and service providers new ways to measure their effectiveness.  I have starting researching things like sentiment analysis which is an example of how access to customers and data explosion provides insight into product / service perception.

With regard to the challenges, performing analysis on this data requires tools, methodologies, and resources that are very unique and unconventional.  For most organizations, it will take some time to align the resources to perform meaningful analysis.  That is not even taking into the account the budget that needs to be set aside for this activity.

While the technology industry is thrilled with their new story filled with magical elephants and all the promise of a new reality, the boots-on-the-ground in data management must feel like a deer caught in the headlights of an on coming 18 wheeler at 90 mph.  To some the data explosion must feel like fireworks against the warm summer sky, to others the explosion must feel like the pounding of cannon fire against the office wall.

What I think it is very important to realize is that this data explosion is really both at the same time.  We need to be realistic and remember that while there is a lot of data out there and with it comes the promise of gaining new insights, this presents significant challenges to organizations in just how they are going to roll this into the mix of things that already need to do.

I intend on keeping my eye on what organizations come out winners and, maybe even more so, what organizations come out as losers in this new data frontier.  One of the things I intend on paying particular attention to is ROI.  What it costs to do this well and what it produces.

Until I see what that looks like, I am going to hold off getting giddy about big data / no sql … what about you?  Are you “all in” or waiting to see how this goes?

In the article, Brian Proffitt of IT World, details Hadoop VP Aran Murthy’s Strata presentation which describes how Hadoop is expanding the types of applications that Hadoop’s MapReduce will be able to support.

This expansion was compared to an operating system.  While I can’t quite see that analogy all the way through, I do see the strategic impact of being able to build big data applications against a Hadoop framework.

This type of expansion could free software developers to concentrate on the more front-end, user facing aspects of application features.  Something I have long thought would be a significant challenge.

I am eager to see this framework in action.  I’ll hold off on judgement until then …

Next Hadoop confirms data as a platform | Business Intelligence – InfoWorld.