Did you ever wonder why the data error was entered into the system, database, or report that’s right there in front of you?
You can look at it a hundred different ways.
-- The customer entered the data wrong on the website.
-- The call centre rep, entered the data wrong on the website.
-- The sales rep, forgot to enter his sales for the month of September and keyed it in for October.
-- The programmer entered the wrong statement in the data integration script.
-- The programmer put ‘greater than’ instead of ‘less than’ statement in the summarization script.
-- The business analyst did not provide the correct data retention requirements, and that’s why you have 6 months of summarized data vs. 16 months.
-- No one could come to a consensus on the definition of the value, that’s why we have 187 values for that field.
These are just a few reasons bad data is where it is.
What I’d like to know is what are your reasons, or the reasons you’ve heard. Feel free to let me know, send me a list of reasons why bad data existed in your system, application, database or report. I don’t want high level reasons, let’s have the granular reasons. When I get to 101 I’ll publish the list for all to see (no names or course).
However, here’s another reason to think about it. It’s apathy.
Really, really.
To have good, quality, accurate data all you need is a little TLC. For data to be accurate people, need to care just a little more about what they are doing. In the above examples, if people gave a little TLC there would be no bad data.
We live in a rushed, hurried world where everything is needed yesterday so a little TLC is hard to come by.
Showing posts with label Data Quality. Show all posts
Showing posts with label Data Quality. Show all posts
Friday, December 4, 2009
Friday, October 2, 2009
September Festival del IDQ Bloggers
With the month of September come and gone, the changing colour of the leaves starting, hockey season starting and the wind getting colder by the day, we found another month filled with interesting posts about data quality. This month I am happy to say, I'm hosting September's "Festival del IDQ Bloggers".
An annual data quality blogging carnival held by the International Association for Information and Data Quality, an international not-for-profit association dedicated to the development of the data and information and data quality profession.
The following is a quick list, I'll say quick but it actually took an excruciating long time to compile, split coffee on my keyboard, hit my head on the light and I stubbed my toe in the process ;-)
On with the data quality blog round-up...
From the DoBlog (http://obriend.info/) the personal blog of Daragh O Brien, IAIDQ Director and Information Quality consultant and writer. Since 2006 Daragh has been writing about Information Quality related topics (amongst other things) on this blog and has even won an Obsessive Blogger award for his writing on Information Quality topics.
We find two posts of interest one about the Law and the other about Market Research.
Blog Post: http://obriend.info/2009/09/25/finding-red-herrings-or-missing-a-trick/
Market Research often falls foul of poor quality information about the target sample population. In this post Colin Boylan (a freelance Market Researcher) discusses some of the issues that can lead to you chasing Red Herrings or just Missing a Trick.
Colin Boylan is a freelance market researcher living and working in Ireland. He has worked with many of the leading market research firms in Ireland and the UK, with particular experience in Pharmaceutical studies (where good quality data is essential).
Also in the same blogging journal we have an interesting tale about the law looking at data quality...
We find two posts of interest one about the Law and the other about Market Research.
Blog Post: http://obriend.info/2009/09/25/finding-red-herrings-or-missing-a-trick/
Market Research often falls foul of poor quality information about the target sample population. In this post Colin Boylan (a freelance Market Researcher) discusses some of the issues that can lead to you chasing Red Herrings or just Missing a Trick.
Colin Boylan is a freelance market researcher living and working in Ireland. He has worked with many of the leading market research firms in Ireland and the UK, with particular experience in Pharmaceutical studies (where good quality data is essential).
Also in the same blogging journal we have an interesting tale about the law looking at data quality...
Blog Post: http://obriend.info/2009/09/29/a-game-changer-ferguson-v-british-gas/
For about 4 years, Daragh has been hammering on about how poor quality information can and could get an organization sued. In January it happened, with a very clear and explicit ruling in the Court of Appeal for England and Wales that sets a very interesting legal precedent (binding in England and Wales and persuasive in all other Common Law jurisdictions such as Ireland, Canada, Australia, India, USA, Pakistan....). This post (based on an article Daragh wrote for the IAIDQ in April) looks at that case and the implications for Information Quality professionals.
Daragh O Brien is a Director of IAIDQ, a Fellow of the Irish Computer Society and, after escaping from indentured servitude in a leading Irish Telco after 12 years is in the process of establishing a specialist Information Quality Management and training practice. He is also writing a book on legal issues in Information Quality with Fergal Crehan, a prominent Irish barrister (lawyer).
No Blog Carnival is complete without a post from the Obsessed Jim Harris in this short but sweet post Jim talks about knowledge and the fact that we know what we know, and we don’t know the rest. Something to think about as you read Jim’s post.
For about 4 years, Daragh has been hammering on about how poor quality information can and could get an organization sued. In January it happened, with a very clear and explicit ruling in the Court of Appeal for England and Wales that sets a very interesting legal precedent (binding in England and Wales and persuasive in all other Common Law jurisdictions such as Ireland, Canada, Australia, India, USA, Pakistan....). This post (based on an article Daragh wrote for the IAIDQ in April) looks at that case and the implications for Information Quality professionals.
Daragh O Brien is a Director of IAIDQ, a Fellow of the Irish Computer Society and, after escaping from indentured servitude in a leading Irish Telco after 12 years is in the process of establishing a specialist Information Quality Management and training practice. He is also writing a book on legal issues in Information Quality with Fergal Crehan, a prominent Irish barrister (lawyer).
No Blog Carnival is complete without a post from the Obsessed Jim Harris in this short but sweet post Jim talks about knowledge and the fact that we know what we know, and we don’t know the rest. Something to think about as you read Jim’s post.
Jim’s OCDQ blog is an independent blog offering a vendor neutral perspective on data quality. A place where he offers a diversity of viewpoints in a collaborative style environment. Jim himself is an independent consultant, speaker, writer and blogger with over 15 years of professional services and application development experience in data quality (DQ), data integration, data warehousing (DW), business intelligence (BI), customer data integration (CDI), and master data management (MDM). Jim has worked with Global 500 companies in finance, brokerage, banking, insurance, healthcare, pharmaceuticals, manufacturing, retail, telecommunications, and utilities.
Jumping across the pond and over to Sweden, where I’ll take a moment and say hi to the Ericsson’s, Brit, Mikael, Max, Guztav and Hanah, I hope all is well. Then a quick move to Denmark where, we have DQ blogger Henrik Liliendahl Sørenson. A man of many talents, who has worked over 20 years in applications, databases and data in general. Henrik has demonstrated his expertise in business directory matching and international aspects of data quality improvement and master data management.
Henrik’s blog, Liliendahl on Data Quality, is a collection of his personal opinions, experiences and observations around data quality. Accumulated over decades and I do mean decades of experience.
Jumping across the pond and over to Sweden, where I’ll take a moment and say hi to the Ericsson’s, Brit, Mikael, Max, Guztav and Hanah, I hope all is well. Then a quick move to Denmark where, we have DQ blogger Henrik Liliendahl Sørenson. A man of many talents, who has worked over 20 years in applications, databases and data in general. Henrik has demonstrated his expertise in business directory matching and international aspects of data quality improvement and master data management.
Henrik’s blog, Liliendahl on Data Quality, is a collection of his personal opinions, experiences and observations around data quality. Accumulated over decades and I do mean decades of experience.
Henrik discusses the multi-use potentials of data quality...could it be...can data quality be used for increasing revenues, and for marketing...read on and find out.
The post has a follow up post sparked by the comments: http://liliendahl.wordpress.com/2009/09/27/process-of-consolidating-master-data/
Going across the globe and going to that lovely local known as Australia we have Vincent McBurney a manager with Deloitte Consulting in Australia and has 15 years as an application programmer, database programmer, ERP implementer and information management consultant. The blog is dedicated to a tool based approach to data integration with news and tips on IBM InfoSphere, Informatica, Oracle, Microsoft and any breaking data integration news.
This particular post is interesting because it talks about something we all like - fudge. But not this fudge, no way, this fudge is actually fudging moments and then having to apply some kludge techniques or go in and kludge the situation to fix it. Fudge, Kludge all around a great read.
Blog Post: http://it.toolbox.com/blogs/infosphere/the-data-quality-and-how-to-fudge-it-34289
From the Data Quality Pro…Dylan Jones provides us an excellent interview with Ken O’Connor who discuss the a means in creating a data issue assessment process, something everything DQ Team should have in place, and that’s why it’s here. Coming from the trenches this is something any data quality analyst can use over and over again.
Going across the globe and going to that lovely local known as Australia we have Vincent McBurney a manager with Deloitte Consulting in Australia and has 15 years as an application programmer, database programmer, ERP implementer and information management consultant. The blog is dedicated to a tool based approach to data integration with news and tips on IBM InfoSphere, Informatica, Oracle, Microsoft and any breaking data integration news.
This particular post is interesting because it talks about something we all like - fudge. But not this fudge, no way, this fudge is actually fudging moments and then having to apply some kludge techniques or go in and kludge the situation to fix it. Fudge, Kludge all around a great read.
Blog Post: http://it.toolbox.com/blogs/infosphere/the-data-quality-and-how-to-fudge-it-34289
From the Data Quality Pro…Dylan Jones provides us an excellent interview with Ken O’Connor who discuss the a means in creating a data issue assessment process, something everything DQ Team should have in place, and that’s why it’s here. Coming from the trenches this is something any data quality analyst can use over and over again.
DataQualityPro is an online community resource that is solely dedicated to the needs and development of data quality professionals everywhere. Dylan Jones, is the founder and editor of Data Quality Pro and Data Migration Pro, leading online knowledge centre and community sites for their respective professions.With a 15 year background in data quality and data migration Dylan now supports a global community of several thousand professionals who actively collaborate and contribute to help increase the collective knowledge in these fields.
Here’s an interesting read about data governance from Gwen Thomas. Gwen Thomas is Founder and President of The Data Governance Institute, which is the premier provider of in-depth, vendor-neutral information about, and assistance with, tools, techniques, models, and best practices for the governance/stewardship of data and information. This is Gwen’s personal blog from the Data Governance Institute.
Here’s an interesting read about data governance from Gwen Thomas. Gwen Thomas is Founder and President of The Data Governance Institute, which is the premier provider of in-depth, vendor-neutral information about, and assistance with, tools, techniques, models, and best practices for the governance/stewardship of data and information. This is Gwen’s personal blog from the Data Governance Institute.
This post is here because data quality is a big piece of data governance. Data governance provides guidance in defining quality. There is a symbiotic relationship between the two.
With this post we get a high level view of the net-centric governance and the potential issues of control one may have with it. It gets the wheels turning when you begin to think of the implications that may and could very well follow.
With this post we get a high level view of the net-centric governance and the potential issues of control one may have with it. It gets the wheels turning when you begin to think of the implications that may and could very well follow.
Blog Post: http://datagovernancematters.com/2009/09/14/net-centric-data-governance-not-for-sissies/
Finally…
After hearing about the actions of an old friend of mine working hard in a data quality team, I was surprised to learn she is still having to justify the existence of the data quality team. It’s always good to know about what you’re doing and the value you bring to the table, but this being the 3rd time within a year, I believe enough is enough…so here it is. Who am I, well I am this guy, business analyst by day, data quality analyst by night. My blog, Data Quality Edge, is really a place to voice my opinions and what I hope will provide a grassroots look at data quality, something really for the data quality analyst in the trench. Because they are the ones that get the job done.
Finally…
After hearing about the actions of an old friend of mine working hard in a data quality team, I was surprised to learn she is still having to justify the existence of the data quality team. It’s always good to know about what you’re doing and the value you bring to the table, but this being the 3rd time within a year, I believe enough is enough…so here it is. Who am I, well I am this guy, business analyst by day, data quality analyst by night. My blog, Data Quality Edge, is really a place to voice my opinions and what I hope will provide a grassroots look at data quality, something really for the data quality analyst in the trench. Because they are the ones that get the job done.
Wishing you all the best in the cooler months ahead! Good reading.
Dan
Wednesday, September 16, 2009
Stop Justifying Data Quality Programs and Do the DQ Work Already!
In a recent discussion with a good friend, I learned that they are in the middle of justifying their work in a data quality team. This being said, a few months ago they were doing it as well, and at the beginning of the year they had just wrapped up another justification project, in the beginning of the economic downturn, it was being done as well. I also know that a few years ago when I was with the team, we also had to do it.
It's a shame. A terrible shame! Some organizations understand the importance of data quality, sometimes that understanding has come at a cost:
• Lost thousands to millions;
• Faced national embarrassment;
• Or made significantly big policy screw-ups.
While other organizations, are more pro-active and have established a data quality team and program to prevent such events from happening. An activity that is considered a best practice and essential to any information technology/business intelligence structure.
However, in either case, you may have someone, traditionally a senior manager, who sees data quality as a cost, a black hole. Yes there is a cost, however the benefits outweigh the costs in a variety of ways.
• Reduction in re-work due to good data quality;
• Improved incoming data quality and data processing due to pro-active initiatives with incoming data migration and integration projects;
• Proactively preventing data quality issues from occurring;
• Improved decision making, using quality data, and more.
To my old team and senior management:
Stop with the justification exercises and begin looking at the benefits and what this dedicated group of data quality analysts have accomplished year after year.
• Recognized Finalist Best Practice by TDWI in DQ;
• Hundreds of data modelling, metadata, data processing and data corrections to incoming projects per year;
• Proactively seeks data processing improvements to improve data loads - ultimately reducing costs;
• Client support to decision makers who really don't understand the technology aspects of the data and its routines;
• Dozens of change management practices each year to improve data quality and data processing which collectively prevents lost revenues, increases sales and manages maintenance costs by reducing reruns and supporting programs such as customer profitability, and other CRM initiatives.
• The estimated benefits weigh in at an average of $1-1.5 million a year if not more.
Another justification exercise only takes the team away from doing what needs to be done, data quality.
So to the senior management in this organization and any other, yes there is a cost to any data quality program. Just remember a data quality team is your vanguard to any organization that deals heavily in data. They bring in benefit. They enable your decision makers. They protect your greatest asset - data!
A good DQ team = Great Value!
It's a shame. A terrible shame! Some organizations understand the importance of data quality, sometimes that understanding has come at a cost:
• Lost thousands to millions;
• Faced national embarrassment;
• Or made significantly big policy screw-ups.
While other organizations, are more pro-active and have established a data quality team and program to prevent such events from happening. An activity that is considered a best practice and essential to any information technology/business intelligence structure.
However, in either case, you may have someone, traditionally a senior manager, who sees data quality as a cost, a black hole. Yes there is a cost, however the benefits outweigh the costs in a variety of ways.
• Reduction in re-work due to good data quality;
• Improved incoming data quality and data processing due to pro-active initiatives with incoming data migration and integration projects;
• Proactively preventing data quality issues from occurring;
• Improved decision making, using quality data, and more.
To my old team and senior management:
Stop with the justification exercises and begin looking at the benefits and what this dedicated group of data quality analysts have accomplished year after year.
• Recognized Finalist Best Practice by TDWI in DQ;
• Hundreds of data modelling, metadata, data processing and data corrections to incoming projects per year;
• Proactively seeks data processing improvements to improve data loads - ultimately reducing costs;
• Client support to decision makers who really don't understand the technology aspects of the data and its routines;
• Dozens of change management practices each year to improve data quality and data processing which collectively prevents lost revenues, increases sales and manages maintenance costs by reducing reruns and supporting programs such as customer profitability, and other CRM initiatives.
• The estimated benefits weigh in at an average of $1-1.5 million a year if not more.
Another justification exercise only takes the team away from doing what needs to be done, data quality.
So to the senior management in this organization and any other, yes there is a cost to any data quality program. Just remember a data quality team is your vanguard to any organization that deals heavily in data. They bring in benefit. They enable your decision makers. They protect your greatest asset - data!
A good DQ team = Great Value!
Monday, August 10, 2009
Pooh, Data Analyst
Obviously Data Quality is an important issue. But so is good communication and having a good data analyst around!
After story time with my daughter, taking some elements from a classic that everyone loves…saying no more…let’s stop in and take a look at what’s happening in the “Hundred Acres Woods Data Centre” with our friend Pooh, Data Analyst!
In ran Piglet on his short stubby feet, “Rabbits coming, and he is doesn’t have happy feelings with him”.
“Pooh bear, you’re not thinking again!” bellowed Rabbit as he marched into the Hundred Acres Woods data centre.
“Well, thinking is often a hard thing to do, without a smackerel of honey.” commented Pooh Bear.
“POOH BEAR!”
“Oh I think I have a headache!” said Pooh.
“How is it bear that you have it wrong, the price of honey bottomed out, it’s less valuable then the pebbles on the floor. You said honey was the commodity of choice. That it is was going up. Eeyore invested everything, and now he doesn’t have anything.”
“Oh that is so sad! What is it that he lost everything of?”, asked Pooh.
“EVERYTHING!”
“We need to hoosh, Eeyore!” interrupts Piglet.
“Yes”, agrees Pooh bear, “but maybe a little less hooshing this time then last time, would you not agree Piglet?”
“No”, said Piglet, “Yes”, replied Piglet quickly afterwards.
“What, NO!”, cried Rabbit, “ You can’t hoosh Eeyore, he’s suing us for more sticks so he can build a new house.”
“Suing?”, interrupted Pooh and Piglet together.
“Yes suing, Eeyore’s going to sue us and we’ll have to work for him, looking for sticks and other stuff. Now tell me why did you say honey. It’s an important thing to know.”
“Yes that’s what I saw on the computer screen.”
“Show ME!”
Pooh, waltzed over to the terminal and showed Rabbit the honey dripping off the computer terminal.
“Poor Bear, you spilt honey on your screen, the server, the computer. It’s everywhere! That is not good analysis of the data. Now print up the report and bring it to me. And I’ll analyze it!” said Rabbit, as he stormed out of the data centre.
A few minutes later, “Here’s the report Pooh”, stated Piglet as he hands the report to Pooh Bear.
“Mmmm!” said Pooh
“What’s mmmm?” asks Piglet.
“I think Rabbit is wrong, I have honey on this paper report too. I think we should stay with honey.”
“Pooh, you have been eating honey, it’s on your paws. A smackerel smudged your report.”
“Yes, Piglet! I believe that’s a good thing.”
After story time with my daughter, taking some elements from a classic that everyone loves…saying no more…let’s stop in and take a look at what’s happening in the “Hundred Acres Woods Data Centre” with our friend Pooh, Data Analyst!
In ran Piglet on his short stubby feet, “Rabbits coming, and he is doesn’t have happy feelings with him”.
“Pooh bear, you’re not thinking again!” bellowed Rabbit as he marched into the Hundred Acres Woods data centre.
“Well, thinking is often a hard thing to do, without a smackerel of honey.” commented Pooh Bear.
“POOH BEAR!”
“Oh I think I have a headache!” said Pooh.
“How is it bear that you have it wrong, the price of honey bottomed out, it’s less valuable then the pebbles on the floor. You said honey was the commodity of choice. That it is was going up. Eeyore invested everything, and now he doesn’t have anything.”
“Oh that is so sad! What is it that he lost everything of?”, asked Pooh.
“EVERYTHING!”
“We need to hoosh, Eeyore!” interrupts Piglet.
“Yes”, agrees Pooh bear, “but maybe a little less hooshing this time then last time, would you not agree Piglet?”
“No”, said Piglet, “Yes”, replied Piglet quickly afterwards.
“What, NO!”, cried Rabbit, “ You can’t hoosh Eeyore, he’s suing us for more sticks so he can build a new house.”
“Suing?”, interrupted Pooh and Piglet together.
“Yes suing, Eeyore’s going to sue us and we’ll have to work for him, looking for sticks and other stuff. Now tell me why did you say honey. It’s an important thing to know.”
“Yes that’s what I saw on the computer screen.”
“Show ME!”
Pooh, waltzed over to the terminal and showed Rabbit the honey dripping off the computer terminal.
“Poor Bear, you spilt honey on your screen, the server, the computer. It’s everywhere! That is not good analysis of the data. Now print up the report and bring it to me. And I’ll analyze it!” said Rabbit, as he stormed out of the data centre.
A few minutes later, “Here’s the report Pooh”, stated Piglet as he hands the report to Pooh Bear.
“Mmmm!” said Pooh
“What’s mmmm?” asks Piglet.
“I think Rabbit is wrong, I have honey on this paper report too. I think we should stay with honey.”
“Pooh, you have been eating honey, it’s on your paws. A smackerel smudged your report.”
“Yes, Piglet! I believe that’s a good thing.”
Wednesday, August 5, 2009
DQ Alert: Easy Savings by Removing Dups
Poor data quality costs over $600 billion annually as per TDWI surveys and studies.
Whether you are the VP looking over your latest sales, the secretary compiling a mailing list, the data quality analyst rummaging through data sets, the business analyst working on a data integration project, or an accountant going over the projected budget.
Whether you are in a large Fortune 500 company, in the government, or a small community association you will benefit.
Duplicate records
Duplicate records of customers cause discontented customers and multiple mail-outs. How is it possible to have duplicate records?
-- Records are manually entered twice;
-- Processes create record twice;
-- Participants registered under multiple names;
-- Participants registered at multiple locals.
So before you send out a mailing list to your community members, or potential program participants for marketing campaigns or program sign-ups for this year’s sports, arts, sales, and/or membership drive seasons:
-- 1. Sort that list by name. Why because you may have the child on the list twice, or three times; after sorting by name;
-- 2. Sort by address, you may have the same family household multiple times because one the parents registered under their name, and/or siblings are registered with your organization as well.
Don’t know how to sort…here’s a good way to start if you’re using Microsoft Excel…highlight your records and click on this little icon 
, it’s easy as breaking eggs.
Or
New SQL type programming here’s a simple query that will identify duplicate records.
SELECT attribute1, attribute2, attribute3, attributen… count(*)
FROM dbo.tablex
GROUP BY attribute1, attribute2, attribute3, attributen…
HAVING (COUNT(*) > 1)
I hope this helps anyone working with customer lists..
Monday, July 20, 2009
Sun Tzu and the Art of Data Quality (Part 3)
Concluding a three-part series of several Sun Tzu teachings in the Art of War and how they may apply to the Art of Data Quality.
Sometimes it’s all about the troops in your camp.
“To begin by bluster, but afterwards to take fright at the enemy’s numbers, shows a supreme lack of intelligence.”
Are you the type of data quality analyst who wants everything. Do you take all challenges on so you look good, or don’t trust others to do the job right; well perhaps you should think before you take it on. Because if you ever ask for it, and can’t deliver the goods you will look like a fool, and be considered unreliable, a definite career shaker.
“He who exercises no forethought but makes light of his opponents is sure to be captured by them.”
Closely linked to the previously mentioned teaching. In this one we learn that if you do not plan (use forethought), you may find yourself in dire situation. Remember the general rule of thumb. Eighty percent planning-twenty percent execution.
“Hence the experienced soldier, once in motion is to set up one standard of courage which all must reach.”
It always pays well to have experienced staff in your data quality competency center. They are active, they know the data, they know the business, they’ve helped developed the standards required to make your data a high quality asset. Another aspect of an experienced analyst, they are the ones to establish the governance that can be applied to almost any situation. In their absence, bad data can be targeted and the result will be a corrected situation with quality data at your disposal; all based on their governance input. A lot of people won’t do this, for fear of job security, it’s those rare individuals that do it, that will move your organization forward. So to you I say, keep the courageous.
“Success in warfare is gained by carefully accommodating ourselves to the enemy’s purpose.”
Data in general always has a purpose. Bad data has no purpose, it is created because someone, or something, somewhere ‘screwed up’. You cannot accommodate yourself to thepurpose of bad data, but you can accommodate yourself to ‘high-traffic’ areas, areas that are most likely to see bad data come-in. To sum it up, be prepared and recognize those areas in your model where bad data will rear it’s ugly head and be ready to chop it off.
In conclusion, I’d like to quote Jim Harris’s comment from the first Sun Tzu article. A paraphrase from Sun Tzu.
“Hence it is only the enlightened executive and the wise leader who will use the highest intelligence of the enterprise for purposes of data quality, and thereby they achieve great results. Quality is an important element in data, because upon it depends an enterprise’s ability to succeed.”
Monday, July 6, 2009
Sun Tzu and the Art of Data Quality (Part 2)
Continuing with the ancient teachings of Sun Tzu and the Art of War, we can gain further insight in the actions needed for building a solid foundation for the Art of Data Quality.
Sometimes it's about timing.
"The quality of decision is like the well-timed swoop of a falcon which enables it to strike and destroy its victim."
Place the data quality analyst in the position of a falcon and your enemy is that lovely data error that needs to be utterly removed from the system. Ultimately, the data will ony be removed by a decision made by the data quality analyst or promoted to higher-ups to decide upon, again identified and promoted by the data quality analyst. The quality of your decision and your analysis has significant impact on how your data repository is viewed by others.
"Rouse him, and learn the principle of his activity or inactivity, Force him to reveal himself, so as to find out his vulnerable spots."
Taking this teaching into account, we can state that Sun Tzu may have been telling his military students to be proactive. A proactive approach to seeking out and removing bad error to improve your data quality is essential.
Taking into account the definition of proactive, "Taking the initiative by acting rather than reacting to events."
Jim Harris, talks about a hyperactive data quality and mentions a proactive approach to handling problems from coming in. That approach looks at prevention. Prevention is a significant key in proactive data quality. Incorporate Sun Tzu's teachings in proactive data quality and you will be vigorously looking for errors, you'll be data profiling like you've never profiled before. Nothing wrong with that either, it's may lead to unwanted discoveries or great opportunities for data cleansing.
"Do not repeat the tactics which have gained you one victory, but let your methods be regulated by the infinite variety of circumstances."
I'm all for standards, governance, policies and processes to get the job down. The way the job should be down. BUT, you also need to be flexible and adaptable. You may have just saved the company a great deal of money with some fancy DQ manoeuvre. However, just remember the next problem that comes along does not necessarily follow the same trail in. So, you must know your systems, processes and be prepared to make adjustments to your tactics to clean that data quickly and effectively.
"We cannot enter into alliances until we are acquainted with the designs of our neighbours."
Not so much about timing, but an important message nevertheless. When developing a service agreement with internal client groups, external customers, vendors, reporting teams, or anyone that needs your data. Be prepared to get 'acquainted' with them. Wine and dine them, understand their needs, desires and limitations. What is it that they want and why they want it? If they are providing provisioning statistics to regulators why do they need marketing campaign statistics. Why do they want the data on a weekly basis when they only analyze it monthly. Get to know them, so you can better serve them. Remember data quality is not just about bad data or good data, it's also about relevant data.
In conclusion for this post, I'd like to say,
"Ponder and deliberate before you make a move."
Remember "haste makes waste", plan and time yourself carefully and know your facts.
Monday, June 29, 2009
Sun Tzu and the Art of Data Quality
Having read Sun Tzu’s Art of War I’d like to say its genius in its simplicity. This is a book that is referred to by many people to many things. Politicians, business leaders, professionals and even educators refer to the man’s ancient wisdom. This time I’d like to refer to it for - the Art of Data Quality.
Sun Tzu’s book has many teachings one can examine, we’ll only draw out a few and how they can be related to data quality.
“The art of war is of vital importance to the State.”
Liken State to Enterprise, and we come to understand that data quality is a critical and vital importance to any organization that handles data. Pointing out the following three examples and we see just how bad data can impact the organization and/or it’s customers, the life blood of any organization.
1. Millions of dollars giving away to a pair of customers who flee the country.
2. A database that holds 7 versions of one customer.
3. A unique health number assigned to two children living 90 miles apart.
“It is a matter of life and death, a road either to safety or to ruin. Hence it is a subject of inquiry which can on no account be neglected.”
It can not be neglected! You know that, the people that read this post know that. When things are a matter of life and death, it must become viral, it must be communicated. Bring it to the forefront of this week’s status meeting, bring it to the board room. If you are IT talk to the business people about it, if you are business talk to your IT team. It’s everyone’s job.
People like Jim Harris, Dylan Jones, Daragh O’brien and so many more. They all get it, they talk about it, they live and breathe it. After reading their posts do YOU say great post and leave it at that, or take it and tell others in your enterprise about it, spread the word, tweet it, present it, talk about it. Remember, Data Quality cannot be neglected. Billions of dollars are wasted each year because of poor data.
“In war, then, let your great object be victory, not lengthy campaigns.”
Sun Tzu dedicated a section to the length of military campaigns, and the importance of having shortened campaigns vs. long ones. This can be applied to data quality as well. The importance here is every organization has limited resources, (people, money and time). You must use your resources effectively so that your teams’ moral does not become demoralized because the bad data is still occurring or re-occurring after a lengthy cleansing project. You must work efficiently and effectively so that the bad data can be isolated and removed, whether it is a small billing adjustment that went wrong or a terrible boondoggle that was created, and let’s face it we’ve all seen boondoggles.
“If you know neither the enemy nor yourself, you will succumb in every battle.”
In conclusion I’d like to remind you all that,
“The art of data quality is of vital importance to the Enterprise.”
Monday, June 1, 2009
Entry Point: Change is a Constant
How many times have you received bad data from an upstream ‘stable’ database environment for no reason what so ever?
1…
2…
10…
13…
76…
Never…?
How’s this for a reply…no environment is stable! PERIOD.
Each and every data warehouse environment is subject to change, subject to growth, subject to budget constraints, and other external conditions (i.e., political changes). Their will always be change, THAT you cannot control.
Remember SOx. One recent example in Ontario is the Harmonized Sales Tax move. This means for those organizations tracking tax in their internal systems, they must make changes to their databases and systems to incorporate the HST and alter their PST and GST taxation collection and tracking in Ontario. This is a nice example of an externally forced upon change. This particular impact will impact both Operational and Decision support systems.
Personal Experience:
The data files were coming in just fine from a source system that was considered stable (i.e., no data issues from them since project delivery).
Then one day, the volume dropped by more then 70% on file feeds received daily.
After some investigation the source system (System B) identified that their volumes had changed as well. They did not even know their data volume had decreased. The investigation was escalated to their source (system A).Who identified that all the records where being sent to system B. There were no change to System A, “more on that shortly”. Back to System B, they do not have the data. Open the source files and their the data was, the records that were missing were in the source files with blanks in the identifying fields.
A scheduled software (note scheduled) upgrade on System A, could process French characters and inserted blanks in the initial fields and subsequent ID fields. So when System B arrived to pick up the record IDs it found nothing to insert.
A simple software upgrade that resulted in wasted time, money and missing data.
Ignore the potential for change and you will be left holding an empty bag. Never get comfortable.
Remember to inform all your upstreamers of how their changes may potentially become a critical impact to your environment.
Some of the most common forms of changes in systems are the result of the following items:
Data integration
Mergers and acquisitions
Politics, laws and regulations,
Software changes
Web interfaces (change of portals)
Hardware changes
I’m certain you may be able to add to this short list, and I welcome you to.
If you are someone making change, remember to practice proper Change Management technics, a topic for future discussions.
1…
2…
10…
13…
76…
Never…?
How’s this for a reply…no environment is stable! PERIOD.
Each and every data warehouse environment is subject to change, subject to growth, subject to budget constraints, and other external conditions (i.e., political changes). Their will always be change, THAT you cannot control.
Remember SOx. One recent example in Ontario is the Harmonized Sales Tax move. This means for those organizations tracking tax in their internal systems, they must make changes to their databases and systems to incorporate the HST and alter their PST and GST taxation collection and tracking in Ontario. This is a nice example of an externally forced upon change. This particular impact will impact both Operational and Decision support systems.
Personal Experience:
The data files were coming in just fine from a source system that was considered stable (i.e., no data issues from them since project delivery).
Then one day, the volume dropped by more then 70% on file feeds received daily.
After some investigation the source system (System B) identified that their volumes had changed as well. They did not even know their data volume had decreased. The investigation was escalated to their source (system A).Who identified that all the records where being sent to system B. There were no change to System A, “more on that shortly”. Back to System B, they do not have the data. Open the source files and their the data was, the records that were missing were in the source files with blanks in the identifying fields.
A scheduled software (note scheduled) upgrade on System A, could process French characters and inserted blanks in the initial fields and subsequent ID fields. So when System B arrived to pick up the record IDs it found nothing to insert.
A simple software upgrade that resulted in wasted time, money and missing data.
Ignore the potential for change and you will be left holding an empty bag. Never get comfortable.
Remember to inform all your upstreamers of how their changes may potentially become a critical impact to your environment.
Some of the most common forms of changes in systems are the result of the following items:
Data integration
Mergers and acquisitions
Politics, laws and regulations,
Software changes
Web interfaces (change of portals)
Hardware changes
I’m certain you may be able to add to this short list, and I welcome you to.
If you are someone making change, remember to practice proper Change Management technics, a topic for future discussions.
Monday, May 25, 2009
DQ is 1/3 Process Knowledge + 1/3 Business Knowledge + 1/3 Intuition
Sitting in any data quality competency centre, or working on any data warehousing support team, there are times when you come across data that just screams at you, and not in a good way. (Kind of like the ghost from the Grudge.) The data's gone through all data processing without a hitch, business says it's good data, but something tells you it's just not right. That my friends is called intuition.
My recommendation would be to use it.
Intuition does not come naturally, it comes to you over time. After you've gained an understanding of the processes and learned about the business, intuition will set you above the other data analysts.
Business Knowledge
Know the business, I've mentioned this before in my post about Data Quality Analyst Attributes.
What do you do when you have 23,000 service cancellations on one order. This would be red flagged immediately in most service oriented companies. It might even cause a job to stop processing.
The process tells you there's a problem. The business would rightfully tell you to investigate, it can't be right. However, someone in the business knows what the truth is. Check the data, you just may find the indicator to identify the type of customer it is, and easily learn that the business is justified in cancelling 23,000 service items. Perhaps the business lost a call centre customer.
Process Knowledge
Processes will establish checks and validation points that tell you when the data is good and bad. It will tell you where the data is and what happens to it at specific system touch points and more. You must know this in order to understand the process and the data that goes through the processes. Having a good understanding of the processes will allow you to identify where and when errors could occur with your data sets. Process knowledge will guide you to where you need to correct the data or processes.
Intuition
Your knowledge, your understanding of the data will guide you and your intuitive feeling to determine what is right. There is no substitute for it. Intuition will give you that sixth sense and you will be able to differentiate true data issues from false data issues when the tools and processes set in place cannot make that differentiation. Use it when the tools just don't cut it.
Example:
The process says the CRM application must match the records, the business believes the records should be matched and even want them to match. You know that even if John Smith, and John Smith living in the same city aren't necessarily the same person, two positives matches does not necessarily make a single profile.
My recommendation would be to use it.
Intuition does not come naturally, it comes to you over time. After you've gained an understanding of the processes and learned about the business, intuition will set you above the other data analysts.
Business Knowledge
Know the business, I've mentioned this before in my post about Data Quality Analyst Attributes.
What do you do when you have 23,000 service cancellations on one order. This would be red flagged immediately in most service oriented companies. It might even cause a job to stop processing.
The process tells you there's a problem. The business would rightfully tell you to investigate, it can't be right. However, someone in the business knows what the truth is. Check the data, you just may find the indicator to identify the type of customer it is, and easily learn that the business is justified in cancelling 23,000 service items. Perhaps the business lost a call centre customer.
Process Knowledge
Processes will establish checks and validation points that tell you when the data is good and bad. It will tell you where the data is and what happens to it at specific system touch points and more. You must know this in order to understand the process and the data that goes through the processes. Having a good understanding of the processes will allow you to identify where and when errors could occur with your data sets. Process knowledge will guide you to where you need to correct the data or processes.
Intuition
Your knowledge, your understanding of the data will guide you and your intuitive feeling to determine what is right. There is no substitute for it. Intuition will give you that sixth sense and you will be able to differentiate true data issues from false data issues when the tools and processes set in place cannot make that differentiation. Use it when the tools just don't cut it.
Example:
The process says the CRM application must match the records, the business believes the records should be matched and even want them to match. You know that even if John Smith, and John Smith living in the same city aren't necessarily the same person, two positives matches does not necessarily make a single profile.
Monday, April 20, 2009
When Bad Data Becomes Acceptable Data
Sometimes you need to weigh the options and determine if all your effort to fix bad data is actually worth it.
Yes, bad data costs the company money, added expenses and so much more.
Yes bad data may critically impact decision making at your organization.
Yes, it will take effort to get rid of bad data.
However, when you start your efforts to clean the bad data think of the following decisions you must make when you take the mountainous task to scrub the data clean.
1) Is the ROI worth it?
a. If you have two bad records that are wrong, do you spent the same amount of time working on a $3 error vs. a $30,000 error. I think it’s obvious, but some of us will burn the midnight oil to get ride of the $3 error just as vehemently as the $30,000 error.
b. BUT remember any error that makes it to a customer’s bill will become a big topic for the customer. Your evil over-billing practices may become the twitter topic of the day. And nobody wants that now do they!
2) How long before the error disappears?
a. With archiving and deletion jobs, that record may be gone before you know it. Saving you some valuable time to concentrate on more pressing matters.
3) What’s the impact of keeping it?
a. Yes, what is the impact of keeping that bad, bad data. BUT ask yourself who is using the data and what type of decision is being made with it?
4) Can you communicate the bad data to the users?
a. This may sound strange to you, but if you have to get the data out and you know the exact problem, duplicate records, over-billing, etc., etc. then make the knowledge workers aware of what they are looking at and reporting.
b. Sometimes just letting the knowledge workers in on the problem will prevent a disastrous decision from being made. They can identify the situation in the foot notes, or remove it all together from their reports and inform the decision maker that the issue exists.
On a personal note, I say clean it, scrub it, polish it, make the data shine, like it has never shined before. BUT, we don’t live in an ideal world and you may have to keep the rotten data there. It may be primarily for budgetary reasons or the size of the impact may be very minimal. So when this is the case ask yourself what’s the impact, what’s the ROI, how long will it be there and can you communicate the situation to the users. When this happens the data is still bad data but becomes acceptable data.
Leaving bad data in there may be viewed as the lazy-man's solution, but when you're swamped with 13 other data quality issues to tackle you need some methodology to identify the key situations and those that can be pushed aside.
Yes, bad data costs the company money, added expenses and so much more.
Yes bad data may critically impact decision making at your organization.
Yes, it will take effort to get rid of bad data.
However, when you start your efforts to clean the bad data think of the following decisions you must make when you take the mountainous task to scrub the data clean.
1) Is the ROI worth it?
a. If you have two bad records that are wrong, do you spent the same amount of time working on a $3 error vs. a $30,000 error. I think it’s obvious, but some of us will burn the midnight oil to get ride of the $3 error just as vehemently as the $30,000 error.
b. BUT remember any error that makes it to a customer’s bill will become a big topic for the customer. Your evil over-billing practices may become the twitter topic of the day. And nobody wants that now do they!
2) How long before the error disappears?
a. With archiving and deletion jobs, that record may be gone before you know it. Saving you some valuable time to concentrate on more pressing matters.
3) What’s the impact of keeping it?
a. Yes, what is the impact of keeping that bad, bad data. BUT ask yourself who is using the data and what type of decision is being made with it?
4) Can you communicate the bad data to the users?
a. This may sound strange to you, but if you have to get the data out and you know the exact problem, duplicate records, over-billing, etc., etc. then make the knowledge workers aware of what they are looking at and reporting.
b. Sometimes just letting the knowledge workers in on the problem will prevent a disastrous decision from being made. They can identify the situation in the foot notes, or remove it all together from their reports and inform the decision maker that the issue exists.
On a personal note, I say clean it, scrub it, polish it, make the data shine, like it has never shined before. BUT, we don’t live in an ideal world and you may have to keep the rotten data there. It may be primarily for budgetary reasons or the size of the impact may be very minimal. So when this is the case ask yourself what’s the impact, what’s the ROI, how long will it be there and can you communicate the situation to the users. When this happens the data is still bad data but becomes acceptable data.
Leaving bad data in there may be viewed as the lazy-man's solution, but when you're swamped with 13 other data quality issues to tackle you need some methodology to identify the key situations and those that can be pushed aside.
Tuesday, April 7, 2009
DQ Problems? Start a Data Quality Recognition Program!
Having spent some time in an Enterprise Data Warehouse Competency Centre, one item I noticed was that we received a lot of input files, and we’re talking about hundreds of files a day coming from dozens upon dozens of different sources.
Needless to say we had over 80 source systems, legacy systems, web based systems and a few PC files as well. Each of those systems was responsible for sending us anywhere between 1 and 60 files per week. So needless to say there were lots of data related issues that came across my and my colleagues’ desks on a daily basis.
Is your EDW similar in nature? Do you have overworked data quality analysts?
Do you have source systems that just don’t care about the contents of the files they send you?
Remember some of the goals of an Enterprise Data Warehouse is to create one version of the truth, to reduce redundancy, to streamline the decision making process and to reduce redundancy and improve data quality.
Apart from implementing data profiling tools, performing data assessments, monitoring thresholds, correcting code and asking for executive support for data quality in general. What else can you do?
If your organization is large enough and during these trying times, the budget permits it, and the organizational culture accepts it introduce a Data Quality Recognition Program.
What, a data quality recognition program, you said? Yes, I said, a data quality recognition program. The following list contains a few steps you will need to establish such a program:
1. Executive buy-in, (primarily needed for budget approval).
2. Advertise the program to all your source systems.
3. A means to track data quality by source system.
4. Identify qualifying systems (those with an owner).
5. Track the data issues and identify the source systems.
6. Compile the results annually.
7. Hand out the award or awards.
8. Advertise the winner(s) and their results.
You can also establish multiple awards, such as most improved data quality, best data quality, most timely data, most accurate data entry clerk, call centre with the best data quality or even a DQ troll award (for the worse data, if they have a sense of humor) and more. Really the type of awards and the number of awards is entirely up to you and of course the budget.
Make sure your award winners have a great day. Provide the team responsible or owner of the source system with a tangible award, or a plaque. It makes for an interesting conversation piece and it gives them bragging rights, especially on the resume.
Once your program is established and awards handed out, watch the data quality of your EDW improve over time, as everyone does their best to be named in the next award ceremony.
Monday, March 30, 2009
Data Quality: A Cause and Effect Story
While reading a classic story to my kids, "Because a bug went ka-choo!" I thought to myself that some CEO's who just don't get data quality might learn a little cause and effect along the same lines as this great story. So here's a lesson about the importance of bugs in your code and the data quality issues it may cause and the effects that pour forth...enjoy!
Because of that chat, the coworker made an anti-company website,
Because of that website people shared similar bad service stories.
Because of those stories, the competition smiled.
Because the competition smiled, they decided to entice.
Because they were enticed,
The customers started to leave.
Because they started to leave, sales came down, and it hit CEO Brown.
And that drop got stuck in his head. Because it got stuck, CEO Brown phoned for help.
Because of his call, the CFO began counting.
Because the CFO was counting, they noticed the shares.
And so the execs sold those shares all alone.
Because they sold those shares, the share price came down.
Because it came down, employees were let go.
Because they were let go, they decided to sue.
Because of that suit the shares price started to sink,
Because it started to sink the executives were investigated for insider trading.
Because they were investigated, they headed to court,
And you may not believe it, it’s true I’m afraid they ran right into executive protest parade.
And that started something they’ll never forget, and as far as I know it is going on yet.
And that’s how it happened, believe me it’s true, because, just because a little bug got scripted.
You may not believe it,
But here’s how it happened.
One long business day,
A little bug was scripted.
But here’s how it happened.
One long business day,
A little bug was scripted.
And because of that script,
A character was dropped.
Because that character dropped
A record got lost.
Because the record was lost,
Provisioning was not informed.
Because they weren’t informed
No service was issued.
Because of no service,
that customer got mad
because he got mad.
He talked to his neighbour,
Because he talked to his neighbour,
A character was dropped.
Because that character dropped
A record got lost.
Because the record was lost,
Provisioning was not informed.
Because they weren’t informed
No service was issued.
Because of no service,
that customer got mad
because he got mad.
He talked to his neighbour,
Because he talked to his neighbour,
the neighbour named Paul chatted to his coworker.
Because of that chat, the coworker made an anti-company website,
Because of that website people shared similar bad service stories.
Because of those stories, the competition smiled.
Because the competition smiled, they decided to entice.
Because they were enticed,
The customers started to leave.
Because they started to leave, sales came down, and it hit CEO Brown.
And that drop got stuck in his head. Because it got stuck, CEO Brown phoned for help.
Because of his call, the CFO began counting.
Because the CFO was counting, they noticed the shares.
And so the execs sold those shares all alone.
Because they sold those shares, the share price came down.
Because it came down, employees were let go.
Because they were let go, they decided to sue.
Because of that suit the shares price started to sink,
Because it started to sink the executives were investigated for insider trading.
Because they were investigated, they headed to court,
And you may not believe it, it’s true I’m afraid they ran right into executive protest parade.
And that started something they’ll never forget, and as far as I know it is going on yet.
And that’s how it happened, believe me it’s true, because, just because a little bug got scripted.
Thursday, March 26, 2009
Put Data Quality in Those Requirements, Already!
Business Analysts look at all kinds of requirements.
1. Networking interface;
2. System design;
3. Database design;
4. Business needs;
5. and, much more...
How often will a business analyst look at data quality requirements?It is a missing. Period!
Rarely will you see a heading in your Business and System Requirements documentation for Data Quality. The business has such a hard time understanding what Data Quality actually means that they ignore the subject. This is not the fault of the business analyst, but often an oversight from the beginning of project conception. Quite often the reason for the oversight is the business' inability to quantify the benefits of data quality and the actions needed to prevent bad data.
I don't want to harp on the business, but lets face it, it is the rare mandate in any IT project to ensure data quality, or even to provide data quality action items. It is often left to the support analysts at the end of the day to step in and engage the business to address these issues. Unless, the IT project has a data steward to engage, then it is very likely that data quality will become an important issue.
Not to far ago in the past, I did a search on a well known career search engine to find out how often data quality is seen as a requirement or function of the Business Analyst role. My results were disappointing. Only 19 positions with the term Business Analyst in the position name held the term 'data quality' in its' content. That translates into less than 5% of the postings that contained the term business analyst. Now I didn't go and look for data integrity, or MDM. It is what it is, a search done out of curiousity with not too much in depth data mining...eh, I had kids to put to bed!
Out of curiousty, closer to the day, a free form search on 'data quality' and 'business analyst'. My results: 25% of business analyst type postings contained data quality in the body of their content.
Needless to say data quality is still a bit off the radar for many companies when it comes to defining the role of a business analyst.
With that said the aspect of including data quality in the role of a business analyst is there, but to what degree remains relatively unanswered. So to those business analysts and business analysts want to be's...remember Data Quality in your requirements, it's just good business.
Data Quality must be inseparable from the data. Good data quality at the requirements stage will positively impact:
1.Project success;
2. Data Integrity;
3. Efficient and effective support;
4. Sound decision making;
5. reduce business costs from errors and corrective actions; and
6. project longevity (something any good BA wants).
So if the business is not interested in Data Quality, make it your interest. It won't hurt, and the business will love you for it!
Wednesday, March 4, 2009
Five Attributes for the Data Quality Analyst
In my years working in data quality I have seen many skills and attributes of what a good data quality analyst must have under their belt.
As a result I believe a data quality analyst must...
Be proactive: Many times the media catches a snip-it of some disastrous event which was the result of bad data quality. Or the masses that are your company's customers have congregated on Facebook to strike down the corporate beast you work for, because of bad billing or bad customer service.
One must ask...why is there bad billing or bad customer service? One fundamental reason is the fact that the data is not quality data. Ultimately, if you bill someone more then they should be billed, you will have a bad reputation and be considered to have poor customer service. It's all about perception. So what to you do to prevent such things. Be proactive, don't sit back and wait for someone to say "I think the order provisioning system has bad data?" or "We billed the Smith family $3000.00 when we should have billed them $30.00!" You don't want those statements to come to you. You want to be proactive and look for the bad data, you want to trend the data so when it finally breaks the trend you know you have something worth looking into. You want to look at the ETL that project ABC is bringing into your data warehouse. You want to review the data dictionary and ensure that all the "t's" are crossed and "i's" dotted. You want to establish quality checks up front, before the data is loaded. You want to be proactive.
Be relentless: When a data issue emerges tackle it. Be ruthless, be relentless, when the developers say they don't know the business and the business says they don't know the script or code. Bring the two together and resolve the issue. They'll be a time when someone passes the buck or tries to brush you off. Don't take no for an answer. Escalate if you want to get answers. Be relentless.
Be technically savvy: Knowing basic SQL in a data warehouse environment is worth your weight in gold. At the least you must know a little SQL so that you can look at the data in different patterns and omit specific values to perform a better analysis of the data to ensure it's quality. You do not want to rely entirely on a predefined report that someone created before your arrival. Data is always changing and you must be able to adapt and change with it. You must have some basic SQL skills and be technically savvy. Eventually your expertise will increase.
Be personable: You might ask why if I worked with data do I have to be personable? Let's face it. You are a data quality analyst, if you are questioning the data coming into the data warehouse from the orders department, do you think they want to correct the issue for you. No they want you to fix the problem. So be personable! Scratch a few backs and someone will scratch yours. It's all about being a team, no matter how large of an organization you are working in, you are all part of the same team, with the same goal and that is to succeed.
Know the business: A data quality analyst who isn't business oriented would be someone who really reads reports and says, "Ah! Bad data again! We'd better fix it." An analytical data quality expert would be intimate with the data. They can look at the trends, outliers and more and have an understanding of what the data is saying about the business. So when there's a quality threshold spike on a data object, you immediately know it's related to what the business is doing. This will save you investigation time and money.
As a result I believe a data quality analyst must...
Be proactive: Many times the media catches a snip-it of some disastrous event which was the result of bad data quality. Or the masses that are your company's customers have congregated on Facebook to strike down the corporate beast you work for, because of bad billing or bad customer service.
One must ask...why is there bad billing or bad customer service? One fundamental reason is the fact that the data is not quality data. Ultimately, if you bill someone more then they should be billed, you will have a bad reputation and be considered to have poor customer service. It's all about perception. So what to you do to prevent such things. Be proactive, don't sit back and wait for someone to say "I think the order provisioning system has bad data?" or "We billed the Smith family $3000.00 when we should have billed them $30.00!" You don't want those statements to come to you. You want to be proactive and look for the bad data, you want to trend the data so when it finally breaks the trend you know you have something worth looking into. You want to look at the ETL that project ABC is bringing into your data warehouse. You want to review the data dictionary and ensure that all the "t's" are crossed and "i's" dotted. You want to establish quality checks up front, before the data is loaded. You want to be proactive.
Be relentless: When a data issue emerges tackle it. Be ruthless, be relentless, when the developers say they don't know the business and the business says they don't know the script or code. Bring the two together and resolve the issue. They'll be a time when someone passes the buck or tries to brush you off. Don't take no for an answer. Escalate if you want to get answers. Be relentless.
Be technically savvy: Knowing basic SQL in a data warehouse environment is worth your weight in gold. At the least you must know a little SQL so that you can look at the data in different patterns and omit specific values to perform a better analysis of the data to ensure it's quality. You do not want to rely entirely on a predefined report that someone created before your arrival. Data is always changing and you must be able to adapt and change with it. You must have some basic SQL skills and be technically savvy. Eventually your expertise will increase.
Be personable: You might ask why if I worked with data do I have to be personable? Let's face it. You are a data quality analyst, if you are questioning the data coming into the data warehouse from the orders department, do you think they want to correct the issue for you. No they want you to fix the problem. So be personable! Scratch a few backs and someone will scratch yours. It's all about being a team, no matter how large of an organization you are working in, you are all part of the same team, with the same goal and that is to succeed.
Know the business: A data quality analyst who isn't business oriented would be someone who really reads reports and says, "Ah! Bad data again! We'd better fix it." An analytical data quality expert would be intimate with the data. They can look at the trends, outliers and more and have an understanding of what the data is saying about the business. So when there's a quality threshold spike on a data object, you immediately know it's related to what the business is doing. This will save you investigation time and money.
Subscribe to:
Posts (Atom)
