TwitterLinkedin
GetMapping@TheMapWorks.com
The MapWorks The MapWorks The MapWorks The MapWorks
  • Home
  • What MapWorks Does
    • Business Glossary
    • Importing your Metadata
    • Industry Solutions
  • Pricing
  • Learn MapWorks
    • FAQ’s & Support Request
    • Sample Project
    • Importing your Metadata
  • Blog
  • Request a Demo
  • Sign in
  • FREE Trial
  • Home
  • What MapWorks Does
    • Business Glossary
    • Importing your Metadata
    • Industry Solutions
  • Pricing
  • Learn MapWorks
    • FAQ’s & Support Request
    • Sample Project
    • Importing your Metadata
  • Blog
  • Request a Demo
  • Sign in
  • FREE Trial
  • metadata-maturity-curve
    Permalink
    Gallery

    Mind the Gap – Metadata Maturity in Organizations

    Data Quality, MapWorks, Metadata, Spreadsheets, Tools

Mind the Gap – Metadata Maturity in Organizations

When it comes to metadata management, organizations progress along a standard path, driven by the individual organization’s specific needs and how much it values metadata.  As an organization matures, it applies a higher priority and more resources to extracting value from metadata in order to make the organization more agile and to improve its data and its processes.

Metadata Maturity Curve

In the “Infant” stage, a project team (or a department, or even the entire organization) recognizes that documenting metadata is important.  Fields in a system need to be defined before they can be created, source-to-target mappings have to be developed and communicated to developers and QA, and appropriate approvers need to sign-off on everything before it is deployed.  Unless the project team has robust tools handy, it resorts to using spreadsheets or a collection of isolated tools for maintaining their metadata during the life of the project.

Business requirements (with their definitions) are written in a Word doc or an Excel workbook.  Data models are developed in modeling tools with the DDL output directly applied to the databases.  Data analysts maintain the business rules and mapping logic in another spreadsheet and QA develops the test cases in its own testing software.  Teams are constantly passing spreadsheets around by email and using VLOOKUP’s to link everything together.

When the email and VLOOKUP’s get unmanageable, someone suggests that “Collaboration” is what’s needed, so the organization moves to the “Child” stage where it keeps its Word documents and Excel workbooks in SharePoint (or some other collaboration software).  The PMO develops standard templates and policies for how SharePoint and the documentation should be maintained.  This stage is definitely an improvement over the email headaches of the “Infant” stage and collaboration and standardization make projects more efficient and less risky.

At the “Child” stage, metadata documentation is usually focused on project execution.  End users typically don’t have access to the information and the metadata isn’t readily available for data quality or data governance initiatives.

Moving from the “Child” stage to the “Adolescent” stage is where things get really interesting.  There’s a “Metadata Gap” that the organization has to cross.  In order to start generating value from metadata, the metadata developed during the project lifecycle must be (re-)used in different ways and with different tools.  Crossing the “Metadata Gap” requires acquiring and utilizing new tools and developing the resources and processes to make them effective.  It requires time, effort, and prioritization.

Once an organization realizes that prioritizing metadata improves data quality and makes users more effective, it has to make choices about exactly how cross the gap.  How much should it spend?  What new tools should it purchase?  How many people should be devoted to the effort?  What disruption will it cause to the current initiatives?  Is the improvement worth the cost?

Once the organization has crossed the gap by allocating time and resources to metadata management, it has entered the “Adolescent” phase.  In this phase, metadata across the organization is being managed centrally.  It’s accessible to a wide audience of both technical and business users.  A business glossary is established that provides the common vocabulary that everyone can speak.  Reference data and valid values are developed for key data elements.  Data lineage is established and the organization now has tools to see the impact of upstream and downstream changes.  History is maintained to provide a rear-view look while new or changing business rules are folded into the process.

As the processes and tools mature and become operationalized, the organization is ready to enter the “Adult” phase of metadata management.  In this phase, data governance initiatives begin to establish standards and prioritize efforts to improve data quality.  With the results of these efforts, the business makes better decisions and customers have a better experience as a result of how the organization manages data.

And they all lived happily ever after . . .

If only it were that easy.

It’s not.  But at MapWorks, we’re providing the tools that make it easier.  To many organizations, especially smaller or newer companies with limited resources, the gap between the “Child” and “Adolescent” phases is very difficult to cross.  The value of robust metadata management and data governance is often out of reach because of the cost (time, budget, etc.) of moving away from spreadsheets to something better.

MapWorks makes that gap a lot easier to cross.  Our software gives companies the benefits that more expensive solutions provide, but at a fraction of the cost and in a fraction of the time.  We’re cloud-based – no hardware to purchase, no software to install.  Just sign up and get started.  Right out of the gate, our users get:

  • Business Glossary –  Align your organization to a common vocabulary and definitions
  • Business and Technical Metadata –  Manage business names, technical names, formats, business rules and mapping logic.
  • Data Lineage and Impact Analysis – Quickly see upstream and downstream impacts for every data element.
  • Reference Data Management – Maintain your LOV’s in a single place.
  • Import from External Sources – Import metadata from your existing databases and tools and avoid manual data entry.
  • Works for Small Teams and Large Organizations – Collaborate with users across the company.
  • Successful Data Governance – MapWorks gives your team the tools to get your data governance program moving.

Try MapWorks risk-free for 30 days and you’ll see what it can do for you.

 

  • mapworks-import-metadata
    Permalink
    Gallery

    Automating your Metadata Catalog with MapWorks

    MapWorks, Metadata, Simple, Tools

Automating your Metadata Catalog with MapWorks

Since we launched MapWorks (and even earlier as part of our data consulting practice) we’ve talked to many people who are searching high and low for a cost-effective answer their metadata chaos.  Spreadsheets and quick/dirty “solutions” to data mapping and lineage aren’t solutions when they can’t be maintained. When we ask about the current state of affairs we get answers ranging from “Yeah, we should really do that…we don’t know where anything is!” to “We tried that a while ago . . . and it’s not going very well.” and everything in between. After the weeping and cursing subsides, we start talking about how they can get a handle on their metadata.

One of the first questions we get is about how to get existing metadata into a tool like MapWorks.  Most data shops already have stockpiles of metadata – spreadsheets, a tool for the business users, another tool for the technical teams. There are usually dozens of systems, each with potentially hundreds of data points and it’s a big task to get all of that into a new tool. Manually entering all that information into a new metadata repository is a costly and time consuming project, not to mention the time it will take to periodically reconcile what currently exists in your organization’s databases to what has been documented in your metadata repository. We’ve been there, we know that pain and that’s why we’ve created a simple, three-step metadata import processes that will make it easier. What takes hours or days takes now takes minutes with some simple steps.

Three simple steps for importing your existing metadata into MapWorks:

  1. Download the template spreadsheet from the Export/Import menu in the MapWorks Mapping module
  2. Fill in the information about your data structures
  3. Upload the spreadsheet to MapWorks

To assist you with getting the metadata for the spreadsheet, we created a series of SQL queries that you can run to automatically extract the metadata directly from your databases. Just copy the results of the query and paste them into the spreadsheet and voilà!

You can access the SQL scripts from this link.

These scripts will need to be run using an account/login that has access to the necessary metadata tables or views for the desired databases. This will vary according to the database being used. If you run into issues or are looking for similar queries for other databases, please drop us a line at answers@themapworks.com and we will see what we can do.

Also, note that these scripts will extract the data used by MapWorks starting with the 05/23/2017 MapWorks release. If a new metadata format is added to MapWorks (i.e. new columns added to the template spreadsheet), we’ll keep the scripts updated.

Avoiding Duplicates

Not sure if something has already been imported into MapWorks? No problem. When you import the information, MapWorks will compare what you are importing to what already exists within MapWorks and allow you to either import new items or update existing items in MapWorks. This is very useful for a periodic reconciliation of what exists in your organization to what is documented in MapWorks. Just pull the metadata from your organization’s databases and upload it to MapWorks. The differences will be highlighted for you.

For ongoing maintenance, we recommend that you use MapWorks as a tool to capture any upcoming changes to your metadata and mappings prior to them being implemented. You can then use the MapWorks Releases module  to track when those changes are implemented and keep a historical record of the changes that have occurred over time. By doing this, you will be proactively updating your metadata repository in MapWorks and it will minimize the need to reconcile the changes after the fact.

  • louvre
    Permalink
    Gallery

    Night at the Museum – The Metadata Curator

    Data Quality, MapWorks, Metadata, Tools

Night at the Museum – The Metadata Curator

Not too many years ago I was in Paris on business.  As is not uncommon in France, there was a strike in the city while I was there and my meetings were cancelled.  So, what’s a guy to do in the City of Lights during the daytime?  Yep, headed over to the Louvre.

Great place, that.  It was thrilling to see artifacts from 4,000 BC.  And the art!  I hadn’t realized how large some of the masterpieces were – that just made them more grand.  The only disappointing thing about the museum was The Mona Lisa.  She was perched high on the wall behind a thick sheet of protective glass . . . You’ll get a better sense of Leonardo’s masterpiece from photos on the internet, but I digress . . .

Did you know that the Louvre has over 380,000 items in its collection?  Only about 35,000 are displayed at any one time.

Okay, okay . . . I’ll get to the point already.  What does the Louvre’s collection have to do with metadata management?  Plenty.  And it all comes down to curation.

Someone (or in the case of the Louvre, a team) has to be responsible for picking through more than 380,000 artifacts and works of art to display something that can be consumed by museum goers in a reasonable amount of time.  These curators pick the finest, the best, the most interesting, the most beautiful and organizes them to elevate and enlighten every person who walks down the parquet hallways.

When it comes to metadata management, someone (or some dedicated team) has to elevate and educate every user who searches for a data element or who is trying to figure out what product code 3347 really means.  The metadata curator doesn’t pare down the collection, but he or she does keep it organized and up-to-date and accessible and useful to everyone who searches it.

In complex data ecosystems, new data elements are coming in all the time.  Lists of values need constant updating.  Source systems are upgraded and replaced.  New reports and dashboards are built daily.  As business requirements evolve, data mappings need have to keep pace.  Lots of people are involved in these processes – architects, modelers, developers, analysts, testers, report writers, data scientists, and end-users.  But besides the data itself, what’s the one thing that’s constant through that whole chain?  The metadata and the curators who keep it clean and useful.

What makes a good metadata curator?

  • Dedicated time to do the curation – maybe it’s part-time, maybe it’s full-time, but the job needs to be part of someone’s official job description
  • Data knowledge – understanding data structures and formats, data models, SQL, and how the data gets used by everyone in the value chain.  Besides technical skills, needs to have a good grasp of the principles of master data management and data governance.
  • Business knowledge – if you’re in healthcare, you’d better know what SNOMED and medDRA mean.  Banking?  1C2A had better mean something to you.
  • Excellent written communicator – Metadata management and data mapping tools are all about presenting written information to others.  Someone’s got to make sure that it’s formatted, written in a clean, standardized way.
  • Self-motivated, works independently – documentation isn’t usually anyone’s favorite pastime, but someone has to knock this out of the park.
  • Great collaboration and teamwork skills – will be working with and across several teams (often at the same time) to make sure the metadata strategy is effectively carried out by each of these teams.  Needs to help disparate groups come to consensus (try getting bankers to agree on the definition of a “loan”)
  • Anal retentiveness is a good thing to have – incomplete lists of values or missing definitions should drive the curator into orbit.

The metadata curator isn’t just the administrator of the metadata management software.  She’s the librarian that knows how to find everything.  He’s the collector that hunts facts about every system and field he can find.  He’s the editor, constantly clarifying and making things more readable.  She’s the steward who knows that the only thing worse than not having metadata is having incorrect metadata. 

  • Albert Einstein
    Permalink
    Gallery

    As Simple as Possible, but not Simpler

    MapWorks, Simple, Tools

As Simple as Possible, but not Simpler

Whether we’re paraphrasing Albert Einstein or restating Occam’s razor, the idea of simplicity resonates.  We’ve all been in those environments where everything was more complicated than it needed to be.  And those systems that morph into Rube Goldberg machines drive us all crazy.   A few bonus features here, add a few political considerations there, and sprinkle in some bells and whistles just because we can; and we end up with something bigger than we need and more costly than we can afford.

Metadata and data mapping tools are no exception to this routine.  We’ve been building data warehouse and reporting systems for 20 years and have seen both ends of the spectrum.  At some clients, it’s too simple.  “Just throw it into Excel and figure out how to share it with the team.”  And when that doesn’t work it becomes “Can someone knock out an Access database this week and get everyone to put their mappings in it?”  Trying to short-cut things causes rework and more headaches for everyone.

Or . . .

Everyone agrees that spreadsheets aren’t the way to go and we launch into a months-long requirements process to document every conceivable need the organization has.  Business glossary, business and technical names, mapping and lineage are no-brainers but then someone wants to make sure it is fully integrated with the data warehouse (never mind the endless meetings just to figure out what “fully integrated” means to everyone).  The project managers want it to have configurable workflows and the development team wants it to generate ETL code.

It takes 3 months just to run the RFP and the vendor bake-off and then you still have to install the hardware, configure and customize the software, test all of the integrations, and put the entire organization through a week of onsite training.

Before you know it, you’re a year into the effort and have spent a hefty war chest before you’ve mapped a single field.

We aim to be the Anti-Rube-Goldberg:  Provide a simple tool that does the basics of metadata management and data mapping really, really well.  Simple, but not too simple.  The alternative to Rube Goldberg is this:

  • Provide a central repository for curating and cataloging definitions, reference values, and mappings
  • Be accessible and usable to the widest audience possible – for technical and business users
  • Provide a historical view of definitions/mappings and easily allow for  future changes (releases, new development, business changes, etc.)
  • Simplify setup, configuration, training, and maintenance
  • Avoid developing features and complexities that limit any of the above

If you have the time and the money, there’s nothing wrong with having robust integration with every tool in your stack.  Generating code is a great feature for your data mapping tool – but it only benefits part of your organization.  Too many nice-to-have features limit what you set out to do in the first place.

Photo Attribution:  This image is available from the United States Library of Congress‘s Prints and Photographs division under the digital ID cph.3b46036.

FOLLOW US AND REACH-OUT

Stay up to date by following us on Twitter

About Us

Fine print

 Terms & Conditions, Policies, etc.
You can always email us answers@themapworks.com

Try MapWorks for free!

Sign-up today! No credit card or obligation required!

Copyright 2017 The MapWorks, Inc. | All Rights Reserved
TwitterLinkedin