Skip to content

Creating a Metadata Architecture From the Ground Up

Metadata

  • Author: Marcelo Alves Baratela
  • Full Title: Creating a Metadata Architecture From the Ground Up
  • Category: #Type/Highlight/Article
  • URL: https://medium.com/p/ff05707ce630

Highlights

  • Creation of data silos, lack of proper documentation, diverging metrics about the same topic, inadequate access privileges to sensitive data. Unless all the challenges that a company might face are crystal clear right from the beginning (which is very unlikely to happen in real life), it is natural that the items on this list start to pop up as the business grows. Natural but still undesirable. That is the reason data governance is such a relevant topic in current times and why QuintoAndar is looking at it with special care
  • Data Quality Management: We must define our data quality processes, as well as its architecture and the necessary tools to implement it;
  • Data Lake Management: It is our responsibility to take care of the governance on how we store and model our data. We define who should deliver each layer’s data, as well as its owners and users;
  • BI Tools Management: We are responsible for maintaining and managing our BI and data visualization tools, as well as their data structures;
  • Reference & Master Data Management: We must define the source of truth for our most important KPIs, so everybody uses the same metrics. We also define the classification of our tables (i.e. gold, silver, bronze, etc).
  • Metadata, Document, Record & Content Management: This pillar focuses mainly on the documentation of our data assets, the creation of data lineages, and the evangelization of other teams about data governance.
  • Having our pillars and priorities well defined, the starting point for this new team was to choose a governance platform that would facilitate the creation of data catalogs, in addition to allowing better metadata management. This was a crucial step because the choice made at that point would impact every other project in the future.