Fun fact: working with big data is relatively easy (once it’s all setup), the very difficult thing is working with small data. Or worse with non-existing data.
Process — Project Every Tiny Detail
The process is the most important step. You will define what, where and how data are collected, transformed and loaded. Though we hear hearing everyday of AI and its endless possibilities there is still at least one thing they cannot do yet: decide the pipeline process.
This means that you’ll need to manually pick every field, table, data source, transformation, join, etc. The good news is that if you do it right you’ll have to do it just once. Afterwards everything will be automated.