Data Modelling
At Kleene, we use the Ingest-Master-Insights (IMI) framework when modelling data. The point of the framework is to be simple and allow the analyst/engineer to always keep in mind the overarching stages of the data modelling process.
The Ingest-Master-Insights (IMI) framework, with the discovery of data taking place before it. The process is to be read from left to right, where the data itself is put through a discovery phase to determine the different data sources, the metrics and dimensions required from the data and how to later master that data. The IMI framework then follows the discovery phase, where data is ingested into the data warehouse, mastered and then the insights layer is created, where any business intelligence tool can be plugged in to create dashboards.
Below is a summary of the many modelling steps used at Kleene, which can be navigated via the left-hand side menu or clicking on the links at the bottom of this page.
| Chapter | Modelling Step | Substeps |
|---|---|---|
| 1 | Raw Data | Where raw data comes from |
| How Kleene handles the raw data | ||
| Naming convention | ||
| 2 | Cleaned | Cleaning field names |
| Data type casting | ||
| Parsing JSON | ||
| Naming convention | ||
| 3 | Subentity | Beginning of true transformation |
| Flattening arrays | ||
| Naming convention | ||
| 4 | Mastering | What the mastering step is |
| Naming convention | ||
| 5 | Entity | What an entity is |
| Renaming columns | ||
| Naming convention | ||
| 6 | Domain | What a domain is |
| Domains on the DAG | ||
| Naming convention | ||
| 7 | Reporting | What a reporting table is |
| Naming convention | ||
| 8 | Views | Views vs tables |
The journey from ingest to insight involves multiple steps of transformation in the data modelling process. But it all begins with handling raw data.
Updated about 1 year ago