Data Modelling
At Kleene, we use the Ingest-Master-Insights (IMI) framework when modelling data. The point of the framework is to be simple and allow the analyst/engineer to always keep in mind the overarching stages of the data modelling process.
Below is a summary of the many modelling steps used at Kleene, which can be navigated via the left-hand side menu or clicking on the links at the bottom of this page.
Chapter | Modelling Step | Substeps |
---|---|---|
1 | Raw Data | Where raw data comes from |
How Kleene handles the raw data | ||
Naming convention | ||
2 | Cleaned | Cleaning field names |
Data type casting | ||
Parsing JSON | ||
Naming convention | ||
3 | Subentity | Beginning of true transformation |
Flattening arrays | ||
Naming convention | ||
4 | Mastering | What the mastering step is |
Naming convention | ||
5 | Entity | What an entity is |
Renaming columns | ||
Naming convention | ||
6 | Domain | What a domain is |
Domains on the DAG | ||
Naming convention | ||
7 | Reporting | What a reporting table is |
Naming convention | ||
8 | Views | Views vs tables |
The journey from ingest to insight involves multiple steps of transformation in the data modelling process. But it all begins with handling raw data.
Updated 10 months ago