Data Modelling

At Kleene, we use the Ingest-Master-Insights (IMI) framework when modelling data. The point of the framework is to be simple and allow the analyst/engineer to always keep in mind the overarching stages of the data modelling process.


2000

The Ingest-Master-Insights (IMI) framework, with the discovery of data taking place before it. The process is to be read from left to right, where the data itself is put through a discovery phase to determine the different data sources, the metrics and dimensions required from the data and how to later master that data. The IMI framework then follows the discovery phase, where data is ingested into the data warehouse, mastered and then the insights layer is created, where any business intelligence tool can be plugged in to create dashboards.



Below is a summary of the many modelling steps used at Kleene, which can be navigated via the left-hand side menu or clicking on the links at the bottom of this page.

ChapterModelling StepSubsteps
1Raw DataWhere raw data comes from
How Kleene handles the raw data
Naming convention
2CleanedCleaning field names
Data type casting
Parsing JSON
Naming convention
3SubentityBeginning of true transformation
Flattening arrays
Naming convention
4MasteringWhat the mastering step is
Naming convention
5EntityWhat an entity is
Renaming columns
Naming convention
6DomainWhat a domain is
Domains on the DAG
Naming convention
7ReportingWhat a reporting table is
Naming convention
8ViewsViews vs tables

The journey from ingest to insight involves multiple steps of transformation in the data modelling process. But it all begins with handling raw data.