Filestore Options with Kleene

Kleene Managed Warehouse and Filestore

Data Management: Fully managed by Kleene.
Data Retention: Default 7-day retention policy, modifiable on request.
Storage: Over 1TB is chargeable.

Kleene Managed Filestore, Own Warehouse

Data Management: Filestore managed by Kleene, warehouse managed by the customer.
Data Retention: 7-day retention policy, modifiable on request for the filestore.
Storage: Over 1TB is chargeable for the filestore.

Own Warehouse and Filestore

Data Management: Fully managed by the customer.

How should I think about data retention?

Understanding the tradeoffs in managing your own filestore is crucial for effective data management. While storage costs are generally not prohibitively high, it's important to consider the balance between data retention and deletion for each system.

Keeping Data

Pros:

Historical Analysis: Enables long-term trend analysis and forecasting.
Compliance Requirements: Necessary for meeting legal and regulatory data retention needs.

Cons:

Higher Storage Costs: Though not typically high, these costs can accumulate with large datasets, especially with Change Data Capture (CDC) data and full data loads.
Increased Management Complexity: More resources are required to manage and maintain large datasets.

Deleting Data

Pros:

Reduced Storage Costs: Saves money by lowering storage expenses.
Lower Management Overhead: Simplifies data management processes.

Cons:

Loss of Historical Insights: Limits the ability to perform long-term analysis.
Potential Compliance Issues: Risk of non-compliance with data retention regulations.

Common Scenarios

Long-term Data Analysis: Retaining data for comprehensive trend analysis and forecasting.
Cost Management: Deleting older data to manage and reduce storage costs.
Compliance: Keeping data to meet legal and regulatory requirements.

Lifecycle Policies

Your data storage and retention configuration for your filestore should align with your organisation's overall data lifecycle policies.

Common strategies customers apply on a data source by data source basis include:

Reference Backup

The filestore is the reference backup for the system (especially if the system is going offline and being replaced with a new system). This scenario is also covered when the filestore is in active use as a data lake.

Reasonable Recovery

Data is retained for enough time to cover a reasonable likelihood of covering any outages or issues before they are discovered. Allowing recovery from the filestore.

Minimal

Data is removed rapidly and the filestore is just a temporary landing zone.

Considerations

Customers should evaluate these tradeoffs for each system to establish a sensible data management policy based on their specific data and business requirements. It is also important to consider that Change Data Capture (CDC) data and full data loads can significantly increase storage requirements.