Data Quality
Approach
Maintaining correctness of data and features is a top level design goal for Fennel.
While Fennel has best-in-class diagnostic and monitoring levers too, unlike many other systems out there, Fennel's approach leans heavily on preventive measures that prevent failures from happening in the first places.
Here are some of the key ideas that help prevent/diagnose data quality issues:
Type | Method | Details |
---|---|---|
Preventive | Strong Typing | Link |
Preventive | Immutability & Versioning | Link |
Preventive | Unit Testing | Link |
Preventive | Compile time lineage validation | Link |
Preventive | Structured metadata & ownership | Link |
Diagnostic | Data Expectations | Link |
Diagnostic | Feature Drift Detection | Link |
Each of these methods is already powerful on their own. And their preventive/diagnostic power further amplifies when applied together.