Data Validation

In my previous article, I presented my first project, in which I assessed the data quality of an Argentine aviation dataset. While it is necessary to perform an initial, thorough analysis when working with a new data source that we consider critical, the reality is that, in day-to-day operations, we need different solutions that can be integrated into a data pipeline. This is because we should never assume that a dataset is of good quality. And this is not only because the data source may have introduced errors or because it may be more or less reliable. Nor is it necessarily because the file may have been unexpectedly modified during the ingestion process. We must never forget that data quality depends on the context, and what we consider to meet the expectations for a data source should be verified continuously. This is where tools such as Great Expectations become especially useful. ...

A practical Data Stewardship project demonstrating how Great Expectations can automate business rule validation for airport data using reusable expectations and Data Docs.