Introduction It can be overwhelming when trying to set-up or develop a monitoring system for your data-pipelining / processing jobs. However, it is an important aspect of any recurring or scheduled data-pipeline. Ideally, your process can be broken down into single steps. Commonly, they can be broken down into Extraction steps (pulling down or reading in data), Transformation steps (pre-processing or manipulations of the data for the desired output), and Load steps (loading the now processed data into a database, or exporting flat files to a shared folder).

Continue reading

Author's picture

Douglas MacDonald

A Data Scientist in Marketing Science and Higher Education

Associate Director, Marketing Science

Philadelphia, PA