Onnowpurbo: Created page with " Managing Data Consistency Across the Data Science Environment It might seem like a simple task to ensure data consistency across the different parts of the data science envir..."

2021-04-07T02:44:48Z

Created page with " Managing Data Consistency Across the Data Science Environment It might seem like a simple task to ensure data consistency across the different parts of the data science envir..."

New page

Managing Data Consistency Across the
Data Science Environment
It might seem like a simple task to ensure data consistency across the different
parts of the data science environment, but it’s much more difficult than it seems.
First off, this area tends to be more complex than it needs to be, eating up more
time and resources than originally estimated. The need for consistency includes
aspects such as data governance and data formats, but also the labeling of data
consistently —using customer IDs across many different sources to enable cor-
relation of different data types related to the same customer, for example.
The challenge is that there is a built-in contradiction in terms infrastructure
between enabling usage of special tools to allow data scientists and data engineers
to be innovative and productive and at the same time ensuring consistency in the
data. This is because specialized tools are optimized to focus on solving certain
problems but either don’t keep the format consistent or don’t interface well with
other tools needed in the end-to-end flow. Optimized, specialized machine learn-
ing tools are simply not good at playing together with other, similar specialized
tools that are addressing comparable or other adjacent problems.
But is it really that bad? Well, it can lead to real problems, depending on how
much freedom is allowed in the architectural implementation and among the
teams. Some examples of problems that can stem from a lack of consistency across
the AI environment are described in this list:
» » Ad hoc solutions: Every case is treated as an isolated problem that needs to
be solved this instant in order for the team to move forward. The result? No
long-term solution and no learning between teams.
44
PART 1 Optimizing Your Data Science Investment» » Increased cost: When you have to duplicate tool capabilities in order to
manage a lack of consistency or when you have to build capabilities into
purchased tools to secure just the basic consistency, those costs add up.
» » End-to-end not working: Inconsistencies can occur when the infrastructure is
implemented across several cloud vendors, which then makes it difficult or
impossible to transfer data and keep data consistent across different
virtualized environments.
Because corporate management cannot enforce, and may not want to enforce,
data consistency across the organization as a company policy, they have to use
other means to preserve data consistency end-to-end. One way is to ensure that
all teams follow proper and relevant guidelines for evaluating and purchasing new
tools that incorporate specific directives related to data consistency. Clearly moti-
vating why this is key to a successful data science strategy execution.
It’s also vital to consider which limits are needed for each individual company,
depending on the type of business, their objectives, and so on. Hold the line when
it comes to data consistency: Otherwise, you may end up with a cumbersome and
costly implementation of data science, one far removed from the productive data
science environment you were hoping for.

Data Science Strategy: Memanaged Konsistensi Data - Revision history

Onnowpurbo: Created page with " Managing Data Consistency Across the Data Science Environment It might seem like a simple task to ensure data consistency across the different parts of the data science envir..."