Overview

This article gives you an overview of the 10 key skills you need to become a better data engineer. I have seen many engineers ask how to become a data engineer and what skills they need. This post is an answer to that question.

If you are overwhelmed by everything you need to learn to become a data engineer, start with the first topic and proceed through the list.

1. Linux

        Most applications are built on linux systems so it is crucial to understand how to work with them. The key concepts to know are 

1. File system commands, such as ls, cd, pwd, mkdir, rmdir 
2. Commands to get metadata about your data, such as head, tail, wc, grep, ls -lh
3. Data processing commands, such as awk, sed 
4. Bash scripting concepts, such as control flow, looping, passing input parameters

2. SQL

        SQL is crucial to access your data weather it be for running analysis or for use by your application. The key concepts to know are

1. Basic CRUD, such as select, where, join (all types of joins), group by, having, window functions
2. SQL internals, such as index: different types and how they work, transaction concepts such as locks and race conditions
3. Data modeling, such as OLTP schemas like star and snowflake schemas, OLAP schemas like denormalization, key value store, facts and dimensions.

3. Scripting

        Knowledge of a scripting language such as bash scripting or python is very helpful to automate multiple steps required for processing data. The key concepts to know are

1. Basic DS and concept, such as list, dictionaries, map, filter, reduce
2. Control flow and looping concepts, such as if, for loop, list comprehension
3. Popular data processing abstraction library such as pandas or Dask in Python

4. Distributed Data Storage

        Knowledge of how distributed data store such as HDFS or AWS S3 works. Concepts like data replication, serialization, partitioned data storage, file chunking

5. Distributed Data processing

        Knowledge of how data in processed in a distributed fashion. The key concepts to know are

1. Distributed data processing concepts, such as Mapreduce, in memory data processing such as Apache Spark
2. Different types of joins across data sets, such as map side and reduce side joins
3. Common techniques and patterns for data processing such as, partitioning, reducing data shuffles, handling data skews on partitioning
4. Optimizing data processing code to take advantage of all the cores and memory available in the cluster

6.Building data pipelines

       Knowledge of how to connect different data systems to build a data pipeline. The key concepts to know are

1. A data orchestration tool, such as airflow
2. Common pitfalls and how to avoid them, such as data quality checks after processing
3. Building idempotent data pipelines
4. Common patterns for batch and stream processing

7. OLAP database

        Knowledge of how OLAP database operates and when to use them. The key concepts to know are

1. what is a column store and why it is better for most types of aggregation queries
2. Data modeling concepts such as paritioning, fact and dimensions, data skew
3. Figuring out client data query pattern and designing your database accordingly‍

8. Queuing systems

        Knowledge of queuing systems and when and how to use them. The key concepts to know are

1. What is a data producer and a consumer
2. Architecture of the system (e.g Kafka)
3. Knowledge of at least once delivery, offsets and log compaction‍

9. Stream processing

        Knowledge of what stream processing is and how to use them. The key concepts to know are

1. What is stream processing and how it is different from batch processing
2. Different types of stream processing such as Event based processing and micro batching
3. Common patterns of stream processing such as change data capture, etc

10. JVM language

        Knowledge of a JVM based language such as Java or Scala will be extremely useful, since most open source data processing tools are written using JVM languages. e.g Apache Spark, Apache Flink, etc. And these JVM based languages are type safe and easy to program for parallel processing and have implementations of the Actor model which is used in fast and scalable processing.