Laid-Back Approaches to Clean Data

The importance of data in the output results of the model means that we need to carry out further handling of the input data. One way that can be done to improve data quality is to carry out data cleaning techniques.

What is Data Cleaning?

Data cleaning is a technique that aims to improve data quality by identifying and eliminating errors and inconsistencies in data.

Here I will share some data cleaning methods and providing simple implementation in how to do it.

There are 4 commonly methods in data cleaning:

Scalling Feature Value
Handling Extreme Outlier
Binning
Scrubbing

Scalling Feature Value

Feature scaling is the process of normalizing the range of features in a dataset. In real cases, the range of feature values varies greatly. If one of the features has a wide value, then that feature will greatly influence the calculations of the algorithm used. Therefore, the range of all features must be normalized so that each feature can provide a comparable contribution. There are several techniques that can be used to perform feature scaling, including:

Absolute Maximum Scaling

Absolute Maximum Scaling is a scaling technique that is carried out based on the absolute maximum value of each feature. The stages of this technique are:

Determine the maximum absolute value of the feature in the data set.
Divide all values in the column by the maximum value.

import matplotlib.pyplot as plt

def max_absolute_scaling(data):
    # Determines the maximum absolute value
    max_abs_value = max(map(abs, data))

    # Divide each value by the maximum absolute value
    scaled_data = []
    for x in data:
        scaled_data.append(x/max_abs_value)

    return scaled_data

data = [3, -1, 6, 2, -4]
scaled_data = max_absolute_scaling(data)
print("Data:", data)
print("Scaled Data:", scaled_data)
plt.plot(data, "red", label="Original Data")
plt.plot(scaled_data, "blue", label="Scaled Data")
plt.legend()
plt.show()

Output:

Data: [3, -1, 6, 2, -4]
Scaled Data: [0.5, -0.16666666666666666, 1.0, 0.3333333333333333, -0.6666666666666666]

Min-Max Scaling

Min-Max Scaling is a scaling technique that is carried out by reducing each value in the dataset by the minimum value and then dividing by the range of the dataset (maximum-minimum). By applying this technique, all feature values will be between 0 and 1. The weakness of this technique is also the same, namely that it is susceptible to outliers.

import matplotlib.pyplot as plt

def min_max_scaling(data):
    # Determine the maximum and minimum values
    max_value = max(data)
    min_value = min(data)

    # Reduces each value by the minimum value
    # then divided by the range of dataset values
    scaled_data = []
    for x in data:
        scaled_data.append(
        (x-min_value)/(max_value-min_value)
        )

    return scaled_data

data = [3, -1, 6, 2, -4]
scaled_data = min_max_scaling(data)
print("Data:", data)
print("Scaled Data:", scaled_data)
plt.plot(data, "red", label="Original Data")
plt.plot(scaled_data, "blue", label="Scaled Data")
plt.legend()
plt.show()

Output:

Data: [3, -1, 6, 2, -4]
Scaled Data: [0.7, 0.3, 1.0, 0.6, 0.0]

Normalization

Normalization is a scaling technique that is similar to min-max scaling, but each feature value is reduced by the average value of the dataset. The results of the reduction are then divided by the range of dataset values.

import matplotlib.pyplot as plt
from statistics import mean

def normalization(data):
    # Determine maximum, minimum, average values
    max_value = max(data)
    min_value = min(data)
    mean_value = mean(data)

    # Subtract each value by the average value
    # then divided by the range of dataset values
    scaled_data = []
    for x in data:
        scaled_data.append(
        (x-mean_value)/(max_value-min_value)
        )

    return scaled_data

data = [3, -1, 6, 2, -4]
scaled_data = normalization(data)
print("Data:", data)
print("Scaled Data:", scaled_data)
plt.plot(data, "red", label="Original Data")
plt.plot(scaled_data, "blue", label="Scaled Data")
plt.legend()
plt.show()

Output:

Data: [3, -1, 6, 2, -4]
Scaled Data: [0.18, -0.22000000000000003, 0.48, 0.08, -0.52]

Standardization (Z-score Normalization)

Standardization is a scaling technique that is carried out by reducing each feature value by the average and dividing by the standard deviation value or what is usually called the z-score. The result of this technique is data that is scaled so that it has features centered on the average and a standard deviation of 1. This technique is suitable if the features have a normal distribution such as salary or age

import matplotlib.pyplot as plt
from statistics import mean, stdev

def standardization(data):
    # Determine the average value, standard deviation
    mean_value = mean(data)
    stdev_value = stdev(data)

    # Subtract each value by the average value
    # then divided by the standard deviation
    scaled_data = []
    for x in data:
        scaled_data.append(
        (x-mean_value)/(stdev_value)
        )

    return scaled_data

data = [3, -1, 6, 2, -4]
scaled_data = standardization(data)
print("Data:", data)
print("Scaled Data:", scaled_data)
plt.plot(data, "red", label="Original Data")
plt.plot(scaled_data, "blue", label="Scaled Data")
plt.legend()
plt.show()

Output:

Data: [3, -1, 6, 2, -4]
Scaled Data: [0.469476477861571, -0.5738045840530313, 1.2519372742975226, 0.20865621238292043, -1.3562653804889828]

Robust Scaling

In the Robust Scaling technique, each data is reduced by the median value and then divided by the Inter Quartile Range (IQR) value. IQR is the difference between the upper quartile (Q3) and the lower quartile (Q1).

import matplotlib.pyplot as plt
import pandas as pd

def robust_scaling(data):
    series = pd.Series(data)
    # Determine the median value, IQR
    q1, median, q3 = series.quantile([0.25, 0.5, 0.75])
    IQR = q3 - q1

    # Subtract each value by the median value
    # then divided by the IQR value
    scaled_data = []
    for x in data:
        scaled_data.append(
        (x-median)/IQR
        )

    return scaled_data

data = [3, -1, 6, 2, -4]
scaled_data = robust_scaling(data)
print("Data:", data)
print("Scaled Data:", scaled_data)
plt.plot(data, "red", label="Original Data")
plt.plot(scaled_data, "blue", label="Scaled Data")
plt.legend()
plt.show()

Output:

Data: [3, -1, 6, 2, -4]
Scaled Data: [0.25, -0.75, 1.0, 0.0, -1.5]

Scaling to Vector Unit Length

Scaling to Vector Unit Length is a scaling technique that is carried out by transforming the components of a feature vector so that the transformed vector has a length of 1. In this technique, each feature value is divided by the vector length.

This technique can only be done if the value ||X||>0

import matplotlib.pyplot as plt
import numpy as np

def vector_normalization(data):
    vector = np.array(data)
    # Determines the length of the vector
    magnitued = np.linalg.norm(vector)

    # Normalize vectors to unit length
    scaled_data = vector/magnitued

    return scaled_data

data = [3, -1, 6, 2, -4]
scaled_data = vector_normalization(data)
print("Data:", data)
print("Scaled Data:", scaled_data.tolist())
plt.plot(data, "red", label="Original Data")
plt.plot(scaled_data, "blue", label="Scaled Data")
plt.legend()
plt.show()

Output:

Data: [3, -1, 6, 2, -4]
Scaled Data: [0.3692744729379982, -0.12309149097933272, 0.7385489458759964, 0.24618298195866545, -0.4923659639173309]

Handling Extreme Outlier

Outliers are values that are much different from the majority of the data in the data set. Outliers may represent natural variation in the population. However, in most cases outliers are caused by an error in the data collection process such as entering incorrect data, equipment failure, or other measurement errors. If outliers are not handled, they can affect the results of statistical analysis and the accuracy of the model being developed.

Outlier Detection

Outlier Detection with Sorting Methods

The sorting method is the simplest method that can be used to detect outliers. Quantitative data can be sorted from low to high and manually detected for data with very low or very high values. In the Python language, sorting can be done using the sorted function.

data = [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
sorted_data = sorted(data)
print("Data:", data)
print("Sorted Data:", sorted_data)

Output:

Data: [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
Sorted Data: [16, 17, 18, 19, 20, 21, 22, 23, 24, 26, 27, 100]

A value of 100 is very high when compared to other data. So 100 can be detected as outliers. However, this method is less accurate because outliers are not determined using statistical calculations.

Outlier Detection using the Histogram Method

Histograms can be used to help visualize data and find out whether there are outlier values in a set of data. Data that is outside the data curve is detected as an outlier. The disadvantages of this method are the same as the average method where outliers are determined only from visual observation of the data.

import matplotlib.pyplot as plt

data = [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
plt.hist(data)
plt.show()

Output:

In the histogram visualization above, it can be seen that the data values are more centered between the values 0 to 40. However, there is data that is separated from other data, namely 100 and this data can be categorized as an outlier.

Outlier Detection with Box-Plot

Box-Plot is a summary of the sample distribution presented graphically which can describe the shape of the data distribution (skewness), a measure of central tendency and a measure of the spread (diversity) of observational data. There are 5 statistical measures that can be read in the box plot, namely minimum, maximum, Q1, median, and Q3. Values outside the box and whisker can be categorized as outliers.

import matplotlib.pyplot as plt

data = [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27] plt.boxplot(data, vert=False)
plt.title("Detecting outliers using Boxplot")
plt.xlabel('Data')
plt.show()

Output:

Value 100 that is outside the whisker can be categorized as an outlier.

Deteksi Outlier dengan Z-Score

The criteria for determining outliers with z-score is that every data point that has a z-score value that is outside the 3rd standard deviation is an outlier. The stages carried out using this technique include:

For all data points, calculate the z-score value using the formula (Xi-mean)/std.
Initialize the threshold value=3 and mark data points that have an absolute z-score value greater than the threshold as outliers.

import statistics as s

def detect_outliers_zscore(data):
    # Threshold initialization
    thres = 3

    # Determine the average value and standard deviation
    mean = s.mean(data)
    std = s.stdev(data)

    outliers = []
    # Determine the z-score value for each data
    for i in data:
        z_score = (i-mean)/std

        # Check whether the data is outliers
        if (abs(z_score) > thres):
            outliers.append(i)

    return outliers

data = [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
outliers = detect_outliers_zscore(data) 
print("Outliers:", outliers)

Output:

Outliers: [100]

Outlier Detection with Inter Quartile Range (IQR)

Based on the Inter Quartile Range (IQR) value, outliers can be detected if the data point is located 1.5 times the IQR above Q3 and below Q1. The stages in determining outliers with IQR are:

Sort asset data in ascending order
Calculate 1st and 3rd quartiles (Q1, Q3)
Calculate the value of IQR=Q3-Q1
Calculate the lower limit value = (Q1–1.5*IQR) and the upper limit = (Q3+1.5*IQR)
For all data in the data set, check whether any data is below the lower limit and above the upper limit. Then mark the data as an outlier.

import pandas as pd

def detect_outliers_iqr(data):
    series = pd.Series(data)

    # Determine Q1, Q3, IQR values
    Q1, Q3 = series.quantile([0.25, 0.75])
    IQR = Q3-Q1

    # Determine lower bound and upper bound
    lower_bound = Q1 - 1.5 * IQR
    upper_bound = Q3 + 1.5 * IQR

    outliers = []
    # Determine the z-score value for each data
    for i in data:
        # Check whether the data is outliers
        if (i < lower_bound or i > upper_bound):
            outliers.append(i)

    return outliers

data = [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
outliers = detect_outliers_iqr(data) 
print("Outliers:", outliers)

Output:

Outliers: [100]

Outlier Handling

After successfully detecting outlier data in the dataset, the next stage is to handle the outlier data. There are several ways that can be done to handle outlier data that has been detected.

Trimming

Outlier data detected using this technique will be removed from the dataset. However, this method is not the best practice to do.

def trimming(data, outlier):
    new_data = []

    # remove data that includes outliers
    for i in outlier:
       new_data = [x for x in data if x != i ]

    return new_data

data = [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
outlier = [100]
new_data = trimming(data, outlier)
print("Data:", data)
print("New Data:", new_data)

Output:

Data: [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
New Data: [16, 17, 18, 19, 20, 21, 22, 23, 24, 26, 27]

Quantile Based Flooring and Capping

Handling of outliers in this technique is carried out by limiting outliers to certain values above the 90th percentile value or placed in factors below the 10th percentile value.

import pandas as pd

def handle_quantile_outlier(data):
    series = pd.Series(data)

    # Determine the 10th and 90th percentiles
    P10 = series.quantile(0.1)
    P90 = series.quantile(0.9)
    new_data = []

    # Replace the data value with P10 for data < P10
    # and with P90 for data > P90
    for x in data:
        if x < P10:
            x = P10
        elif x > P90:
            x = P90
        new_data.append(x)

    return new_data

data = [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
new_data = handle_quantile_outlier(data)
print("Data:", data)
print("New Data:", new_data)

Output:

Data: [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
New Data: [17.1, 17.1, 18, 19, 20, 21, 22, 23, 24, 26.9, 26, 26.9]

Mean/Median Imputation

The average value is greatly influenced by the presence of outliers, so it is recommended to replace these outliers with median values.

import statistics as s

def handle_median_outlier(data, outlier):
    # Determine median value
    median = s.median(data)

    # Change outlier with median value
    new_data = []
    for i in outlier:
        for x in data:
            if x == i:
                new_data.append(median)
            else:
                new_data.append(x)

    return new_data

data = [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
outlier = [100]
new_data = handle_median_outlier(data, outlier)
print("Data:", data)
print("New Data:", new_data)

Output:

Data: [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
New Data: [16, 17, 18, 19, 20, 21, 22, 23, 24, 21.5, 26, 27]

Log Transformation

Log transformation is a common technique used to reduce the skew in a distribution and make it more symmetric. In this way, the occurrence of extreme values can be reduced and the data becomes more normally distributed.

import math
import matplotlib.pyplot as plt

def handle_log_outlier(data):
    new_data = []
    # Transformasi setiap nilai data dengan log
    for x in data:
        new_data.append(math.log(x, 10))

    return new_data

data = [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
new_data = handle_log_outlier(data)
print("Data:", data)
print("New Data:", new_data)
plt.plot(data, 'red', label='Data')
plt.plot(new_data, 'blue', label='New Data')
plt.legend()
plt.show()

Output:

Data: [16, 17, 18, 19, 20, 21, 22, 23, 24, 100, 26, 27]
New Data: [1.2041199826559246, 1.2304489213782739, 1.2552725051033058, 1.2787536009528289, 1.301029995663981, 1.322219294733919, 1.3424226808222062, 1.3617278360175928, 1.380211241711606, 2.0, 1.414973347970818, 1.4313637641589871]

Binning

Data binning is a method of separating or grouping continuous numerical values into discrete intervals called "bins" or "groups". Data grouping methods can be used to simplify data distribution and assist in statistical analysis and visualization. There are several techniques that are often used to group data, including:

Equal Width Binning

This technique groups data into intervals or bins with the same width and has been determined previously. Even though this method is simple, it cannot be applied to data with a skewed distribution.

import pandas as pd 

data = {'age': [16, 17, 18, 19, 20, 21, 22, 23, 24, 20, 26, 27]}

df = pd.DataFrame(data)

num_bins = 4

# Calculate the bin width
bin_width = (df['age'].max() - df['age'].min()) / num_bins

# Create bin limits
bin_edges = [df['age'].min() + i * bin_width for i in range(num_bins + 1)]

df['age_bins'] = pd.cut(df['usia'], bins=bin_edges, include_lowest=True, right=True)

print(df)

Output:

    age        age_bins
0     16  (15.999, 18.75]
1     17  (15.999, 18.75]
2     18  (15.999, 18.75]
3     19    (18.75, 21.5]
4     20    (18.75, 21.5]
5     21    (18.75, 21.5]
6     22    (21.5, 24.25]
7     23    (21.5, 24.25]
8     24    (21.5, 24.25]
9     20    (18.75, 21.5]
10    26    (24.25, 27.0]
11    27    (24.25, 27.0]

Equal Frequency Binning

In this technique, data is grouped into bins with each bin having approximately the same number of data points. This technique is useful when maintaining the same frequency or distribution across bins if it is important. This binning method can also effectively deal with outlier data and skewed data.

import pandas as pd 

data = {'age': [16, 17, 18, 19, 20, 21, 22, 23, 24, 20, 26, 27]}

df = pd.DataFrame(data) 
df['age_bins'] = pd.qcut(df['age'], q=3)

print(df)

Output:

    age          age_bins
0     16  (15.999, 19.667]
1     17  (15.999, 19.667]
2     18  (15.999, 19.667]
3     19  (15.999, 19.667]
4     20  (19.667, 22.333]
5     21  (19.667, 22.333]
6     22  (19.667, 22.333]
7     23    (22.333, 27.0]
8     24    (22.333, 27.0]
9     20  (19.667, 22.333]
10    26    (22.333, 27.0]
11    27    (22.333, 27.0]

Quantile Binning

In this technique, data is grouped based on percentile values. The limits of a bin are based on certain percentile values (e.g. 25th, 50th, and 75th percentiles).

import pandas as pd 
import numpy as np

data = {'age': [16, 17, 18, 19, 20, 21, 22, 23, 24, 20, 26, 27]}
df = pd.DataFrame(data) 

# Defines percentiles for bin boundaries
percentiles = [0, 25, 50, 75, 100] # In this case, quartiles are used

# Defines percentiles for bin boundaries
bin_edges = np.percentile(df['age'], percentiles)

df['age_bins'] = pd.cut(df['age'], bins=bin_edges, include_lowest=True)

print(df)

Output:

    age        age_bins
0     16  (15.999, 18.75]
1     17  (15.999, 18.75]
2     18  (15.999, 18.75]
3     19    (18.75, 20.5]
4     20    (18.75, 20.5]
5     21    (20.5, 23.25]
6     22    (20.5, 23.25]
7     23    (20.5, 23.25]
8     24    (23.25, 27.0]
9     20    (18.75, 20.5]
10    26    (23.25, 27.0]
11    27    (23.25, 27.0]

Scrubbing

Data scrubbing is a process for changing or deleting incomplete, incorrect, inaccurate, or repetitive data in a dataset. By carrying out this process, it can help improve data consistency, accuracy and reliability.

Deleting Repetitive Data

Deleting duplicate data events is one way to perform data scrubbing. Repeated data often appears if the dataset used comes from several different sources.

import pandas as pd

data = {'age': [15, 17, 23, 22, 17],
        'height': [155, 162, 165, 170, 162]}
df = pd.DataFrame(data)

print("Data:")
print(df)

# Check if data is duplicated
duplicate_data = df[df.duplicated()]

print("Duplicate Data:")
print(duplicate_data)

# Delete duplicate data
df = df.drop_duplicates()
print("New Data:")
print(df)
Output:
Data:
   age  height
0    15     155
1    17     162
2    23     165
3    22     170
4    17     162
Duplicate Data:
   age  height
4    17     162
New Data:
   age  height
0    15     155
1    17     162
2    23     165
3    22     170

Handling Missing Data

In real cases, usually there is a lot of missing data in a data set. The causes of this data loss are very varied, ranging from data corruption to device failure when recording measurements.

Deleting Missing Data

Missing data can be resolved by deleting rows or columns of data that have NULL values.

import pandas as pd
import numpy as np

data = {'age': [15, 17, 23, np.nan, 17],
        'height': [155, 162, np.nan, 170, 162]}
df = pd.DataFrame(data)

print("Data:")
print(df)

# Deletes rows of missing data
df.dropna(axis=0, inplace=True)

print("New Data:")
print(df)

Output:

Data:
   age  height
0  15.0   155.0
1  17.0   162.0
2  23.0     NaN
3   NaN   170.0
4  17.0   162.0
New Data:
   age  height
0  15.0   155.0
1  17.0   162.0
4  17.0   162.0

Pros:
• A model trained by removing all missing values will produce a robust model.
Cons:
• Losing a lot of information.
• Works poorly if the percentage of missing values is too large compared to the data set.

Fill in missing data with mean/median/mode values

Columns in a dataset that have numeric values can be replaced with the mean, median, or mode of other data in that column. This technique will prevent data loss like the previous method.

import pandas as pd
import numpy as np

data = {'age': [15, 17, 23, np.nan, 17],
        'height': [155, 162, 165, 170, 162]}
df = pd.DataFrame(data)

print("Data:")
print(df)

df_filled_mean = df.copy()
df_filled_median = df.copy()
df_filled_mode = df.copy()

# Determine mean/median/mode value
mean = df['age'].mean() 
median = df['age'].median()
mode = df['age'].mode().values[0]

# Fill in the data with the mean/median/mode value
df_filled_mean['age'].fillna(mean, inplace=True)
df_filled_median['age'].fillna(median, inplace=True)
df_filled_mode['age'].fillna(mode, inplace=True)

print("New Data:")
print("Filled mean:")
print(df_filled_mean)
print("Filled median:")
print(df_filled_median)
print("Filled mode:")
print(df_filled_mode)
Output:
Data:
   age  height
0  15.0     155
1  17.0     162
2  23.0     165
3   NaN     170
4  17.0     162
New Data:
Filled mean:
   age  height
0  15.0     155
1  17.0     162
2  23.0     165
3  18.0     170
4  17.0     162
Filled median:
   age  height
0  15.0     155
1  17.0     162
2  23.0     165
3  17.0     170
4  17.0     162
Filled mode:
   age  height
0  15.0     155
1  17.0     162
2  23.0     165
3  17.0     170
4  17.0     162

Pros:
• Prevent data loss resulting in deleted rows or columns
• Works well with small data sets and is easy to implement.
Cons:
• Only works with numeric continuous variables.
• May cause data leaks

Fill in Missing Data in Categorical Columns

When missing data is found in a categorical column of either character or number type, the missing data can be filled in with the highest frequency of the category. If there is a lot of missing data, the data is replaced with a new category.

import pandas as pd
import numpy as np

data = {'age': [15, 17, 23, 20, 17],
        'impression': ['good', 'fair', 'fair', 'very good', np.nan]}
df = pd.DataFrame(data)

print("Data:")
print(df)

most_category = df['impression'].mode().values[0]

# Fill in the data with the highest category frequency
df['impression'].fillna(most_category, inplace=True)

print("New Data:")
print(df)

Output:

Data:
   age        impression
0    15         good
1    17         fair
2    23         fair
3    20    very good
4    17          NaN
New Data:
   age        impression
0    15         good
1    17         fair
2    23         fair
3    20    very good
4    17         fair

Pros:
• Prevent data loss resulting in deleted rows or columns
• Works well with small data sets and is easy to implement.
• Eliminate data loss by adding unique categories
Cons:
• Only works with categorical variables.
• Adding new features to the model while coding may result in poor performance

Data Type Conversion

Most machine learning models cannot be run on categorical data. Therefore, categorical data needs to be converted into numerical data. One technique that can be used is one-hot-encoding. One-hot-encoding is a representation of categorical variables in binary vector form.

import numpy as np

# Categorical data to be converted
colors = ["red", "green", "yellow", "red", "blue"]

# Color list
total_colors = ["red", "green", "blue", "black", "yellow"]

# map each color to numeric
mapping = {}
for x in range(len(total_colors)):
  mapping[total_colors[x]] = x

one_hot_encode = []

# Convert the numeric value of each data
for c in colors:
  arr = list(np.zeros(len(total_colors), dtype = int))
  arr[mapping[c]] = 1
  one_hot_encode.append(arr)

print(one_hot_encode)

Output:

[[1, 0, 0, 0, 0], [0, 1, 0, 0, 0], [0, 0, 0, 0, 1], [1, 0, 0, 0, 0], [0, 0, 1, 0, 0]]

Deleting Irrelevant Data

Data is said to be irrelevant when the data does not match the problem being researched.

import pandas as pd

data = {'age': [15, 17, 23],
        'email': ['ahmad@gmail.com', 'putra@yahoo.com', 'tegar@gmail.com']}
df = pd.DataFrame(data)

print("Data:")
print(df)

# Remove irrelevant attributes
df = df.drop('email', axis=1)

print("New Data:")
print(df)

Output:
Data:
   age              email
0    15   ahmad@gmail.com
1    17   putra@yahoo.com
2    23   tegar@gmail.com
New Data:
   age
0    15
1    17
2    23

Avoiding Structural Errors

Structural errors include typos, incorrect naming conventions, incorrect use of capital letters, and so on. The following is an example of improvements to letter capitalization in categorical features:

import pandas as pd
import numpy as np

data = {'age': [15, 17, 23, 20, 17],
        'impression': ['good', 'Fair', 'fair', 'Very good', 'Good']}
df = pd.DataFrame(data)

print("Data:")
print(df)

# Fixed letter capitalization
df['impression'] = df['impression'].str.lower()

print("New Data:")
print(df)

Output:

Data:
   age    impression
0    15         good
1    17         Fair
2    23         fair
3    20    Very good
4    17         Good
New Data:
   age    impression
0    15         good
1    17         fair
2    23         fair
3    20    very good
4    17         good

Closing

In conclusion, employing effective data cleaning methods not only enhances the reliability of your analyses but also paves the way for informed decision-making, ensuring that your data-driven journey is built on a solid foundation of accuracy and integrity

Blog

Laid-Back Approaches to Clean Data

Alit Darma Putra

What is Data Cleaning?

Scalling Feature Value

Absolute Maximum Scaling

Min-Max Scaling

Normalization

Standardization (Z-score Normalization)

Robust Scaling

Scaling to Vector Unit Length

Handling Extreme Outlier

Outlier Detection

Outlier Detection with Sorting Methods

Outlier Detection using the Histogram Method

Outlier Detection with Box-Plot

Deteksi Outlier dengan Z-Score

Outlier Detection with Inter Quartile Range (IQR)

Outlier Handling

Trimming

Quantile Based Flooring and Capping

Mean/Median Imputation

Log Transformation

Binning

Equal Width Binning

Equal Frequency Binning

Quantile Binning

Scrubbing

Deleting Repetitive Data

Handling Missing Data

Deleting Missing Data

Fill in missing data with mean/median/mode values

Fill in Missing Data in Categorical Columns

Data Type Conversion

Deleting Irrelevant Data

Avoiding Structural Errors

Closing

Join Our Newsletter. No Spam, Only the good stuff.

Related