Waylon Walker
Posted on July 8, 2021
Kedro versioned datasets can be mixed with incremental and partitioned datasets to do some timeseries analysis on how our dataset changes over time. Kedro is a very extensible and composible framework, that allows us to build solutions
from the individual components that it provides. This article is a great example of how you can combine these components in unique ways to achieve some powerful results with very little work.
How does our dataset change over time??
This was a question presented to me at work. We had some plots being produces as the output of our pipeline and the user wanted the ability to compare results over time. Luckily this was asked early in the project so we were able
to proactively setup versioning on the right datasets.
To enable this all we needed to do now was to add versioned: true and we will be able to compare results over time. Yes kedro makes it that easy to setup.
🤷♀️ What is Kedro (The Parts)
Waylon Walker ・ Feb 24 '20
set up a project
Set up a new project just as usual. note I like using pipx for global cli packages. You can pick a specific version of kedro or opt for the latest while simply globally installing kedro and running kedro new is purely dependent on the last time you chose to update kedro.
pip install pipx
pipx run kedro new
cd versioned-partitioned-kedro-example
conda create -n versioned-partitioned-kedro-example python=3.8 -y
conda activate versioned-partitioned-kedro-example
pip install kedro
kedro install
git init
git add .
git commit -m "init project from pipx run kedro new"
I called my project versioned-partitioned-kedro-example. You can call your project whatever you like. If you try to use some special characters where they don't belong, kedro will catch you. Under the hood, kedro is using a
library called cookiecutter
⚠️ Please do not skip out on using a virtual environment. You may use whichever virtual environment tool you prefer, but please do not skip out. Wrecking a running project for learning is not fun.
update dependencies
I popped open my dependencies, added kedro[pandas] and find-kedro. Since those are extra packages our example will require.
aiohttp
black==21.5b1
find-kedro
flake8>=3.7.9, <4.0
ipython
isort~=5.0
jupyter_client>=5.1, <7.0
jupyterlab~=3.0
jupyter~=1.0
kedro-telemetry~=0.1.0
kedro==0.17.4
kedro[pandas]
nbstripout~=0.4
pytest-cov~=2.5
pytest-mock>=1.7.1, <2.0
pytest~=6.2
requests
wheel>=0.35, <0.37
note I created find-kedro, and I like using it to create my pipeline object. Think of how pytest automatically picks up everything named test, find-kedro does the same thing for kedro. It picks up everything with node or pipeline in the name and creates pipelines out of it.
Install new dependencies
After adding our additional dependencies to the requirements.in, we can tell kedro to install everything and compile the dependencies. Behind the scenes --build-reqs uses a library called pip-compile to create a requirements.txt file with hard pinned dependencies, which is ideal for creating reproducible projects. You and your future colleagues may not thank you for this, but they sure as heck won't be cursing your name when they can't
get the project to run.
kedro install --build-reqs
git add .
git commit -m "added additional dependencies"
create a node
For this example, we need a node to do much. This node will
pass the cars.csv from a URL to a parquet file. I am going to use a lambda to build my identity function inline.
# pipelines/cars_nodes.py
from kedro.pipeline import node
nodes = []
nodes.append(
node(
func=lambda x:x,
inputs='raw_cars',
outputs='int_cars',
name='create_int_cars',
)
)
🗒️ note
find-kedro' will automatically pick up these nodes for us after we set up ourpipeline_registry.py`.
bash
git add .
git commit -m "add create_int_cars node"
implement find-kedro
Next, we need to tell kedro where our nodes are. This is where find-kedro comes in. Once we point to the directory where our modules of nodes/pipelines are, it creates the pipelines dictionary for us automatically. It will even separate each module into a pipeline and stitch them all into one default pipeline.
` python
pipeline_registry.py
"""Project pipelines."""
from typing import Dict
from pathlib import Path
from kedro.pipeline import Pipeline
from find_kedro import find_kedro
def register_pipelines() -> Dict[str, Pipeline]:
"""Register the project's pipelines.
Returns:
A mapping from a pipeline name to a "Pipeline "object.
"""
pipeline_dir = Path(__file__).parent / 'pipelines'
return find_kedro(directory= pipeline_dir)
`
🗒️ This is very similar to the default ` pipeline_registry'except the last two
lines.
git add .
git commit -m "implement find-kedro"
create a baseline catalog
Once we have a pipeline setup, the kedro cli can automatically fill in missing catalog entries with MemoryDataSet's. Thus, using the cli helps consistently scaffold the catalog and ensure we don't end up with a typo in our dataset name.
kedro catalog create --pipeline cars_nodes
Kedro will kick out the following catalog file to base/catalog/cars_nodes.yml
for us to get started with.
raw_cars:
type: MemoryDataSet
int_cars:
type: MemoryDataSet
🔥 use the kedro cli to fill in any missing datasets from the automatically catalog.
make a versioned dataset
Kedro has scaffolded MemoryDataSet 's for us. We will convert them to the appropriate dataset type and turn on versioning for our int layer, which is the first point we save in our environment.
raw_cars:
type: pandas.CSVDataSet
filepath: https://waylonwalker.com/cars.csv
int_cars:
type: pandas.ParquetDataSet
filepath: data/int_cars.parquet
versioned: true
Commit your changes to the catalog.
git add .
git commit -m "create catalog"
run the pipeline
Once we have the nodes and catalog setup, we can run the pipeline a few times to get some versioned data. Each time we run, it will save a new version inside the int_cars.parquet directory.
kedro run
kedro run
kedro run
kedro run
kedro run
🗒️ we put our data in the data directory. By default, this directory is included in the
.gitignoreand will not be picked up by git.
inspect the data
Listing the files in data/int_cars.parquet shows that I now have five different datasets available. I can load old ones, but by default, kedro will load the latest one.
ls data/int_cars.parquet
2021-07-05T15.24.53.164Z
2021-07-05T15.29.56.144Z
2021-07-05T15.30.23.101Z
2021-07-05T15.30.26.555Z
2021-07-05T15.31.12.688Z
🗒️ kedro sets the version at the timestamp that the session starts. All datasets created within the same run will have the same version.
stack on an incremental dataset
This is where things get interesting. Kedro comes with an incremental dataset that will load all of the files from a particular directory into a dictionary where the keys are the filename of the dataset. To load up all datasets into
this dictionary all we need to do is add a new catalog entry that is a type: PartitionedDataSet, with a path pointing to the same place as the original, and a dataset type the same as the original.
int_cars_partitioned:
type: PartitionedDataSet
dataset: pandas.ParquetDataSet
path: data/int_cars.parquet
catalog list
Listing the catalog entries confirms that we have successfully added our new PartitionedDataSet.
In [17]: context.catalog.list()
Out[17]:
['raw_cars',
'int_cars',
'int_cars_partitioned',
'parameters']
loading an incremental dataset
Now we can easily load the datasets from every run we just did into a single dictionary, simply by running context.catalog.load('int_cars_incremental').
In [19]: context.catalog.load('int_cars_incremental')
2021-07-05 11:32:40,534 - kedro.io.data_catalog - INFO - Loading data from `int_cars_incremental` (IncrementalDataSet)...
Out[19]:
{'2021-07-05T15.29.56.144Z/int_cars.parquet': Unnamed: 0 mpg cyl disp hp drat wt qsec vs am gear carb
0 Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4
1 Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4
2 Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1
3 Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1
4 Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2
5 Valiant 18.1 6 225.0 105 2.76 3.460 20.22 1 0 3 1
6 Duster 360 14.3 8 360.0 245 3.21 3.570 15.84 0 0 3 4
7 Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2
8 Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2
9 Merc 280 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4
10 Merc 280C 17.8 6 167.6 123 3.92 3.440 18.90 1 0 4 4
11 Merc 450SE 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 3
12 Merc 450SL 17.3 8 275.8 180 3.07 3.730 17.60 0 0 3 3
13 Merc 450SLC 15.2 8 275.8 180 3.07 3.780 18.00 0 0 3 3
14 Cadillac Fleetwood 10.4 8 472.0 205 2.93 5.250 17.98 0 0 3 4
15 Lincoln Continental 10.4 8 460.0 215 3.00 5.424 17.82 0 0 3 4
16 Chrysler Imperial 14.7 8 440.0 230 3.23 5.345 17.42 0 0 3 4
17 Fiat 128 32.4 4 78.7 66 4.08 2.200 19.47 1 1 4 1
18 Honda Civic 30.4 4 75.7 52 4.93 1.615 18.52 1 1 4 2
19 Toyota Corolla 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 1
20 Toyota Corona 21.5 4 120.1 97 3.70 2.465 20.01 1 0 3 1
21 Dodge Challenger 15.5 8 318.0 150 2.76 3.520 16.87 0 0 3 2
22 AMC Javelin 15.2 8 304.0 150 3.15 3.435 17.30 0 0 3 2
23 Camaro Z28 13.3 8 350.0 245 3.73 3.840 15.41 0 0 3 4
24 Pontiac Firebird 19.2 8 400.0 175 3.08 3.845 17.05 0 0 3 2
25 Fiat X1-9 27.3 4 79.0 66 4.08 1.935 18.90 1 1 4 1
26 Porsche 914-2 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 2
27 Lotus Europa 30.4 4 95.1 113 3.77 1.513 16.90 1 1 5 2
28 Ford Pantera L 15.8 8 351.0 264 4.22 3.170 14.50 0 1 5 4
29 Ferrari Dino 19.7 6 145.0 175 3.62 2.770 15.50 0 1 5 6
30 Maserati Bora 15.0 8 301.0 335 3.54 3.570 14.60 0 1 5 8
31 Volvo 142E 21.4 4 121.0 109 4.11 2.780 18.60 1 1 4 2,
'2021-07-05T15.30.23.101Z/int_cars.parquet': Unnamed: 0 mpg cyl disp hp drat wt qsec vs am gear carb
0 Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4
1 Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4
2 Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1
3 Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1
4 Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2
5 Valiant 18.1 6 225.0 105 2.76 3.460 20.22 1 0 3 1
6 Duster 360 14.3 8 360.0 245 3.21 3.570 15.84 0 0 3 4
7 Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2
8 Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2
9 Merc 280 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4
10 Merc 280C 17.8 6 167.6 123 3.92 3.440 18.90 1 0 4 4
11 Merc 450SE 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 3
12 Merc 450SL 17.3 8 275.8 180 3.07 3.730 17.60 0 0 3 3
13 Merc 450SLC 15.2 8 275.8 180 3.07 3.780 18.00 0 0 3 3
14 Cadillac Fleetwood 10.4 8 472.0 205 2.93 5.250 17.98 0 0 3 4
15 Lincoln Continental 10.4 8 460.0 215 3.00 5.424 17.82 0 0 3 4
16 Chrysler Imperial 14.7 8 440.0 230 3.23 5.345 17.42 0 0 3 4
17 Fiat 128 32.4 4 78.7 66 4.08 2.200 19.47 1 1 4 1
18 Honda Civic 30.4 4 75.7 52 4.93 1.615 18.52 1 1 4 2
19 Toyota Corolla 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 1
20 Toyota Corona 21.5 4 120.1 97 3.70 2.465 20.01 1 0 3 1
21 Dodge Challenger 15.5 8 318.0 150 2.76 3.520 16.87 0 0 3 2
22 AMC Javelin 15.2 8 304.0 150 3.15 3.435 17.30 0 0 3 2
23 Camaro Z28 13.3 8 350.0 245 3.73 3.840 15.41 0 0 3 4
24 Pontiac Firebird 19.2 8 400.0 175 3.08 3.845 17.05 0 0 3 2
25 Fiat X1-9 27.3 4 79.0 66 4.08 1.935 18.90 1 1 4 1
26 Porsche 914-2 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 2
27 Lotus Europa 30.4 4 95.1 113 3.77 1.513 16.90 1 1 5 2
28 Ford Pantera L 15.8 8 351.0 264 4.22 3.170 14.50 0 1 5 4
29 Ferrari Dino 19.7 6 145.0 175 3.62 2.770 15.50 0 1 5 6
30 Maserati Bora 15.0 8 301.0 335 3.54 3.570 14.60 0 1 5 8
31 Volvo 142E 21.4 4 121.0 109 4.11 2.780 18.60 1 1 4 2,
'2021-07-05T15.30.26.555Z/int_cars.parquet': Unnamed: 0 mpg cyl disp hp drat wt qsec vs am gear carb
0 Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4
1 Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4
2 Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1
3 Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1
4 Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2
5 Valiant 18.1 6 225.0 105 2.76 3.460 20.22 1 0 3 1
6 Duster 360 14.3 8 360.0 245 3.21 3.570 15.84 0 0 3 4
7 Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2
8 Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2
9 Merc 280 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4
10 Merc 280C 17.8 6 167.6 123 3.92 3.440 18.90 1 0 4 4
11 Merc 450SE 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 3
12 Merc 450SL 17.3 8 275.8 180 3.07 3.730 17.60 0 0 3 3
13 Merc 450SLC 15.2 8 275.8 180 3.07 3.780 18.00 0 0 3 3
14 Cadillac Fleetwood 10.4 8 472.0 205 2.93 5.250 17.98 0 0 3 4
15 Lincoln Continental 10.4 8 460.0 215 3.00 5.424 17.82 0 0 3 4
16 Chrysler Imperial 14.7 8 440.0 230 3.23 5.345 17.42 0 0 3 4
17 Fiat 128 32.4 4 78.7 66 4.08 2.200 19.47 1 1 4 1
18 Honda Civic 30.4 4 75.7 52 4.93 1.615 18.52 1 1 4 2
19 Toyota Corolla 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 1
20 Toyota Corona 21.5 4 120.1 97 3.70 2.465 20.01 1 0 3 1
21 Dodge Challenger 15.5 8 318.0 150 2.76 3.520 16.87 0 0 3 2
22 AMC Javelin 15.2 8 304.0 150 3.15 3.435 17.30 0 0 3 2
23 Camaro Z28 13.3 8 350.0 245 3.73 3.840 15.41 0 0 3 4
24 Pontiac Firebird 19.2 8 400.0 175 3.08 3.845 17.05 0 0 3 2
25 Fiat X1-9 27.3 4 79.0 66 4.08 1.935 18.90 1 1 4 1
26 Porsche 914-2 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 2
27 Lotus Europa 30.4 4 95.1 113 3.77 1.513 16.90 1 1 5 2
28 Ford Pantera L 15.8 8 351.0 264 4.22 3.170 14.50 0 1 5 4
29 Ferrari Dino 19.7 6 145.0 175 3.62 2.770 15.50 0 1 5 6
30 Maserati Bora 15.0 8 301.0 335 3.54 3.570 14.60 0 1 5 8
31 Volvo 142E 21.4 4 121.0 109 4.11 2.780 18.60 1 1 4 2,
'2021-07-05T15.31.12.688Z/int_cars.parquet': Unnamed: 0 mpg cyl disp hp drat wt qsec vs am gear carb
0 Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4
1 Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4
2 Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1
3 Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1
4 Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2
5 Valiant 18.1 6 225.0 105 2.76 3.460 20.22 1 0 3 1
6 Duster 360 14.3 8 360.0 245 3.21 3.570 15.84 0 0 3 4
7 Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2
8 Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2
9 Merc 280 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4
10 Merc 280C 17.8 6 167.6 123 3.92 3.440 18.90 1 0 4 4
11 Merc 450SE 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 3
12 Merc 450SL 17.3 8 275.8 180 3.07 3.730 17.60 0 0 3 3
13 Merc 450SLC 15.2 8 275.8 180 3.07 3.780 18.00 0 0 3 3
14 Cadillac Fleetwood 10.4 8 472.0 205 2.93 5.250 17.98 0 0 3 4
15 Lincoln Continental 10.4 8 460.0 215 3.00 5.424 17.82 0 0 3 4
16 Chrysler Imperial 14.7 8 440.0 230 3.23 5.345 17.42 0 0 3 4
17 Fiat 128 32.4 4 78.7 66 4.08 2.200 19.47 1 1 4 1
18 Honda Civic 30.4 4 75.7 52 4.93 1.615 18.52 1 1 4 2
19 Toyota Corolla 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 1
20 Toyota Corona 21.5 4 120.1 97 3.70 2.465 20.01 1 0 3 1
21 Dodge Challenger 15.5 8 318.0 150 2.76 3.520 16.87 0 0 3 2
22 AMC Javelin 15.2 8 304.0 150 3.15 3.435 17.30 0 0 3 2
23 Camaro Z28 13.3 8 350.0 245 3.73 3.840 15.41 0 0 3 4
24 Pontiac Firebird 19.2 8 400.0 175 3.08 3.845 17.05 0 0 3 2
25 Fiat X1-9 27.3 4 79.0 66 4.08 1.935 18.90 1 1 4 1
26 Porsche 914-2 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 2
27 Lotus Europa 30.4 4 95.1 113 3.77 1.513 16.90 1 1 5 2
28 Ford Pantera L 15.8 8 351.0 264 4.22 3.170 14.50 0 1 5 4
29 Ferrari Dino 19.7 6 145.0 175 3.62 2.770 15.50 0 1 5 6
30 Maserati Bora 15.0 8 301.0 335 3.54 3.570 14.60 0 1 5 8
31 Volvo 142E 21.4 4 121.0 109 4.11 2.780 18.60 1 1 4 2}
👆 notice that incremental datasets are all loaded for you, its a dict of
filepath:dataset
stack on a partitioned dataset
Let's take a look at a similar type of dataset called PartitionedDataSet. We can add it to the catalog in a very similar way to how we added the IncrementalDataSet.
int_cars_incremental:
type: IncrementalDataSet
dataset: pandas.ParquetDataSet
path: data/int_cars.parquet
loading a partitioned dataset
Note that we get a dict with the same keys as before, but this time the values are a load function rather than loaded data. Partitioned datasets can be helpful if you are operating on datasets that take up more memory than you have available. In our case of coupling this with versioned datasets, its likely to grow quite large, so PartitionedDataSet 's are likely a better option for this use.
In [18]: context.catalog.load('int_cars_partitioned')
2021-07-05 11:31:11,253 - kedro.io.data_catalog - INFO - Loading data from `int_cars_partitioned` (PartitionedDataSet)...
Out[18]:
{'2021-07-05T15.29.56.144Z/int_cars.parquet': <bound method AbstractVersionedDataSet.load of <kedro.extras.datasets.pandas.parquet_dataset.ParquetDataSet object at 0x7f4bb1570820>>,
'2021-07-05T15.30.23.101Z/int_cars.parquet': <bound method AbstractVersionedDataSet.load of <kedro.extras.datasets.pandas.parquet_dataset.ParquetDataSet object at 0x7f4bb1570850>>,
'2021-07-05T15.30.26.555Z/int_cars.parquet': <bound method AbstractVersionedDataSet.load of <kedro.extras.datasets.pandas.parquet_dataset.ParquetDataSet object at 0x7f4bb1570910>>,
'2021-07-05T15.31.12.688Z/int_cars.parquet': <bound method AbstractVersionedDataSet.load of <kedro.extras.datasets.pandas.parquet_dataset.ParquetDataSet object at 0x7f4bb15709a0>>}
incremental vs. partitioned
IncrementalDataSet 's and PartitionedDataSet 's are very similar as they give you access to a whole directory of data that uses the same underlying dataset loader. The significant difference is whether you want your data pre-loaded or if you want to load and dispose of it as you iterate over it.
- incremental loads the data
- partitioned give a load function
creating nodes with partitioned datasets
Let's create a node with this PartitionedDataSet to collect stats on our dataset over time. This node does a dict comprehension to get the length of each version that we pulled.
def timeseries_partitioned(cars: Dict):
return {k:len(car()) for k, car in cars.items()}
nodes.append(
node(
func=timeseries_partitioned,
inputs='int_cars_partitioned',
outputs='int_cars_timeseries_partitioned',
name='create_int_cars_timeseries_partitioned',
)
)
🗒️ note that inside of the dict comprehension car is a load function that we need to call.
creating nodes with incremental datasets
Doing the same node with our IncrementalDataSet looks very similar, except this time car is loaded data inside of the dict comprehension, not a function that we need to call.
def timeseries_incremental(cars: Dict):
return {k:len(car) for k, car in cars.items()}
nodes.append(
node(
func=timeseries_incremental,
inputs='int_cars_incremental',
outputs='int_cars_timeseries_incremental',
name='create_int_cars_timeseries_incremental',
)
)
More catalog entries
After adding those nodes, we can add the catalog entries again with the command line. This will not overwrite any of the datasets we just created. It will only add to it.
kedro catalog create --pipeline cars_nodes
int_cars_timeseries_partitioned:
type: MemoryDataSet
int_cars_timeseries_incremental:
type: MemoryDataSet
int_cars_timeseries_partitioned:
type: pickle.PickleDataSet
filepath: data/int_cars_timeseries_partitioned.parquet
int_cars_timeseries_incremental:
type: pickle.PickleDataSet
filepath: data/int_cars_timeseries_incremental.parquet
Loading the new datasets
Loading the two dtasets that we just created show that we have the ended up with the same result using both incremental and partitioned datasets. This result is a dictionary of filepaths mapped to the size of the dataset. Since the default filepaths are timestamps we could start doing some time series analysis to see how our dataset is changing over time.
In [32]: context.catalog.load('int_cars_timeseries_incremental')
2021-07-05 12:00:55,014 - kedro.io.data_catalog - INFO - Loading data from `int_cars_timeseries_incremental` (PickleDataSet)...
Out[32]:
{'2021-07-05T15.29.56.144Z/int_cars.parquet': 32,
'2021-07-05T15.30.23.101Z/int_cars.parquet': 32,
'2021-07-05T15.30.26.555Z/int_cars.parquet': 32,
'2021-07-05T15.31.12.688Z/int_cars.parquet': 32,
'2021-07-05T16.43.43.088Z/int_cars.parquet': 32}
In [33]: context.catalog.load('int_cars_timeseries_partitioned')
2021-07-05 12:01:03,223 - kedro.io.data_catalog - INFO - Loading data from `int_cars_timeseries_partitioned` (PickleDataSet)...
Out[33]:
{'2021-07-05T15.29.56.144Z/int_cars.parquet': 32,
'2021-07-05T15.30.23.101Z/int_cars.parquet': 32,
'2021-07-05T15.30.26.555Z/int_cars.parquet': 32,
'2021-07-05T15.31.12.688Z/int_cars.parquet': 32,
'2021-07-05T16.43.43.088Z/int_cars.parquet': 32,
'2021-07-05T16.50.46.686Z/int_cars.parquet': 32}
☝️ I have a full article on creating datasets that are not tabular datasets using pickle.
This post was primarily built live on https://twitch.tv/waylonwalker, give me a follow and join in the live show if that is something that interests you.
Posted on July 8, 2021
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.
Related
