Creating new Data Pipelines from the command line
Waylon Walker
Posted on September 27, 2021
Kedro new is simply a wrapper around the cookiecutter templating library. The kedro team maintains a ready made template that has everything you need for a kedro project. They also maintain a few kedro starters, which are very similar to the base template.
Unsure what kedro is, Check out yesterdays post on What is Kedro.
pipx
I recommend using pipx
when running kedro new. pipx
is designed for system level cli tools so that you do not need to maintain a virtual environment or worry about version conflicts, pipx
manages the environment for you.
The kedro team does not recommend pipx
in their docs as they already feel like there is a bit of a tool overload for folks that may be less familiar with.
pipx kedro new
I like using pipx
as it gives you better control over using a specific version or always the latest version, unlike when you run what you have on your system depends on when you last installed or upgraded.
Kedro New
The kedro team also has a set of starters, by passing in --starter
you can start with a different template. Here is an example with the kedro spaceflights starter.
pipx run kedro new --starter spaceflights
=============
Please enter a human readable name for your new project.
Spaces and punctuation are allowed.
[New Kedro Project]: Spaceflights Complete
Repository Name:
================
Please enter a directory name for your new project repository.
Alphanumeric characters, hyphens and underscores are allowed.
Lowercase is recommended.
[spaceflights-complete]:
Python Package Name:
====================
Please enter a valid Python package name for your project package.
Alphanumeric characters and underscores are allowed.
Lowercase is recommended. Package name must start with a letter
or underscore.
[spaceflights_complete]:
Change directory to the project generated in /home/u_walkews/git/spaceflights-complete
A best-practice setup includes initialising git and creating a virtual environment before running ``kedro install`` to install project-specific dependencies. Refer to the Kedro documentation: https://kedro.readthedocs.io/
Other versions of kedro with pipx
pipx
not only ensures that you run the latest version, it can also run a very specific version.
pipx run --spec kedro==0.16.6 kedro new
https://waylonwalker.com/kedro-environment/
The next post in this series will help you create your virtual environment for your new kedro project.
Check Out These Related Posts
Data Engineering in The Terminal | How I Kedro
Waylon Walker ・ Aug 18 '21
Connect with Me
I am trying to build my YouTube@waylonwalker channel I would greatly appreciate a sub.
Connect with me on twitter@_waylonwalker.
I stream a few times per week on twitch@waylonwalker.
Check out all of my public repos on github@waylonwalker.
Stay up to date by joining the newsletter if that's your thing.
Connect on LinkedIn@waylonwalker.
Follow me on Dev.to@waylonwalker.
Posted on September 27, 2021
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.