TheLazy.Dev
Portfolio GitHub

Blog

javascript |scraper |node |tutorial

How to make a CLI based movie scraper using NodeJS

raptorcentauri

Dave Shilander

Posted on May 2, 2022

How to make a CLI based movie scraper using NodeJS

How to make a CLI based movie scraper using NodeJS

This guide will instruct you on how to make a CLI application that will ask the user for movie, and present info about that movie in return. To be most successful you should have at least a beginers understanding of Node and NPM. You should also have a basic understanding of using bash to navigate files. Although you are free to use a GUI application if you wish, only terminal examples will be provided.

Prerequsites

  • A MovieDB API Key

    You will need a free acount with TheMovieDB to obtain and API key. You can sign up for one here
    Once you have a key, you can consult the API Documentation

  • Node JS

    You will need to have Node JS installed. Instruction for various platforms can be found here

  • A text editor or IDE of your choice.

  • A terminal that is capable of running bash commands.

Part One (Setup Folder & File Structure, Install NPM Packages)

First, create the root folder for the project. This guide will call it movie-scraper. Then navigate into the newly created folder.

mkdir movie-scraper && cd movie-scraper
Enter fullscreen mode Exit fullscreen mode

Within the movie-scraper directory run the following command.

npm init -y
Enter fullscreen mode Exit fullscreen mode

This will create the package.json file. You need to make two modifications to this file.
Add "type": "module" to the top level, and "start": "node src/index.js" to the scripts section.

It should look like this. (You may have some slight differences depending on your pesoral npm settings, just focus on the changes needed above)

{
  "name": "movie-scraper",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "type": "module",
  "scripts": {
    "start": "node src/index.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
  }
}
Enter fullscreen mode Exit fullscreen mode

Run this next command in your terminal to install the npm packages axios dotenv and inquirer

npm i axios dotenv inquirer
Enter fullscreen mode Exit fullscreen mode

You should now see a node-modules directory in your root folder.

.
├── node_modules
├── package-lock.json
├── package.json
Enter fullscreen mode Exit fullscreen mode

You should also see the newly installed packages listed in your package.json dependcies

"dependencies": {
    "axios": "^0.26.1",
    "dotenv": "^16.0.0",
    "inquirer": "^8.2.2"
}
Enter fullscreen mode Exit fullscreen mode

Now you are going to create a .env file that will hold your API key for TheMovieDB. If you are using git or any other version control for this project, remember to not commit this file.

Replace PUT_YOUR_API_KEY_HERE with your actual API key. Keep the backslashes and quotation marks.

echo TMDB_API_KEY=\"PUT_YOUR_API_KEY_HERE\" >> .env
Enter fullscreen mode Exit fullscreen mode

You can verify the file was created using the cat command

cat .env
Enter fullscreen mode Exit fullscreen mode

You will see th following output with your API key in place of the Xs

TMDB_API_KEY="XXXXXXXXXX"
Enter fullscreen mode Exit fullscreen mode

Create the src directory that will contain the logic for the program, and navigate into it.

mkdir src && cd src
Enter fullscreen mode Exit fullscreen mode

Create the the following files in the src directory:

Movie.js
config.js
getByID.js
index.js
prompt.js
searchMovie.js

You can do this in one command in your terminal

touch {Movie,config,getByID,index,prompt,searchMovie}.js
Enter fullscreen mode Exit fullscreen mode

Your project should now have the following file stucture

.
├── .env
├── node_modules
├── package-lock.json
├── package.json
└── src
    ├── Movie.js
    ├── config.js
    ├── getByID.js
    ├── index.js
    ├── prompt.js
    └── searchMovie.js
Enter fullscreen mode Exit fullscreen mode

Part Two (Javascript Files)

Copy/Paste the following code to each correspodning file

src/config.js 

import dotenv from "dotenv";
dotenv.config();

export default{
    tmbdkey: process.env.TMDB_API_KEY,
};
Enter fullscreen mode Exit fullscreen mode

src/getByID.js 

import config from "./config.js";
import axios from 'axios'

const getByID = async (id) => {
        const options={
            params: {
                api_key: config.tmbdkey,
                language: "en-US",
                append_to_response: `credits`
            }
    }

    let response = await axios.get(`https://api.themoviedb.org/3/movie/${id}`, options)
    return response.data

}

export default getByID;
Enter fullscreen mode Exit fullscreen mode

src/index.js 

import inquirer from 'inquirer';
import Movie from './Movie.js'
import moviePrompts from './prompt.js'

const movieResponse = await inquirer.prompt(moviePrompts)
const selectedMovie = await new Movie(movieResponse.movieID);
selectedMovie.summary()
Enter fullscreen mode Exit fullscreen mode

src/Movie.js 

import getByID from './getByID.js'

class Movie {
    constructor(id) {
        return (async () => {
            this.details = await getByID(id)
            return this
        })()
    }

    get title() {
        return this.details.title
    }

    get tagline() {
        return this.details.tagline
    }

    get overview() {
        return this.details.overview
    }

    get directors() {
        const directors = this.details.credits.crew.filter(
            (obj) => obj.job === 'Director',
        )
        return directors.map((director) => director.name)
    }

    get writers() {
        const writers = this.details.credits.crew.filter(
            (obj) => obj.job === 'Screenplay',
        )
        return writers.map((writer) => writer.name)
    }

    get cast() {
        const cast = this.details.credits.cast.slice(0, 5)
        return cast.map((castMember) => ({
            name: castMember.name,
            role: castMember.character,
        }))
    }

    summary() {
        const summary = 
`
${this.title} - ${this.tagline}
Directed By: ${this.directors}  Written By: ${this.writers}

${this.overview}

Starring:
${this.cast.map(castMember => ` ${castMember.name} as ${castMember.role}`)}
`
        console.log(summary) 
    }
}

export default Movie
Enter fullscreen mode Exit fullscreen mode

src/prompt.js 

import searchMovie from './searchMovie.js'

const moviePrompts = [  
    {
        name: "name",
        type: "input",
        message: "Enter a movie to search:"
    },
    {
        name: "movieID",
        type: "list",
        message: "Select a movie:",
        choices: (answers) => searchMovie(answers.name)
    }
]

export default moviePrompts
Enter fullscreen mode Exit fullscreen mode

src/searchMovie.js 

import config from "./config.js";
import axios from 'axios'

const searchMovie = async (movie) => {
        const options={
            params: {
                api_key: config.tmbdkey,
                language: "en-US",
                query: movie,
            }
    }

    const simpleList = movieObj => ({name: `${movieObj.title} (${movieObj.release_date.slice(0,4)})`, value: movieObj.id})

    const res = await axios.get('https://api.themoviedb.org/3/search/movie', options);

    const list = res.data.results.map(simpleList).slice(0,5)

    return list;

}

export default searchMovie;
Enter fullscreen mode Exit fullscreen mode

Part 3 (Execution)

At the root of your project run

npm start
Enter fullscreen mode Exit fullscreen mode

You will be preseted with the following.

> movie-scraper@1.0.0 start
> node src/index.js

? Enter a movie to search:
Enter fullscreen mode Exit fullscreen mode

Follow the prompts as given by the application.

💖 💪 🙅 🚩
raptorcentauri
Dave Shilander

Posted on May 2, 2022

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

How to make a CLI based movie scraper using NodeJS
javascript How to make a CLI based movie scraper using NodeJS

May 2, 2022