How to Run Llama-3.1🦙 locally in Python using Ollama, LangChain

In this article, we will learn how to run Llama-3.1 model locally on our PC using Ollama and LangChain in Python

Outline

Install Ollama
Pull model
Serve model
Create a new folder, open it with a code editor
Create and activate Virtual environment
Install langchain-ollama
Run Ollama with model in Python
Conclusion

Install Ollama

Follow the instructions based on your OS type in its GitHub README to install Ollama:

https://github.com/ollama/ollama

I am on a Linux-based PC, so I am going to run the following command in my terminal:

curl -fsSL https://ollama.com/install.sh | sh

Pull model

Fetch the available LLM model via the following command:

ollama pull llama3.1

This will download the default tagged version of the model. Typically, the default points to the latest, smallest sized-parameter model. In this case, it will be llama3.1:8b model.

To download another version of the model, you can go to: https://ollama.com/library/llama3.1 and select the version to install, and then run the ollama pull command with the model and its version number. Example: ollama pull llama3.1:70b

On Mac, the models will be downloaded to ~/.ollama/models

On Linux (or WSL), the models will be stored at /usr/share/ollama/.ollama/models

Serve model

Run the following command to start ollama without running the desktop application.

ollama serve

All models are automatically served on localhost:11434

Create a new folder, open it with a code editor

Create a new folder on your computer and then open it with a code editor like VS Code.

Create and activate Virtual environment

Open the terminal. Use the following command to create a virtual environment .venv and activate it:

python3 -m venv .venv

source .venv/bin/activate

Install langchain-ollama

Run the following command to install langchain-ollama:

pip install -U langchain-ollama

The above command will install or upgrade the LangChain Ollama package in Python. This package allows users to integrate and interact with Ollama models, which are open-source large language models, within the LangChain framework. The -U flag ensures that the package is upgraded to the latest version if it is already installed.

Run Ollama with model in Python

Create a Python file for example: main.py and add the following code:

from langchain_ollama import OllamaLLM

llm = OllamaLLM(model="llama3.1")

response = llm.invoke("The first man on the moon was ...")
print(response)

The above code imports the OllamaLLM class from the LangChain library and initializes an instance of the language model "llama3.1". We pass a prompt about the first man on the moon, and store the generated response in the variable response. When we run the above code we get the following response from the model:

...Neil Armstrong!

On July 20, 1969, Neil Armstrong became the first person to set foot on the lunar surface during the Apollo 11 mission. As he stepped off the lunar module Eagle and onto the moon's surface, he famously declared: "That's one small step for man, one giant leap for mankind."

Conclusion

Thanks for reading.
You can view the Ollama documentation for more commands.

Blog