Local GenAI LLMs with Ollama and Docker

Local GenAI LLMs with Ollama and Docker

Author: Bret Fisher June 14, 2024 Duration: 50:08

Bret and Nirmal are joined by friend of the show, Matt Williams, to learn how to run your own local ChatGPT clone and GitHub Copilot clone with Ollama and Docker's "GenAI Stack," to build apps on top of open source LLMs.

We've designed this conversation for tech people like myself, who are no strangers to using LLMs in web products like chat GPT, but are curious about running open source generative AI models locally and how they might set up their Docker environment to develop things on top of these open source LLMs.

🙌 The Agentic DevOps Guild has launched! It's a training + community + mentorship program for engineers wanting to learn the latest CI/CD automation and dive into Agentic DevOps. Meetups are happening now, with new course videos dropping every few weeks. Join the Guild and become your team's leader in AI for infrastructure automation https://www.bretfisher.com/theguild 🍾

Matt Williams is walking us through all the parts of this solution, and with detailed explanations, shows us how Ollama can make it easier on Mac, Windows, and Linux to set up LLM stacks.

Be sure to check out the video version of this episode for any demos.

This episode is from our YouTube Live show on April 18, 2024 (Stream 262).

★Topics★

Creators & Guests

  • (00:00) - Intro
  • (01:32) - Understanding LLMs and Ollama
  • (03:16) - Ollama's Elevator Pitch
  • (08:40) - Installing and Extending Ollama
  • (17:17) - HuggingFace and Other Libraries
  • (19:24) - Which Model Should You Use?
  • (26:28) - Ollama and Its Applications
  • (28:57) - Retrieval Augmented Generation (RAG)
  • (36:44) - Deploying Models and API Endpoints
  • (40:38) - DockerCon Keynote and LLM Demo
  • (47:44) - Getting Started with Ollama

You can also support this podcast by subscribing to my YouTube channel and my weekly newsletter at bret.news!

Grab the best coupons for my Docker and Kubernetes courses.
Join my cloud native DevOps community on Discord.
Grab some merch at Bret's Loot Box
Homepage bretfisher.com


Bret Fisher hosts DevOps and Docker Talk: Cloud Native Interviews and Tooling alongside Nirmal Mehta, pulling conversations directly from their live audience sessions. This isn't a theoretical lecture series; it's grounded in the practical, often messy realities of building and running modern systems. Each episode digs into the specific tools and philosophies shaping infrastructure today, from the foundational elements like Docker and container orchestration with Kubernetes or Swarm to the broader cultural shifts of DevOps, SRE, and platform engineering. You'll hear discussions that span the entire software lifecycle, touching on the integration of security with DevSecOps, the automation promises of GitOps, and what "Cloud Native" truly means for development teams. The dialogue stays focused on actionable insights and real-world use cases, offering a genuine sense of what works and what doesn't in the field. For anyone navigating the container and cloud ecosystem, this podcast serves as a regular, insightful check-in with practitioners and experts, hosted by someone deeply embedded in the community. The natural, interview-driven format makes complex topics accessible, ensuring listeners come away with more than just buzzwords-they get context and clarity.
Author: Language: English Episodes: 100

DevOps and Docker Talk: Cloud Native Interviews and Tooling
Podcast Episodes
Heroku + MCP = The Fastest Way to Run AI Agents in the Cloud [not-audio_url] [/not-audio_url]

Duration: 43:41
Julián Duque from Heroku joins me to explain and demo their new AI platform.Check out the video podcast version here https://youtu.be/BGqlLZHdRDs🙌 The Agentic DevOps Guild has launched! It's a training + community + ment…
Is AI ready for DevOps? [not-audio_url] [/not-audio_url]

Duration: 29:27
Launching our new Podcast: https://agenticdevops.fmBret and Nirmal are at KubeCon London and record their ideas about how AI Agents will change DevOps, platform engineering, SRE, automation, troubleshooting, and more.🙌 T…
What you missed at KubeCon [not-audio_url] [/not-audio_url]

Duration: 41:39
At KubeCon EU 2025 in London, Nirmal and I discussed the important (and not-so-important) things you might have missed. 🙌 The Agentic DevOps Guild has launched! It's a training + community + mentorship program for engine…
Docker Model Runner [not-audio_url] [/not-audio_url]

Duration: 15:24
Docker launched "Docker Model Runner" to run LLMs through llama.cpp with a single "docker model" command. In this episode Bret details examples and some useful use cases for using this way to run LLMs. He breaks down the…
What's Coming in 2025? [not-audio_url] [/not-audio_url]

Duration: 19:00
This episode is about what I'm seeing and what I'm doing right now, and then for the rest of the year. There are three parts. First, I talk about what's about to happen for me for the next few weeks re going to London fo…
Docker Build the best way with Docker Bake [not-audio_url] [/not-audio_url]

Duration: 17:22
The Docker Bake Build tool just went general availability, and I'm excited about what this means for creating reproducible builds and automation that can run anywhere. In this video I'm gonna break down some of the featu…
Is Swarm at EOL? [not-audio_url] [/not-audio_url]

Duration: 18:41
I've been a big fan of Swarm since it was launched over a decade ago and I've made multiple courses on it that still sell. But, we recently got some news out of Mirantis that might be bad news. So I talked about it last…
Best of Cloud Native 2024 [not-audio_url] [/not-audio_url]

Duration: 1:11:42
Bret and Nirmal reunite for their traditional annual Holiday Special episode of breaking down the most significant developments in cloud native from 2024 and sharing predictions for 2025.💥 This episode is brought to you…
Aikido: Is a Single DevSecOps Tool Possible? [not-audio_url] [/not-audio_url]

Duration: 1:04:13
Or watch the video version on YouTube. Bret is joined by Willem Delbare and Roeland Delrue to discuss Aikido, a security tool consolidation platform designed specifically for smaller teams and solo DevOps practitioners.…
DevOps Cert Prep using AI, with the KodeKloud Team [not-audio_url] [/not-audio_url]

Duration: 1:14:41
Bret is joined by Mumshad Mannambeth and Vijin Palazhi of KodeKloud for Q&A on what we should be studying and certifying for in 2025. 🙌 The Agentic DevOps Guild has launched! It's a training + community + mentorship prog…