A Beginner's Guide to Running AI Models Locally

I recently started learning about local AI and I’m surprised at how easy it is to set up once you understand the basics. In this post, my goal is to explain what local models are, why it is useful to have models on your own computer, what kind of hardware you need, what model weights actually mean, and how to start running models on your machine.

If you’ve ever seen terms like 7B, quantization, tokens, or GGUF and immediately closed the tab, this guide is for you. We’re going to break it down in plain English so you can stop guessing and start experimenting.

What is Local AI?

Local AI means running AI models on your own hardware - be it a laptop, SSD, mac mini, thumbdrive. It is your ability to download a freely available AI model so you can start interacting with AI in the privacy of your own system.

With ai tools that you pay for such as ChatGPT, Claude, Copilot, etc. whenever you make a request or send a chat prompt to the model, it gets processed in a cloud based environment and your data is often used for model training and enhancement.

Local AI offers a completely free, private and secure way to interact with AI systems for work and life without paying hefty fees. With increased token costs and Frontier Labs increasing prices, having a local ai setup is very attractive. It also works completely offline, once you download a model.

What’s not to love?!

P.S. - when I say “locally” it means, “on your hardware - laptop, desktop, whatever you use.”

Benefits of running AI Locally

So why would you even want to run an AI model locally when tools like ChatGPT, Claude, and Gemini already exist?

A few reasons:

• Privacy - your prompts stay on your machine instead of being sent to a cloud service
• Offline access - once the model is downloaded, you can use it without an internet connection
• No monthly subscription - the model is free to run, outside of the hardware you already own
• More control - you choose the model, the tool, and how you want to use it
• Great for learning - you start to understand how models actually work instead of only using the polished app version

This doesn’t mean local models are always better than frontier models. Cloud tools are still powerful, fast, and convenient. But local AI gives you another option. You can experiment, learn, build, and use AI without needing to send every prompt somewhere else.

That is a pretty big deal.

For the first time, regular people can download powerful AI models and run them on their own machines for free.

Requirements

To run an AI model on your machine, you need at least 8GB of RAM (most laptops fit) with at least 5GB+ free to store the model. You can check your machine’s spec using this site: https://www.canirun.ai/ it will tell you in 15 seconds what models you can load on your machine.

One thing you will learn quickly is that running models on your machine requires memory. The model is doing all the work on your system, so you need enough memory for it to process your prompt and return an output. Running models on your machine can sometimes cause your machine to cry, get overheated or just work super hard.

Let me explain why models need space and memory, and what that looks like across different model sizes.

Why do models need space/memory?

AI models are basically big files full of learned patterns. When you download a model, you’re downloading the “brain” of that model onto your machine. That file has to live somewhere, so it takes up storage space just like an app, video, or game would.

But storage is only one part of it.

When you actually run the model, your computer needs to load that model into memory so it can think and respond to your prompts. This is why RAM matters. If the model is too big for your available memory, it may run very slowly, fail to load, or crash altogether.

Think of it like opening a huge Photoshop file or running a big game. The file might be saved on your SSD, but when you open it, your computer needs enough working memory to actually use it.

The larger the model, the more capable it usually is - but it also needs more space and more memory.

This is also why you’ll see model files with sizes like 4GB, 8GB, or even 40GB+. That number tells you how much room the model needs on your machine, and it gives you a clue about whether your laptop can comfortably run it.

So the short version is:

• Storage is where the model lives after you download it
• Memory/RAM is what your computer uses to run the model
• Bigger models usually need more of both
• Smaller models are easier to run and still very useful for everyday tasks

You don’t need the biggest model to get started. In fact, I recommend starting small. A lightweight model that actually runs well on your machine is way better than a massive model that makes your laptop sound like it’s about to take flight.

Understanding Model Weights

When you browse local AI models, you’ll see names with numbers like 3B, 7B, 13B, 30B, or 70B.

That B stands for billion parameters.

Parameters are the tiny learned values inside the model. You can think of them like the model’s memory of patterns it learned during training. The more parameters a model has, the more “room” it has to understand language, follow instructions, write code, summarize text, and reason through problems.

So a 7B model has around 7 billion parameters. A 70B model has around 70 billion parameters. THAT’s pretty HUGE!s

This is why model size is tied directly to hardware. More parameters usually means the model needs:

• More storage space to download
• More RAM or VRAM to run
• More processing power from your CPU, GPU, or Apple Silicon chip
• More time to respond if your machine is not powerful enough

A small 3B or 7B model can run on many modern laptops. Bigger models like 13B, 30B, or 70B need much stronger hardware.

Make sense?

This doesn’t mean bigger is always better for you. If you’re just getting started, a smaller model is perfect. It will be faster, easier to run, and still good enough for everyday tasks like writing, brainstorming, summarizing, and simple coding help.

My advice: start with the smallest model that works well for your use case. Once you understand how it feels on your machine, then try larger models and compare the difference.

So, how many free models are there?

There are a lot of open source models. The best place to browse them is Hugging Face - think of Hugging Face as GitHub for AI models. It is the place to browse, download, and even upload AI models that anyone can use.

As of this writing, there are over 2.8 million models on Hugging Face, and they vary by task, language, app, and provider. There are models that can generate images, videos, and code. There are models that can help you write, summarize, translate, brainstorm, and so much more.

If a frontier model can do something, chances are there is an open source model that can do a version of it too. For free (plus your hardware’s memory 😁).

Key Terms to Understand When Working With Open Source Models (and frontier models too!)

Before you start downloading models, there are a few terms you will see over and over again. You don’t need to memorize all of them, but understanding the basics will make everything feel way less confusing.

Parameters

Parameters are the learned values inside a model. When you see 3B, 7B, or 70B, that number is talking about parameters.

A 7B model has around 7 billion parameters. A 70B model has around 70 billion parameters. More parameters usually means the model can handle more complex tasks, but it also needs more storage, memory, and processing power.

Model Size

Model size usually refers to how large the model file is on your computer. You might see a model that is 4GB, 8GB, 20GB, or even bigger.

This matters because your machine needs enough storage to download the model and enough memory to run it. A model can be free and still be too heavy for your laptop. Ask me how I know.

Quantization

Quantization is a way to make a model smaller and easier to run.

Think of it like compressing a large image. You keep most of the quality, but reduce the file size so it loads faster and takes up less space. With models, quantization reduces how much memory the model needs.

This is why you might see model versions with names like Q4, Q5, or Q8. In general, lower numbers use less memory, but may lose a little quality. Higher numbers keep more quality, but need more hardware.

If you’re just getting started, a quantized model is usually your best friend.

Inference

Inference is what happens when you actually use the model.

Training is how a model learns. Inference is when you ask it a question and it gives you an answer.

So when you run a local model and type, “explain this code to me,” the model is doing inference on your machine. That is why your RAM, CPU, GPU, or Apple Silicon chip matters.

Tokens

Tokens are the pieces of text that models read and generate.

A token can be a word, part of a word, punctuation, or even a space depending on the model. When you send a prompt, the model breaks your text into tokens, processes them, and then generates new tokens as the response.

This is why AI tools talk about token limits. The model can only handle so much text at one time.

Context Window

The context window is how much text the model can keep in mind at once.

If a model has a small context window, it may forget earlier parts of a long conversation or struggle with long documents. If it has a larger context window, it can work with more text at the same time.

For example, a model with a bigger context window is better for summarizing long PDFs, reviewing large code files, or keeping track of a longer chat.

RAM and VRAM

RAM is your computer’s general memory. VRAM (Video RAM) is memory on a graphics card.

For local AI, both matter. Some models run mostly on your CPU and RAM. Others can use your GPU and VRAM to run much faster.

If you’re on a Mac with Apple Silicon, the system uses unified memory, which means the CPU and GPU share the same memory. This is one reason newer Macs can be really nice for running local models.

GGUF

GGUF is a common file format for running local models, especially with tools like Ollama and llama.cpp.

You don’t need to understand the deep technical details here. Just know that if you see a .gguf file, it is usually a model file packaged in a format that local AI tools can load and run.

This is the file format that you download to your machine - it signifies a quantized model.

Embeddings

Embeddings are a way to turn text into numbers so a model or app can understand meaning and similarity.

This is useful when you want to search your own documents, build a chatbot over your notes, or create an AI tool that can find related information. Instead of matching exact words, embeddings help match meaning.

Evals

Evals are tests used to measure how well a model performs.

They can test things like reasoning, coding ability, math, writing quality, safety, or how well the model follows instructions. Evals are helpful, but they are not the whole story.

A model can score well on evals and still feel awkward for your specific use case. Always test models with the kind of prompts you actually care about.

Benchmark

A benchmark is a standard test used to compare models.

You will see benchmarks when people say one model is “better” than another. They can be useful, but don’t treat them like gospel. A model that wins on a benchmark might not be the best model for your laptop, your workflow, or your actual task.

My rule: benchmarks are helpful, but your real-world use case matters more.

How to Run Models in your machine

Ok, now that we have a baseline understanding of model weights and key terms, let’s talk about how to actually run an AI model on your machine.

To actually run an AI model on your machine, you need a runner. A runner is essentially the tool that allows you to interact with the model on your computer. There are quite a few runners in the wild, but here are some of the most popular ones:

Runner	Best For	Source	Notes
Ollama	Beginners	Open source(ish)	Simple, popular, and easy to use from the command line
LM Studio	Beginners	Closed source	Great if you want a visual app instead of using the terminal
llama.cpp	Advanced users	Open source	The lower-level project that tools like Ollama and LM Studio are built on top of
Jan	Beginners/intermediate users	Open source	A local AI desktop app with a clean interface
Msty	Beginners/intermediate users	Closed source	Another friendly desktop app for running and chatting with local models

The app you choose to use will vary depending on your level of experience with local AI. In this demo, we will use LM Studio to get started.

Download LM Studio (or your runner of choice)

Go to LM studio’s site and download the desktop app. The app is free for home and work use. Read their termsfor more info.

Double click the dmg file downloaded and move to your Applications folder (if on a mac).

Now open up the LM Studio app and you should see this lovely message

Click the “Get Started” button and the app will automagically check for an open source model that fits your machine’s specs. For me, it chose “Gemma4: gemma-4-e4b” a 4 billion parameter model that needed about 7GB of RAM.

See how the explainers above come in handy?! Now you can read the model descriptors with understanding.

For context, I’m using a MacBook Pro with an Apple M1 chip and 16GB of unified memory.

Click the download button to install your first open source model on your machine! The speed at which the model downloads depends on how fast your Wi-Fi connection is.

But once it’s downloaded, click load model, hit continue and begin your inference session - aka, start chatting.

And that’s it. You’ve successfully installed and run your first local AI model - private, secure, 100% free, and most importantly, fully yours.

Wrapping up

Local AI can feel intimidating at first, but once you break it down, you start to see that it’s a lot simpler than it appears.

Remember, you do not need the biggest model. And you do not need the most expensive laptop. You just need a machine that can handle a small model and a little curiosity to start testing things out.

Start small, try a few prompts, compare different models, and pay attention to how your machine responds. That is how you learn what works for you.

Happy building!

Kedasha