Foundry Local: Why to Start Your AI Build Here

You can have an AI model answering questions on your own machine with no Azure subscription, no API key and no invoice. We timed it: once the SDK was installed, the quickstart script took 17 seconds from start to finish, including the first-time downloads.

That is Foundry Local, and it is a low-cost way to start learning what AI models can and can't do for your product. This post covers why we think teams should start there, what happened when we ran it, two things that went wrong, and where it is the wrong tool.

What it is

Microsoft describes Foundry Local as "an end-to-end local AI solution for shipping applications that run entirely on your device." The documentation lists these parts:

• An SDK for C#, JavaScript, Rust and Python.
• A lightweight runtime that handles model download, hardware acceleration, model management and inference through ONNX Runtime. Microsoft says it adds about 20 MB to an application package.
• A curated model catalog. It covers chat models (Microsoft names GPT OSS, Qwen, DeepSeek, Mistral and Phi) and audio transcription (Whisper).
• Automatic hardware acceleration. It picks GPU, NPU or CPU for you.
• An OpenAI-compatible API, including the OpenAI Responses API format, so code written for the OpenAI SDK can be pointed at a Foundry Local endpoint with minimal changes.

It runs on Windows, macOS (Apple silicon) and Linux.

Five reasons to start here

Reasons 1 and 3 are Microsoft's claims. We confirmed the first by not needing a subscription, and we did not inspect network traffic to test the third. Reason 2 is our measurement. Reasons 4 and 5 are our judgment.

1. There is nothing to buy to try it. Microsoft says there are no per-token costs and no backend infrastructure to maintain, and no Azure subscription is needed. We needed none of those.
2. You get a first answer fast. Our script ran in 17 seconds end to end, downloads included. That excludes the one-off pip install.
3. Prompts stay on the device. Microsoft says inference runs locally, so prompts and outputs are processed on the machine. That makes a first experiment simpler to set up. It does not settle your own data rules, which are a question for your security and legal teams.
4. The skills carry over. The API follows OpenAI's request format and Microsoft publishes tutorials for a multi-turn chat assistant, tool calling, a document summarizer and a RAG application. Messages, streaming and tool calling are patterns OpenAI-style APIs share. Whether a specific app moves to a hosted model without changes is something to test, not assume.
5. It shows you what small models can't do, before you've committed budget. More on that below, because it happened to us.

What we did

We followed the Python quickstart on an Apple-silicon Mac, in a fresh virtual environment (Python 3.12):

python3 -m venv v
./v/bin/pip install foundry-local-sdk openai
./v/bin/python app.py

The app.py is Microsoft's sample with the progress printing shortened and a timer added. It downloads the model qwen2.5-0.5b, loads it, streams an answer to "What is the golden ratio?" and unloads it. We installed foundry-local-sdk 2.1.0.

What happened:

• The SDK downloaded and registered a WebGPU execution provider, then the model. About 550 MB of files were cached on disk afterwards.
• The whole run, downloads included, took 17 seconds on our connection.
• The answer streamed in 1.6 seconds.

Here is the answer, cut to its opening sentence:

Model loaded and ready.
Assistant: The golden ratio is a mathematical concept where a line segment is
divided into two parts such that the ratio of the whole length to the longer
part is exactly equal to the ratio of the shorter part to the whole segment.
[...]
Model unloaded.

Two things that went wrong

The answer was wrong. That definition is backwards. In the golden ratio, the whole divided by the longer part equals the longer part divided by the shorter part. The model also went on to bring in the Pythagorean theorem and right triangles, which don't belong there. The quickstart uses a 0.5-billion-parameter model (qwen2.5-0.5b), and it behaved like a small model.

This is the most useful result of the exercise. A model that fluently produces a wrong definition is a risk in any product, and you want to learn that before you design around it. Swap the alias for a larger model from the catalog, ask it questions you know the answers to, and score it. That is an evaluation, and it belongs at the start of the project, not the end.

The quickstart code already carries a deprecation. Calling get_chat_client(), as the quickstart does, printed:

DeprecationWarning: The OpenAI direct client is deprecated and will be
removed at the end of 2026; use ChatSession.

The quickstart page was updated in July 2026 and still uses that call. If you copy it into something you plan to keep, use ChatSession instead. We also saw a one-time warning at startup that two native libraries (ONNX Runtime GenAI and Foundry Local) both define the same Objective-C class. Nothing crashed in our run, but we did not test beyond it.

When Foundry Local is the wrong tool

• Many users at once. Microsoft says Foundry Local is built for one user at a time on constrained hardware, and points to server inference frameworks for concurrent users.
• Enterprise-scale inference on your own infrastructure. Microsoft's page points to Foundry Local on Azure Local, which it describes as Kubernetes-native with Azure Arc management. We have not tested it.
• Model choice beyond the catalog. The catalog is deliberately curated. Microsoft says models are tested on a range of consumer hardware and small enough to distribute to end users.
• Quality-critical work with a small model. See above.

Where this leaves a platform team

A local model is a good way to learn and prototype, and a reasonable fit for offline or on-device features. It is not an AI platform. A platform needs identity on every model call, evaluation before release, spend controls, and a landing zone to run in. Starting local tells you what you want to build, and the platform decisions come after.

Frequently Asked Questions

What is Foundry Local?

Foundry Local is Microsoft's runtime for running AI models on your own device. Microsoft describes it as an end-to-end local AI solution for shipping applications that run entirely on the device, with an SDK for C#, JavaScript, Rust and Python, a curated model catalog and automatic hardware acceleration.

Do I need an Azure subscription to use Foundry Local?

No. Microsoft's documentation says Foundry Local runs entirely on local hardware and no Azure subscription is required. It supports Windows, macOS on Apple silicon, and Linux.

Does Foundry Local send my prompts to Microsoft?

Microsoft says inference runs entirely on the device, so prompts and outputs are processed locally. The network is still used to download models and execution providers, and users may choose to share logs. The product terms and the Microsoft Privacy Statement govern any diagnostic data, so read them before you rely on it.

Can Foundry Local serve many users at once?

It is not designed to. Microsoft says it is optimized for hardware-constrained devices where one user accesses the model at a time, and recommends a dedicated server inference framework for concurrent users.

Sources

1. What is Foundry Local?: Microsoft Learn, page last updated 4 August 2026; the source for the feature list, privacy answer, platforms and limits
2. Get started with Foundry Local: Microsoft Learn, updated 14 July 2026; the quickstart we ran

Timings, file sizes and output are from our own run on 12 October 2026 and will differ on your hardware and connection. This is general information, not legal, security or compliance advice.

Where to go next

• Prototype working, platform not? Take the free Agentic AI Readiness Scorecard. It takes a few minutes and needs no email.
• Moving to the cloud? Use the free Azure AI Landing Zone Readiness Checklist and the Agent Platform Production Checklist.
• Want to build in the hosted service? See A Hands-On Introduction to Microsoft Foundry.
• Want a second pair of eyes on your agent platform? Bring your score to a Discovery Call and we will scope the Agent Platform Readiness Review with you.