Over the past year, a growing number of the advisers and fund managers we work with have asked the same question: “Can we run an AI model in-house, so client data never leaves the building?”
The answer is yes, and the technology side is more approachable than most firms expect. The harder part is running it in a way that holds up under Rule 206(4)-7, Reg S-P, and Rule 204-2. This article walks through what a local deployment actually looks like: the hardware, the software, a real workflow, and the compliance program you’d need to wrap around it.
(We cover the full governance picture in The SEC RIA AI Framework. This piece is about the practical side: what’s on the desk, what’s on the screen, and what has to be in the compliance manual.)
Why local?
When an employee pastes a client question into a public AI chatbot, that text leaves your perimeter and lands on a third party’s infrastructure. Depending on the plan, the settings, and the contract, it may be retained, logged, or used for training. For a firm handling nonpublic personal information, that’s a safeguards risk under Reg S-P that most firms have never assessed. In many cases it’s happening today, quietly, outside any written policy.
A local LLM inverts the model: the AI runs on a machine you own, inside your office network. Configured correctly (local model, local document index, no cloud connectors), the questions your employees ask and the documents they ask about stay on that machine. There’s no cloud inference provider receiving your prompts, and no data-processing addendum standing between your client data and someone else’s retention policy. The trade-off is that you now own the machine, the configuration, and the oversight, which is exactly what the rest of this article covers.
The hardware: one machine, roughly $3,000–$5,000
You do not need a server rack. For a small-to-mid-size RIA, a single Apple Mac Studio is a particularly practical option, for one specific reason: unified memory. Apple Silicon lets the graphics processor address the same large memory pool as the rest of the machine, which is exactly what large language models need. Dedicated GPU workstations can match or beat it on raw speed, but for a firm without an in-house IT department, the Mac wins on simplicity, power draw, and price-per-gigabyte of usable AI memory.
Representative configurations as of this writing (August 2026):
| Configuration | Approx. price | What it runs | |—|—|—| | Mac Studio, M4 Max (16-core CPU / 40-core GPU), 64GB unified memory | ~$3,000 | Mid-size open models (27B–32B class) with comfortable headroom, strong enough for the document Q&A workflow described below | | Mac Studio, M3 Ultra, 96GB unified memory | ~$5,300 | Large open models (70B class): noticeably better reasoning, plus room for longer documents and context |
Two pricing notes worth flagging. First, the 64GB memory option on the M4 Max requires stepping up to the higher chip configuration: the $1,999 base model tops out at 36GB, which is workable but tight. Second, the M3 Ultra model launched at $3,999 but currently sits around $5,300, and Apple has trimmed its higher-memory configurations, reflecting industry-wide memory price pressure. Check current pricing when you spec the machine; this market is moving.
Either configuration handles the workflow in this article; the larger one buys headroom. One realistic caveat: a single machine serves a small office well for this kind of use, but it isn’t infinite. If several people hit it at the same moment, responses queue. For a 5–25 person firm asking reference questions, that’s rarely noticeable. For heavier ambitions, the sizing conversation gets more involved.
Practical notes:
- The machine is quiet and compact, sits on a desk or shelf, and plugs into your existing network. No server room, no special cooling.
- Leasing is an option. Apple offers business financing and leasing directly (Apple Financial Services), and most Apple business resellers (CDW, Insight, SHI) will structure a 24–36 month lease. That converts a capital purchase into a predictable monthly operating expense and builds in a refresh cycle, useful since local AI hardware is improving quickly.
- Employees don’t sit at this machine. They reach it through a browser, like any internal tool.
The software: Ollama + AnythingLLM
Two free, widely used pieces of software turn that Mac into an internal AI service:
Ollama is the engine. It downloads and runs open-weight models (Meta’s Llama, Google’s Gemma, Alibaba’s Qwen, Mistral, and others) entirely on the machine. Installing it is a single download; pulling a model is one command. The mainstream open-weight models permit commercial use, though each has its own license, and reviewing it should be part of adopting the model. (One configuration note that matters: Ollama also offers optional cloud-hosted models. A privacy-driven deployment uses local models only, and that choice should be deliberate and verified, not assumed.)
AnythingLLM is the interface. Deployed in its self-hosted server (Docker) edition on the same machine, it gives your team a familiar chat window in the browser and adds the piece that makes this genuinely useful: document workspaces. You upload firm documents (PDFs, Word files, policies) into a workspace, and the system indexes them so the model answers from your documents, pointing back to the source material it relied on. The technique is called retrieval-augmented generation (RAG): instead of answering from its general training, the model retrieves the relevant passages from your uploaded documents and answers from those.
The self-hosted server edition also provides the administrative basics a firm needs: individual user accounts, role-based permissions (admin, manager, and chat-only roles), and workspace-level chat history. These are useful controls, but they’re application features, not a compliance program. More on that below.
Setup, realistically: a working demonstration is an afternoon. A production rollout the whole office relies on (accounts, permissions, backups, update procedures, testing) is a project measured in days, and it’s worth doing as a project.
A concrete workflow: “Ask the Operations Manual”
Before we walk through this, an important framing note: the workflow below is purely a demonstration. We chose it because it’s simple, low-risk, and immediately useful, but it’s one example of a pattern, not the ceiling. The same architecture supports far more sophisticated workflows: separate permissioned workspaces for the ADV and client agreements, vendor contract review, fund document Q&A, due-diligence file analysis, drafting support against approved templates, and multi-step internal processes that would be genuinely powerful to automate. Once the machine, the software, and the governance exist, adding workflows is incremental. Start simple; the platform grows with you.
Here’s the demonstration deployment we’d sketch for a typical adviser:
1. Create a workspace in AnythingLLM called Operations Manual and upload the firm’s compliance manual, code of ethics, trading procedures, and BCP: current versions only, uploaded by one accountable owner. 2. Pin a system prompt to the workspace: “Answer only from the uploaded documents. Always identify the section you’re relying on. If the documents don’t address the question, say so and direct the employee to the CCO.” 3. Give every employee a login with chat-only access. The CCO and one backup hold admin rights. 4. Employees ask questions in plain language:
- “What’s our pre-clearance requirement for personal trades?”
- “How long do we keep trade confirmations?”
- “Where’s our procedure for verifying a client’s wire request?”
The assistant answers in seconds, pointing to the relevant manual section so the employee can open it and verify. Note what the assistant does in that third example: it finds the procedure. The procedure (callback verification, dual authorization) is what protects the client. The assistant’s job is making sure no employee ever says “I didn’t know where that was written.”
5. The CCO reviews the chat logs periodically. This is where the tool quietly becomes a compliance asset: the questions employees ask reveal exactly where the manual is unclear, where training is thin, and which procedures nobody can find. That’s real input for your annual review under Rule 206(4)-7.
Two honest limitations. First, a system prompt is an instruction, not a guarantee: models can still answer from general knowledge or paraphrase loosely, which is why testing and human verification stay in the picture. Second, local open-weight models trail the frontier hosted models on hard reasoning. For this narrow job, finding and restating what your own documents say, they’re well suited. You’re trading some raw capability for privacy and control, and for this use case that’s a good trade.
The compliance program around it
Running the tool is the easy half. An examiner who learns you’ve deployed an internal AI assistant will want to see that it was adopted deliberately, governed in writing, and actually supervised. None of this is exotic. It’s your existing compliance program, extended to cover a new system. Here’s the shape of it:
1. A written AI use policy. Rule 206(4)-7 doesn’t say the words “artificial intelligence.” It requires policies and procedures reasonably designed to prevent violations, and an AI assistant is squarely the kind of new risk those policies have to address. Amend the manual before deployment: what the tool is approved for (internal reference questions against firm documents), what it isn’t (client communications, investment advice, anything leaving the firm without human review), and who administers it. And make sure anything the firm says publicly about its AI use matches reality: the SEC has brought “AI-washing” enforcement actions against advisers whose claims outran their systems.
2. Human review of anything that leaves the building. AI output used internally as a reference is low-risk. AI-drafted text that reaches a client is your firm’s statement, full stop, subject to your fiduciary and antifraud obligations, and to the Marketing Rule (206(4)-1) whenever the final communication meets the definition of an advertisement. The clean rule of thumb: the tool drafts, a human owns.
3. Books and records (Rule 204-2). Decide in writing how chat logs are handled. Not every chat is a required record, but some will fall into required categories, and the logs are also your supervision evidence. Your procedure should cover retention period, export (application chat history is not an archive; plan for getting records out in producible form), and who can delete.
4. Safeguards (Reg S-P). The machine holds firm documents and sits on your network, so it inherits your safeguards program. Note that the amended Reg S-P, with its incident-response, customer-notification, and service-provider oversight requirements, is now in force for all RIAs (compliance dates were December 3, 2025 for larger advisers and June 3, 2026 for everyone else). Concretely: unique named accounts, no shared logins, disk encryption, no inbound exposure from the internet (remote access via your VPN only), patching on a schedule, the machine in your backup and incident-response plans, and its logins on your offboarding checklist. Local deployment shortens your vendor list (no cloud AI provider holds your data), but it doesn’t empty it: Apple, Ollama, AnythingLLM, and the model publishers are still part of your supply chain and belong in your due-diligence file.
5. Accuracy controls. Language models can be confidently wrong. Before rollout, test the system with a bank of real questions checked against the manual by the CCO, including questions the documents don’t answer, where the correct behavior is “I don’t know, ask the CCO.” Re-test when you swap models or overhaul documents, and log the testing. Under 206(4)-7 the standard isn’t that you adopted a policy; it’s that the controls work, and the test log is how you show that.
6. Document currency. The assistant is only as current as its workspace. Assign an owner (usually the CCO) to swap in revised documents and remove superseded ones whenever the manual or ADV changes, as a step in your document-revision checklist. An assistant citing last year’s policy alongside this year’s is worse than no assistant.
7. Training and attestation. A short session before rollout: what the tool is for, what it isn’t for, and the standing rule that it assists but doesn’t decide. Have employees acknowledge the AI use policy the same way they acknowledge the code of ethics.
8. Annual review. Fold the system into your 206(4)-7 annual review: Is the policy followed? What do the logs show? Did testing happen? Is the document set current? Fifteen minutes of a well-run review, but only if the pieces above exist.
The honest summary
For roughly $3,000–$5,000 in hardware (or a lease payment), free software, and a short implementation project, a firm gets an internal assistant that answers employee questions from the firm’s own documents, points to its sources, and, configured correctly, keeps every question and every document inside your walls. And remember: the operations-manual workflow is just the demonstration. The same foundation supports a growing set of genuinely complicated workflows as the firm’s comfort and governance mature. The recurring cost isn’t money. It’s the modest, ongoing discipline of the program above.
That’s also the honest dividing line. Installing Ollama and AnythingLLM is something any capable IT person can do in an afternoon. Turning that installation into a supervised, tested, examinable system (one you’d be comfortable walking an SEC examiner through) is compliance work, and it’s the part firms consistently underestimate.
If you’re weighing a deployment like this, or you suspect employees are already using public AI tools and want to get ahead of it, reach out. We’re happy to walk you through a live version of this exact workflow (your questions against a real operations manual) and pressure-test what it would look like at your firm.
—
MTradecraft LLC is an independent cybersecurity compliance consultancy in Dallas, TX, advising SEC-registered investment advisers, broker-dealers, and private fund managers. We sell no hardware and take no vendor commissions. Our recommendations are our own. Questions about this article: brian@mtradecraft.com.