Private document intelligence for a family office.

Afamily office came to us wanting its documents searchable in plain English. The office manages the family’s wealth through a private investment and management company. The documents span long agreements and contracts, investment and financial reporting, and a historical archive made up largely of scanned, image-based PDFs. The legal folder alone holds roughly 200,000 files, and everything lives in SharePoint Online, the document store the office had chosen years earlier.

The office runs on Microsoft 365. Identity starts in its on-premises Active Directory, which syncs to Microsoft Entra ID, and SharePoint grants access to those users and groups. In a family office, who can see a document matters as much as what it says.

The office’s documents record the family’s private financial affairs, and the rule was absolute: nothing in them could be sent to an outside AI provider. Leadership had already acted on it before bringing us in: a set of Mac Studios with M3 Ultra chips, a local model, and Open WebUI in front of it. Separately, the office had licensed Microsoft 365 Copilot for everyday work, and pointing it at the documents was the obvious next step. Neither was the answer, for different reasons.

Copilot is the right place to start for an office already on Microsoft 365. It works inside SharePoint and honors each user’s existing permissions, which is the hardest part of this problem, already solved. For these documents it was ruled out on policy before performance came into it. Every prompt and every retrieved passage is processed in Microsoft’s cloud, and the office drew a line between a vendor that stores its files and a vendor that reads them. A limited trial on non-sensitive files pointed the same way: answers on long agreements and dense financial reports lacked the depth and precision the work required, and the scanned archive stayed largely out of reach.

The local setup had a quieter flaw. By the office’s own estimate, it captured roughly 80% of the content in complex documents, and gave no signal about the 20% it missed. A clause on a scanned page, a table that failed to parse, an exhibit that never made it into the index: each one simply disappeared from the results, indistinguishable from a clause that was never there.

So the staff kept working the way it always had: SharePoint keyword search, browsing sites and libraries, or asking the colleague who remembered where something was filed. A typical contract lookup took about 30 minutes. SharePoint’s own OCR had indexed part of the scanned archive, but not well enough to locate a clause inside it.

Any system had to clear all four before it could be trusted with the documents.

• 01
  No outside providers.

  No document and no prompt may reach an external model provider, Microsoft included. All inference runs on the office’s own Mac Studios.

• 02
  VPN only.

  Nothing is exposed to the public internet. Microsoft Graph change notifications require a public webhook endpoint, so they cannot reach the system, and every sync has to be initiated from inside the network.

• 03
  Permissions survive the move.

  Files synced from SharePoint to local machines do not carry SharePoint’s permissions, and those permissions are layered: sites and libraries, folders and files with broken inheritance, Microsoft 365 groups, nested Entra security groups, and sharing links. All of it has to be rebuilt and enforced locally. This was the gate to prove before anything else was built.

• 04
  Privacy over speed.

  The office accepted slower answers in exchange for complete control of the data path. That trade shaped every model and hardware decision that followed.

The third constraint is the one that decides whether the rest is worth attempting. Copilot gets permissions for free because it never leaves Microsoft 365. Any system that syncs files out has to rebuild that guarantee from nothing, and on SharePoint that means rebuilding every layer of it. A system that rebuilds it imperfectly is worse than no system at all.

We deployed Chronicle, our document intelligence system, adapted to run across the office’s Mac Studios, with ingestion and serving split across them. The only outbound traffic goes to Microsoft Graph and the Entra ID sign-in endpoints, through an allowlist.

The first deployment served the office’s legal team, led by its General Counsel. Once the permission and reliability gates were proven there, the system expanded to the office’s investment and financial documents, under the same permission rules and on the same machines.

Active Directory is the single source of truth for users and groups, and Entra ID carries those identities into Microsoft 365. Open WebUI signs people in through Entra ID single sign-on, so every query arrives carrying a real directory identity, the same one each person already uses for Microsoft 365.

A harvester walks every SharePoint site and document library in scope through Microsoft Graph and reads the effective permissions on each file, including the folders and files where inheritance has been broken. Group grants are expanded to their members, Microsoft 365 groups and nested Entra security groups alike. Sharing links are read conservatively: a link to named people maps to those people, and a link open to the whole organization or to anyone is not treated as an access grant. Files carrying the office’s most restrictive sensitivity label are excluded from the index entirely. Every chunk is stamped with the list of principals allowed to see it, and those stamps decide both what a person can search and what the model is allowed to draw on when it answers.

Filtering happens inside the index query, before ranking, so restricted content never enters the model’s context in the first place. Changes are picked up by polling Microsoft Graph delta queries on a saved token, sharing changes included, which suits a network with no inbound access. A nightly full reconciliation catches anything the polling missed. As a second check, top results can be re-verified live against Graph before an answer is generated.

This layer has its own test suite: four test accounts and a set of questions whose only correct answer is nothing at all. One account reaches a folder where inheritance has been broken; another has access only through a nested group.

The ingestion machine downloads files through Microsoft Graph into a local staging area. SharePoint remains the system of record; nothing about the source of truth changes.

Every file is then entered in a coverage ledger with a recorded status and a reason. Nothing is dropped silently.

Digital PDFs and Office files go through native text extraction. Scans go through Apple Vision OCR, which runs quickly on the M3 Ultra. Low-confidence pages and complex tables are routed to a local vision-language model for a second pass.

Chunking follows the structure of legal documents rather than a fixed token count: articles, sections, clauses, defined terms, and exhibits. Each chunk carries its document type, parties, dates, and its SharePoint site, library, and path. The embedding model was selected by evaluation against the office’s own documents, not by benchmark ranking.

The serving machine holds a hybrid index: dense vectors alongside BM25 keyword search, with permission metadata on every chunk. Keyword search earns its place in legal text, where party names and clause numbers carry exact meaning that embeddings blur. Each query runs the same sequence: permission filter, hybrid search, then a local cross-encoder reranker.

Generation runs on a large open-weight mixture-of-experts model, quantized to fit within a single machine’s 256 GB of memory and chosen by benchmarking answer quality against speed. Open WebUI remains the interface. Its built-in retrieval is replaced by a custom pipeline that calls the Chronicle retrieval service.

We built a golden question set with the legal staff, including questions where the correct behavior is to refuse. Retrieval is scored separately from answer quality, and a person grades the answers rather than the system grading itself. The full evaluation reruns on every change to models, chunking, or OCR settings.

Graph access runs through a dedicated Entra app registration with read-only rights, scoped to the sites in the index, which started with the legal sites and widened as the rollout did. Audit logs record each query, the user who asked it, and the documents retrieved. Models and packages are installed through approved manual updates. The office’s IT team has runbooks, a health dashboard covering sync lag, ledger failures, and permission drift, and backups of both the index and the ledger.

The machines are split by role, and each can cover another’s role in a degraded mode. No machine is a single point of failure.

99% capture on complex documents, contract lookups under two minutes, and no permission leaks in testing.

The coverage figure was measured by running the old and new pipelines on the same set of difficult documents and having a person compare the output. The improvement in capture matters, but the ledger matters as much: the remaining 1% is no longer invisible. Every file that could not be fully processed is listed with a reason, so the staff knows exactly what the system has read and what it has not.

The drop in lookup time follows from that. A question that once meant half an hour of browsing and asking around now returns a cited answer in under two minutes, including questions about scanned documents that keyword search had never been able to answer.

The permission result is what made the rest usable. Every question designed to cross a permission boundary returned nothing, across all four test accounts. That is what allowed the General Counsel to put the system in front of the legal team, and then to extend it to the office’s investment and financial documents under the same rules.

The documents had always been in the office’s SharePoint. Everything in the index is now within reach of the people entitled to see it.