Models whose weights can be downloaded have progressed to the point of making private assistants, document processing, code, and certain multimodal interactions credible without sending all the data to a public API. At the same time, quantization and runtimes allow smaller models to run on compact servers, workstations, and sometimes edge devices.
This development does not mean that any model can be installed on a laptop or that self-hosting is automatically cheaper. One must analyze license, quality, memory, throughput, security, updates, and operation.
Models, licenses, and offerings are evolving rapidly; any reference must be revalidated before publication or architecture selection.
"Open" can refer to several realities
A model can publish:
- its weights;- his inference code;- his training recipe;- a part of the data;- a license authorizing certain uses;- only an API.
The terms "open source," "open weights," and "available for download" are not interchangeable. The license must be read for commercial use, redistribution, fine-tuning, restricted sectors, and attribution obligations.
A proper template sheet describes architecture, limits, languages, assessments, and terms of use. It does not exempt from internal evaluation.
Why self-host
The possible motivations are:
- sensitive data;- offline operation;- local latency;- stable cost at high volume;- personalization;- version control;- reversibility;- industrial or geographical requirements.
These benefits come at a price: GPU capacity, operations team, monitoring, security, updates, and peak management.
The most realistic use cases
Classification and extraction
A specialized compact model can classify, normalize, or extract fields with a constrained format. The quality is evaluated on a business corpus.
Private RAG
The model generates from an internal index. The embeddings, the search, and the model can all be deployed in a controlled environment.
Code assistance
A local model can complete, explain, or transform code without sending the repository to a third party. Long agentic tasks require more capacity and sandboxing.
Edge processing
Visual inspection, transcription, detection, or assistance can operate close to the source to reduce latency and maintain a network-free service.
Degraded mode
An application can use a local model for essential functions and a more capable API for complex cases when connectivity and policy allow.
Size without relying solely on the number of parameters
The memory footprint depends on precision. As a rough guide, a 16-bit parameter uses two bytes before overhead; a 4-bit parameter, about half a byte. The runtime, buffers, attention cache, and concurrency add significant consumption.
The context length increases the KV cache. The throughput depends on the memory bandwidth, the architecture, and the batch. Two models of the same size can have very different profiles.
Qualification, quantification, deployment, monitoring, and updating chain of an open model.
Quantification: compromise, not free compression
Quantization reduces memory and can speed up inference. It can also degrade certain tasks, languages, computations, or structured outputs. The impact varies depending on the model and the method.
It is necessary to compare the target version on the evaluation set, with the same prompts and parameters. Aggressive quantization may remain acceptable for classification and insufficient for precise synthesis.
The quantified artifacts have a verified provenance, footprint, and license.
CPU, GPU and edge accelerators
The CPU may be sufficient for a low volume or a compact model, with higher latency. The GPU provides throughput and response time, but creates a memory and operational constraint. Mobile or industrial accelerators often require specific formats and operators.
The benchmark must be carried out on the target hardware. Public figures use batches, contexts, and optimizations that do not always correspond to actual usage.
Edge AI: bringing computation closer to the data
Edge is relevant when:
- The network is intermittent;- the latency must be very low;- the data must not leave the site;- The gross volume is too high;- A local decision must continue during a failure.
The architecture can preprocess locally and then synchronize results. Models and policies are signed, versioned, and deployed in waves.
An edge device has thermal, energy, and storage constraints. Critical functions must retain a deterministic fallback when the model is unavailable.
Supply chain security
Downloading weights and a runtime adds external artifacts. It is necessary to check:
- source ;- imprint;- format ;- code executed;- dependencies;- vulnerabilities;- license;- network behavior;- update model.
Formats capable of executing code are handled with caution. Conversion is done in an isolated environment.
Operational safety
The inference service is authenticated, limited, and segmented. Prompts, outputs, and caches may contain sensitive data. Logs minimize them and access is audited.
Self-hosting does not eliminate prompt injection, exfiltration, or tool abuse. It gives more control over the components, provided they are secured.
Update and rollback
A new version of a model can improve a benchmark and change rejections, formats, or uses of language. It goes through:
- license qualification;2. scan and conversion;3. offline evaluation;4. load test;5. canary ;6. observation;7. promotion;8. possibility of return.
Edge devices can be offline. The update mechanism must handle resume, signing, disk space, and minimum version.
Full cost
Compare:
- purchase or rental of capacity;- utilization rate;- energy;- cooling;- storage;- network;- support ;- security;- engineering;- renewal;- peak capacity.
An API can remain more cost-effective for a low and irregular volume. A private capacity becomes interesting when usage is stable, the data is sensitive, or control is strategic.
Hybrid architecture
A gateway can route according to sensitivity, complexity, and availability. A small local model handles standard cases; an external model takes on authorized complex tasks. Evaluations ensure that routing does not silently degrade quality.
The format of prompts, tools, and outputs is sufficiently portable to switch models, while retaining explicit differences.
Evaluate an open solution
The protocol must measure:
- accuracy;- language;- format ;- security;- latency to the first token;- flow;- memory;- energy;- stability;- cost per successful task.
Long, ambiguous, and adverse cases are included. The public benchmark does not replace this campaign.
Model governance
The registry contains name, version, license, origin, authorized uses, owners, evaluation results, environments, and review date. An unapproved model cannot be deployed on sensitive data.
Abandoned or vulnerable models are withdrawn with a migration plan.
A new architecture option
Open models and the edge expand choices. They make more local, controlled, and resilient AI possible, but require real product and operational engineering.
Partitech can compare the models, build the benchmark, size the infrastructure, integrate the runtimes, and set up the versioning and monitoring chain. The goal is an open deployment justified by need, not by a promise of free access.
Let's talk about your project
Evaluate and prototype an open or edge AI architecture with Partitech.Contact Partitech.