Plausible is not exact
A language model predicts the most probable text. It verifies nothing. As soon as several rules have to be chained, its error rate rises sharply.
Research · The Jev model
Letting AI make business decisions, and being able to verify them
At the heart of ProSuite, the VAILS server and its VVL language bring together two families of artificial intelligence: language models, which understand text, and symbolic reasoning, which applies rules exactly and verifiably. This page presents our modelling choices and our first measurements.
60%correct answers for a language model alone when six rules have to be chained
100%for the VAILS symbolic engine on the same facts
≈ 0.4 sper decision settled by the Jev decision-maker
2,800questions in our benchmark
The problem
Why a language model alone is not enough to run a business
Entrusting part of management to AI means entrusting it with thousands of small decisions every day: sorting a request, qualifying a customer, checking a document, applying a price. A large language model, used on its own, is not designed for this.
No degree of certainty
It returns an answer, never a usable probability. Yet that is exactly what a process needs to decide on its own, request a check or hand over to a person.
A cost per decision
Every sort or check becomes a slow, billed call. At the scale of a business, latency and cost become the main obstacle.
Invisible rules
When the business rule is hidden in an instruction written in natural language, no one can audit it, version it or prove that it was applied.
Our principle
The model proposes, the symbolic layer decides
Neuro-symbolic AI combines a neural network, which provides language understanding, with a symbolic layer, which provides logic, structure and proof. In VAILS, this combination follows four layers.
- 01Understand
The language model reads the particular case: a request, a quote, a CV or an invoice is transcribed into the company's schema. The model transcribes; it does not reason.
- 02Model
The ontology carries the general knowledge: categories, rules, exceptions and reference profiles, written once and compiled into logical rules.
- 03Decide
Rule, then Jev, then language model, then human: each decision goes to the simplest engine able to make it with enough confidence.
- 04Verify and trace
Contracts, proofs and a decision log: which engine decided, with what confidence, on which rule.
The language
Meaning as a value of the VVL language
In most tools, calls to AI are added on top of a conventional programming language. In VAILS, it is the language itself, VVL, that knows how to handle meaning.
By default, a VVL value behaves like an ordinary value: a text, a number, a list. The program remains deterministic and fast. When a value is marked as semantic, the usual operators start judging by meaning: equality becomes equivalence of meaning, membership becomes inclusion of concepts, ordering becomes a comparison of rank.
This separation is an essential design choice: the unpredictable part of the system stays small, visible and localised. VVL has 21 operators of this kind, an instruction that chooses a branch by meaning, decision tables, projections onto an ordered scale and fuzzy-logic degrees measured on a text.
[semanticMode "jev"]
[print ([§ "a baby penguin"] == [§ "a young penguin"])] # 1
[print ([§ "a salmon"] in [§ "freshwater and saltwater fish"])] # 1
[print ([§ "a colonel"] < [§ "a captain"])] # 0Jev
A fast decision-maker whose confidence is measured
Jev is a decision model that does not write: it decides. It is given a text and typed questions of three forms: yes or no, a choice among options, a position on a scale. It answers in a single pass, with a calibrated probability for each answer.
We use it as what cognitive psychology calls a System 1: a fast and economical mode of decision. The large language model plays the role of System 2, slower and more costly, called upon only when Jev hesitates.
Let p be the probability given by Jev and τ the threshold chosen by the business, 0.8 by default.
If p ≥ τ, the answer is yes. If p ≤ 1 − τ, the answer is no. In between, the question goes to the language model, then to a person if doubt persists.
The threshold thus becomes a management parameter: the higher it is, the more delicate cases are escalated to the language model or to a person. Several questions about the same text are sent in a single call. Each decision is logged with the engine that made it, its probability and its response time.
| Time for one Jev decision | 0.35 to 0.45 s, of which 2 ms is VAILS processing |
|---|---|
| Classification of questions put to the router | 16 out of 16 correct, confidence ≥ 0.99 |
| Reference routing scenarios | 11 out of 13 decided by Jev, all correct; the 2 deliberately ambiguous cases left to the language model |
| End-of-task judge for an agent | 10 out of 10, about 0.4 s |
| Three decisions on the same ticket, preloaded | 1 call instead of 3, 0.5 s instead of 1.1 s |
Tuning measurements on small samples. They will be extended to several hundred cases per type of decision.
Model rather than guess
Every piece of knowledge has its place
The company's general knowledge is written in an ontology; the language model only transcribes the particular case into that ontology; reasoning is entrusted to the symbolic engine.
Ontologies compiled into logic
A VAILS ontology describes classes, typed attributes and relations. It is automatically translated into logical clauses, executed by a Prolog-style engine built into the server. Inheritance becomes a rule: anything that is a building-works quote is also a quote. A relation declared transitive, such as “is located in”, is computed by a closure that remembers the elements already visited, which guarantees that it terminates even when the data contain cycles. Exception rules are written under the closed world assumption: whatever is neither stated nor deducible is considered false.
rel(X, r, Z) :- tc(X, r, Z, [X]).
tc(X, r, Z, _) :- dedge(X, r, Z).
tc(X, r, Z, V) :- dedge(X, r, Y), not(member(Y, V)), tc(Y, r, Z, [Y|V]).Reference profiles: abstract objects
To represent a profile, an archetype or a standard, VAILS relies on the theory of abstract objects of the logician Edward Zalta. An abstract object encodes the properties that define it. Comparing a candidate, a customer or a file with a profile gives a degree of satisfaction between 0 and 1: the share of the encoded properties that the instance possesses.
[semanticMode "jev"]
$rh = [ontology:define rh
[ontology:class Candidat [attributes
[nom String] [competence String] [remote String] [niveau String]]]
[ontology:abstract ProfilDevConfirme : Candidat
[encode competence "ingenieure logicielle"]
[encode remote "teletravail complet"]
[encode niveau "confirmee"]]]
$bob = [new Candidat [nom "Bob"] [competence "boulanger"]
[remote "sur site uniquement"] [niveau "apprenti"]]
[print ($bob ~~ "ProfilDevConfirme")] # ~0.04Contracts and proofs
Borrowed from the software engineering of critical systems, a contract specifies what must be true before and after an operation. An ontological contract accepts an extraction from the language model only if it satisfies the ontology above a threshold; otherwise, the extraction is rerun with the error found, then cleanly rejected. For propositional logic, the model may propose a proof, but it is a small verifying kernel that accepts or rejects it.
We are precise about what this brings: a contract does not make the model infallible. It guarantees that whatever gets through has been checked against explicit rules.
Measuring
Our benchmark
To compare a language model alone with neuro-symbolic modelling rigorously, we built a generator of small worlds: facts, rules, distractors and a question whose answer is yes or no. The names are invented, so that no prior knowledge of the model can help it.
VAILS engine, exact factsModel alone, step by stepModel alone, direct answer
- Three families of reasoning: transitive relations with or without cycles, membership of nested categories, chains of rules with or without exceptions.
- A depth k from 1 to 6: the number of sentences that must be chained to answer.
- Reference answers computed by a reasoner independent of VAILS, then cross-checked by a second computation.
- The same language model, gpt-4.1, with temperature zero, under all conditions.
| k | Model alone, direct | Model alone, step by step | VAILS engine |
|---|---|---|---|
| 1 | 97% | 99% | 100% |
| 2 | 87% | 95% | 100% |
| 3 | 75% | 88% | 100% |
| 4 | 69% | 86% | 100% |
| 5 | 64% | 84% | 100% |
| 6 | 60% | 82% | 100% |
| Overall | 74% | 88% | 100% |
The weak point of the model alone
Exception rules: even when reasoning step by step, it answers only 72% of these questions correctly. It cites the right sentences, then concludes that it cannot be known, instead of applying the closed world assumption stated in the text.
The complete chain
The language model extracts the facts, VAILS reasons: a first pilot of 52 questions gives 98% correct answers, against 65% and 87% for the model alone on the same questions. All remaining errors come from extraction and are measured separately. The full campaign is under way.
What we learned
When general knowledge is left in the text passed to the extraction model, the model starts reasoning in place of the engine. Told that a relation is transitive, our extractor produced 369 unrequested deductions over 52 cases, despite an explicit instruction to deduce nothing. Removing this knowledge from the text and placing it in the ontology made the problem disappear. The same benchmark, replayed in simultaneous sessions, also enabled us to detect and fix a session isolation defect in the server: measuring is also a way of making things reliable.
For management
What this changes for the business
A management rule written in VVL remains readable by the business. Here is a customer request router: three typed decisions, entrusted to Jev in a single call, and an escalation branch to a person when nothing matches.
[semanticMode "jev"]
$route = [match [§ $ticket]
`[[§ "urgent, time-critical"] [§ "Payments, invoicing, refunds"]] "on-call billing"
`[[§ "urgent, time-critical"] [§ "Bugs, outages, integrations"]] "on-call technical"
`[_ ?team] ("queue " + $__team)
"human escalation"]Customer relations
Request routing, urgency, customer mood, spam and fraud detection, escalation.
Documents
Schema-guided extraction, classification, consistency checks before any processing.
Quotes and prices
Exact application of pricing rules modelled in an ontology, across hundreds of quotes.
Qualification
Comparison of candidates, customers or files with reference profiles, with a degree of satisfaction.
Transparency
Limitations and work in progress
We prefer to say precisely where we stand.
- Sovereignty: the language model is currently called from an external provider, and Jev is served by Cloudflare. We are working towards running both of these stages within the customer's perimeter.
- End-to-end traceability: every decision is logged, but the complete path, from the source to the conclusion and the rule that justifies it, has yet to be assembled.
- Broader measurements: the full campaign for the neuro-symbolic chain and larger samples for Jev are under way.
- Explicit failures: when an external engine does not respond, the decision must fail visibly rather than return a misleading score.
References
- Kahneman, D. Thinking, Fast and Slow, 2011.
- Zalta, E. N. Abstract Objects: An Introduction to Axiomatic Metaphysics, 1983.
- Meyer, B. Applying Design by Contract, IEEE Computer, 1992.
- Garcez, A. d'A. and Lamb, L. C. Neurosymbolic AI: the 3rd wave, Artificial Intelligence Review, 2023.
- Dinu, M.-C. et al. SymbolicAI: a framework for logic-based approaches combining generative models and solvers, CoLLAs 2024.
- Regulation (EU) 2024/1689 on artificial intelligence, Articles 12 to 14.
Putting it into practice
Apply this work to your processes?
These tools are instructions of the VVL language: our partners implement them for their clients, on the ProSuite platforms.