Introduction
Archi—AI Augmented Research Chat Intelligence—is an open-source, configurable framework for building AI assistants for research and education. It combines retrieval over an organization’s documents and tickets with agents that can call external tools, and packages those capabilities into independently deployable services such as a chatbot, data manager, and ticketing integrations.
Within CERN, Archi is currently being adapted for the CMS Computing Operations (CompOps) team. CompOps handles operational questions and incidents across a large and constantly changing computing infrastructure. The Archi deployment brings together internal documentation, historical support material, JIRA tickets, and tools that can query live systems. This gives operators and users a conversational way to find relevant information, investigate problems, and prepare actionable answers with supporting evidence.
My GSoC work focused on the next step in that deployment: bringing Archi into the support workflow itself, then building a rigorous way to measure whether its answers are correct and useful there.
Work summary
The project produced two connected capabilities. The first allows Archi to participate directly in JIRA support work. The second makes that automation measurable against a curated golden dataset, including questions whose answers depend on changing operational data.
Automatic JIRA ticket responses
A standalone service polls selected projects, asks Archi to answer eligible tickets or mentions, and publishes the result through the JIRA API.
Evaluation platform
A shared CLI and browser console allow to evaluate Archi over a set of “golden” questions written by operators, run repeatable evaluations, inspect evidence, and score answers claim by claim.
The implementation is represented by four main contributions:
- JIRA service for autonomous ticket answers (PR #573)
- Responses to JIRA mentions with ticket context (PR #586)
- Atom-based evaluation CLI and browser console (PR #596)
- Evaluation of answers backed by live data (PR #608)
JIRA ticket responding feature
Why it was needed
Before this work, a user could ask the Archi chatbot an operational question, but answering a support ticket still required a person to move between the ticket and the chatbot. That extra step limits the value of the assistant for repetitive questions and makes it harder to integrate Archi naturally into the team’s existing support process.
The goal was therefore not to build another chat interface, but to make Archi available where support conversations already happen. A JIRA integration can reduce the delay before a useful first answer, apply the same operational knowledge consistently, and give CompOps members a concrete response that they can review or build upon.
How it was implemented
The responder was implemented as a service separate from the chatbot. This boundary keeps the JIRA polling lifecycle and credentials independent from interactive chat, prevents either workload from blocking the other, and allows a deployment to run the ticket responder on its own.
The service authenticates with a personal access token belonging to a dedicated Archi service account. At a configurable interval, it uses the JIRA API to find recently updated tickets in selected projects and statuses. For each eligible trigger, it assembles the issue title, description, and relevant discussion context, runs the configured Archi agent, and posts the answer back as a JIRA comment.
A small persistent state table records which issues and mention comments have already been handled. This avoids duplicate answers across polling cycles and normal service restarts. The follow-up mention workflow lets users explicitly call Archi from a ticket comment; it ignores the service account’s own messages and can restrict eligible mention authors by JIRA project role. These controls keep the behavior predictable without coupling the responder to the chatbot application.
The main architectural choice was to treat JIRA answering as an independent operational service, while reusing the same Archi agent and knowledge capabilities available elsewhere.
For CompOps, this provides a faster first response to common questions and a consistent route from the team’s knowledge base to its ticket queue. It also creates the real-world workload that motivated the second half of the project: determining, over time, whether those automatic answers are actually good enough.
Evaluation process for static and live tickets
Why an evaluation platform was needed
An answer can sound convincing while omitting one critical condition, contradicting a documented value, or failing intermittently. Those failures matter in an operations setting. A useful evaluation therefore has to measure more than general textual similarity: it must identify the individual facts or actions that a correct answer is expected to contain.
The project also had a second complication. Some CompOps questions have stable answers that can be recorded once in a golden dataset, while others ask for live state—for example, the current size of a data container or which file contains a particular run and luminosity section. Treating a months-old value as permanent ground truth would evaluate Archi against an answer that is no longer true.
Atom-based evaluation
I designed the evaluation around gold atoms: small, independently judgeable obligations extracted from a canonical answer. An atom represents one fact, qualification, or required action. It can be marked required, when omitting it makes the whole answer fail, or optional, when it adds useful information without being essential.
The workflow has three shared phases:
- Prepare: validate the dataset and turn each canonical answer into a fixed, reviewable set of gold atoms.
- Run: ask a fresh Archi agent each question for one or more independent attempts, without revealing the answer or atoms.
- Score: classify every atom as entailed, not mentioned, or contradicted, then aggregate the results.
This decomposition makes a result explainable. Instead of receiving only a single opaque score, a reviewer can see exactly which required claims were supported, missing, or contradicted. Repeating questions also exposes answer instability that would be hidden by a single successful run.
The main outputs include the overall pass rate, atom score, required-atom recall, per-question and per-atom results, lifecycle counts, and execution latency. The system also preserves the tested agent configuration, answers, judgments, tool traces, and integrity-checked artifacts needed to understand or reproduce a run.
CLI, jobs, and the browser console
The evaluation engine is available through the archi eval qa CLI command suite for local experiments, automation, and staged runs. The complete prepare–run–score lifecycle can be executed as one command or phase by phase when the dataset or intermediate answers need inspection.
The browser console exposes the same validation, runtime, scoring logic, and artifacts as the CLI. It provides persistent catalogs for datasets and evaluator profiles, an interface for generating and manually editing atoms, and immutable reviewed dataset versions. Evaluation jobs run in the background, can use parallel workers, and record progress and failure states instead of tying the work to a browser request.
From the dashboard, an operator can:
- import and review a golden dataset;
- generate or edit the atoms associated with each answer;
- select an agent specification, evaluator profile, and number of attempts;
- start, monitor, cancel, or retry evaluation jobs;
- compare aggregate metrics and latency graphs;
- inspect individual answers, atom judgments, reasoning evidence, and tool traces;
- return to persistent run history after the job has completed.
This shared architecture was important: the console is not a second evaluator with subtly different behavior. It is a human-facing way to operate and inspect the same workflow used by the CLI.
The reason behind these design choices is that I wanted to keep the whole evaluation logic as simple as possible. Because the main need was to produce traceable artifacts each and every run, results are not stored in a relational database but rather as standalone files signed with a hash. This allowed the console to reuse most of the logic of the CLI without engineering a whole different set of classes and code to maintain.
Evaluating questions that depend on live data
The most challenging part of the project was extending this model to time-sensitive questions. For these items, the golden dataset stores a controlled oracle recipe rather than a permanent answer. The recipe tells the evaluator how to query a trusted data source through evaluator-only MCP tools and which fields from the response define the canonical answer.
At preparation time, the evaluator runs the recipe, normalizes the selected data, records bounded evidence and provenance, and derives atoms from the resulting current answer. This creates a reviewed snapshot without exposing the oracle, canonical answer, or gold atoms to the Archi agent being tested. The evaluator’s tool configuration is deliberately separate from the tested agent’s tools so that evaluation knowledge cannot leak into the answer.
Live truth may change during a long evaluation. To distinguish that situation from an agent failure, the workflow checks the oracle before agent execution and again after all attempts have completed. If the value changed or the source became unavailable, the affected result enters an attention_required state rather than being scored against stale truth. Reviewers can then refresh the snapshot, continue with unaffected questions, or retry after the dependency recovers.
This mechanism makes it possible to evaluate both halves of a realistic CompOps answer set:
- Static tickets, whose expected answers come from stable procedures or documentation.
- Live tickets, whose expected answers must be materialized from systems such as Rucio or DBS at evaluation time.
Together, the atom model and live-data checks turn the golden dataset into more than a list of example answers. It becomes a reviewable and reproducible contract for what Archi must know, which parts of each answer are essential, and which evidence was current when a result was produced.
Evaluation Results
During the summer, a debate arised on whether Archi should keep using LangChain integrated tools or move to an MCP-support based deployment. On 22 August 2026, I ran a controlled comparison to decide whether Archi should move from its original LangChain-tool architecture to an MCP-based one. The old deployment used the built-in vector-store and catalog search tools, together with MONIT searches for operational data. The new deployment replaced document retrieval with the CMS knowledge graph (OKG) and exposed external capabilities through MCP, including a standalone Rucio MCP and DBS tools.
Both runs used the same reviewed golden dataset created for CompOps of 22 static and live questions (currently expanding), four independent attempts per question, the same resolved agent configuration, and the same evaluator profile. All 88 attempts in each run were scored successfully, and all six pre- and post-run oracle data reliability checks succeeded. The only intentional difference captured by the artifacts is the agent specification and its available tool architecture.
| Metric | Old: LangChain tools | New: MCP and knowledge graph | Change |
|---|---|---|---|
| Attempts passing every required atom | 32/88 (36.4%) | 55/88 (62.5%) | +26.1 percentage points |
| Contradiction-sensitive atom score | 41.4% | 61.4% | +20.0 percentage points |
| Required-atom recall | 46.6% | 72.8% | +26.1 percentage points |
| Entailed atom judgments | 132/336 (39.3%) | 189/336 (56.3%) | +57 judgments |
| Contradicted atom judgments | 57/336 (17.0%) | 30/336 (8.9%) | -27 judgments |
| Technical failures | 0/88 | 0/88 | no change |
| Mean attempt latency | 108.1 s | 85.3 s | -21.0% |
| Median attempt latency | 101.5 s | 84.0 s | -17.3% |
Some examples of gains that came from switching deployment, which resulted in the model being able to perform better with MCP tools and OKG:
- The old MONIT-based tools failed to locate requested tape replicas in some questions. The new Rucio and DBS MCP calls consistently identified the object as a 17-dataset container, returned its size, and found the correct tape; all eight attempts across those two questions changed from fail to pass.
- Document and event retrieval returned plausible files for the run but was less consistent in locating exactly the files containing the requested data. Direct DBS MCP queries returned the exact file in all attempts, which means the model was able to use those tools better.
- For a question asking for the largest tape endpoint, static retrieval chose the wrong tape in three of four attempts, failing the correct answer because the vector search results were inconsistent and bloated. Live Rucio capacity queries consistently returned the correct one, improving the question to 100% pass rate.
The questions that did not pass are equally useful for deciding what to build next. A query about deleted Rucio rules failed in both deployments: the MONIT searches did not find the required matching history, while the current Rucio MCP could return active rules and locks but did not expose the deleted-rule history requested by the question. Several static questions were close but failed some of the required atoms. Answers about site readiness, long-running replication rules, incomplete DAS blocks, and the RSE capacity script often explained the main procedure but omitted one required URL, edge case, exact command flag, or provenance detail. This is a completeness and source-coverage problem rather than an execution failure.
Overall, the experiment supports the architectural shift. The MCP-based deployment produced substantially more complete and less contradictory answers, eliminated the need to infer current Rucio and DBS state from MONIT events, and was faster on average without reducing run reliability.
Improvements and future work
The immediate next step is to expand and maintain the golden dataset with CompOps domain experts. The quality of the evaluation ultimately depends on representative questions, reviewed canonical answers, and atoms that reflect what operators consider essential. Periodic human review will also help measure evaluator agreement and identify ambiguous questions rather than encoding that ambiguity as an Archi failure.
Further improvements include:
- scheduling regular regression runs and notifying maintainers when a model, prompt, tool, or knowledge update reduces quality;
- Implement a solution to evaluate Archi’s tool usage and expected retrieved content, to tell if the agent failed due to the tool itself or its improper usage.
- adding first-class side-by-side comparison views for agent configurations and tracking metrics across releases;
- strengthening the responder for larger or distributed deployments, including a standard database migration path and coordinated trigger ownership;
- using evaluation findings to prioritize documentation gaps as well as agent improvements.
The two project areas are intended to reinforce each other. JIRA integration gives Archi a concrete operational role, while the evaluation platform provides the feedback loop needed to deploy that role responsibly. The long-term goal is not simply to automate more answers, but to make their quality visible, explainable, and continuously improvable.