Designing an Agentic Stakeholder System for Organizational Knowledge Work

Knowledge only creates value when people can use it at the point of work.

Most organizations already have plenty of knowledge. It sits in policy documents, internal platforms and the experience of colleagues. The problem starts when a project team needs an answer now. Finding the right page takes time. Applying it may still require a call with the person who knows how the rule works in practice.

My thesis looks at a practical way to reduce that dependency. The system stores approved knowledge in readable files, retrieves the relevant material for a question and follows task-specific instructions when it prepares advice or a first draft. It handles routine preparation while the stakeholder remains responsible for exceptions and decisions.

01

Why stakeholder knowledge remains difficult to use

Take a common question: “Does this GenAI idea count as an experiment or a formal use case?” The answer may exist somewhere in the organization. That does not make it easy to use. Someone still has to find the right system, check whether the guidance is current and translate a formal definition into a decision about the work in front of them.

Some of the most useful knowledge was never written down. A product owner has a preferred PBI structure. An advisor knows which risk to raise first. A stakeholder knows when the standard rule does not quite fit. This tacit knowledge is often shared in meetings or messages, then lost when the conversation ends.

The business cost is familiar: work pauses, people search and specialists repeat answers they have already given. A traditional knowledge base helps with storage. It does less to connect the stored information to the task someone is trying to finish.

02

The proposed artifact architecture

The prototype makes one clear distinction. Context files describe what the system should know. Skill files describe how recurring work should be performed. A policy change can update the knowledge without rewriting the PBI process. A better way of drafting advice can change the skill without changing the underlying facts.

context/
  ai-way-of-working.md
  monitoring-guidelines.md
  stakeholder-explanations.md

skills/
  trusted-advisor.md
  create-pbi.md

index.md

Context files hold policies, team facts and approved stakeholder explanations. Skill files hold the working method. They tell the system how to give advice, structure a PBI or keep an answer concise.

When a request arrives, the system reads an index and opens only the relevant context and skills. The user can inspect those source files next to the answer. This makes the result easier to review than a generic chatbot response based on hidden model knowledge.

Why retrieval alone is not enough

RAG gives a language model access to external documents. That solves part of the problem. It does not tell the model how a team wants a recurring task performed. It also does not keep those documents current when a stakeholder corrects the answer.

03 · Research design

An iterative Design Science Research approach

I developed the prototype through three demonstrations. Each demo tested one practical question. The outcome shaped the next version, so the final system grew from observed problems rather than from a fixed feature list.

1. 01

   Demo 1 · Grounding

   Grounded question answering

   The first version used one Markdown knowledge file. One question could be answered from this file, while a second question deliberately concerned missing knowledge. Equivalent phrasings were used to test whether the behavior remained stable.

   What happened

   The supported answer remained correct and the missing-knowledge question consistently returned “unknown” across five wording variations.

   What changed next

   The knowledge base expanded. Procedural skill files were added to support recurring work.

2. 02

   Demo 2 · Task support

   Maintained-wiki retrieval and skill-guided generation

   A maintained wiki supported information retrieval. Skill files guided advisory answers and PBI creation. Three colleagues completed each task through their normal workflow and through the prototype.

   What happened

   Across the nine timed tasks, total task time was 11:39 for the human baseline and 4:02 for the system-supported condition. The size of the benefit differed by task.

   What changed next

   The final demo tested whether corrections to content and working style could persist.

3. 03

   Demo 3 · Maintenance

   Conversational ingest and reuse

   Stakeholders corrected the system in ordinary language. It identified the relevant context or skill file and stored the lesson there. A later answer then used the updated file.

   What happened

   Across 15 controlled ingestion runs, the feedback was interpreted consistently and routed to the intended file. Unrelated files were left unchanged.

   What remained unresolved

   Uncertainty was not visible enough. Persistent updates also require explicit human approval.

The evaluation combined controlled tests with realistic work. Repeated prompts tested stable behaviour. Timed comparisons showed where the system saved effort. Stakeholder reviews checked whether the answers reflected their reasoning without taking over their authority.

04 · Evaluation results

Where the system created value

The prototype was useful when the normal task involved searching across systems or starting a structured document from scratch. It offered less advantage when an experienced colleague already knew the answer. The value depended on the work, not on the mere use of AI.

Aggregate time across the nine Demo 2 tasks. The difference between task types is more informative than the total alone.

Information retrieval reduced search and navigation effort

In the first demo, two colleagues took 75 and 154 seconds to find and interpret an organizational fact. The system answered in 16 and 8 seconds. When asked something that was not in its maintained knowledge, it returned “unknown”. It did not fill the gap with a plausible answer. That made the missing knowledge visible to the team.

PBI generation accelerated first-draft creation

PBI creation showed the clearest benefit in the second demo. The manual route involved opening Azure DevOps, finding or creating an item, reusing an earlier structure and writing acceptance criteria. The system produced a structured first draft in roughly half a minute. The draft still needed review and resizing. Even so, the colleague could start with a concrete proposal instead of an empty PBI.

Advisory support showed more limited efficiency gains

Colleagues who already knew the relevant organizational position could answer the advisory question quickly, sometimes faster than the system. Although the generated advice reached the expected recommendation, participants often considered it too long. A fast response does not save time if the recipient must shorten it before it becomes useful.

05

Keeping the knowledge base current through normal work

Knowledge bases tend to become outdated for a simple reason: maintenance is a separate task. A stakeholder answers the urgent question first. Updating the wiki comes later, if it happens at all. The useful correction remains in a meeting or chat while the formal source stays unchanged.

The ingestion loop removes that separation. When a stakeholder corrects an answer, the system treats the correction as maintenance input. It identifies the relevant context or skill file and prepares the update there. The conversation solves the immediate problem while also improving the source used for future answers.

There is no separate documentation session. The stakeholder does not need to find the right wiki page, rewrite the lesson or create a maintenance ticket. The useful input already exists in the chat. Once the proposed change has been reviewed and approved, later users benefit from it automatically.

The system also distinguishes facts from ways of working. A clearer definition of an experiment belongs in a context file. A request for shorter, compliance-focused advice belongs in an advisory skill. A preference for smaller, copy-ready PBIs belongs in the PBI creation skill.

Across 15 controlled ingestion runs, differently worded feedback reached the intended file without changing unrelated material. The next answer used the new instruction. Participants gave the maximum mean score for capture accuracy and represented reasoning. They were also comfortable with teams using the maintained answer before contacting them.

06

Control stays with the stakeholder

The system does not reproduce a person or take over an organizational role. It maintains the knowledge and working instructions needed for a limited set of tasks.

Ingestion needs a clear control point. A proposed change should show what will be removed and what will be added. The stakeholder can then approve or reject it before the knowledge becomes available to other users. Each file should retain its source, owner, date and change history.

The evidence comes from three participants in one team at a Dutch bank. The knowledge base was small and curated. The results show value for the tested tasks, not a proven organization-wide effect. A larger deployment still needs access control, ownership rules and a process for conflicting sources.

07

A practical layer between documentation and people

The system gives teams a useful first answer or draft before they need to interrupt a stakeholder.

It creates value in two ways. First, it makes approved knowledge easier to retrieve and apply. Second, it turns corrections from normal chat interactions into reusable updates. That second part is important. The knowledge base can improve as people use it, without relying on a separate round of manual documentation.

The result is not autonomous organizational decision-making. The sources remain visible. Updates require review. Missing knowledge must be stated plainly. Decisions with real consequences stay with the accountable person.

In practice, this means less time spent searching, fewer repeated explanations and a knowledge base that improves while the work is already being discussed.