From a 300-Advisor Pilot to 98% Team Adoption: Morgan Stanley’s Five Steps to Put AI into Production

The first gate in enterprise AI deployment is for management to identify a business use case or problem worth validating, then use a demo to turn it into a workflow that all stakeholders can observe together. In March 2023, Morgan Stanley announced that it was building an internal AI assistant for its Wealth Management business using GPT-4, with the goal of giving advisor teams internal knowledge-retrieval capabilities. The use case described in Key Milestone in Innovation Journey with OpenAI was unusually specific: the target users were Financial Advisors, and the target task was to help them find relevant content more quickly across Morgan Stanley’s internal research, product, and process materials. AI provided retrieval and content assistance, while Financial Advisors retained professional judgment and responsibility in client-facing work.

The starting point for this AI use case looks simple, but it reduced three common uncertainties before the project moved forward. First, the users and pain point were clear: Financial Advisors needed to search a large internal knowledge base every day. Second, the knowledge boundary was clear: the system retrieved and generated only from firm-approved internal content. Third, responsibility for the output was clear: answers preserved links to source documents, while Financial Advisors reviewed the results and decided how to use them. Who would use the system, what information the AI could rely on, and who would remain accountable for the result could all be stated clearly at project launch.

Morgan Stanley’s choice also echoes the conclusion we proposed in Issue 1, The Real Inflection Point for Enterprise AI: Governance and Collaboration Become the New Constraints After Standardized Workflows Reach Production: standardized operating processes with tight constraints, strong auditability, and clear metrics tend to enter production earlier. Here, “standardized” includes both established SOPs and knowledge-based tasks where users, inputs, outputs, and responsibility boundaries can be clearly described. For enterprises, usage frequency, the impact of errors, and initial value metrics should also be outlined at this stage, because they determine whether the project is worth taking into technical validation.

Once the use case has been selected, the demo needs to turn those judgments into a workflow that all parties can observe together. Public materials do not disclose the specific format, duration, or formal name of Morgan Stanley’s internal demo, but the functionality described at project launch already reveals a minimum task chain:

Advisor asks a question → System retrieves controlled internal content → System generates an answer with source references → Advisor reviews the answer and decides how to use it

Through this task chain, the business team can confirm whether the system addresses a real source of friction, the technical and compliance teams can confirm data sources and access scope, and management can decide whether the project should move into the next round of validation. A smooth demo still cannot answer questions about stability under complex inputs, critical error types, or acceptable red lines. At this stage, the value of the demo is to give all parties a shared understanding of the same core workflow and a basis for deciding whether to proceed to PoC.

The core output of Stage One should therefore be a “use case card” that supports management decision-making. It should include the use case and accountable owner, data boundaries, expected outputs and review method, initial value indicators, and questions that still need validation. At this point, value indicators can remain expected values—for example, whether document-search time can be reduced, internal knowledge coverage expanded, or answer traceability improved. These expectations will then be tested against real evidence in later stages.

Stage One leaves behind a clearly defined business use case and a minimum workflow that can be demonstrated. These outputs tell the enterprise what problem the AI is intended to solve. Once the project enters PoC, management needs to answer a second question: can the AI pass a technical feasibility test for the core task under standards accepted by business experts? For Morgan Stanley, this meant that the internal AI assistant needed to perform knowledge retrieval, content synthesis, and answer generation on controlled business samples, while allowing Financial Advisors to judge whether the results were accurate, relevant, and coherent.

In its collaboration story, Morgan Stanley Uses AI Evals to Shape the Future of Financial Services, OpenAI disclosed that Morgan Stanley tests every AI use case before deployment and measures model performance against real-world business tasks. Public materials do not explicitly call this process a PoC, but its role is clear: it uses a structured evaluation framework to determine whether an AI capability can pass pre-deployment technical validation.

Business experts provide the most important reference point in the PoC. Financial Advisors know the internal materials, what information a qualified answer should cite, which caveats need to be preserved, and which errors could affect subsequent client communication. Their judgment can create a human baseline: for the same question, what answer would a professional typically give, and how far is the AI still from that baseline? The technical team can then adjust the system around concrete gaps, while management can see what evidence underpins technical feasibility.

Failure cases that appear during evaluation are equally important. Morgan Stanley reviews the questions entered and the outputs produced, then works with OpenAI on how to adjust prompts and retrieval methods to achieve the level of accuracy the firm requires. Within a complete PoC process, that iteration path can be summarized as:

Input real business questions → Inspect retrieved content and generated results → Experts score outputs and record failure cases → Adjust prompts or retrieval methods → Retest on comparable tasks

Through this cycle, the team can further identify where failures occur: the system may fail to retrieve the correct material, omit critical conditions, summarize content incorrectly, or introduce information that is not present in the internal knowledge base. Failure cases help the enterprise determine whether the project still has room for correction and create a record of risks that should be watched closely during the subsequent pilot.

At the end of the PoC, management should therefore receive a “technical validation package” that supports a go/no-go decision. It should include a representative test set covering common questions, complex questions, and high-risk edge cases; a human baseline and scoring rubric provided by business experts; key metrics such as accuracy, relevance, completeness, and traceability; and the failure cases and unresolved issues identified during testing. Morgan Stanley’s public materials confirm expert participation, evaluation dimensions, and iteration methods, but parameters such as exact sample size, formal passing scores, and error thresholds need to be set by each enterprise according to its own use case and risk level.

This analysis also connects with the “AI accountability gap” discussed in Issue 3, Enterprise AI Enters the Operational Asset Management Stage: From Deployment Expansion to Value Measurement and Dynamic Trade-Offs. Enterprises often struggle to measure AI after usage expands because they did not establish reusable standards and feedback mechanisms early enough. Moving this work forward into the PoC allows subsequent pilots and production operations to continue accumulating evidence against the same set of metrics.

After technical validation, Morgan Stanley first placed the internal AI assistant with approximately 300 Financial Advisors for trial use. According to Morgan Stanley Is Testing OpenAI’s Chatbot Technology, which relayed CNBC reporting, the firm planned to expand the system more broadly after this trial period. For management, this stage needs to answer one question: can the AI operate in a controlled real-world environment?

Those 300 advisors exposed the system to the real distribution of questions that appear in day-to-day work. Questions used in a PoC can be prepared in advance; questions in a pilot emerge from the client needs, market changes, and internal research tasks advisors face every day. The same knowledge base encounters different ways of asking questions, different levels of information depth, and different time constraints. Advisors also decide, based on answer quality, whether to ask follow-up questions, return to source documents, consult colleagues, or abandon the tool altogether. The limited pilot therefore begins to test whether the system can fit into advisors’ existing work habits.

Morgan Stanley’s 2023 OpenAI collaboration announcement had already indicated that the stream of advisor questions and feedback would be used to refine the offering. Later, the scale of that feedback became more concrete in real-world use. In Big Banks Are in a Race to Get AI Right, Axios cited Morgan Stanley AI executive Jeff McMillan as saying that the firm spent months refining the assistant to a usable level and processed more than 20,000 pieces of advisor feedback.

That volume of feedback shows that even after technical validation, real users can still expose new retrieval gaps. The system may fail to find relevant material, retrieve the right material but omit important conditions, or generate an answer that appears complete but does not adequately support an actual client-service task. Other problems emerge from the usage process itself: how much time advisors need to spend checking sources, how they need to phrase questions to get useful answers, and under what circumstances they revert to legacy search and communication methods.

A limited pilot should therefore turn user feedback into a continuously updated issue register. The first category is unanswered questions, which require investigation of the knowledge scope or retrieval method; the second is incomplete answers, which require recording what was omitted and under what conditions; the third is usage friction, where the system has produced a result but the advisor still chooses manual search, asks a colleague, or abandons the tool. Each issue should retain the scenario in which it occurred, the materials involved, the human workaround, and the remediation status. Representative real-world problems should then be added to subsequent evaluation sets.

For enterprise management, the output at the end of the pilot should be a “real-world operating record”: which users used AI for which tasks, and whether usage persisted; which questions required human intervention; which answers were abandoned or re-searched; which frictions affected efficiency, trust, or willingness to keep using the system; and which issues would block expansion.

Management can then make three types of decisions:

1. The system can operate in real work, the main issues have been identified and are controllable, and the project moves into pre-production evaluation and acceptance;
2. The problems remain fixable but the current evidence is insufficient, so the project stays within a limited scope;
3. Adoption is low, real demand diverges from the original use case, or critical problems cannot be controlled, so the project returns to use case selection or PoC for redesign.

The pilot leaves behind real operating evidence. The next gate is to determine whether that evidence is strong enough to support production entry.

After the limited pilot, management needs to answer a stricter question: does the system meet the production-entry criteria defined in advance? Positive pilot feedback, high usage, and task improvements may show that the solution is worth continuing, but they are not sufficient on their own to authorize production deployment. Pre-production evaluation and acceptance need to consolidate fragmented pilot evidence into a formal decision—a defined set of passing criteria, responsibility boundaries, and risk conclusions.

The first criterion is that core outputs meet pre-agreed quality and traceability requirements. In Key Milestone in Innovation Journey with OpenAI, Morgan Stanley stated that the AI assistant generates answers exclusively from firm-approved internal Wealth Management content and provides Financial Advisors with links to the corresponding source documents. OpenAI further disclosed in Morgan Stanley Uses AI Evals to Shape the Future of Financial Services that the firm integrated quality assurance into its evaluation framework and used a regression suite of sample questions to identify system weaknesses and improve compliant outputs. Enterprises can use this approach to build an acceptance set covering high-frequency questions, boundary cases, and high-risk questions, separately testing answer accuracy, relevance, consistency with sources, and critical error types. The system should pass this gate only when it meets agreed thresholds and does not cross any error red lines that would prohibit production entry.

The second criterion is that data flows, knowledge scope, and access permissions remain within approved boundaries. Morgan Stanley constrained answers to approved internal content and used OpenAI’s zero data retention policy to address privacy concerns around proprietary data. In acceptance engineering, enterprises therefore need to verify item by item which data the system can read, which users can access it, whether inputs and outputs are retained, where logs are stored, and how unauthorized access attempts are handled. This also echoes the conclusion from Issue 11, AI Being Useful Is Not Enough: Enterprise AI Must First Pass the Secure Deployment Gate: before high-risk use cases enter production, the operating environment, data flows, and traceability mechanisms all need to be clearly explained. Any unexplained data path, permission gap, or retention risk should be fixed before launch or removed from the initial production scope.

The third criterion is that human review and exception handling have become formal operating procedures. Morgan Stanley has consistently kept Financial Advisors at the center of client service and professional judgment. For enterprises, “a human reviews it” needs to be translated into executable rules: which outputs require human confirmation, what evidence reviewers must use, who has authority to pause the system, where errors are reported, and who carries final business responsibility. Acceptance should also test this response chain using scenarios such as incorrect answers, missing evidence, and abnormal access, confirming that responsible personnel can identify the problem and escalate it correctly.

The fourth criterion is that the initial production scope and residual risks receive explicit management approval. Once the first three checks are complete, the enterprise still needs to lock down which users, tasks, data, and functions are approved for production, preventing pilot conclusions from being generalized to scenarios that were never validated. Each unresolved risk should be categorized as resolved, accepted under defined conditions, required to be fixed before launch, or temporarily excluded from the production scope. Supporting evidence, accountable owners, remediation deadlines, and triggers for re-checking should also be retained. Morgan Stanley’s public materials do not disclose these internal decision details, so this criterion is a methodology we derive from production-readiness requirements and should not be presented as a publicly confirmed Morgan Stanley process.

Once the system is in production, management must continue answering a further question: as models, knowledge, users, and business scope keep changing, can the system remain effective, controllable, and worth operating? Passing acceptance only means that the system was authorized for production under the scope and conditions that existed at that time. As internal content changes, user questions become more complex, or models, retrieval methods, and access permissions evolve, the original acceptance conclusion can gradually become invalid.

In Morgan Stanley Uses AI Evals to Shape the Future of Financial Services, OpenAI states that more than 98% of Morgan Stanley Wealth Management advisor teams now actively use AI Assistant, with many teams integrating it into daily work. At the same time, Morgan Stanley runs daily testing with a regression suite to continuously identify weaknesses and improve compliant outputs. Together, these two facts show the core logic of stable operations: as deployment scope expands, quality control must become part of the daily operating process as well.

Enterprises can condense this continuous operating mechanism into a closed loop:

Monitor → Correct → Revalidate Changes → Expand or Roll Back

Continuous monitoring should use the pre-production acceptance results as the baseline. Enterprises should periodically check answer quality, source traceability, compliance errors, system availability, and abnormal access, while also recording behaviors such as human rewrites, issue escalations, and discontinued use. Once metrics deteriorate or new error types appear, the issue should be classified by root cause—knowledge content, retrieval method, model or prompt, permission configuration, business process, or user behavior—and assigned to a clearly accountable owner.

Any change that can affect system performance should trigger relevant revalidation. OpenAI’s case study shows that Morgan Stanley’s evaluation framework evolved as experience accumulated, adding translation evals and adjusting retrieval methods as the document corpus expanded. In enterprise practice, changes to the model, prompts, knowledge base, permissions, or use cases should therefore trigger reruns of the relevant tests. Higher-impact changes can first be released to a small group and expanded only after results are confirmed.

Management ultimately needs to decide whether to expand or roll back based on operating evidence. Morgan Stanley’s adoption rate of more than 98% shows that the system has achieved broad real-world use, but adoption alone does not prove return on investment, output quality, or acceptable risk. Enterprises still need to combine quality trends, exception records, and business value to determine which functions can expand further, which issues need time-bound remediation, and when the system should be paused or rolled back to a previous stable version.

Looking back across the full deployment path, the five stages answer five progressively deeper questions: Use Case Selection & Demo confirms whether the business problem is worth solving; PoC verifies whether the technology can deliver; the Limited Pilot observes whether the system can operate in a real environment; Pre-Production Evaluation & Acceptance determines whether it meets production-entry criteria; and Production Deployment & Continuous Operations ensures those criteria remain valid as conditions change. The evidence created at each step then becomes the basis for the enterprise’s decision to proceed to the next stage.

Morgan Stanley devoted business experts, technical teams, compliance personnel, and ongoing evaluation resources to this deployment process. Small and medium-sized enterprises are unlikely to replicate the same staffing model or volume of feedback. What is worth learning is the discipline of making each stage answer a specific question and retaining enough evidence to support the next decision. The cycle can be shorter, and the number of participants can be smaller, but the key judgments around use case selection, technical validation, real-world piloting, production acceptance, and continuous operations still need to be preserved.

When SMEs actually begin executing these five stages, the difficulty often comes from organization and method. Business, technical, data, and security responsibilities may sit with only a handful of people. Even if management understands the complete framework, it may still struggle to determine which gate the project is currently at, what evidence is missing for the next decision, and which activities can reasonably be combined. As a result, the process may collapse into a one-off tool purchase and trial, or alternatively become over-engineered beyond what the existing team can support.

SMEs need a lightweight path aligned with their current foundations: first identify which gate is blocking progress, then retain only the materials required to support the next decision. An enterprise may need nothing more than a use case card to begin, or it may need to add a test set, a pilot log, a production-acceptance checklist, or a production operating register. The number of documents is not important. What matters is that each artifact answers a management question and helps the enterprise decide whether to continue, correct, or pause.

NextAI+ Praxis can enter at this execution gap. The first step is to assess the enterprise’s process maturity, knowledge and data foundations, organizational responsibilities, and current project stage, then identify the conditions that need to be filled first. The second is to turn business objectives, technical validation, data permissions, and risk requirements into a minimum executable path. The third is to help the team complete the first round of validation and delivery, while turning the use case, testing, acceptance, and operating methods produced along the way into reusable templates for future projects.

This collaboration does not need to begin with a company-wide transformation program. Enterprises with weak process and knowledge foundations can start with a deployment diagnostic and mapping of the first workflow; enterprises with a clearly defined use case can start with evaluation and pilot design; enterprises already running pilots can prioritize production-entry criteria and continuous operating mechanisms. The role of NextAI+ Praxis is to compress the complete deployment framework into the next step the enterprise can realistically support, so that limited investment gradually accumulates into reusable organizational capability.

• Morgan Stanley, Key Milestone in Innovation Journey with OpenAI, published March 14, 2023, accessed July 23, 2026. Useful for understanding the initial use case selection, internal knowledge boundaries, links to source documents, and human responsibility.
• Morgan Stanley, Morgan Stanley Wealth Management Announces Latest Game-Changing Addition to Suite of GenAI Tools, published June 26, 2024, accessed July 23, 2026. Useful for understanding the formal rollout of Assistant, the 98% team adoption rate, and how Debrief entered the advisor meeting workflow.
• OpenAI, Morgan Stanley Uses AI Evals to Shape the Future of Financial Services, page does not state a publication date, accessed July 23, 2026. Useful for understanding evaluation sets, expert feedback, daily regression testing, zero data retention, and iteration of the evaluation framework.
• OpenAI, AI in the Enterprise: Lessons from Seven Frontier Companies, PDF does not state a publication date, accessed July 23, 2026. Useful for further understanding how enterprises can begin with evals and move into production through iterative deployment.

• Ryan Heath, Big Banks Are in a Race to Get AI Right, Axios, published April 12, 2024, accessed July 23, 2026. Useful for understanding the more than 20,000 pieces of feedback, content relabeling, final human checks, and organizational constraints banks encounter when deploying AI.

The five stages in this article also extend three conclusions developed in previous issues: enterprises should begin with workflows whose boundaries are clear and measurable; before AI enters high-risk production scenarios, the operating environment, data flows, and responsibility boundaries need to be explicit; once a system enters continuous operation, it must also be subject to long-term measurement and dynamic adjustment.

• Issue 1, The Real Inflection Point for Enterprise AI: Governance and Collaboration Become the New Constraints After Standardized Workflows Reach Production, discussed why highly constrained, auditable standard operating processes with clear metrics are more likely to enter production first, and why governance and collaboration become new operating constraints as AI deployment expands. Best read alongside the “Use Case Selection & Demo” section of this article.
• Issue 3, Enterprise AI Enters the Operational Asset Management Stage: From Deployment Expansion to Value Measurement and Dynamic Trade-Offs, treated continuously operating AI workflows as operational assets requiring long-term observation and management, focusing on how enterprises can track adoption, resource consumption, and business outcomes, then decide whether to expand, adjust, or stop. Best read alongside the “Production Deployment & Continuous Operations” section of this article.
• Issue 11, AI Being Useful Is Not Enough: Enterprise AI Must First Pass the Secure Deployment Gate, discussed how enterprises can choose deployment models based on use-case risk, and why data location, system connections, permission auditing, and responsibility boundaries need to be clarified before production deployment. Best read alongside the “Pre-Production Evaluation & Acceptance” section of this article.