Technology Pilot Lifecycle Toolkit

Steps

1. Identify a clear problem area as indicated through data, strategic priorities or patient/staff experiences.
2. Learn more about the problem, leveraging Catalyst techniques such as interviewing, observing, and conducting surveys with stakeholders. Meet with your quality improvement and data departments to determine what data or reports you need to quantify the extent of the issue.
3. Create a first draft of the outcomes and metrics you’re trying to impact. Consult with your operations and finance teams. What strategic objectives would solving this problem support?
4. Refine and reframe the problem with new insights.
5. Share findings with stakeholders and leadership and agree upon final problem statement.

Steps

1. Outline constraints and final problem statement for the solution in a project charter.
2. Collaborate with stakeholders (executives, patients, relevant staff, etc.) to generate ideas for approaches to the problem given constraints.
3. Look to external inspiration to develop ideas: learn how other similar or analogous organizations dealt with similar challenges.
4. Prioritize 2-3 solution options to flesh out further, using prototypes, storyboards, and user feedback. To gain visibility and buy-in from leadership, consider creating and presenting a 7-part narrative pitch at a leadership meeting.
5. After exploring and discussing with leadership, decide:
   • To pursue a digital solution — continue on to the next step.
   • If desired solution does not have a technology or digital component, continue designing the solution using the Catalyst method. This will not require a tech pilot.

Requirements, broadly defined, are a list of criteria, specifications, and/or functional needs that a technology platform or vendor would ideally fulfill. We often re-frame these requirements as vendor-facing questions to support information gathering in the vetting process. Consider meeting with the pilot metrics consultant to understand how your metrics might align with your requirements.

Steps

1. Before defining requirements, it is important to understand (or, in some cases, create) your organization’s process for selecting a technology solution. This may involve working closely with leaders from a variety of departments like legal, purchasing, compliance, as well as strategic leaders.
2. Develop and document a set of business, functional, and technical requirements that are “must haves” and “nice to haves.” Examples of requirements to consider might include: EHR integration, languages available, literacy level, geographic coverage, target populations, role-based interfaces (e.g MA v. MD), reporting capability, security assessment, service level agreements.

Steps

1. Based on your requirements, create scoring criteria by “weighting” requirements in order of importance. This will help you assess and compare vendor solutions.
2. In the initial solution landscape scan, consult recommendations from partner organizations to narrow the range of vendor solutions in your problem area. This will help you zone in on more sustainable partners.
   • Review related CCI tech vendor demos, case studies, and landscapes.
   • Attend electronic health record (EHR) user group meetings and conferences to learn about new products.
   • Leverage technology ecosystem resources like Prime Health, health technology accelerators, social impact investors, etc.
3. Select between 2-5 vendors suited to your problem and research them more deeply. You might do this by:
   • Conducting phone calls with and requesting demos from account managers at your top vendors.
   • Reaching out to vendor account managers to review and respond to your requirements list.
   • Using a tool like Valify (free for safety net providers) to invite vendors to submit responses to your requirements, in a process similar to an RFP.
4. Document your findings and host a meeting with internal stakeholders (CIO, leadership, end users) to discuss the top options. Consider having all stakeholders review and “score” each vendor against the requirements list.
5. For each shortlisted vendor, create a rough budget and cost-benefit analysis to understand the financial commitment. This could be considered an initial conversation around sustainability beyond the pilot project.

In this step, you’ll compare the vendors on your shortlist, decide on one to move forward with, and begin the contracting process for the pilot period (not a long-term contract.) We recommend limiting the pilot duration to less than 6 months if possible—remember, the idea is to learn fast!

Steps

1. Based on scoring of vendors, alignment with vendor team, and long-term feasibility of company, select single vendor for a pilot. Communicate decision to leadership and key stakeholders.
2. With your internal team, define non-negotiable contract deal terms. Reach out to CCI for additional insights on reasonable deal terms.
3. Define and finalize success metrics for your pilot evaluation. Success metrics and benchmarks will help your team decide whether to scale the product or cease use after the pilot concludes.
4. Reach out to vendor to discuss a pilot. If there is mutual interest, work with vendor account manager to structure deal, contract, and pilot terms.
5. Inform legal, purchasing, and compliance teams of desire to work with vendor and prepare necessary documents (agreements, consent, etc.)
6. Finalize contract or memo of understanding (MOU) with vendor.

Steps

1. 1. Organize internal pilot team and define roles. Ensure leadership is willing to protect staff time.
   2. Develop and review pilot metrics with internal team and vendor.
   3. Set up any reports or systems needed to collect data on pilot effectiveness.
   4. Plan pilot with vendor, project manage pilot implementation according to contract terms, and conduct necessary staff trainings.
   5. Monitor the ongoing pilot using reports or systems created. Provide regular feedback to vendor regarding implementation for course correction.
   6. At the end of the pilot, evaluate the success of the pilot to inform your next steps:
      

      Result	Suggested Next Step(s)
      Need more information	Extend pilot and continue data collection.
      Pilot did not yield expected outcomes.	End pilot and vendor partnership. If considering a second pilot, return to vendor vetting shortlist to consider other partners for second pilot.
      Pilot met success benchmarks and vendor partnership was positive.	Negotiate new contract and begin planning to scale and spread solution across organization.
      Pilot met success benchmarks, but vendor partnership could have been better.	Host feedback meeting with partner, or return to vendor vetting shortlist to consider other partners for second pilot.

   7. Present results and recommended next steps to leadership.

Steps

1. Complete the Case Study Template using your Project Charter as the backbone for your narrative.
2. Share findings with peers in CCI community through Q&As and other presentations.
3. Identify external opportunities — such as conferences, EHR user groups, coalition meetings — to present findings beyond the CCI community.