The inaugural “ShopTalk” column (“Hell Is Not a Night Club,” February 2007) outlined a process that, if diligently adhered to, greatly reduces the risks inherent in acquiring mission-critical, third-party software components. The process includes steps such as issuing an RFP, visiting vendors, and talking with references, which most carriers implement when undertaking a software search and evaluation. But these steps in and of themselves are inadequate risk mitigators for such a critical decision for the following reasons:
• The results rely exclusively on what people say, not on what people do.
• The greatest risk mitigator is observed, measurable performance—both for software and for the people who support and customize the software.
• The opportunity to gain important additional information and insight is lost if the dimension of performance is not included in the assessment.
The software selection process, or “road map,” that I encourage clients to use emphasizes performance-based activities both as a way of reducing the field of potential vendors and as the ultimate verification step. This ultimate verification step is a proof of concept (POC). I believe this to be the single most important exercise any carrier acquiring a significant piece of software can undertake—and it should be mandatory. And yet many carrier teams, often tired and stressed at the end of a long search project, feel they don’t have the time to “delay” any longer and must get on with the implementation.
The common carrier objections to performing a POC are:
• “It takes a long time [usually three to six months], and my senior management won’t wait any longer.”
• “A proof of concept costs a significant amount of money [usually six figures], and we don’t have funding approval.”
• “The proof of concept produces a throw-away result.”
The reality is as follows:
• The wrong decision (or even a decision that is less than well understood) can cost millions of dollars, divert and waste the time and energy of critical IT and business resources, and come at a large opportunity cost to the business.
• Only a POC can answer questions concerning how the software performs in a carrier’s environment, how the vendor performs on a joint project with no opportunity for “behind-the-curtain magic,” and whether the vendor will be a comfortable partner to work with.
• The POC should be designed to produce a usable result that moves the carrier toward a production implementation rather than be a throw-away exercise.
• The POC should have two major objectives: first, to provide the needed verification the vendor and its software are an appropriate choice; and second, to produce a usable product that becomes (at least part of) an implementation deliverable.
So, what does a POC consist of, and how do you set one up and execute it?
Like every other structured undertaking, the POC should start with a statement as to its purpose and how the outcome will be measured. Common objectives for a POC include:
• Verification of a set of function points. For example, verify the system can accurately structure, rate, and issue a specified insurance product, e.g., an umbrella.
• Verification the system has substantial configuration capabilities for screen design, workflow, and product rules definitions.
• Verification the system will run in a specified technical environment.
• Verification of the system’s ability to integrate with various legacy applications.
• Verification the system will perform and scale acceptably on an agreed hardware configuration running agreed meaningful transactions.
• Assessment of the vendor’s implementation capabilities, including methodology, resources, and general project capabilities.
• Assessment of the ease of doing business with and partnership fit of the vendor.
These objectives need to be documented, negotiated, and agreed between the carrier and the vendor. It is important both parties clearly understand the definition of success and how it will be measured.
Once the objectives for the POC are established, other key activities include:
• Negotiate and agree on terms and conditions for both software and services. The vendor may or may not require some form of payment for the use of its software during the POC. If a usage fee is required, it should be offset against the “final” license price. So, if the carrier pays a license fee for use of the software during the POC and then goes forward with the vendor and signs a full license, there effectively will have been no software charge for the POC. If, however, the POC does not lead to a full implementation with that vendor, the usage fee may be nonrefundable. Regardless of whether a usage fee is required, the vendor probably will insist the carrier sign a software license agreement to protect the vendor’s rights relative to the software.
The other area with legal and financial implications is that of implementation services. In order to support the POC, the vendor will need to commit a group of resources to the project for some period of time, usually between three and six months. This will require the signing of a services agreement and associated work orders to define the activities and deliverables for which the vendor is responsible. Negotiating these terms and conditions also provides the carrier with a good sense of how well it can work with the vendor.
• Schedule the POC project. Most POC projects are arbitrarily time boxed. It is common for a POC to be a 90-calendar-day exercise. This generally is considered to be long enough to install the software, conduct needed training, build a small but meaningful function slice, and verify the viability of the software and vendor. I have, however, seen significantly longer time frames, depending upon the size, complexity, and sophistication of the carrier involved. In one case, a policy administration vendor with a minimal track record won the implementation of a new PAS platform for a top-10 national carrier following a 15-month POC. Often there is a need to reconcile the functional definition of the project with a fixed time line.
• Create the carrier team. The vendor team that is committed to the POC will be focused on executing those tasks that relate to the gathering of requirements, the configuration of the software to meet those requirements, and the vendor side of the legacy integration effort. In other words, the vendor’s focus is on the software and its performance. This leaves some large areas of the project, such as quality assurance, overall legacy integration, user education, training, and system rollout in the hands of the carrier.
So, the carrier has to marshal a team of professionals who can run the overall project, including the coordination of the vendor team’s tasks and deliverables; learn and work with the vendor’s software; and handle major, discreet areas of the project with minimal, if any, input from the vendor.
• Define who does what. While the vendor may be appropriately focused on the software-related aspects of the POC project, some vendors do more than others. The more a vendor does, the less the carrier needs to do, and vice versa. Thus, it is important to agree and document project responsibilities. This is not just good project management, it also is good practice for the contractual side of the relationship that the carrier and vendor can describe clearly and consistently the boundaries of the services they will be providing and what the vendor will be compensated for.
• Execute the POC. Like all projects, a POC finally must be executed. And like any other project, it should be adequately monitored and controlled. While the POC is a relatively short-duration project with a smaller-than-usual scope statement, it still is a project, and one with a great deal riding on it. The POC is not a “CYA” exercise and cannot be treated as such.
One good practice, assuming staffing affords, is to bring in some new faces and fresh perspectives to work on the POC. Otherwise, there tends to be a “Stockholm Syndrome” effect, where the team that selected the vendor already has bought into the success of that vendor and will be somewhat less demanding and diligent during this final, critical step.
• Measure the results. One of the most important aspects of a proof of concept is ensuring the criteria for success are well understood. As noted earlier, there are two major objectives in the POC: first, to verify the software selection decision; and second, to build something of lasting worth. If a choice must be made between the two, the former always takes precedence.
For example, if the POC calls for five integration points to be built but only two are completed, the major question is whether the completion of two interfaces sufficiently proved the integration capabilities of the software. If the answer is yes, the lack of the remaining three interfaces is not troubling or worrisome but is merely work to be completed at a later date. In other words, verifying the integration capabilities of the software is more important than building all five interfaces in the time allocated.
Again, the measurements of success are the:
• measurement of specific functional capabilities;
• verification of generic capabilities such as integration or configurability;
• measurement of specific scalability and performance criteria; and
• assessment of the vendor’s service-delivery capabilities.
There also should be a “quality” dimension to the exercise. As we all learn, usually with accompanying pain, we get the behavior we encourage. If the vendor is required to deliver a function on a given date, that is likely to be exactly what it will do. The key question is, Was the function delivered to a high standard of quality, or was it simply delivered because it was due?
In order to avoid the latter, quality standards should be agreed to and enforced at the start of the POC. “Smoke tests” can be defined and agreed to so functionality is not considered delivered until it has passed the smoke test and is accepted into the carrier’s test environment.
The planning, execution, and analysis of a disciplined proof of concept is the difference between thinking and knowing you have made an appropriate and important choice on behalf of your company. Given the stakes, why would you not want to take a few months to ensure the next 15 to 20 years will be relatively painless and productive?