Methodology

Fall SolutionMap 2020 methodology updates

  1. Rankings to reflect number of customers, FTEs, and revenue per customer  

In the past, a vendor’s “bubble size” in SolutionMap’s charts solely indicated its number of customers. While this demonstrates some level of market penetration, it did not adequately convey a vendor's experience and resources to handle complex global deployments. Going forward, the bubble size calculation methodology will consider the following:

    • SolutionMap ranking bubblesmarket scale based on the global number of customers 
    • number of full-time employees focused on the relevant solution(s) at the provider
    • revenue per customer in the relevant solution(s)

These factors were iterated by the Spend Matters analyst team and validated by SolutionMap participants. 

2. Revenue Estimations for Insider Vendor Summary Reports Explained

Insider subscribers have access to scoring summary reports of all ranked vendors. This includes company background information for each vendor including annual revenue. If a vendor did not disclose this information, we have applied the following method to determine an estimated revenue range:

# of Full-time Employees (FTEs) times $150-$300K/Employee

i.e. A company with 270 FTEs would have a revenue range of $40.5M - $81M

Previous Methodology Updates for Reference - Revamp in Q2, 2019

The first 'major revamp' of SolutionMap methodology by the Spend Matters analyst team took place for the Q2 2019 release (the first major “revamp” since the SolutionMap’s Q2 2017 launch), introducing in-depth platform scoring.

Having a platform-level understanding allows potential technology users to discern the level of integration between product modules and suites, overall interoperability with ERP and broader technologies, as well as the quality and effectiveness of applied advanced technologies such as artificial intelligence (AI).

All ranked providers were required to re-do the SolutionMap review and scoring process in its entirety. Although the core methodology as outlined below hasn’t changed, the RFI requirements were expanded to well over 1,000 and divided into platform and functional components across source-to-pay and services procurement modules and suites.

Adding such a level of granularity allows for deeper functionality and platform capability assessments and ensures objective scoring scales (see example below).

To illustrate the objective scoring format, here is a sample common configurability, globalization platform requirement for Source-to-Pay (S2P):

Multi-Lingual
How extensive are the multi-lingual support capabilities?
0 - not currently supported / not applicable
1. flat file menu mappings for a small set of languages;
2. dynamic mappings of menu options and help text based on standard translations and regional linguistic variances and support for over a dozen languages;
3. override features that allow a buyer to override mappings on documents / menu options being shared with a supplier and over twenty languages supported;
4. Includes capability beyond which is previously addressed and beyond peers

As before, validated user/peer reviews (1,000+) are also factored into the transparent vendor ratings. Analyst scoring makes up the left side of the quadrant, customer summary scoring determines the right side of the quadrant. SolutionMap rankings are updated and published semiannually. Procurement professionals consult the free rankings, subscription research and data to drive procurement technology selection decisions.

3. General Methodology Overview

Here’s how the rankings are created:

  1. Every SolutionMap has a dedicated lead analyst supported by the greater analyst team plus operational support staff
  2. The lead analyst develops the RFI in collaboration with the internal analyst team, customers and consultants to ensure requirements are comprehensive and relevant to practitioners following a set methodology
    1. Capabilities definitions, personas and rating weights defaults are determined ‘up front’ before issuing the RFI
  3. During the RFI process, vendors need to:
    1. Provide detailed company and customer demographics
    2. Self-score solution requirements (and offer detailed descriptions to illustrate functionality)
      1. Scoring scales are pre-determined by Spend Matters analysts and fully capability-based (ensuring objectivity)
    3. Describe all solution related services available to customers
    4. Provide customer references
    5. Supply documentation regarding:
      1. Revenue models
      2. Buyer and supplier fees
      3. New vs. replacement customers
      4. Customer TCO, ROI and success metrics
      5. Case studies
  4. Customer (user) references complete a detailed survey focused on key provider differentiators, delivery against expectations/requirements, strengths, promoter score, collaborativeness, improvement areas, problem solving skills, features/capabilities improvement needs, shortcomings and more (making up 50% of the ranking score). Any vendor newly participating in SolutionMap must obtain minimally 3 references to participate and then continuously obtain additional ones.
    1. Individual customer inputs are never shared with the vendor community
    2. Customer identities are known to the core Spend Matters project team only
    3. Customer inputs are aggregated to form the basis of the customer value ratings
      1. The center of the SolutionMap rankings chart always represents the average vendor score
      2. Vendors visible in the top and bottom right quadrants of the ranking chart exceed the average vendor score
      3. The average customer reference score fluctuates with each release based on newly added vendors and additionally obtained references for ranked providers
      4. NOTE: A vendor's position on the ranking chart may alter based on two factors:
        1. Improved or decreased customer reference scores (due to added or expired references)
        2. A provider's score relative to the category average (where the vendor's reference score may be unchanged)
      5. Spend Matters also directly solicits references from the procurement practitioner community
        1. Entries are manually reviewed and validated by the Spend Matters team by checking the identity of the submitter and ensuring the responses do not seem suspicious (extreme scoring throughout the survey out of line with the average for that provider)
        2. Upon approval, the survey participants receives a participation reward of choice
  5. Analysts participate in a 90-minute technology demo (per module) for each RFI participant to validate the self-scoring and adjust the scores up and down based on the technology demonstrations
  6. Analysts rate providers against strict functional / solution definition requirements after reviewing vendors’ self-scoring RFI inputs and demonstrations
    1. Demonstration notes and documentation may also be reviewed and extra spot checks may be performed
    2. Solution draft scoring is finalized for each vendor
  7. Vendors are invited to fact check their scores on a granular level against analyst scoring and request clarification
    1. Vendors have the option to appeal individual functional scores at a granular level via a defined process involving additional written documentation, calls and demonstrations
    2. SolutionMap placements are never shared prior to final publications with providers
  8. Final SolutionMap rankings are created and produced in due course based on finalized data and previously defined criteria, weightings and inputs
  9. Participating vendors are required to review and update their full RFI scoring inputs every 12 months (‘general’ platform-level inputs as well as category-specific self-scores)
    1. Vendors may submit intermittent category updates (max. twice every 12 months for each single RFI category)
    2. Any scoring updates require validation demos. Participation in new categories require the minimum number of customer references (3).

The following assessment elements are taken into account:

  • All elements of a technology*
  • Integrations
  • Supplier experience
  • Front end users and administrators’ user experience

* The technology elements are very detailed and vary by the technology being ranked