#14 - How to Compare Software

At some point in time, a CISO will need to purchase new security technology.  Whether it's antivirus, firewalls, or SIEMs you need to understand how to choose a product that will benefit your organization for years to come.  This podcast discusses 5 different techniques that CISOs can apply to help with product selection

1. Perform Market Research to learn the players 
   • Gartner Magic Quadrant
   • Forrester Wave
2. Leverage Vendor Comparison Tools to spot the features
   • Mitre ATT&CK Evaluation
   • AV-Comparatives
   • MoSCoW Method (Must Have, Should Have, Could Have, & Will not Have)
   • Pugh Matrix
3. Use Predictive Analysis tools to see the trends
   • Google Trends
   • OpenHub.Net
   • Stack Overflow
   • DB-Engines
4. Apply Problem Framing to understand the limitations and politics 
   • Define the Problem: List the current problem you are facing.
   • State the Intended Objective: Identify the goal an organization is trying to achieve so that a consensus can be made when the original problem has been solved
   • Understand the Status Quo: If you take no action, does the current problem get worse, get better, or remain the same.
   • List any Implied Solutions: List early solutions that appear to address the initial problem. Likely these solutions may come from your direct boss who has a certain way of doing things.
   • Identify the Gap- The gap is roughly the difference between the intended objective and the status quo. Essentially this is the opportunity cost your organization must use when comparing this against other problems in the organization. 
   • Identify the Trap- For each of the implied solutions imagine how you might build the product or service as directed and still not solve the intended objective.
   • Explore Alternatives- Are there other solutions that avoid traps or gaps to address a problem that have not been previously evaluated?
5. Execute an Analytical Hierarchy Process (AHP) to remove bias
   • AHP is a structured process that helps remove politics or bias from decision-making.  It relies on creating relative weights among decision criteria, and possibly decomposing those into sub-criteria resulting in a weighted formula for all inputs.  Those become the equation that is used to evaluate alternatives; each alternative is scored on its sub-criteria then summed up by relative weight, resulting in a relative scoring based on numeric analysis.  For example, selecting a new product might involve evaluating three major criteria:  cost, functionality, and maintenance.  These are ranked pairwise on a relative scale of 1x-9x.  For this example, cost is twice as important as maintenance; functionality is twice as important as maintenance; cost is equally important to functionality.  From that comes a 40% - 40% - 20% ranking (all must sum to 100%).  Next, sub-criteria may be identified and weighted, e.g., initial cost is 1/3 the importance of ongoing cost.  Thus, the 40% global weighting for cost would consist of local weighting of 1 part initial cost [25%] to 3 parts ongoing cost [75%] (1:3 ratio).  So, initial cost becomes 25% of the 40% of total cost = 10% of overall decision, and ongoing cost becomes 75% of the 40% of total cost = 30% of overall decision.  This may be repeated for other criteria at as many levels deep as desired, resulting in an overall weighting of input criteria based on simple pairwise comparisons.  Each candidate choice is now be scored for each criterion on a selected scale (e.g., Option A scores 4 of 10 for initial cost, Option B scores 8 of 10 for initial cost), and the weighted products are summed for a final score.

•      References for Analytic Hierarchy Process (AHP):
  • Everyman's link
  • Shorter explanation link (pitches productized version)
  • Online calculator link
  • Expensive eBook
  • Not-so-expensive reference