Home Page for the Information Technology (IT) Discipline

"Test Suite Management"

Table of Contents

Introduction: Introduction: Introduction to Test Suite Management
Framework: Using This Artifact as a "Test Suite Management Framework"
Glossary of Related Terms: The "Test Suite Management Glossary"
Capabilities: Test Suite Management as an Enterprise Capability
Ownership: Clearly Defined Test Suite Management Ownership is Critical for Success
Verbs and Actions: Understanding Why Verbs and Actions are Important to Test Suite Management
Roles: Key Verb and Action Driven Roles For Test Suite Management
Taxonomy: Understanding Test Suite Management Classifications or Categorizations
Life Cycle (Lifecycle): Data Lifecycle Phases for Test Suite Management
Metrics: Test Suite Management Metrics
Services: Test Suite Management as a Set of Services (a.k.a. Test Suite Management Services)
Principles & Best Practices: Common Principles and Best Practices for Test Suite Management
Further Reading and Reference Material for Test Suite Management

Introduction: Introduction to Test Suite Management

This document represents an aggregated, ordered and contextualized view of the material we've been able to compile and publish that is related to the topic of "Test Suite Management." The goal is to make this page a landing and launch point for all things related to this topic. As our content becomes more complete and more accurate, this page should become a very useful and powerful knowledge base for this topic and all parties interested in it.

You'll find that the content for this document is consistent with that of other discipline related documents. This is intentional. The consistency is based on a knowledge pattern that helps individuals learn more about different topics, quicker and more efficiently. We hope you find the material useful and easy to learn.

It's important to realize that content in this document and any related sub-documents are constantly evolving. Therefore, we recommend you check for updates, regularly, to keep up with the latest material.

The Foundation always welcomes your feedback and suggestions for improvement, as we're always looking for ways to improve our solutions and offerings to the general community.

All solutions published by the Foundation are subject to the terms and conditions of the Foundation's Master Agreement.

Framework: Using This Artifact as a "Test Suite Management Framework"

This document or artifact, along with everything in it, is intended to act as a "Framework" that addresses various aspects of Test Suite Management.

The readers will notice that most sections in the Table of Contents (TOC) use a format where the TOC entry is prefixed with a topic name, followed by a short descriptive title (i.e. "TOPIC_NAME: TOPIC_RELATED_SECTION_TITLE"). This is intentional and represents a format by which the Foundation may achieve things like the identification of appropriate topic areas, the segregation of distinct topic areas from each other, the appropriate ordering of topic areas, and achieve the maintenance of consistency, both, within and across different IT Disciplines.

To elaborate, this artifact is intended to:

1. Organize different areas of the discipline known as Test Suite Management into clear and compartmentalized areas that allow the Foundation to more effectively and productively collect, document and publish information that pertains to this discipline.
2. Decompose each area of Test Suite Management into smaller and, therefore, more digestible units for more efficient learning and understanding.
3. Document common industry wisdom about each area, piece or subcomponent of Test Suite Management
4. Act as a set of Test Suite Management related best practices and guidelines that have been collected, documented, and published for the benefit of IT Professionals, regardless of their specific industry, line of business, or area of expertise.
5. Act as a consistent and repeatable pattern for documenting, publishing and learning, both, within this Discipline and across "all" Disciplines.

From the Foundation's perspective, if done correctly, all of the above will allow the Foundation to properly decompose, document and publish content related to each sub-area or sub-topic for each IT Discipline, including this specific discipline (i.e. "Test Suite Management").

From the reader's perspective, if done correctly, all of the above will allow him or her to easily find and learn about specific areas of interest associated with this and all other IT Disciplines in a manner where the reader may effectively consume and digest material in small atomic segments that act as repeatable and more effective learning units.

As this artifact evolves and progresses, the reader will see it address key areas of the professional IT Discipline "Test Suite Management" that range from its detailed definition through closely related terms, phrases and their definitions, to its detailed specification of Test Suite Management Capabilities, and all the way through to defining, delivering, operating and supporting Test Suite Management Services.

As mentioned previously, this document will continue to evolve and the Foundation recommends the reader check back, regularly, to stay abreast of modifications and new developments. It is also important to understand that the structure of this artifact may change to meet the needs of such evolution.

Glossary of Related Terms: The "Test Suite Management Glossary"

Language between IT professionals and the businesses we serve is often a significant barrier to success, as we often spend countless hours trying to interpret each other's meanings. This is often also true between IT professionals who are taught to use certain terms and definitions as part of the organizations and industries they serve. It's when you start to jump from organization to organization, from enterprise to enterprise, and from industry to industry that you realize how much time and effort is wasted on just getting language and meanings correct. For these reasons, the Foundation puts a great deal of focus on terms and phrases, as well as their corresponding definitions. We highly recommend you spend time learning and understanding all of the related terms and phrases, along with their meanings, for all areas of "Test Suite Management."

Test Suite Management Glossary

Centralized Test Suite Management	Test Suite Management Procedure
Decentralized Test Suite Management	Test Suite Management Process
Enterprise Test Suite Management	Test Suite Management Professional
Federated Test Suite Management	Test Suite Management Program
Regional Test Suite Management	Test Suite Management Project
Test Suite	Test Suite Management Reference Architecture
Test Suite Automation	Test Suite Management Release
Test Suite Capacity Management	Test Suite Management Report
Test Suite Catalog	Test Suite Management Reporting
Test Suite Catalogue	Test Suite Management Roadmap
Test Suite Configuration	Test Suite Management Role
Test Suite Configuration Item	Test Suite Management Schedule
Test Suite Configuration Management	Test Suite Management Security
Test Suite Cost	Test Suite Management Service
Test Suite Data Entity	Test Suite Management Service Assurance
Test Suite Database	Test Suite Management Service Contract
Test Suite Decommission	Test Suite Management Service Level Agreement (SLA)
Test Suite Delivery	Test Suite Management Service Level Objective (SLO)
Test Suite Dependency	Test Suite Management Service Level Requirement (SLR)
Test Suite Deployment	Test Suite Management Service Level Target (SLT)
Test Suite Document	Test Suite Management Service Provider
Test Suite Document Management	Test Suite Management Service Request
Test Suite File Plan	Test Suite Management Software
Test Suite Framework	Test Suite Management Solution
Test Suite Governance	Test Suite Management Stakeholder
Test Suite History	Test Suite Management Standard
Test Suite Identifier	Test Suite Management Strategy
Test Suite Inventory	Test Suite Management Supply
Test Suite Item	Test Suite Management Support
Test Suite Lifecycle	Test Suite Management System
Test Suite Lifecycle Management	Test Suite Management Theory
Test Suite Management	Test Suite Management Training
Test Suite Management Application	Test Suite Management Vision
Test Suite Management Best Practice	Test Suite Management Workflow
Test Suite Management Capability	Test Suite Metadata
Test Suite Management Center of Excellence	Test Suite Migration
Test Suite Management Certification	Test Suite Plan
Test Suite Management Class	Test Suite Portfolio
Test Suite Management Community of Practice (CoP)	Test Suite Portfolio Management
Test Suite Management Course	Test Suite Processing
Test Suite Management Data	Test Suite Record
Test Suite Management Data Dictionary	Test Suite Records Management
Test Suite Management Database	Test Suite Repository
Test Suite Management Demand	Test Suite Reuse
Test Suite Management Dependency	Test Suite Review
Test Suite Management Document	Test Suite Schedule
Test Suite Management Documentation	Test Suite Schematic (Schema)
Test Suite Management File Plan	Test Suite Security
Test Suite Management Form	Test Suite Software
Test Suite Management Framework	Test Suite Strategy
Test Suite Management Governance	Test Suite Support
Test Suite Management Lessons Learned	Test Suite Taxonomy
Test Suite Management Metric	Test Suite Termination
Test Suite Management Operating Model	Test Suite Tracking
Test Suite Management Organization	Test Suite Tracking Software
Test Suite Management Plan	Test Suite Transaction
Test Suite Management Platform	Test Suite Unique Identifier
Test Suite Management Policy	Test Suite Verification
Test Suite Management Portfolio	Test Suite Version
Test Suite Management Principle	Test Suite Workflow
Test Suite Management Procedure

Please refer to the IT Glossary for other terms and phrases that may be relevant to this professional discipline.

Readers may also refer to the Taxonomy of Glossaries for terms and phrases that are semantically grouped according to IT Disciplines or enterprise domains.

Capabilities: Test Suite Management as an Enterprise Capability

A Capability, as it pertains to Information Technology (IT) or to an enterprise that an IT Organization serves, is defined to be "A manageable feature, faculty, function, process, service or discipline that represents an ability to perform something which yields an expected set of results and is capable of further advancement or development. In other words, a Capability is nothing more than "the ability to do something" or, quite simply, a Feature or Function. Therefore, when applied to an enterprise, a Capability represents a critical Enterprise Feature or Enterprise Function.

When it comes to Capabilities, there are multiple types that an enterprise needs to be aware of. Examples include but are not limited to:

• Business Capabilities: A set of Capabilities needed and/or offered by a Business.
• Information Technology (IT) Capabilities: A set of Capabilities needed and/or offered by an IT Organization.
• System Capabilities: A set of Capabilities needed and/or offered by a System
• Service Capabilities: A set of Capabilities needed and/or offered by a Service
• Organizational Capabilities: A set of Capabilities needed and/or offered by one or more Organizations.
• Enterprise Capabilities: A set of Capabilities needed and/or offered by an Enterprise (i.e. the combined set of all of the above).

As can be seen above, there are Capabilities that are associated with Resources, Organizations, and Assets such as Systems. All are important to an enterprise.

In the case of this IT Discipline (i.e. Test Suite Management), we use the word Capability in the context of an Enterprise Capability or an IT Capability, which are both equivalent to Enterprise Disciplines or IT Disciplines, respectively. In short, the Capability of Test Suite Management represents the ability to deal with any and all Test Suite Items and anything relevant that is related to or associated with any Test Suite Items.

If you think about it, a capability is really nothing more than a "verb" or "action that represents "the ability to do something." Understanding this allows us to derive a consistent and highly repeatable set of sub-capabilities for any Noun we're dealing with. For example:

• The ability to architect a specific Noun.
• The ability to copy a specific Noun.
• The ability to create a specific Noun.
• The ability to delete a specific Noun.
• The ability to design a specific Noun.
• The ability to edit a specific Noun.
• The ability to modify a specific Noun.
• The ability to purge a specific Noun.
• The ability to request a specific Noun.
• Etc. (The list is quite long.)

In summary, the implication is that the Enterprise Capability or Enterprise Discipline known as Test Suite Management is the superset of all the above Sub-Capabilities, as they pertain to or are applied to the discipline-specific Noun: "Test Suite." This now translates more specifically to:

• The ability to architect a specific Test Suite Item.
• The ability to copy a specific Test Suite Item.
• The ability to create a specific Test Suite Item.
• The ability to delete a specific Test Suite Item.
• The ability to design a specific Test Suite Item.
• The ability to edit a specific Test Suite Item.
• The ability to modify a specific Test Suite Item.
• The ability to purge a specific Test Suite Item.
• The ability to request a specific Test Suite Item.
• Etc. (The list is quite long.)

For a more complete list of very specific Capabilities/Disciplines, refer to the Foundation's Master Inventory of IT Disciplines. It is important to note that this inventory is in a flat or non-hierarchical form, specifically because "hierarchy" is almost always a matter of personal preference or context (what hierarchy is important to one Resource or Organization may be unimportant to another's needs or requirements). Therefore, the Foundation has published its inventory of Capabilities in a non-hierarchical, flat form.

This now brings us to a very obvious problem that surrounds Capabilities, which is the fact that there are simply too many "granular" or "specific" Capabilities to document and publish in any single Capability Model. The end result is that a Capability Model may become unwieldy because of trying to incorporate so many different specific Capabilities. Also, Capability Modeling "Purists," who all have their own (and very differing) opinions about how Capability Models should or should not be represented, almost always refuse to get into the details. To address this, we recommend using a generic set of Capabilities that map to and are driven by the Systems Development Life Cycle. For example:

• Identify and Solicit Demand for Test Suites
• Create and Set Strategy for Test Suites
• Perform Research on or for Test Suites
• Plan for the Delivery of Test Suites
• Identify Requirements for Test Suites
• Define and Design Test Suites
• Build or Construct Test Suites
• Control Quality of or for Test Suites
• Deliver Test Suites (to Meet Identified Demand or Desire)
• Operate and Support Test Suites

As you can see from the above, we now have a very limited, controlled and manageable set of Discipline-specific Capabilities for the Discipline Test Suite Management.

As a reminder, the above Capability representations are "suggestions" for baselining or initializing your own Enterprise Capability Model (ECM). It's recommended that you take the time to work with your enterprise stakeholders to improve upon and/or customize your own ECM so that you can help meet their needs. However, with that being said, it's always a better idea to go in with a baseline that you can modify rather than building your own solution from scratch, especially if your goals are to standardize, not reinvent the wheel, and not deviate too far from what other enterprises are doing to model their own environments. This is especially true if you've never had any experience building ECMs that have gained and maintained full adoption.

Why do enterprises perform Capability Modeling? Enterprises most often build Capability Models that are associated with Test Suite Management for the following reasons...

• To create a common enterprise language for the Discipline known as Test Suite Management.
• To create a common and reusable Test Suite Management Taxonomy (or a Taxonomy for Test Suite Management). Such Taxonomies can be used to help layout or structure an enterprise's Intranet (often for routing and/or traverse-ability) and tag Documents in Document Management Systems.
• To create a "Knowledge Map" or "Knowledge Model" for Test Suite Management. Such Knowledge Maps or Models can then be translated and used for structuring training programs, establishing Communities of Practice (CoPs), and defining and publishing things like policies, processes, procedures, and standards.
• To help categorize and structure Requirements during Requirements collection and analysis.
• To help map things like Applications, Software, and Hardware to the Capability areas of Test Suite Management.
• To help define enterprise Services, along with helping to understand who provides and consumes such services.
• To help understand what activities go on, in an enterprise, and who performs such activities.
• And much more...

Capability Modeling Recommendations: Some things to consider and keep in mind when working on or creating your Test Suite Management and Enterprise Capability Models...

• Try and keep "context" or "contexts" out of your core model. For example, your Test Suite Management Stakeholders may work hard to get what's important to them into the model (e.g. their "organizations" or organizational contexts), as opposed to what might actually be best for the enterprise. It's always a good idea to keep the model as pure as possible and keep such contexts out.
• While it's a good idea to keep contexts out of your core Capability Model, it will eventually become clear that Capability Contexts are, in fact, very important to your enterprise. You will have to find a way to model and represent such contexts, separately. For example, one context might be an organizational view of what organizations provide or consume specific Capabilities, such as Test Suite Management. Such contexts often represent "cross sectional" or "pivoted" views of your model.
• One of the quickest ways to lose the interest of your enterprise is to keep your model secluded or silo-ed in a spreadsheet or database and not publish it or use it for solving other problems in the enterprise. A good Enterprise Capability Model (ECM) helps get to better Service Management Models, better Document Management Taxonomies, and (most importantly) to better Knowledge Models that can be published as traversable content on your Intranet so that stakeholders can, find, learn about, better understand and use such Capabilities.
• Once you embark on building an ECM, you will undoubtedly run into the challenge that your Business doesn't speak in terms of "Capabilities." Remember, Capability Models are things that were designed by Information Technology (IT) Architects that, very often, have never worked in a Business, built a Business, or run a Business. Some times, it's better to build models using terminology your Business understands, such as "Services," "Features," or "Functions," all terms that most Businesses are far more familiar with. However, this should not be interpreted as a deterrent to keep you from embarking on building an ECM, should it be the right thing to do for your enterprise.
• You will be hard pressed to find many, if any, reusable Capability Models that can be used as solid reference material for your own models, which is why the Foundation has taken the time to document different aspects and issues associated with Capability Modeling and Capability Models.
• You will constantly be under pressure to help define and get your Business Stakeholders to understand what a Capability is. It's for this reason that you may want to use terminology that might be more common with Businesses, such as "Services," "Functions," and/or "Features."
• It's better to start with a model you can go into your Business Stakeholder meetings with, with the intent of improving upon what you have, than it is to go in with nothing at all and build your models from scratch.
• Do your best to work with your Test Suite Management Stakeholders to help verify and improve your Test Suite Management Capability Model.
• Don't assume that tools will help solve your Capability Modeling needs. It's always better to start simple and work your way up to tools, after you know what you want to achieve and can realistically achieve.
• Good Capability Models will be many levels or layers "deep." Modelers that think they can represent their entire enterprise or all its key Capabilities in just a few levels of hierarchy are fooling themselves or trying to fool those that they're trying to sell their models to.
• While Capability Models may look like "hierarchical" structures when building your first few layers, good Capability Models will eventually start to share common Capabilities in different areas of the model, forcing the model to take on the structure of a multidimensional "graph."
• Keep your model published and fresh so that the enterprise can see it and use it, immediately. The more eyes you have on your model, the faster it will improve.

Ownership: Clearly Defined Test Suite Management Ownership is Critical for Success

Here's a very simple fact... If an enterprise does not establish and enforce clearly defined Ownership (i.e. a Resources and his or her Organization are assigned as accountable ownership) for Test Suite Management, the enterprise has automatically set itself up for failure in its implementation of that discipline. Therefore, if you and your enterprise want to implement and maintain a successful solution for Test Suite Management, there must be a clearly defined Owner that can and will be held accountable for getting work done, providing transparency, helping with strategy setting, and coordinating implementation of Test Suite Management as a fully functional and mature enterprise Service.

Having clearly defined Ownership should not be confused with having fully dedicated Resources that spend one hundred percent of their time working on Test Suite Management. In fact, smaller enterprises can rarely afford to dedicate full time Resources, like larger enterprises can, to all enterprise IT Disciplines. This being the case, all IT Disciplines, including Test Suite Management, should "always" have clearly defined Owners so that there is always a clear point of accountability and contact for any issues or work that need to be addressed.

In addition to the common best practice of having clearly assigned Ownership for Test Suite Management, it is also considered a best practice to clearly publish and socialize Test Suite Management Ownership details to a centralized location (often referred to as a "Service Catalog" or an "Enterprise Service Catalog"), along with Ownership details for all other IT Disciplines, so that the entire enterprise has constant access to it.

Figure: How Ownership of the Capability Test Suite Management fits into the Canonical Model for IT

The above figure helps us understand how Capability or Discipline Ownership fits into the Canonical Model for Information Technology (IT) (i.e. "Think," "Deliver," and "Operate"). Owners are assigned to individual Disciplines or Capabilities, such as Test Suite Management, and are instantly made accountable to the enterprise for the results of all Test Suite Management Thinking activities (i.e. Strategy, Research, Planning and Design), all Test Suite Management Delivery activities (i.e. Construction, Deployment and Quality Assurance), and all Test Suite Management Operations activities (i.e. Use, Maintenance and Support). Done correctly, Test Suite Management Ownership is constant and ongoing. It's important to understand that such assigned Ownership should "never" end so that there is clear and constant accountability and transparency for all aspects of the Canonical Model to the enterprise.

Not having clear Ownership for Test Suite Management means that there is no clear understanding of who is accountable for it, who can provide understanding of what's going on within it, who can help the enterprise provide short term and long term descriptions of work being performed within the Discipline area to improve it over time for its customers, and who can help with getting work done that's associated with it. It means your or your enterprise's implementation for Test Suite Management will be highly incomplete and erratic because no one is constantly (or even partially) watching over the Discipline and its needs for maintenance and evolution. Not having clear Test Suite Management Ownership is a recipe for confusion and, sometimes, even chaos.

In summary, if you and your enterprise truly want to be successful with your implementation of Test Suite Management, ensure that a clear and highly accountable owner is identified and assigned to the Discipline. Publish those ownership details, preferably in an enterprise's Service Catalog, and socialize it so everyone knows whom to go to for answers and for help with Test Suite Management related work. In other words, if you want to implement Test Suite Management as an enterprise Service, then you absolutely must start with clearly defined, published and socialized Ownership.

Verbs and Actions: Understanding Why Verbs and Actions are Important to Test Suite Management

Throughout the Foundation's documentation, you will continuously run into the references of "Nouns and Verbs." These concepts are key to consistency and standardization, throughout the IT Industry, down to each and every IT Discipline. Given that we've discussed the impact of "Nouns" on the discipline of "Test Suite Management," this section will start to discuss the importance of "Verbs" or "Actions" that can be performed with or against the key Noun or Nouns associated with this Discipline. To reiterate, Verbs or Actions allow us to clearly understand what can be performed on or with the Noun in question. As will be discussed in the next section, Verbs or Actions will also help us clearly identify whom it is (i.e. the "who" or more specifically the Roles) that performs or executes such Verbs or Actions against a Discipline and its associated Noun or Nouns. As will be discussed later, Verbs or Actions will also help identify key Attributes (i.e. Field Names) that are necessary for the very data definition of the Noun or Nouns for this Discipline and will even help identify which Verbs or Actions can be automated for this Discipline.

As a reminder, the base Noun for the discipline known as Test Suite Management is: "Test Suite," which is sometimes referred to as a the Noun: "Test Suite Item."

By now, it should be becoming apparent that verbs represent a baseline for defining solid functional requirements and sub-capabilities for what would be a part of any good Test Suite Management System or Service. What this means is that if you and/or your Organization is looking for a solution in this space (e.g. the purchasing or building of a software solution or the implementation of a Service to address the needs of Test Suite Management), you could use discipline-related verbs to drive the foundation of what the solution should or shouldn't do, as mapped to specific stakeholders that will use or provide the solution.

Examples of the types of Verbs or Actions that are important to this Discipline include but are not limited to:

• Architect
• Create
• Edit/Modify
• Delete
• Design
• Copy
• Own
• Manage
• Etc.

The above list represents a very small subset of all Verbs or Actions that are relevant for this Discipline. The more complete set can be found in the Roles section of this document, where readers can see the direct correlation of Verb to Noun and to, both, Generic Role and Discipline Specific Role.

Roles: Key Verb and Action Driven Roles For Test Suite Management

An "action" or a "verb" is something that can be performed on or with a specific "noun." The reason it is important to itemize all relevant verbs is because we can now start to determine what we can or cannot do with the noun in question, where in this case the noun is "Test Suite."

Actions/Verbs	Example as Applied to "Test Suite"	Generic Roles	Discipline-Specific Roles
Administrate	Administrate Test Suite	Administrator	Test Suite Administrator
Approve	Approve Test Suite	Approver	Test Suite Approver
Architect	Architect Test Suite	Architector	Test Suite Architector
Archive	Archive Test Suite	Archiver	Test Suite Archiver
Audit	Audit Test Suite	Auditor	Test Suite Auditor
Bundle	Bundle Test Suite	Bundler	Test Suite Bundler
Clone	Clone Test Suite	Cloner	Test Suite Cloner
Code	Code Test Suite	Coder	Test Suite Coder
Configure	Configure Test Suite	Configurer	Test Suite Configurer
Copy	Copy Test Suite	Copier	Test Suite Copier
Create	Create Test Suite	Creator	Test Suite Creator
Decommission	Decommission Test Suite	Decommissioner	Test Suite Decommissioner
Delete	Delete Test Suite	Deletor	Test Suite Deletor
Deploy	Deploy Test Suite	Deployer	Test Suite Deployer
Deprecate	Deprecate Test Suite	Deprecator	Test Suite Deprecator
Design	Design Test Suite	Designer	Test Suite Designer
Destroy	Destroy Test Suite	Destroyer	Test Suite Destroyer
Develop	Develop Test Suite	Developer	Test Suite Developer
Distribute	Distribute Test Suite	Distributor	Test Suite Distributor
Download	Download Test Suite	Downloader	Test Suite Downloader
Edit	Edit Test Suite	Editor	Test Suite Editor
Educate	Educate Test Suite	Educator	Test Suite Educator
Export	Export Test Suite	Exporter	Test Suite Exporter
Govern	Govern Test Suite	Governor	Test Suite Governor
Import	Import Test Suite	Importer	Test Suite Importer
Initialize	Initialize Test Suite	Initializer	Test Suite Initializer
Install	Install Test Suite	Installer	Test Suite Installer
Instantiate	Instantiate Test Suite	Instantiator	Test Suite Instantiator
Integrate	Integrate Test Suite	Integrator	Test Suite Integrator
Manage	Manage Test Suite	Manager	Test Suite Manager
Merge	Merge Test Suite	Merger	Test Suite Merger
Modify	Modify Test Suite	Modifier	Test Suite Modifier
Move	Move Test Suite	Mover	Test Suite Mover
Own	Own Test Suite	Owner	Test Suite Owner
Package	Package Test Suite	Packager	Test Suite Packager
Persist	Persist Test Suite	Persister	Test Suite Persister
Plan	Plan Test Suite	Planner	Test Suite Planner
Purge	Purge Test Suite	Purger	Test Suite Purger
Receive	Receive Test Suite	Receiver	Test Suite Receiver
Record	Record Test Suite	Recorder	Test Suite Recorder
Recover	Recover Test Suite	Recoverer	Test Suite Recoverer
Register	Register Test Suite	Registrar	Test Suite Registrar
Relocate	Relocate Test Suite	Relocator	Test Suite Relocator
Reject	Reject Test Suite	Rejecter	Test Suite Rejecter
Remove	Remove Test Suite	Remover	Test Suite Remover
Replicate	Replicate Test Suite	Replicator	Test Suite Replicator
Report	Report Test Suite	Reporter	Test Suite Reporter
Request	Request Test Suite	Requestor	Test Suite Requestor
Restore	Restore Test Suite	Restorer	Test Suite Restorer
Review	Review Test Suite	Reviewer	Test Suite Reviewer
Save	Save Test Suite	Saver	Test Suite Saver
Search	Search Test Suite	Searcher	Test Suite Searcher
Split	Split Test Suite	Splitter	Test Suite Splitter
Sponsor	Sponsor Test Suite	Sponsor	Test Suite Sponsor
Store	Store Test Suite	Storer	Test Suite Storer
Strategize	Strategize Test Suite	Strategizer (or Strategy Setter)	Test Suite Strategizer (or Test Suite Strategy Setter)
Support	Support Test Suite	Supporter	Test Suite Supporter
Train	Train Test Suite	Trainer	Test Suite Trainer
Upgrade	Upgrade Test Suite	Upgrader	Test Suite Upgrader
Upload	Upload Test Suite	Uploader	Test Suite Uploader
Version	Version Test Suite	Versioner	Test Suite Versioner
View	View Test Suite	Viewer	Test Suite Viewer

At a minimum, the above list of Verbs can be used to help identify, track, and manage the basic "Features" required by and associated with Test Suite Management, even if your enterprise doesn't maintain a Capability Model that lists specific Test Suite Management Capabilities. Application designers, developers, and architects often find such Verb Lists or Feature Inventories to be invaluable.

Taxonomy: Understanding Test Suite Management Classifications or Categorizations

A Taxonomy, in its noun form, is defined as:

...a documented and orderly set of types, classifications, categorizations and/or principles that are often achieved through mechanisms including but not limited to naming, defining and/or the grouping of attributes, and which ultimately help to describe, differentiate, identify, arrange and provide contextual relationships between the entities for which the Taxonomy exists.

From this general definition, we can derive that the definition for a Test Suite Management Taxonomy is:

...a documented and orderly set of types, classifications, categorizations and/or principles that are often achieved through mechanisms including but not limited to naming, defining and/or the grouping of attributes, and which ultimately help to describe, differentiate, identify, arrange and provide contextual relationships between Test Suite Items, Entities or Types.

In short, what this means all means is that a Taxonomy is nothing more than a classification or typing mechanism and that a Test Suite Taxonomy is nothing more than a classification or typing mechanism that helps people and systems distinguish between different Test Suite Items, Entities, Types, Records or any other Test Suite Management element you can think of.

It's important to understand that Taxonomies can be as simple as a list of relevant terms or phrases with respective meanings or definitions or they can take on more complex forms, such as hierarchical and graphical model structures that can be homogeneous and heterogeneous in nature. More complex Taxonomies include examples such as "Visual Taxonomies" and "Audible Taxonomies" but, expect in the case of very special technologies, are typically out of scope for general Information Technology (IT) Operations.

The Foundation directs readers to its ever-evolving Inventory of Taxonomies for Standard Taxonomy suggestions. Specifically, readers may want to start with the Taxonomy of Taxonomies, which helps make it clear that the IT Industry is composed of many hundreds if not thousands of Taxonomies, Classifications, Categorizations or Types.

Life Cycle (Lifecycle): Data Lifecycle Phases for Test Suite Management

Data Lifecycle (or Life Cycle) for any and all data is the period from the "inception" of data through to its ultimately being "purged" from existence. This is no different for Test Suite Management related data.

Like the data associated with any other professional IT Discipline, Test Suite Management related data adheres to the following common Data Lifecycle Phases:

Figure: Test Suite Management Lifecycle Phases

1. Inception: Data is in it's raw idea-like form and is not ready for consumption by the general population because it has not been documented or registered, anywhere, in a formal manner.
2. Creation and Registration: Data is formally put into existence for day-to-day use by appropriate stakeholders.
3. Iterative Maintenance: Data is in a mode of constant use and is updated and modified, as needed, to meet the needs of daily use by various stakeholders.
4. Decommission and Deletion: Data is prepared for deletion and eventually deleted from daily operational use but still exists for administrative or organizational purposes, such as historical auditing. It can be restored to any one of its relevant last states and, therefore, can be brought back into existence for day-to-day use.
5. Purged From Existence: Data is completely removed from an environment with no means to restore or reconstruct it, without recreating it from scratch and with no guarantees that it will match it's previous state.

The above Life Cycle Phases represent the high level transitions that occur from the inception of Test Suite Items or Entities all the way through to their complete elimination from existence. A more detailed breakdown of these transitions or phases represents what are referred to as "Test Suite Management States."

Metrics: Test Suite Management Metrics

As with any professional Discipline, the place to start with when dealing with Test Suite Management specific metrics is with standard metrics categorizations. Standard Metrics Categorizations, or what are commonly referred to as "SMCs," include but are not limited to...

• Quantitative Metrics or, in this case, Test Suite Management Quantitative Metrics
• Qualitative Metrics or, in this case, Test Suite Management Qualitative Metrics
• Time Metrics or, in this case, Test Suite Management Time Metrics
• Utilization Metrics or, in this case, Test Suite Management Utilization Metrics
• Financial Metrics or, in this case, Test Suite Management Financial Metrics

Test Suite Management Quantitative Metrics: Quantitative metrics for Test Suite Management often revolve around the "counting" of key constructs that are associated with the Discipline. For example, the number of Test Suite Items or Entities that have been Created, Edited or Modified, Copied or Cloned, Destroyed, Archived, Restored, etc. (Note the correlations to key Test Suite Management Verbs!). Also, the counts for things like the number of Test Suite Management Stakeholders, such as but not limited to Paying Customers, End Users, Employees, Consultants, etc. are also very useful.

Test Suite Management Qualitative Metrics: Qualitative metrics for Test Suite Management often revolve around concepts such as Test Suite Management Defects, Failures, Problems, Incidents, and/or Issues. So, for example, if we were to capture the number of Test Suite Management Defects (i.e. their counts) over time, we could do things like see if Defect quantities are going up or down, over time, allowing us to explore that area for things like correlating Causes and Effects.

Test Suite Management Time Metrics: When dealing with Test Suite Management Time Metrics, there are usually two forms. The first was introduced in the previous paragraph, which has to do with capturing and measuring things like Quantitative or Qualitative Metrics, over time. In this case, we capture other metric categories, over time, with the intent to see how they change and perform, based on modifications to the Test Suite Management Operating Environment. The second form of Time related metrics has to do with system or operational performance, such as in the case of how long it takes to process a Test Suite Management Request, from the time it is created to the time the Requester gets a satisfactory deliverable that allows him or her to move on with his or her work.

Test Suite Management Utilization Metrics: Utilization Metrics specifically have to do with the consumption of Test Suite Management specific solutions or deliverables. For example, tracking the number of Test Suite Management Service Requests, over periods of time, along with their corresponding Test Suite Management Deliverables, allows one to measure how active Test Suite Management Services are against other Services that may exist within the Enterprise.

Test Suite Management Financial Metrics: As is always the case for any single Discipline, Financial Metrics for Test Suite Management always revolve around things like revenue, expenses, and profits, both, for operators of the Service or Services and for consumers of the Service or Services. For example, if a Test Suite Management Request is invoked by a Test Suite Management Customer (acting as the "Requester"), it becomes important to be able to identify and understand what the cost is to that Customer who is invoking the Request, and it also becomes important to understand why that cost is what it is. In the case of Services that do not yield revenue or profits, measuring costs is a strong way to, at very least, help understand the costs associated with each Service being performed by, within, external to, and for the Enterprise and its Customers.

Note: It's important to understand that, when it comes to metrics, enterprises should take a "Crawl," "Walk," "Run" approach to collecting, working with, and understanding them. That is, you cannot get to complex metrics collection, dissection, analysis, and understanding until you start with basic metrics and slowly work your way to more complex metrics representations.

Services: Test Suite Management as a Set of Services (a.k.a. Test Suite Management Services)

One of the most important concepts you will learn about Test Suite Management (or any Discipline, for that matter) is the notion of implementing the Discipline as an accountable, planned, controlled, transparent, and managed "Service."

In short, Services represent a logically "bounded" and repeatable sets of work types, activities or tasks that are performed by humans and/or machines, with the specific intent to provide outputs or deliverables, in the form of solutions for the requesting Stakeholders who are commonly considered the customers of such Services. In other words, we perform and/or provide a Service to deliver very specific solutions to very specific Stakeholders who are looking for a means to solve a certain problem they have.

Services can be performed manually (i.e. by people), automatically (i.e. by machines such as Computers), or by a combination of the two (i.e. a hybrid that is both manually and automated).

Test Suite Management Service Components: The successful implementation of Test Suite Management as a set of Services for your enterprise usually implies that a number of key components have been established to support it. These components are:

1. A clearly documented and socialized Test Suite Management Service Owner
2. A clearly documented and socialized Test Suite Management Service Provider, Organization or Group who is performing the Service or work
3. A clearly documented and socialized inventory of all Test Suite Management Service Inputs, including Test Suite Management Service Requests and any artifacts necessary to support such Requests
4. For every Test Suite Management Service Input, a clearly documented and socialized inventory of Test Suite Management Service Outputs, making it clear to consumers what they can expect to receive as a result of a successful Service Request
5. For every Test Suite Management Service Input, a clearly documented and socialized inventory of the work being performed by the Service Provider to achieve such Outputs or Deliverables
6. For every Test Suite Management Service Input, a clearly documented and socialized inventory Service Levels that can be used to set expectations and measure actuals against for said Service Outputs

Test Suite Management Ownership: The most important thing to understand about a Test Suite Management Service is that, in order for such a Service to be successful, there must be a clear and accountable Owner for it. That is, there needs to be a very clear and accountable named person or organization that owns and is fully responsible for the Service, all of its sub-Services and, most importantly, all of the Service's "Outcomes." Without clear ownership, Services are almost never successful. And, for those few occasions where Services are successful without clear ownership, you can assume that they're successful because the people working in those Service areas are acting as heroes, or... the those Services are just plain lucky (that kind of luck doesn't last for long).

Test Suite Management Service Inputs: There are typically two types of inputs to any Test Suite Management Service. The first is what is known as a "Test Suite Management Service Request" and the second really represents any and all supporting artifacts that are necessary to support such requests, including but not limited to Data and Information in the form of Documents, either electronic or paper in form. Many would argue that the "money" to pay for the Service execution of the Request would be the third but, for now, we will assume that payment is controlled through the Data and Information provided to the Service Operators, in support of the Request.

Test Suite Management Service Outputs: The outputs of any Service are often referred to as the Service's Deliverables. Therefore, the readers should be aware that the terms "Test Suite Management Outputs" and "Test Suite Management Deliverables" are synonymous and interchangeable. All work performed in any enterprise is, by default, a Service that is being performed for someone else and, therefore, all work or Services yield results. These results are the Service's Outputs or Deliverables and a good Service ensures that such Outputs are appropriately documented to the consumers of said Service. This means that for any given Test Suite Management Service Request Type or Category there will be one or more clearly defined and documented Outputs or Deliverables, making it clear to the consumer what he, she, or they will get in response to their Request. This can be as simple as an answer to a question or as complex as the Merger of two enterprises.

Test Suite Management Service Levels: Service Levels represent "performance agreements," contractual or otherwise, that dictate how well a Test Suite Management Service should perform, most often keeping the Customers, Consumers, Clients or End Users of the Service in mind. Test Suite Management Service Levels can come in many forms and are often worked out by the Customers paying for the Services and the Service Providers who sell or provide the Services. In many cases, Service Levels are also self-imposed by the Service Providers performing the Services as a means to set expectations for Service Customers. In short, Test Suite Management Service Levels are constraints, limitations, and/or expectations that are tied directly to Test Suite Management Service Deliverables. They represent measures for things like quality, efficiency, and cost against said Deliverables or Outputs that allow the consumer of such Services to measure what they actually get against what they expected to get.

Principles & Best Practices: Common Principles and Best Practices for Test Suite Management

A "Principle" is defined as being: "A professed assumption, basis, tenet, doctrine, plan of action or code of conduct for activities, work or behavior." Therefore, we can deduce the definition of "a Test Suite Management Principle" to be:

Test Suite Management Principle: "1. A professed assumption, basis, tenet, doctrine, plan of action or code of conduct for any activities, work or behavior associated with the Discipline known as Test Suite Management."

A "Best Practice" is defined as being: "One or more Activities, Actions, Tasks or Functions that often do not conform with strict Standards and that have evolved, over time, to be considered as conventional wisdom for consistently and repeated achieving Outcomes or Results that can be measured as being equal to or above acceptable norms." Therefore, we can deduce the definition of "a Test Suite Management Best Practice" to be:

Test Suite Management Best Practice: "1. One or more Test Suite Management related Activities, Actions, Tasks or Functions that often do not conform with strict standards and that have evolved, over time, to be considered as conventional wisdom for consistently and repeatedly achieving Outcomes or Results that can be measured as being equal to or above acceptable norms."

The plural form of this term would be "Test Suite Management Best Practices."

Common Test Suite Management related principles and best practices exist to help achieve higher than average expectations of quality and to ease in the implementation, support, operations, and future change associated with the solutions industry professionals put in place to address the needs of this Discipline and all its related stakeholders.

While this entire document is meant to represent and serve as a set of common principles and best practices for Test Suite Management, the following list represents a summary of some very basic examples of what implementers, supporters, and operators of Test Suite Management should constantly be working toward:

Principle or Best Practice	Description
Establish and always have very clear Ownership for Test Suite Management.	Establishing, publishing and socializing clear Ownership for Test Suite Management allows an enterprise and all its Resources, regardless of their geographic location, to assign accountability for all aspects of the Discipline. It also ensures that there's always at least one person that everyone can go to for transparency into the Discipline as well as for handling work that is associated with the Discipline.
Always use standard terminology for Test Suite Management, in order to standardize communications between stakeholders.	It is often argued that the biggest mistake you can make is to create your own words and/or your own definitions, when communicating with others. There is no place where this is more accurate than in the field of Information Technology. IT Stakeholders make up their own words and definitions far too often, or let their business constituents do so. When you make up words or definitions, or you let others do so, you're creating a grave injustice for your organization. Self invented terminology and grammar often leads to poor communications, which in turn leads to redundancy of solutions, higher complexity of environments, slower delivery times, and much higher costs. Therefore, the IF4IT always recommends that you leverage standard terminology for Test Suite Management, whenever possible.
Centralization of Test Suite related data.	While often impossible to centralize and collocate all Test Suite related data and information, especially in a geographically dispersed environment, Test Suite Management related stakeholders should always strive to centralize all data and information. The goals are to eliminate data fragmentation, improve source of truth for data, reduce the number of systems needed to support stakeholders, reduce the complexity of solutions, improve usability, and to ultimately reduce the costs associated with Test Suite Management.
Transparency of Test Suite related data.	Stakeholders should always strive to make any and all Test Suite Management data transparent to all other appropriate stakeholders, at a minimum, and often to the entire enterprises. The exception when private user data must be protected. Many stakeholders often make the mistake of treating internal operational data as private or protected. This often creates a data silo and will often lead to internally silo-ed organizations that revolve around such data silos.
Do not let "perfection" of Test Suite Management solutions stand in the way of "good enough solutions".	Often, Test Suite Management stakeholders "overthink" solutions, leading to the impression that best-of-breed or perfect solutions are more effective than "good enough" solutions. Experience tells us that "good enough" is, almost always, the better path to follow. We live in an age where technologies grow old in the blink of an eye. Even the implementation of something that looks perfect, today, will look antiquated, tomorrow. This is especially true if your enterprise doesn't have a long term funding plan and commitment to improvements and upgrades of the solution(s) put in place.
Follow industry Standards, Best Practices, and Guiding Principles for Test Suite Management, whenever possible".	One of the most common errors many enterprises make is to create solutions from scratch or without the guidance, assistance and/or experience of others who have created such solutions, before them. Whenever possible, the IF4IT recommends that you research existing Standards, Best Practices, and Guiding Principles to avoid the mistakes of others, while also gaining from their successes. Remember, we live in a vast world. Chances are very high that someone else has already experienced the pain you're about to create for yourself. Wise people will always look to learn from such people's experiences before they go down the road of implementing their own solutions.
Work toward and maintain a Single Source of Truth (SSoT), whenever possible.	While it may be impossible to truly maintain a Single Source of Truth (SSoT) for all data items at all times, especially in the case where the same data entity or instance enters an enterprise through unique data channels, it is an accepted, industry-wide best practice to always work toward such a goal.

Further Reading and Reference Material for Test Suite Management

The Information Technology (IT) Learning Framework. A tutorial that helps understand Information Technology and how disciplines, such as this one, fits into the bigger picture of IT Operations.