Combinatorial Analysis: Transforming Software Testing with Mathematics!

"Go down deep enough into anything, and you will find mathematics" - Dean Schlicter

In the dynamic and complex landscape of software engineering, testing accuracy and comprehensiveness are critical to ensuring the delivery of high-quality solutions. As programmers and software engineers, we face the daily challenge of writing tests that effectively address the vast array of possible scenarios in our systems. In this context, traditional tools and approaches can sometimes fail to cover all the nuances, leading to potential gaps in our test coverage.

That's why approximately 8 months ago I started a series of studies on how to improve my daily life when developing software, using something that we can always trust due to its logic and often precision, mathematics!

And it was from these studies that I noticed something curious: combinatorial analysis and its mathematical logic emerge as powerful allies, transforming our approach when dealing with software testing. By applying fundamental principles of counting, combinations, groupings, arrangements, and permutations, we can structure our tests in a more systematic and comprehensive way. These mathematical techniques allow us to quantify and explore the universe of test scenarios, ensuring that no cases are overlooked.

In this article, we will explore how combinatorial analysis can be applied to software test engineering, transforming the way we develop, organize, and prioritize our tests. With practical examples relevant to our daily lives as developers, we will illustrate how mathematics is not just a theoretical pillar, but a practical tool that can guide our efforts to achieve accuracy in software testing. The objective is not to impose something or say that this technique is better, but just to provide insights and strategies that you can apply if you wish, increasing the quality and reliability of the software.

If you like the content, please share and like the post! This helps me continue to bring content in text form!😄

Pure mathematics is, in its own way, the poetry of logical ideas.

- Albert Einstein, German theoretical physicist

Combinatorial Analysis: Fundamentals and Applications

What Is Combinatorial Analysis?

Combinatorial analysis is a branch of pure mathematics that deals with the counting, arrangement, and combination of objects within a defined set, according to specific rules. Essentially, it helps us calculate the number of possible configurations or groupings that can be formed from a set of items, where the order of items may or may not matter. This field of mathematics is fundamental to solving problems in probability and statistics, and it has its roots in ancient mathematics, evolving significantly over the centuries.

Origins and Development

Combinatorial analysis is not a modern concept; its origins can be traced back to ancient studies on permutations and combinations in India and ancient Greece. Over the years, mathematicians such as Blaise Pascal and Pierre de Fermat expanded its applicability, especially in the context of probability calculation. However, it was in the 20th century that combinatorial analysis flourished as a field of study within mathematics, with applications extending across various disciplines, from physics to computer science.

Everyday Applications

In everyday life, combinatorial analysis can be seen in simple actions, such as deciding the order of reading books on a shelf, organizing a project team from a group of colleagues, or even when choosing ingredients for a recipe. In each of these situations, we are unconsciously performing combinatorial calculations.

And How Does It Fit into Software Testing?

In software engineering, combinatorial analysis plays a crucial role in helping identify all possible states or inputs that a system can receive, which can aid in comprehensive test coverage. In an environment where systems are becoming increasingly complex, with numerous variables and interdependent states, applying combinatorial principles allows test engineers to create more efficient and effective test plans, minimizing the likelihood of undetected bugs.

Principles and Techniques for Software Testing

• Fundamental Principle of Counting: This basic rule helps us determine the total number of possible test scenarios when multiple independent variables are involved, providing a solid foundation for structuring tests.

• Combinations and Groupings: These concepts allow us to consider different groupings of inputs without regard to order, essential for testing features that depend on multiple selections or configurations.

• Permutations: The analysis of permutations is relevant when the order of elements affects the outcome. In software testing, this is particularly useful for testing sequences of operations or transactions.

By exploring and applying these principles of combinatorial analysis, we can transform the way we approach software testing, from design to execution, ensuring more robust and reliable systems. In the following topics, we will dive into each of these techniques, highlighting their practical applicability in software testing scenarios, with concrete examples that illustrate their effectiveness.

Fundamental Principle of Counting: The Pillar in Building Efficient Tests

This principle is a cornerstone in understanding how we can structure our unit tests more intelligently and effectively.

Imagine you're at an ice cream parlor, deciding on the layers of a delicious sundae. First, you choose from 3 ice cream flavors. Then, you decide whether you want chocolate, caramel, or strawberry topping. And finally, you choose whether or not to add sprinkles. How many different sundae combinations can you create? The Fundamental Principle of Counting tells us to simply multiply the options for each decision:

3 flavors X 3 toppings X 2 sprinkle options = 18 unique combinations. 

Mathematically, if we have one task that can be performed in n ways and another subsequent task that can be performed in m ways, then together, these tasks can be performed in n×m ways. This principle can be extended to any number of sequential tasks, providing a powerful foundation for counting in scenarios with multiple steps or decisions.

So how to use this in software testing? Well, the first thing you need is to understand all the inputs you have in a method, in addition, we need to take into account the branches within that method. We'll talk about this soon, but first, I want to show a simpler example of the FPC applied in practice.

Imagine we are testing a profile setting function in an application, where users can choose a language (English, Spanish, French), a theme (Light, Dark), and enable or disable notifications. Following the Fundamental Principle of Counting, we have 3×2×2 = 12 possible test scenarios.

Shall we translate this into code? Follow along with the example below:

Consider a UserProfile class with a configureProfile method that accepts language, theme, and notification status as parameters. Let's implement this class and its unit tests using JUnit 5.

public class UserProfile {
    public String configureProfile(String language, String theme, boolean notificationsActive) {
        // Profile configuration logic
        return "Profile configured with " + language + ", " + theme + " and notifications " + (notificationsActive ? "active" : "deactivated");
    }
}

For our tests, we will cover all 12 possible scenarios, applying the Fundamental Counting Principle to ensure complete coverage.

import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;

class UserProfileTest {

    UserProfile profile = new UserProfile();

    @Test
    void configureProfile_English_Light_NotificationsActive() {
        assertEquals("Profile configured with English, Light and notifications active", profile.configureProfile("English", "Light", true));
    }

    @Test
    void configureProfile_English_Light_NotificationsDeactivated() {
        assertEquals("Profile configured with English, Light and notifications deactivated", profile.configureProfile("English", "Light", false));
    }

    @Test
    void configureProfile_English_Dark_NotificationsActive() {
        assertEquals("Profile configured with English, Dark and notifications active", profile.configureProfile("English", "Dark", true));
    }

    @Test
    void configureProfile_English_Dark_NotificationsDeactivated() {
        assertEquals("Profile configured with English, Dark and notifications deactivated", profile.configureProfile("English", "Dark", false));
    }

    @Test
    void configureProfile_Spanish_Light_NotificationsActive() {
        assertEquals("Profile configured with Spanish, Light and notifications active", profile.configureProfile("Spanish", "Light", true));
    }

    @Test
    void configureProfile_Spanish_Light_NotificationsDeactivated() {
        assertEquals("Profile configured with Spanish, Light and notifications deactivated", profile.configureProfile("Spanish", "Light", false));
    }

    @Test
    void configureProfile_Spanish_Dark_NotificationsActive() {
        assertEquals("Profile configured with Spanish, Dark and notifications active", profile.configureProfile("Spanish", "Dark", true));
    }

    @Test
    void configureProfile_Spanish_Dark_NotificationsDeactivated() {
        assertEquals("Profile configured with Spanish, Dark and notifications deactivated", profile.configureProfile("Spanish", "Dark", false));
    }

    @Test
    void configureProfile_French_Light_NotificationsActive() {
        assertEquals("Profile configured with French, Light and notifications active", profile.configureProfile("French", "Light", true));
    }

    @Test
    void configureProfile_French_Light_NotificationsDeactivated() {
        assertEquals("Profile configured with French, Light and notifications deactivated", profile.configureProfile("French", "Light", false));
    }

    @Test
    void configureProfile_French_Dark_NotificationsActive() {
        assertEquals("Profile configured with French, Dark and notifications active", profile.configureProfile("French", "Dark", true));
    }

    @Test
    void configureProfile_French_Dark_NotificationsDeactivated() {
        assertEquals("Profile configured with French, Dark and notifications deactivated", profile.configureProfile("French", "Dark", false));
    }
}

Each of these tests ensures that the configureProfile method of the UserProfile class functions as expected for the different combinations of language, theme, and notification status, covering all 12 possible scenarios.

But the question some might have is the following: can we rely on this? Would the scenario change if we had branching paths?

When we have 12 clearly defined scenarios, as in the example we worked with, there's a solid foundation to trust that we've covered all possible combinations of the variables in question. This is especially true when these variables are independent of each other, like the languages, themes, and notification settings in our example. In such situations, the Fundamental Principle of Counting provides a reliable methodology to ensure that each unique combination has been considered.

Complexity can increase when we introduce branching into the scenario. Branching occurs when the choice in one decision affects the available options in subsequent decisions. In these cases, we cannot simply apply the Fundamental Principle of Counting directly, as the decisions are no longer independent.

For instance, if we add the condition that certain notification settings are only available for specific themes (say, "active" notifications are only available for the "Dark" theme), then the number of valid scenarios might be reduced, and some combinations would become invalid.

Let's modify our example to understand this better:

public class ProfileConfigurer {
    // Method with branching
    public String configureProfile(String theme, boolean notifications, boolean energySaving) {
        if ("Dark".equals(theme) && energySaving) {
            return "Dark Theme with Energy Saving";
        } else if ("Dark".equals(theme)) {
            return "Dark Theme without Energy Saving";
        } else if ("Light".equals(theme) && notifications) {
            return "Light Theme with Notifications";
        } else {
            return "Light Theme without Notifications";
        }
    }
}

For the configureProfile method that presents branches, the traditional approach used to determine the total number of test scenarios does not follow a single direct formula from the Fundamental Counting Principle, due to the conditional dependence between the variables. Instead, the analysis is done in chunks, considering the specific ramifications of the method.

• Input ariables and their possible values:

  • Theme: Has two possible values ("Dark" and "Light").

  • Notifications: As it is a boolean, it has two possible values (true and false).

  • Energy Saving: Also being a boolean, it has two possible values (true and false).

Ramification Analysis:

• Dark Theme with Energy Saving: This condition is specific and requires the theme to be "Dark" and energySaving to be true. It does not depend on the notifications state.

• Dark Theme without Energy Saving: This condition is also specific to the theme being "Dark", but energySaving needs to be false. Again, it does not depend on notifications.

• Light Theme with Notifications: Requires the theme to be "Light" and notifications to be true. It does not depend on energySaving.

• Light Theme without Notifications: This is the default case when none of the other if's are met. It occurs when the theme is "Light" and notifications are false, regardless of energySaving.

Calculation of Test Scenarios:

• For Dark Theme:

  • With energy saving: 1 scenario (Dark Theme + Energy Saving enabled).

  • Without energy saving: 2 scenarios (Dark Theme + Energy Saving disabled, with notifications being either true or false).

• For Light Theme:

  • With notifications: 2 scenarios (Light Theme + Notifications enabled, with energy saving being either true or false).

  • Without notifications: 1 specific scenario (Light Theme + Notifications disabled, and energy saving does not influence).

Totaling:

• Dark Theme: 1 (energy saving) + 2 (without energy saving) = 3 scenarios.

• Light Theme: 2 (with notifications) + 1 (without notifications, energy saving does not influence) = 3 scenarios.

But it's clear that we did not apply any mathematics, just logic and separation of what we are going to test. So how would it look to apply the FPC directly to the method?

To apply the FPC, we consider each independent choice in the method's inputs and multiply the number of available options for each one.

1. Theme: 2 options ("Dark", "Light").

2. Notifications: 2 options (true, false).

3. Energy Saving: 2 options (true, false).

The total number of test scenarios is given by the product of the number of options for each input, which reflects the central idea of the FPC that each choice is independent.

Total Scenarios = Theme Options × Notification Options × Energy Saving Options

Total Scenarios = 2 × 2 × 2 = 8

This number includes all possible combinations of theme, notifications, and energy saving, regardless of the specific branches in the code. This calculation takes a purely combinatorial approach, where each combination of inputs is considered a potential test scenario, before considering the specific conditional logics that would reduce this number based on the business rules implemented in the method.

Let's identify what these 8 scenarios would be, including those that were not explicitly considered in the previous analysis:

Scenarios for "Dark Theme":

1. Dark + Notifications enabled + Energy Saving enabled.

2. Dark + Notifications enabled + Energy Saving disabled.

3. Dark + Notifications disabled + Energy Saving enabled.

4. Dark + Notifications disabled + Energy Saving disabled.

Scenarios for "Light Theme":

5. Light + Notifications enabled + Energy Saving enabled.

6. Light + Notifications enabled + Energy Saving disabled.

7. Light + Notifications disabled + Energy Saving enabled.

8. Light + Notifications disabled + Energy Saving disabled.

In scenarios 1 to 4 with the "Dark" theme, specific code branches are applied directly, considering whether or not to activate power saving, regardless of the status of notifications.

In scenarios 5 to 8 with the "Clear" theme, the differentiation occurs mainly based on the status of notifications, while energy savings, in theory, would not affect the result, according to the conditions specified in the configureProfile method.

Additional Scenarios Identified:

The previous analysis mainly considered the explicit branching in the code and may have omitted the following specific scenarios, which are included in the count of 8 derived from the FPC:

• Scenario 5 and 7 (Light Theme + Energy Saving enabled): These scenarios include the activation of energy saving with the light theme, which, although they do not directly alter the outcome according to the method's logic, are valid input combinations that need to be tested to ensure the system behaves as expected in all possible situations.

• Scenario 6 (Light Theme + Notifications enabled + Energy Saving disabled): This scenario is similar to 5, but with energy saving disabled, which is also a distinct combination of inputs that needs to be verified.

By including these additional scenarios in the testing, we ensure complete coverage, addressing all possible combinations of inputs, which aligns with the comprehensive approach suggested by the FPC. This way, if another programmer were to change this code in the future, they would be prevented from making a logic error; if they err and change the behavior without the business requirements demanding this change, the tests will fail, alerting that something was broken.

import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.springframework.test.context.junit.jupiter.SpringExtension;

import static org.junit.jupiter.api.Assertions.assertEquals;

@ExtendWith({SpringExtension.class})
class ProfileConfigurerTest {

    ProfileConfigurer configurer = new ProfileConfigurer();

    @Test
    @DisplayName("Dark Theme with Notifications enabled and Energy Saving enabled")
    void darkTheme_NotificationsEnabled_EnergySavingEnabled() {
        assertEquals("Dark Theme with Energy Saving", configurer.configureProfile("Dark", true, true));
    }

    @Test
    @DisplayName("Dark Theme with Notifications enabled and Energy Saving disabled")
    void darkTheme_NotificationsEnabled_EnergySavingDisabled() {
        assertEquals("Dark Theme without Energy Saving", configurer.configureProfile("Dark", true, false));
    }

    @Test
    @DisplayName("Dark Theme with Notifications disabled and Energy Saving enabled")
    void darkTheme_NotificationsDisabled_EnergySavingEnabled() {
        assertEquals("Dark Theme with Energy Saving", configurer.configureProfile("Dark", false, true));
    }

    @Test
    @DisplayName("Dark Theme with Notifications disabled and Energy Saving disabled")
    void darkTheme_NotificationsDisabled_EnergySavingDisabled() {
        assertEquals("Dark Theme without Energy Saving", configurer.configureProfile("Dark", false, false));
    }

    @Test
    @DisplayName("Light Theme with Notifications enabled and Energy Saving enabled")
    void lightTheme_NotificationsEnabled_EnergySavingEnabled() {
        assertEquals("Light Theme with Notifications", configurer.configureProfile("Light", true, true));
    }

    @Test
    @DisplayName("Light Theme with Notifications enabled and Energy Saving disabled")
    void lightTheme_NotificationsEnabled_EnergySavingDisabled() {
        assertEquals("Light Theme with Notifications", configurer.configureProfile("Light", true, false));
    }

    @Test
    @DisplayName("Light Theme with Notifications disabled and Energy Saving enabled")
    void lightTheme_NotificationsDisabled_EnergySavingEnabled() {
        assertEquals("Light Theme without Notifications", configurer.configureProfile("Light", false, true));
    }

    @Test
    @DisplayName("Light Theme with Notifications disabled and Energy Saving disabled")
    void lightTheme_NotificationsDisabled_EnergySavingDisabled() {
        assertEquals("Light Theme without Notifications", configurer.configureProfile("Light", false, false));
    }
}

Therefore, the approach here is to know when to segment the analysis based on the logical ramifications of the method, rather than directly applying n×m multiplication to all variables. This strategy is essential when the options available for one variable depend on the value of another, as is the case with the energy saving option that only applies to the "Dark" theme.

Groupings in Combinatorial Analysis

Groupings, known in the mathematics of combinatorial analysis as combinations, refer to the process of selecting items from a set such that the order of the selected items does not matter. In other words, we are interested in the possible subsets of a larger set, without taking into account the sequence in which the elements are chosen.

The central concept behind groupings is the idea of choosing a specific number of items from a larger set, where the only thing that matters is which items are chosen, not the order in which they appear. For example, if we have a set of letters {A, B, C}, and we want to choose 2 of them, the possible groupings would be {A, B}, {A, C}, and {B, C}. Note that {A, B} is considered the same grouping as {B, A}, because order is not relevant.

Let's move to an example in code:

import java.util.Set;

public class PermissionValidator {
    public boolean validatePermissions(Set<String> permissions) {
        // Rule 1: Must include the "read" permission
        if (!permissions.contains("read")) {
            return false;
        }

        // Rule 2: Cannot have "execute" without "write"
        if (permissions.contains("execute") && !permissions.contains("write")) {
            return false;
        }

        // If it passes all checks, the permission set is valid
        return true;
    }
}

For the case of the validatePermissions function, which checks a set of permissions based on specific rules, the concept of groupings (or combinations) helps us understand the number of distinct ways permissions can be grouped, regardless of order. However, due to the specific rules imposed by the function, not all theoretical groupings are valid.

When we talk about groupings in relation to the validatePermissions function example, we are referring to the distinct sets of permissions that can be formed, regardless of whether they comply with the rules set out in the function. Each grouping is a combination of permissions where order does not matter.

Available Permissions:

• Read

• Write

• Execute

Calculation of Groupings:

To calculate the number of possible groupings with these 3 permissions, we consider that each permission can be present or absent, leading to

\(2^3=8\)

possible combinations. This comes from the Fundamental Counting Principle, where we have 2 options (include or not include) for each of the 3 permissions.

Possible Groupings:

Let's list all 8 possible groupings generated by this calculation:

• {} - No permission

• {"read"}

• {"write"}

• {"execute"}

• {"read", "write"}

• {"read", "execute"}

• {"write", "execute"}

• {"read", "write", "execute"}

But there is something IMPORTANT that we need to clarify, so read the topic below:

So these are possible groupings. This includes all possible sets, from the empty set to the full set with all permissions.

Valid Groupings According to the Rules:

However, when we apply the specific rules of the validatePermissions function:

• The permission set must include "read".

• The set cannot include "execute" without including "write".

We identify that not all of the initial 8 groupings are valid. The groupings that comply with both rules and, therefore, are considered valid for the function, are:

• {"read"} - Just "read", complies with rule 1.

• {"read", "write"} - "Read" and "write", complies with both rules.

• {"read", "write", "execute"} - All permissions, complies with both rules.

Therefore, of the 8 possible groupings, only 3 are valid according to the rules established in the validatePermissions function.

Clarification:

• 8 Possible Groupings: Refers to the total number of combinations that can be formed with the available permissions, regardless of the rules.

• 3 Valid Groupings: Refers to the number of combinations that comply with the specific rules of the validatePermissions function.

Understanding the distinction between the total number of possible groupings and the number of valid groupings, which comply with specific rules, is crucial for several reasons that I describe below:

Test Coverage

By identifying all 8 possible groupings, we ensure that we are considering every combination of permissions a user could potentially have. This allows us to design our tests to cover each unique scenario, including those that the function is supposed to invalidate. This is crucial for complete test coverage, ensuring that the software behaves as expected in all possible situations.

Business Rule Validation

By focusing on the 3 valid groupings that comply with the rules of the validatePermissions function, we are directly testing the business logic implemented in the function. This helps us verify if the function is correctly enforcing the intended business rules, such as the need to have the "read" permission and the rule that "execute" requires "write".

Identification of Edge Cases

Differentiating between all possible groupings and those that are valid according to the function's rules allows us to identify edge cases that might not be immediately obvious. For example, a grouping that includes "execute" without "write" is an important edge case that tests the robustness of the function in dealing with invalid inputs.

Deep Understanding of the Application

This exercise of distinguishing between different types of groupings helps us and quality assurance testers (QAs) to gain a deeper understanding of the application and its business rules. This promotes better communication within the team and a more conscious and methodical approach to software development and testing.

In mathematics, the art of posing a question must be considered of greater value than solving it. - Georg Cantor

Why is the FPC the Pillar?

Simple permutation, simple arrangement, and groupings (combinations) are fundamental concepts of combinatorial analysis that help count and organize elements in sets in specific ways. Each of these concepts has a unique application, but all are intrinsically connected to the Fundamental Principle of Counting (FPC).

Let's understand this in a simpler way, but in the footnote, there will also be a mathematical way to understand it with the applied formulas.

Simple Arrangements

Imagine you have a keyring with spaces for exactly 3 keys, and you have 5 different keys to choose from. Simple arrangements would be all the different ways to choose and hang 3 of those keys on the keyring, considering that the order in which you hang them matters (for example, having the house key in the first slot, the car key in the second, and the office key in the third is different from having the car key first, the office key second, and the house key in the third).¹

Now let's relate this to something we as developers see in our day-to-day work, for example, consider you're testing an e-commerce system and need to verify the checkout process. Specifically, you want to test the sequence of adding products to the cart, applying a discount coupon, and selecting a shipping method. Simple arrangements are useful here because the order in which these actions occur can affect the final outcome of the checkout. For instance, applying a coupon before adding all the products might not yield the same result as applying it afterward. Testing different sequences of these actions ensures that the checkout process is robust and functions correctly in various scenarios.

Simple Permutations

Now, imagine you want to organize your 5 favorite books on a shelf that has space for all of them. Simple permutations are all the different ways to arrange these books on the shelf, where the order of the books is significant. If you swap two books around, that counts as a new arrangement.²

Bringing this into software engineering as well, let's consider that you're testing an application that allows users to customize a dashboard by dragging and dropping widgets. Simple permutations come into play when, for instance, we want to ensure that any possible configuration of widgets works smoothly. This means testing all the possible ways to arrange the widgets. If there are 3 widgets, you would test all the ways to arrange them to ensure that the dashboard always displays information correctly, regardless of how the widgets are ordered.

For an example closer to the daily life of programmers, we can consider the order of function execution in a script. Suppose you have 3 critical functions in your script: validateData(), processData(), and saveData(). The order in which these functions are called is crucial for the correct operation of the script. Simple permutations come into play when we want to explore all the possible ways to organize the calling of these functions to ensure the robustness of the script.

Groupings

I want to emphasize groupings again, knowing we've already discussed them, but it's worth reinforcing because it can be confusing when read for the first time.³

Suppose you have a bouquet with 5 different flowers and you want to select just 3 to place in a smaller vase. Groupings, or combinations, refer to the different selections of flowers you can make, but unlike arrangements, the order of the flowers in the vase doesn't matter. If you choose a rose, a lily, and a daisy, it's the same combination as choosing a lily, a daisy, and a rose.

And this can be applied if you are working on a project management system and want to test users' access permissions. Some users can create tasks, others can just view them, and some can do both. Using the concept of groupings, you would create tests for all possible combinations of permissions to ensure that the permissions system is working as expected. This means testing not only users with all permissions, but also those with a specific subset of permissions, ensuring that each permission group is validated.

Fundamental Counting Principle (PFC)

Imagine you are getting dressed in the morning and you have 2 shirts and 3 pants to choose from. The PFC is how you would calculate all possible shirt and pants combinations. You multiply the number of shirt choices by the number of pants choices (2 shirts x 3 pants), giving you the total possible outfit combinations.

How Everything Relies on the FPC

• Simple Arrangements: They are based on the idea of the FPC, where each choice (each space on the keyring, for example) has a specific number of options, and you multiply these options to find all the possible ways to make these choices in sequence .

• Simple Permutations: Are a special case of the FPC where you use all available options (all books, for example), and the multiplication of these options (choices for the first, second, third book, etc.) gives you the total number of ways to organize the items.

• Groupings: Although order doesn't matter here, the initial idea of selecting items from a larger set (choosing 3 out of 5 flowers, for example) still relies on the FPC logic of making sequential choices. However, we adjust the final count to remove the importance of the order of choices.

Therefore, the Fundamental Principle of Counting is the foundation that allows us to understand and calculate the number of ways to make sequential choices, whether considering the order of those choices (arrangements and permutations) or not (groupings). It helps us structure and quantify the possibilities in various situations, from simple daily tasks to complex software testing problems.

But there are some caution points to consider. ⚠️

Independence of Choices

The FPC assumes that each choice or decision is independent of the others. However, in many software systems, actions can be interdependent. For example, a successful withdrawal in a banking system depends on having sufficient balance, which in turn may depend on previous deposits. It's crucial to identify and consider these dependencies when applying the FPC to ensure the validity of the generated test scenarios.

Coverage of Edge Cases

While the FPC can help calculate the total number of test scenarios, it's important to ensure that edge cases are adequately represented. Edge cases are situations that occur at the boundaries of requirements or business rules and often reveal bugs or unexpected behaviors. Make sure that the test scenarios include these important cases.

Complexity and Feasibility

Applying the FPC can lead to a very large number of test scenarios, especially in complex systems with many independent variables. While comprehensive coverage is important, it's also essential to consider the feasibility and efficiency of testing. It may be necessary to prioritize and select a representative subset of scenarios for testing, especially in situations with limited resources.

Sequences and Order of Operations

In many systems, the order of operations is crucial. The FPC can help identify different sequences of actions, but it's important to carefully analyze whether all the generated sequences are valid within the system's context. Some sequences may not make sense from a business logic perspective or may be unreachable due to system constraints.

Initial States and Conditions

The system's initial state and the conditions under which tests are run can significantly affect the results. When applying the FPC, consider how different initial states or conditions might influence the possible choices and outcomes of the tests.

Exception and Error Handling

When I was studying and writing this article, I thought about exception scenarios, how can we plan tests based on the FPC that don't forget to include scenarios that should result in exceptions or errors?

It's as important to test the system's ability to handle invalid inputs or error situations as it is to test its behavior under ideal conditions. Well, I will try to explain my point of view and what I learned from studying and trying to apply.

Regarding exceptions, let's consider the example below:

public class BankAccount {
    private double balance;

    public BankAccount(double initialBalance) {
        if (initialBalance < 0) {
            throw new IllegalArgumentException("Initial balance cannot be negative.");
        }
        this.balance = initialBalance;
    }

    public void deposit(double amount) {
        if (amount <= 0) {
            throw new IllegalArgumentException("Deposit amount must be positive.");
        }
        balance += amount;
    }

    public void withdraw(double amount) {
        if (amount <= 0) {
            throw new IllegalArgumentException("Withdrawal amount must be positive.");
        }
        if (amount > balance) {
            throw new IllegalArgumentException("Insufficient balance for withdrawal.");
        }
        balance -= amount;
    }

    public double getBalance() {
        return balance;
    }
}

Applying the FPC to Plan Unit Tests

To plan our unit tests, we consider the operations and conditions that can throw exceptions:

1. Constructor: Throws an exception if the initial balance is negative.

2. Deposit: Throws an exception if the deposit amount is negative or zero.

3. Withdrawal: Throws an exception if the withdrawal amount is negative, zero, or greater than the balance.

Applying the FPC, we consider each method and the conditions that can lead to an exception:

1. Constructor: 1 failure condition (negative initial balance) → 1 test.

2. Deposit: 1 failure condition (negative or zero amount) → 1 test.

3. Withdrawal: 2 failure conditions (negative or zero amount, amount greater than the balance) → 2 tests.

In total, we have 4 distinct tests focused on exceptions.

import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;

class BankAccountTest {

    @Test
    void constructorWithNegativeInitialBalance_ShouldThrowException() {
        Exception exception = assertThrows(IllegalArgumentException.class, () -> new BankAccount(-100));
        assertEquals("Initial balance cannot be negative.", exception.getMessage());
    }

    @Test
    void depositWithNegativeValue_ShouldThrowException() {
        BankAccount account = new BankAccount(100);
        Exception exception = assertThrows(IllegalArgumentException.class, () -> account.deposit(-50));
        assertEquals("Deposit amount must be positive.", exception.getMessage());
    }

    @Test
    void withdrawWithNegativeValue_ShouldThrowException() {
        BankAccount account = new BankAccount(100);
        Exception exception = assertThrows(IllegalArgumentException.class, () -> account.withdraw(-50));
        assertEquals("Withdrawal amount must be positive.", exception.getMessage());
    }

    @Test
    void withdrawAmountGreaterThanBalance_ShouldThrowException() {
        BankAccount account = new BankAccount(100);
        Exception exception = assertThrows(IllegalArgumentException.class, () -> account.withdraw(150));
        assertEquals("Insufficient balance for withdrawal.", exception.getMessage());
    }
}

But it is clear that calculating the exact number of invalid scenarios in the ContaBancaria situation was not completely straightforward or simple as in other examples. This is due to the interdependent and conditional nature of the operations involved (creating an account, depositing, withdrawing) and the specific rules for each of these operations.

Challenges in Applying the FPC:

State Dependence: The outcome and validity of an operation (especially withdrawals) depend on the current state of the account, which is affected by previous operations. This introduces complexity that goes beyond a simple application of the FPC, where each choice is independent of the others.

Conditional Variation: The number of invalid scenarios for withdrawals can vary depending on the current balance, which is influenced by previous deposits and the initial balance. This creates a situation where the number of possible scenarios isn't simply the product of the available options.

Indirect Application of the FPC:

Although the direct application of the FPC to calculate the total number of valid and invalid scenarios was challenging due to these factors, the principle still provided a logical foundation to structure thought and begin analysis:

• We identified individual operations and the conditions that would lead to invalid outcomes.

• We recognized the need to consider the sequence of operations and their impact on the account's state.

Indeed, we can create test cases for invalid scenarios based on the FPC's mathematics, especially when dealing with a series of independent choices leading to invalid states. The challenge arises when the choices aren't entirely independent or when the system's logic introduces dependencies between choices, as in the BankAccount example.

Applying the FPC to Invalid Scenarios:

For invalid scenarios involving independent choices, the FPC can be applied as follows:

1. List all the independent choices that can lead to an invalid state. For example, in the BankAccount case, choices like starting with a negative balance, depositing a negative or zero value, and withdrawing a negative or zero value are independent of each other, and each can lead to an invalid state.

2. Calculate the number of ways each choice can be made. If each invalid choice is unique (for example, only one way to deposit a negative value), then the calculation is straightforward.

3. Combine the choices to form invalid scenarios. If the choices are independent (one does not affect the possibility of the other occurring), then you can multiply the number of options for each choice to get the total number of invalid scenarios. For example, if there are 2 ways to create an account that would result in exceptions (very negative balance and slightly negative) and only 1 way to make an invalid deposit (zero value), then the total number of invalid scenarios would be the product of these options.

Challenges with Dependencies:

The challenge arises when there are dependencies, as in the case of a withdrawal that exceeds the available balance. In this scenario, the validity of the withdrawal depends on previous operations (account creation and deposits), making the choices interdependent. In such cases:

1. Analyze the dependencies to understand how previous choices affect the available options in subsequent choices.

2. Group related choices and treat them as a single complex step in the calculation, if possible.

3. Consider specific scenarios where interdependency is critical and calculate the possibilities for those scenarios.

Although the FPC offers a systematic approach to calculating the number of test scenarios, both valid and invalid, application in scenarios with complex dependencies may require additional analyses. For these cases, the FPC still serves as a basis for structuring thought and analysis, but it may be necessary to complement it with other combinatorial analysis techniques.

Having said all that, let's take a closer look at how we can use the FPC to arrive at a relevant number of test cases for valid and invalid scenarios when we have branching with logical operators.

Working with Logical Operators

Let's assume we have a Validator class that contains a method validateData which takes two parameters: age and score. This method validates whether the age is within a specific range (for example, 18 to 65 years) and if the score is above a certain value (for example, 50).

public class Validator {
    public boolean validateData(int age, int score) {
        if (age >= 18 && age <= 65 && score > 50) {
            return true; // Validation successful
        }
        return false; // Validation failed
    }
}

Applying the FPC: Calculation Approach for Valid Scenarios

When considering valid scenarios where both conditions linked by the && must be true:

• For the age condition (age >= 18 && age <= 65): We identify 2 representative scenarios that we want to test that satisfy this condition (for example, 20 years as a typical case within the range and 65 years as a boundary case).

• For the score condition (score > 50): Here we identify another 2 representative scenarios for the score that we want to test that satisfy this condition (for example, 55 as a typical case above the limit and 51 as a near-boundary case).

• Combining Conditions: Since we are dealing with the && operator, only scenarios where both conditions are true simultaneously are relevant. If we take 2 scenarios for age that satisfy the first condition and 2 scenarios for score that satisfy the second condition, and each of these scenarios for one condition can be combined with each scenario of the other condition, we would potentially have 2 * 2 = 4 combinations.

However, as we are considering only valid scenarios (where both conditions are true), we need to ensure that the selected combinations of age and score are logically consistent and meet both conditions. Therefore, each valid age scenario should be paired with each valid score scenario, maintaining a total of 4 valid scenarios.

But what about invalid scenarios where the method should return false?

Let's expand the scenarios to include not only when these conditions are met (true) but also when they are not (false).

Applying the FPC: Calculation Approach for Valid Scenarios

When we consider valid scenarios where both conditions linked by && must be true:

• For the age condition (age >= 18 && age <= 65): We identify 2 representative scenarios we want to test that satisfy this condition (for example, 20 years as a typical case within the range and 65 years as a boundary case).

• For the score condition (score > 50): Here we identify another 2 representative scenarios for the score we want to test that satisfy this condition (for example, 55 as a typical case above the limit and 51 as a near-boundary case).

• Combining Conditions: As we are dealing with the && operator, only scenarios where both conditions are true simultaneously are relevant. If we take 2 scenarios for age that satisfy the first condition and 2 scenarios for score that satisfy the second condition, and each of these scenarios for one condition can be combined with each scenario from the other condition, we would potentially have 2 * 2 = 4 combinations.

However, as we are only considering valid scenarios (where both conditions are true), we need to ensure that the selected combinations of age and score are logically consistent and meet both conditions. Therefore, each valid age scenario should be paired with each valid score scenario, maintaining a total of 4 valid scenarios.

• Scenarios for Age Condition:

  • True: We can have 2 representative scenarios for valid tests, as mentioned earlier (for example, 20 and 65 years).

  • False: We can identify 2 more scenarios to test the false condition, like an age below 18 (for example, 16 years) and one above 65 (for example, 70 years). This gives us a total of 4 scenarios for the age condition.

• Scenarios for Score Condition:

  • True: We consider 2 scenarios for when the condition is true (for example, scores of 55 and 51).

  • False: We can add 2 more scenarios for when the score does not satisfy the condition, like 50 (the exact limit, which is technically invalid) and a clearly below value, like 30. So this gives us a total of 4 scenarios for the score condition.

Applying the FPC for Total Calculation:

With 4 possible scenarios for each condition (including valid and invalid), and considering that we need to test all possible combinations of true/false for both conditions:

• If we apply the FPC such that each age scenario (4 possible) can be combined with each score scenario (4 possible), we would have a total of 4×4=16 possible combinations.

Age Scenarios:

1. Valid age within range (e.g., 20 years)

2. Age at the lower limit (e.g., 18 years)

3. Age at the upper limit (e.g., 65 years)

4. Age below the lower limit (e.g., 17 years - invalid)

5. Age above the upper limit (e.g., 66 years - invalid)

Score Scenarios:

5. Score above the limit (e.g., 55 points)

6. Score at the limit (e.g., 51 points - considering > 50 as valid)

7. Score at the exact limit (e.g., 50 points - invalid)

8. Score below the limit (e.g., 49 points - invalid)

Combining Scenarios:

Now, we'll combine the age and score conditions to form complete scenarios. Given that we have 5 age scenarios and 4 score scenarios, theoretically we could consider 20 combinations (5 x 4), but to reach 16 scenarios, we'll focus on the most representative and relevant combinations:

1. Age 20, score 55 (valid)

2. Age 20, score 51 (valid)

3. Age 20, score 50 (invalid)

4. Age 20, score 49 (invalid)

5. Age 18, score 55 (valid)

6. Age 18, score 51 (valid)

7. Age 18, score 50 (invalid)

8. Age 18, score 49 (invalid)

9. Age 65, score 55 (valid)

10. Age 65, score 51 (valid)

11. Age 65, score 50 (invalid)

12. Age 65, score 49 (invalid)

13. Age 17, score 55 (invalid - fails age)

14. Age 66, score 55 (invalid - fails age)

15. Age 17, score 49 (invalid - fails age and score)

16. Age 66, score 49 (invalid - fails age and score)

Including invalid scenarios, we have a total of 16 possible combinations when applying the FPC. However, it's essential to filter these combinations to ensure the test scenarios are logically consistent and relevant to the testing objectives, considering the nature of the conditions and the business rules of the system.

Notes:

Valid Scenarios: The scenarios considered valid are those where the age is within the specified range and the score is above 50.

Invalid Scenarios: Include situations where the age is outside the allowed range and/or the score does not meet the minimum criterion. Even if one of these aspects is valid, failing to meet any one of the conditions results in an invalid scenario due to the use of the && operator.

It's always good to emphasize that with a large number of scenarios, especially in cases more complex than this example, as we know complexity depends on the business context, it's important to consider the applicability and relevance of each scenario to ensure effective allocation of testing efforts.

Does the FPC resemble Code Coverage?

They are two approaches with distinct objectives and methodologies in software engineering and software testing. Understanding these differences can help us refine our testing strategy and ensure more robust software quality.

Code Coverage

• Focus on Execution: Code coverage focuses on quantifying the percentage of source code executed during testing. It measures, for instance, whether a line of code was executed, if all branches of a conditional statement were tested, or if all loops were traversed in their various capacities.

• Automation: Code coverage tools are generally automated and integrated into the development and testing process, providing real-time reports on which parts of the code have been effectively tested.

• Limitations: While code coverage is essential to understand the extent of testing, it does not, by itself, guarantee the quality or completeness of the tests. A high code coverage percentage does not necessarily imply comprehensive testing of business logics or adequate handling of edge cases.

Fundamental Principle of Counting (FPC)

• Focus on Test Logic: The FPC and its special cases, like permutations and combinations, offer a mathematical approach to structuring and planning test cases. They help us understand all the possible combinations of inputs and scenarios a software might encounter, ensuring we consider a full range of situations.

• Deep Analysis: Applying the FPC requires careful and detailed analysis of functionalities and possible interactions within the software, leading to a deeper understanding of requirements and expected behaviors.

• Complementarity: Unlike code coverage, which is an automated metric, the use of the FPC is an analytical activity that can reveal uncovered test scenarios, even in areas of the code that have been "covered" according to coverage tools.

Advantages of FPC for Programmers and QAs

• Logical Test Coverage: By using the FPC to plan tests, we ensure scenarios consider all variables and their interactions, going beyond the mere execution of code lines.

• Identification of Critical Test Cases: Another helpful point we gain is the ability to identify essential test cases that may not be immediately obvious, including edge cases and unique combinations of conditions.

• Complementing Code Coverage: By combining FPC-based analysis with the code coverage metric, we can achieve a more holistic and comprehensive testing approach, ensuring both code execution and business logic are being adequately tested.

What I Concluded in These Last 8 Months of Study?

In the last 8 months of study focused on the Fundamental Principle of Counting (FPC) and its special cases within software testing, I concluded that these mathematical concepts provide a systematic foundation for approaching the design and implementation of software tests. They offer a structured way to think about the different test scenarios, both valid and invalid, ensuring more comprehensive and effective test coverage. Here are some key points I highlight from this learning journey:

Structuring and Organizing Tests

The FPC and its extensions, such as permutations, combinations, and arrangements, help organize the testing process, allowing a clear view of all the possibilities that need to be considered. This leads to better organization of test cases and a more methodical approach to ensuring all relevant combinations of inputs are tested.

Identification of Test Cases

Applying these concepts helps us systematically identify important test cases, including valid scenarios to verify expected functionality and invalid scenarios to test the software's robustness against unexpected or incorrect inputs. This is crucial for developing software that is not only functional but also resilient and secure.

Improvement in Test Coverage

By ensuring we consider all possible combinations of inputs and actions, we can significantly improve test coverage. This reduces the likelihood of undetected bugs, increasing software reliability.

Efficiency in Test Planning

Although the FPC may indicate a large number of potential scenarios, it also allows us to prioritize which ones are most critical or likely, helping to allocate testing resources more efficiently and focus on the most important aspects of the system.

Deep Understanding of the System

The need to consider all possible combinations and sequences of actions to apply these mathematical concepts leads to a deeper understanding of the logic and internal workings of the system, which is invaluable for any software engineer or QA professional.

I can add that combinatorial analysis teaches programmers to think in a structured and systematic way about problems, an essential skill in software engineering, where solving complex problems is a routine part of the job. This analytical approach helps demystify complex problems by breaking them down into manageable parts.

If I were starting my studies again, I would follow Plato's advice and start with mathematics. - — Galileo Galilei

What I don't expect...

These observations are my own, you can disagree and I respect that. Obviously I don't expect everyone to agree and find it a valid approach for their daily lives. I'm trying to propose something that is as accurate as possible to understand the possible test scenarios within a system's behavior, which needs to be put to the test. Perhaps this is not the most intuitive way to think about test scenarios, but the mathematical approach brings more precision to the number of test cases and makes us think systematically and logically about how we should understand the behavior of the functionality we need to test. In large enterprise applications it's easy to miss this, even when we have code coverage tools to provide us with coverage feedback.

Thinking this way is initially strange, but after studying this topic for a while I realized that we can use many mathematical concepts to our advantage! Thinking about test scenarios can be difficult, especially when we are novices or are not aware of all the rules present in a behavior that a functionality covers, covering all scenarios is also exhausting in software, which is why I have been looking for ways to improve and speed up the way that we identify scenarios, test cases both invalid and valid.

For me, math, computer science, and the arts are insanely related. They are all creative expressions.

— Sebastian Thrun

1. To calculate all the different ways to choose and hang 3 out of the 5 keys on the keyring, where order matters, we use the formula for simple arrangements, which is given by:

\(A(n, k) = n! / (n - k)! ​​\)

Where:

• A(n,k) is the number of possible arrangements,

• n is the total number of items to choose from (in this case, 5 keys),

• k is the number of items to be chosen and arranged (in this case, 3 spaces on the keyring),

• n! denotes the factorial of n,

• (n−k)! is the factorial of the difference between n and k.

The final calculation gives 60 different ways to arrange 3 out of the 5 keys on the keyring.

2. To organize the 5 favorite books on a shelf where order matters, we use the concept of simple permutations. The formula for calculating simple permutations is the factorial of the number of items, represented by n!, where n is the total number of items to organize.

\(P(n) = n! \)

In the case of the 5 books:

\(P(5)=5!=5×4×3×2×1=120 \)

Therefore, there are 120 different ways to arrange your 5 favorite books on the shelf, considering that swapping the position of any two books results in a new arrangement. ​​

3. To calculate the number of possible groupings when choosing 3 flowers from a set of 5 different flowers, I used the combinatorial analysis combination formula. The formula for combinations is given by:

\(C(5, 3) = \frac{5!}{3!(5-3)!}\)

Where:

• C(n,k) is the number of possible combinations (groupings),

• n is the total number of items to choose from (in this case, 5 flowers),

• k is the number of items to be chosen (in this case, 3 flowers),

• n! denotes the factorial of n, which is the product of all positive integers up to n,

• k! is the factorial of k,

• (n−k)! is the factorial of the difference between n and k.