It's been some time since I felt a gap in integration testing applications like feature tests in Laravel. At first look, we think running a few isolated tastes without any effects of other actions would be reliable. Let's talk about the pros and cons. I'll bring a way to a real enterprise application that would be enhanced against incoming problems like a chess super grandmaster playing blindfold chess.

Imagine you have a complicated enterprise application with hundreds of feature tests. You should provide a bunch of data with the factory to ensure the tests would be passed. However in reality this might not happen. I mean either the user's behavior or consequently users' data generated by themselves aren't like test processes completely. Therefore users might encounter a few errors that those cases weren't thought of in advance.

Additionally, you can't consider each of the scenarios in detail. You know an enterprise application would have dozens of conditions in every action.Whether we can make a specific test for every one of them?

on the other hand, since some applications don't have feature tests from the beginning, Writing tests for current code is like reading the whole source code to ensure considering everything.

1- Isolated tests are useful for running individually, not continuously

Isolated tests would be a flaw in the case of complex applications. Because end users will send requests to a bunch of endpoints to make their data, they send a request to that endpoint to see their result.

In other words, in production, other's actions would impact your method however you declare a way to pass your test in the feature test Laravel by mocking and factory data. It might lead to false confidence in the code.

In this way, You want to make sure that the code works well without any effect of other methods while real application would impact other actions.

2- Unable to consider all aspects and scenarios

Considering all aspects and scenarios tests in advance or in coding time would be tough. The code might be updated every day then writing tests might be for each change might impossible. Meanwhile, the developer who adds a new change is the best person to refactor or implement the tests.

So generally, other developers don't want to implement tests for other developers' code. Furthermore, how often should we mock other services such as sending emails, notifications, and SMS or third-party endpoints? These days our dependencies have been enhancing these types of services. Therefore mocking them can be deceptive sometimes.

Sicilian Test Orchestra would be more realistic tests

We shouldn't expect feature tests to test all aspects completely although feature tests are really useful However I want to introduce a new way to fill this gap I mentioned.

I've been working on a new package that provides a testing process like the end users' behavior called Sicilian Test OrchestraGithub. It sends a bunch of different requests to every one of the methods continuously without any database truncating then shows in detail and saves all reports in the database.

There's no need to code for every method. You just need to declare routes with its requests and we run randomized to detect what the application reacts. For example :

$ composer require mehdi-fathi/sicilian-test-orchestra

'providers' => [
  /*
   * Package Service Providers...
   */
   SicilianTestOrchestra\SicilianTestOrchestraServiceProvider::class
]

namespace Tests\Feature;

// use Illuminate\Foundation\Testing\RefreshDatabase;
use SicilianTestOrchestra\RequestStrategyTestable;
use Tests\TestCase;

class ExampleTest extends TestCase
{

    use RequestStrategyTestable;

    protected array $testOrchestraRequests = [
      'user_login' => ['auth'],  //auth,quest
      'shuffle_next' => true,
      'next' => [
          [
              'route' => 'save',
              'method' => 'post',
              'data' => [
                  'body' => ['string', 'min:1', 'max:4'],
              ],
              'call' => 1,
              'call_shuffle' => 5,
          ],
          [
              'route' => 'update',
              'method' => 'put',
              'data' => [
                  'body' => ['string', 'min:1', 'max:4'],
              ],
              'param' => [
                  'id' => ['numeric', 'min:1', 'max:1']
              ],
              'call' => 1,
              'call_shuffle' => 2,
          ],
          [
              'route' => 'list',
              'method' => 'get',
              'data' => [],
              'call' => 1,
              'call_shuffle' => 2,
          ],
          [
              'route' => 'show',
              'method' => 'get',
              'data' => [
                  'id' => ['numeric', 'min:1', 'max:1']
              ],
              'call' => 1,
              'call_shuffle' => 2,
          ],
          [
              'route' => 'destroy',
              'method' => 'get',
              'data' => [
                  'id' => ['numeric', 'min:1', 'max:1']
              ],
              'call' => 1,
              'call_shuffle' => 2,
          ],
          [
              'route' => 'show',
              'method' => 'get',
              'data' => [
                  'id' => ['numeric', 'min:1', 'max:1']
              ],
              'call' => 1,
              'call_shuffle' => 2,

          ],
      ]
];

}

Run this command to create migration for the report_tests table:

php artisan vendor:publish --tag=migrations

You declare treat of your methods as above so run your tests :

php artisan test

In the data key, we've taken advantage of Faker lib to generate data. call_shuffle is a key to number call randomly without any order.

You'll see all the results of tests in detail. every time you run tests you can see some new response. In addition, We keep those on a table to check what happened in tests. It should be better to run many times in a task schedule.

About Sicilian Test Orchestra in detail

Since this package is new, I want to foster that gradually. As I mentioned, this package would be used to test Laravel Applications, potentially. This project is released recently in the 1.0 version. What would encourage me to develop is to read your ideas and viewpoints. So Not only I would like to know your thoughts about that gap in the feature test But also your view about the Sicilian Test Orchestra would be helpful. This package has a long way to go. I would like you to comment freely or make an issue in my GitHub.

For more details check out GitHub repository