Magento 2 – Fix Setup Scripts Failing Integration Tests

This works for us but we haven't yet looked into moving them to a separate location to address 6.)

1.) Place your integration tests under dev/tests/integration/testsuite/Vendor
2.) copy dev/tests/integration/phpunit.dist.xml
to
dev/tests/integration/phpunit.xml

and replace

        <directory suffix="Test.php">testsuite</directory>
        <directory suffix="Test.php">../../../update/dev/tests/integration/testsuite</directory>
        <exclude>testsuite/Magento/Test/Integrity</exclude>
        <exclude>testsuite/Magento/MemoryUsageTest.php</exclude>

with

        <directory suffix="Test.php">testsuite/Vendor</directory>

3.) run it ../../../vendor/bin/phpunit or with ../../../vendor/bin/phpunit path/to/tests from the dev/test/integration folder

Please note that the integration tests take longer than 15 seconds, at least on first run as it essentially installs Magento. You can save on subsequent runs if you use

<const name="TESTS_CLEANUP" value="disabled"/>

in your phpunit.xml

You cannot run the core test suite with a customized build like this. But this is not their purpose anyways. The core integration test suite is there to ensure that the core modules work together as intended and to prevent bugs in Magento itself.

If you want to run an integration test for the checkout that is adapted to your changes, you are free to reuse parts of the core tests, but you will have to make adjustments like adding values for custom required attributes in the used fixtures.

But do not try to recreate integration tests just for core functionality. Create tests that ensure, that your code works.

For high level "Magento still works" tests, functional tests are more appropiate than integration tests. That's also why the new Magento Functional Test Framework (MFTF) will give us extensible core tests. This one is meant to be run on custom builds.