Functional tests
are generally used as a synonym for integration tests
. You may be confusing the unit tests
with functional tests
if I understood your question correctly.
In unit tests, aim to test the function itself in isolation, by mocking/stubbing other components.
In integration tests, test different systems working together, to see if they work together correctly to produce the correct output.
For your example send_invite
, in unit tests, you can mock the email and other framework components, and just test if the send invite function correctly calls the email component with correct arguments. For the integration tests, you can set up a testing server and see if the function correctly sends emails and does everything it supposed to do, including the other components.
Unit tests are the smallest testing block, followed by the integration tests.
EDIT:
Let's leave the words aside and approach the problem as why we need tests. We need tests to ensure our code is working correctly.
In the send_email
example, first we need to test the function in isolation to see it does what it supposed to do. We do this by isolating the function from other components by mocking/stubbing to test the function itself. Assuming we have a different component that is responsible for sending emails, and our function is calling this component to send emails, if our email component is broken, we won't want this to affect the test of send_email
, it's not directly related to this function. We just want to make sure the code in the send_email
function is working correctly. That is generally called unit testing
.
After we are sure the function is working correctly in isolation, we need to test if the invitation system is working correctly, with all of it's components. This time we do it by testing the function without mocks/stubs. This is generally called integration tests
.
In some places, there are functional tests with side effects
and functional tests without side effects
, which means calling the function with predefined arguments in tests without mocking/stubbing to see it produces the correct output, and calling the function with predefined arguments in tests with mocking/stubbing other components, respectively. This can translates into unit tests
and integration tests
.
The language can be confusing, and different cultures/communities sometimes call these tests with different words. What needs to be done in testing is, starting testing with the smallest meaningful unit possible, and work from there to whole system.(This can be done in reverse order too, but it's out of the scope of this answer)
I'm going to go ahead and register an answer for what I've come up with so far.
My hypothesis is that for a function with deep coupling and state, the reality is that it's simply going to take a lot of lines to control for its outside context.
Here's what my test case looks like roughly, relying on the standard Mock library:
- I use the standard ORM to set up the sequence of events
- I create my own start
datetime
and subvert the auto_now_add
times to fit a fixed timeline of my design. I thought that the ORM didn't allow this, but it works fine.
- I make sure the function-under-test uses
from datetime import datetime
so that I can patch datetime.now()
in just that function (if I mock the entire datetime
class, the ORM pitches a fit).
- I create my own replacement for
object.key_method()
, with simple but well defined functionality that depends on the arguments. I want it to depend on the arguments, because otherwise I might not know if the logic of the function-under-test is working. In my case, it simply returns the number of seconds between startTime
and endTime
. I patch it in by wrapping it in a lambda and patching directly on to object.key_method()
using the new_callable
kwarg of patch
.
- Finally, I run a series of asserts on various calls of
summary
with different arguments to check equality with expected hand-calculated results accounting for the given behavior of the mock key_method
Needless to say, this is significantly longer and more complicated than the function itself. It depends on the DB, and doesn't really feel like a unit test. But it is also fairly decoupled from the internals of the function--just its signature and dependencies. So I think it might actually be a unit test, still.
In my app, the function is quite pivotal, and subject to refactoring to optimize its performance. So I think the trouble is worth it, complexity notwithstanding. But I'm still open to better ideas on how to approach this. All part of my long journey toward a more test-driven style of development...
Best Answer
At the moment I like this statement : "It’s not important what you call it, but what it does" made by Gojko Adzic in this article.
You really need to specify with the people talking about the tests what you intend to test.
There are a lot of people having different views, depending on what their role is.
For testers a general accepted test methodology in the Netherlands is TMap. TMap makes the following distinction.
They have more specific kind of tests that can be performed within the above mentioned tests. Look at this word doc for an overview.
Wikipedia also has a nice overview.
The book the pragmatic programmer says:
Looking at these different sources and putting in some of my own experiences and opinions I would start by making distinctions by three categories
what is the goal of the test
My list above is just a start and a suggestion but I really do think: "It’s not important what you call it, but what it does"
Hope this helps.
26-10-2016 Edit: Just recently a very nice introduction was placed on YouTube Unit tests vs. Integration tests - MPJ's Musings - FunFunFunction #55