Improving Wetware

Because technology is never the issue

Testing emails when your app is not the sender

Posted by Pete McBreen 08 Mar 2022 at 04:24

When testing emails, when your application is the sender of the email, then tools like mailtrap are good for capturing the SMTP emails from Development and Staging environments, allowing automated tests to grab the emails and check things like the password reset flows.

Problems arise however when other systems are sending emails that are relevant to your application. A recent example I ran across was similar to this One Time Passcode workflow documented by Microsoft. The basic problem is that a third party is emailing to one of your test accounts and you need to extract some information from that email in order to complete an action.

As a tester, it is normally feasible to get a few test accounts setup by you email admins, and then your test cases can reuse that limited set of email addresses for the automated tests. A better way though is to use a service like Mailinator which allows easy access to a reasonable number of randomly generated usernames, say lmykuhuzgfbgim@mailinator.com, on one of the Mailinator domains. On the free tier, the emails show up in a public email box at https://www.mailinator.com/v4/public/inboxes.jsp?to=lmykuhuzgfbgim.

The fun starts as soon as you have an account, then you can use an API to get the contents of your private emails to your own mailinator domain. The Message API allows you to fetch the message identifiers, read the email related to that identifier and then delete the message related to that identifier to keep the mailbox clean.

The way this works is by specifying a wildcard catch-all on the email domain, that will catch all emails not addressed to known usernames. Normally an email server would send back a message saying mailbox not known, but the catch-all just grabs all those unknown emails and forms the basics of the mailinator system. You could roll your own, but much simpler to use Mailinator or one of the competing services.

Test code is not Production code

Posted by Pete McBreen 06 Mar 2022 at 23:52

And as such needs to be held to different standards. Copy and Paste of test code is not as problematic as it could be in Production code. What matters is that the tests are cheap to write and modify, so speculatively scattering the test case code into multiple modules is at best a premature optimization, and potentially a waste of time.

When using playwright and pytest, a new test should be written inline in a single test_xxx without writing any new any helper functions, test utilities or extracting selector strings out to constants. Yes, it can use existing fixtures to handle things like Login/logout and data setup, and exiting helper functions to perform necessary actions, but all the new stuff has to be inline in the test and ideally should be less than 30 lines of code.

The resulting commit to the repository should only be for a single file, with the file touched in at most three places

  • The DocString that describes the tests contained in the file
  • Potentially some new import statements
  • The new test case inside the test_xxx method

The rationale for this is to force the person writing the test to focus just on the test and nothing else. Refactoring to extract out selector strings can happen later, as can extracting reusable bits of test code, either as test utilities or possible fixtures. The initial focus should be on a quick and dirty implementation for the test so that it can run against the application that is being tested in the relevant environments.

Once the test is running and being useful testing the application, then is the time to think about the code quality:

  • If there are now multiple tests that have identical data setup and teardown, then extract that code to a fixture
  • If the same selector is now used in multiple test cases, or multiple places in the same test case, then it can make sense to extract it to a constant and collect the selectors for that UI component into a separate place.
  • If the test case had to hit a web service, it may make sense to extract code to a helper function, but only if there are other existing tests that could make use of the extracted function.

Overall the goal is to have new test cases that read from top to bottom without your team having to investigate what is happening in as myriad of new functions and modules that have been added just to support this new test case.