Overview - agile testing
When developers finish a first draft of a new feature or bug fix, they move the corresponding JIRA issue into the "Needs 1st Level Review" column of the Kanban board and un-assign it. Someone else on the team then assigns the issue to themself and moves the issue to "Reviewing" column. Reviewing involves two types of reviewing: code review (discussed elsewhere) and functional testing. Functional testing involves installing the developers branch-specific installer for the issue and testing its function. If any bugs or errors related to the changes are noted, the tester returns the issue to the To-Do column and re-assigns it to the original developer. Once the issue passes both the code review and functionality review, we move the issue the next lane and un-assign it.
Later, when changes have been merged into the master branch (the main line of development), we double-check that the issue can build under Bitbucket pipelines. If it passes this hurdle, we next move the issue to the Needs Testing lane of the kanban board. We "un-assign" the issue, and it stays in that lane until someone has time to perform functional review with the master branch installer. It's important to perform additional testing at this stage to ensure that the developer's changes interact well with other changes in the code base. At this point, the master branch installer is a pre-release version of IGB.
Overview - release testing
Before making an official release of IGB, we test release candidate installers on the three major platforms - Mac, Windows, and Linux. This is important because operating systems differ in how they handle security, and this changes with new versions. For example, an IGB installer that worked great on older versions of the Apple OS may become in-installable on a newer version. (Here is an example from release 9.0.2.)
For release testing, we may set up some computers side-by-side and go through each smoke testing and systems testing protocol as a gruop, testing each protocol on the three platforms simultaneously. This can save time because it allows comparing behavior across operating systems and uncovers OS-specific problems.
General testing procedures
To check IGB functionality on a system, you should test in two ways:
- Naive system - Test on a system that mimics a completely naive users who has never installed IGB on their computer before. This mimics the situation of new IGB users.
- Current user - Test on a system that is running a functional copy of the current IGB release. This mimics the situation of existing IGB users.
Note that to test IGB and IGB installations, you need to understand how the IGB installation process works on the system you're testing.
IGB installers are using the Install4J tool to install the compiled IGB code together with a "sandboxed" Java virtual machine that is used only to run IGB. When a user installs IGB, a folder gets created in the part of the file system dedicated to individual users' applications. Administrator (root) privileges are not required to install IGB, so this location is nearly always located in the user's personal or home directory. In addition, be aware that the Install4j installer software runs every time IGB launches. When IGB launches, the Install4J runs, too. It sends a request to the IGB server to read a file called "updates.xml." When we release a new version of IGB, we edit this file residing on the server, incrementing the release number as required. Then, when the installer reads this file the next time a user launches IGB, it will invite the user to download and install a new version if available.
Also note that IGB saves information about user preferences to the local system. IGB is using the Java preferences API for this. This means that to mimic a completely naive system, you must clear the user preferences. The easiest way to do this is by launching IGB and running the Reset Preferences to Defaults command, available in the Other Options tab of the Preferences window.
Last but not least, IGB stores IGB-specific data in a folder named .igb ("dot-ig-bee") in the user's home folder, which is different depending on the operating system. To mimic an entirely naive system, you should delete this or move it a new location for safekeeping.
Naive system quick start
To remove all traces of IGB and mimic a naive system:
- If IGB is already installed, launch it and clear the IGB preferences. (See User's Guide for details.)
- Open the installation directory and run the uninstall script.
- Delete the .igb ("dot-IGB") directory in the ~/home (Mac and Linux) directory or its Windows equipment on your computer.
Current user system quick start
To mimic an existing user of IGB:
- If IGB is already installed, check the version. Is it the latest released version?
- Visit the latest Arabidopsis and human genomes.
- Use the checkboxes to open data files and then load them by clicking the Load Data button.
- Change the foreground and background colors to mimic a user customizing how data look.
- Create bookmarks for testing.
Installing the test target
The test target is the version of IGB that you are planning to test. This is either a branch-specific installer (in the case of agile testing) or a release candidate (for release testing.)
Download the test target installer and run it.
Note: Instead of installing it in the default location, install the test target version (branch build or release candidate) on your computer in a directory named for the release. In addition, answer "no" when asked if you want to create a shortcut. Following this convention will reduce the chance for conflicts that may appear to be errors but really are only artifacts of the testing process.
What to do if you find an error
The first thing you should do is check: Is this error also present in earlier versions of IGB? Is this a new bug introduced by a recent change, or is it an older error or flaw that is already known.
- Search JIRA for mention of similar behavior. The bug you have found may be a known issue.
- If you find a JIRA report, read it carefully. Was the bug thought to have been fixed? Consult the testing manager or colleagues to decide what to do next. There are many options. Typically, you will re-open the old bug report and move it to the top of the backlog or to the current sprint, depending on severity.
- If you can't find a prior bug report, this means the error could be newly introduced or could be a pre-existing error that has never been reported. Using the target version, develop step-by-step instructions on how to trigger the observed error. If we do not have step-by-step instructions on how to reproduce the error, there is no way we can fix it. So this aspect is essential.
- Next, use your new, step-by-step instructions to find out: Does the error exist in previous versions of IGB? If yes, then you have found an old bug that may not have been reported. If not, it is new. This will provide valuable information about when and how the error was introduced.
- Finally, create an issue for the error in JIRA. If this is a new issue, title it "Investigate and fix ..." (and describe the error). Include the steps required to reproduce the bug. Move the issue to the top of the backlog or to the current spring. If doing release testing (see above) make a note of the error you found in the share testing document. Include a link to the newly created issue.