visualAssert/Verify (image@conf), visualSearch (image@conf, number of matches)

The visual UI testing commands allows you to write automated visual tests with Kantu - this makes Kantu the first and only Chrome and Firefox extension (and Selenium IDE) that has "πŸ‘πŸ‘ eyes". A huge benefit of doing visual testsβ€Šis that you are not just checking one element or two elementsβ€Šat a time, you’re checking a whole section or page in one visual assertion.

The visual test commands will take a screenshot of the page, and search it against the provided image. If the comparison says that the new image is found on the screenshot, then visionFind succeeds, otherwise it fails. You can supply a confidence value behind the @ symbol. Its value ranges from 1 (exact match) to 0.1 (almost everything matches). The visual test commands wait !timeout_wait seconds for the image to appear. If the image is not found, an error is triggered (you can overwrite this with !errorignore).

The difference between visualAssert, visualVerify and visualSearch is what happens if no image if found:

  • πŸ‘ visualAssert - Macro stops with error if no matching image if found.
  • πŸ‘ visualVerify - Macro logs warning if no image is found, but macro execution continues.
  • πŸ‘ visualSearch - The number of matches is filled into a variable, just like the sourceSearch command does. If no image if found, the number of matches is 0.

Tip: You can use the resize command to resize the browser window to emulate various screen resolutions.

Visual Testing and Monitoring

The image search runs 100% local in your web browser. It uses C++ powered computer-vision image processing, running directly in your Chrome and Firefox as WebAssembly code. The input images for the visual tests stored in the "πŸ‘ Visual" tab:

Visual testing and visual assert with the new visionFind command

When you select a visual test command, the normal Selenium IDE "Select" and "Find" buttons turn into "πŸ‘ Select" and "πŸ‘ Find". "πŸ‘ Select" allows you to take new screenshots for use with visionFind, and "πŸ‘ Find" starts an image search. The found areas are marked along with their confidence value. The best match is marked with a pink frame.

At the top of the "πŸ‘ Visual" tab you find buttons to take screenshots, and export and import all images. And next to every image is the option to rename it, to add its name to the visual command in the table above (+ symbol), to view it in original size (πŸ‘ symbol) and to delete it.

visionLimitSearchArea (viewport/full/element:locator)

The visionLimitSearchArea allows you to change the search area from viewport (the part of the website that is currently visible in the browser) to full page. And with element:locator you can restrict the image search/comparison to a specific DOM element. This is especially useful for testing canvas elements like Google Maps, Youtube videos or e. g. seat reservation applets.

The screenshot that is used as input for the image search commands is available as "_lastscreenshot" in the screenshot tab. This can be useful for debugging, so you can see on what image the image search was done.

Click on Elements with #elementFromPoint(${!imageX}, ${!imageY})

The coordinates of the center of the best found match are available in the internal variables !imageX and !imageY. Together with the elementFromPoint (x,y) function of the web browser, you can find out which element (locator) is at the image, and then use this locator as input for all commands that take a locator as input. For the most common use of elementFromPoint with the !imageX/Y we introduced a shortcut notation of just using #efp, which of course e-f-p stands for e_lement_f_rom_p_oint. This short video shows how to use #efp to click on an instagram like button:

For UI testing and automating canvas elements we recommend to use ClickAt with elementFromPoint (instead of Click), since ClickAt can click on specific coordinates inside a canvas element. By contract, Click #efp always clicks in the middle of the DOM element.

Command Target Comment
visualVerify (image name here) This sets !imageX and !imageY if image is found
ClickAt #elementFromPoint (${!imageX}, ${!imageY}) comment: same as using #efp, see below:
ClickAt #efp shorthand notation
Type #efp elementFromPoint can be used with any command that takes a locator as input
ClickAt #elementFromPoint (${anyOtherVariable}, ${var2}) Here we need to full command as we use different variables as input.

Note that the elementFromPoint function of the browser is not always reliable, on complex websites with overlapping elements it might return an element that can not receive a click. If you find such cases, report them to us in the forum, there are probably workarounds available.

Visual UI Test Example

Quite literally, seeing is believing. The best way to understand visionFind is to run the included DemoComputerVision macro. It is installed with every fresh Kantu install and shows a visual test of the best online OCR website.

Command Target Pattern/Text
open https://a9t9.com/
visualSearch buttonImage matches

Works in

Kantu for Chrome Selenium IDE, Firefox IDE Classic

Related Demo Macros

DemoCanvas, DemoComputerVision, DemoVisualUITest

See also

storeImage, storeText, Web Automation Extension User Manual, Selenium IDE commands, Classic Firefox Selenium IDE.

Anything wrong or missing on this page? Suggestions?

...then please contact us.

Kantu Selenium IDE for Chrome and Firefox - Web Test Automation
Subscribe to the a9t9 automation software newsletter . We'll send you updates on new releases that we're working on.