Converting programs to visual representation by intercepting screen draws

Gathering visual context the easy way.

Problem

This was the problem that everyone said was impossible to do, gather the definition of the app screens, not snapshots. We tried various methods, and this was the easiest to implement. Some programming languages have commands that take a set of defined graphic elements and paint them onto the screen. By intercepting those commands, one can capture the definition of the graphic elements, and ultimately convert them into a visual representation of what the user sees in a web browser.

Solution

For this to work, we add a code library into the app to watch for screen draws while the app runs in a simulator. As the developer develops their app and runs it in a simulator, the code library captures what the screen looks like and where the strings and images are displayed. This technique works for all programming environments that include screen draw callbacks.

Impact

It is far easier to capture screen definitions using this technique than to decompile the compiled binary, which is another patent.

Status

US Patent Application Number 61928927
Filed 20-JAN-2015