Converting programs to visual representation by intercepting screen draws
Gathering visual context the easy way.
Gathering the visual representation of an application’s screens is non-trivial. Some programming languages have commands that take a set of defined graphic elements and paint them onto the screen. By intercepting those commands, one can capture the definition of the graphic elements, and relay those definitions to a system that converts them into a visual representation of what the user sees on screen.
This covers a technique where a code library in the app watches for screen draws on apps running in a simulator. As the developer is developing their app and running it in a simulator, the code library captures what the screen looks like and where the strings and images are displayed. This can work for all programming environments that include screen draw callbacks.
It is far easier and accurate to capture screen definitions using this technique than to decompile the compiled binary.
US Patent Application Number 61928927