The visual patent retrieval methods are based on algorithms for analyzing the often binary (black and white) or grayscale images available in patents. In these APs, preliminary work of the TIB based on Deep Learning is adapted to the special properties of patent images: The representation and quality of the images as well as their schematic and sketchy character require specific solution approaches.

First, algorithms for recognizing the type of illustrations in patents are necessary (typical ones are e.g. bar charts, data flow diagrams, flow charts, technical drawings and graphical representations of functions (x-/y-axis). Second, recognition of text and numbers in figures (OCR), recognition of the orientation of a figure (portrait, landscape format of the figure), and recognition of arrows and lines that establish relationships between graphical information and text are required. In addition, image-descriptive text passages in the document are identified and their relation to the image is qualified.

The approaches are iteratively adapted according to the requirements of the patent experts to ensure actual usability by the target group.

Since the project is not technology-driven but user-oriented, it is essential to know the working environment and information behavior of future users. In order to validate the insights gained from the literature in this regard in practice and to capture them in their complexity, the various needs are surveyed through observations and interviews. Within the scope of the project, IWIST analyzes the domain-specific (subject) information behavior and designs a generalization concept. Central in this context is the link between text and image when searching for patents/disclosures or when examining a corresponding result set. The analysis results of the expert behavior form the basis for the identification of the functional scope and the presentation form during prototype development. They also serve to be able to fall back on authentic material during the evaluation.

IWIST creates a requirements catalog for the design of the functionality and the user interface, which is created in close relation to the elicited information needs. The development of the prototype for the GUI is carried out iteratively according to the user-centered design approach and holistically from the user's perspective. Developers and later users work together in early phases of specification. A first version is already developed in the first year. Based on experience and empirical evaluations as well as progress in the development of suitable search methods, a revised and optimized version is developed in the second year of the project. In each case, the requirements are presented and visualized in a low-fidelity prototype. 

The prototypical system must enable a patent expert to use the tools without detailed knowledge of the algorithms behind them and to use them for his questions. The user-centric approach chosen in the project ensures that the system is easy and intuitive to use.  The approach optimally adapts the resulting prototype to the needs of the target group. Above all, it must provide a suitable division of tasks between the system and the expert (or human and machine) so that the user can effectively and efficiently achieve the desired result during the interaction. Different visualization and interaction approaches are tested to work out the optimal mix for the experts' workflow.

The basis for the graphical user interface (GUI) are the requirements identified during the information behavior analysis, which result in prototypical tasks and questions. These are implemented in the GUI through appropriate functionalities that underlie the image analysis algorithms. The strategies and solution paths are developed en détail with the users; they are based on the specifications provided by the data (patent images and texts, their semantic annotation schemes, similarities, etc.). For the realization of the functions, the most efficient design possible is chosen (object-function schema), which is used throughout on the GUI. This is based on usability and user experience guidelines.

For the development of the prototypical tool and the formative evaluation of the prototype, a sufficiently large set of prototypical questions and their corresponding answer sets are required. The ongoing, holistic evaluation of the system from the user's perspective, related to the patent experts' work tasks, requires similar empirical and partly qualitative methods as the information needs analysis. Thus, evaluation is formative, accompanying the individual development steps to realize meaningful functionality, but also summative, to assess the overall performance of the prototype at the end. Interim evaluations of two iterations of the prototype are planned, as well as an overall evaluation at the end of the project.

The summative evaluation examines the prototype in its entirety towards the end of the project. The system and the methods developed (such as the cross-domain and cross-industry analyses) are to be evaluated holistically and from the perspective of the users. For this purpose, users from the target group who have not been involved so far will also work with the system, be interviewed and trained. Here, too, special emphasis will be placed on task-specific access.

To integrate the visual search, we first examine how the extracted visual features and the references between images and text can be represented and used for an effective semantic search. For this purpose, the results obtained in preprocessing serve as a basis, because the data acquired here form the information base for the visual semantic search. Based on this information, it is investigated whether the extracted mid- & high-level features can be adequately represented as simple features of the indexed patent documents, or whether a representation in a graph-based data model is more appropriate.

For a keyword-based search, it is investigated how the keywords can be used - possibly using the lexicons developed for the different domains - for a combined textual and visual search. In addition to keyword-based and Boolean search and refinement of search results, similarity-based search is another important part of this work package. It is investigated how to define a multimodal similarity that uses both textual and visual features. These aspects form the basis for performing cross-linguistic and multimodel analyses to test cross-domain and cross-industry exploitations, respectively.

The aim of the "System Integration" work package is to design the technical components developed in the context of this project and their interactions from a software engineering perspective in such a way that the functional and non-functional requirements are supported as well as possible.

Starting from the pre-processed database, it is investigated how the patent documents can be read into the search system as efficiently as possible, pre-processed and adequately indexed. The combination of text and image data, the different preprocessing of these data as well as the subsequent indexing of the extracted features for the common semantic search require an orchestration in order to be able to track and monitor the progress and status of the individual work steps.

In addition to orchestration of data processing, the expected large data volumes necessitate scalability of processing and storage. Here we will investigate how to parallelize the processing of the text and image data using computer clusters, and what persistence will best support the subsequent search. One focus here will be the indexing of the extracted visual features, as this is essential for efficient search.

Another area of work for system integration is the interface for the UI as well as the implementation of the common semantic search. In developing this common search interface, attention will be paid to important design principles from software architecture such as loose coupling and data encapsulation, but also runtime efficiency.

The aim of this work package is to develop and evaluate an exploitation strategy that includes the communication of the project results, the consulting of potential users and the technology transfer of the developed tool as well as its use. In the transfer offices of the participating institutions, competences in the field of patent research with a focus on patent retrieval tools are to be built up, so that in a further step potential users and user groups for the developed tool can be identified and advised. Measures from the areas of start-up promotion, technology marketing and networking will be used. These include, for example, individual start-up consultations, webinars, training courses and workshops. In addition, the organizational conditions will be created so that the patent retrieval tool can be made available to Leibniz institutions and other interested parties after the project has ended. This package of measures is intended to make knowledge and innovative technology quickly and permanently available for research and society.