CoVA

Context-Aware Visual Attention-based end-to-end pipeline for Webpage Object Detection (CoVA) aims to learn function f to predict labels y = [$y_1, y_2, ..., y_N$] for a webpage containing N elements. The input to CoVA consists of:

1. a screenshot of a webpage,
2. list of bounding boxes [x, y, w, h] of the web elements, and
3. neighborhood information for each element obtained from the DOM tree.

This information is processed in four stages:

1. the graph representation extraction for the webpage,
2. the Representation Network (RN),
3. the Graph Attention Network (GAT), and
4. a fully connected (FC) layer.

The graph representation extraction computes for every web element i its set of K neighboring web elements $N_i$. The RN consists of a Convolutional Neural Net (CNN) and a positional encoder aimed to learn a visual representation $v_i$ for each web element i ∈ {1, ..., N}. The GAT combines the visual representation $v_i$ of the web element i to be classified and those of its neighbors, i.e., $v_k$ ∀k ∈ $N_i$ to compute the contextual representation $c_i$ for web element i. Finally, the visual and contextual representations of the web element are concatenated and passed through the FC layer to obtain the classification output.