## Github上热门图深度学习（GraphDL）源码与工业级框架

3 月 15 日 专知

【导读】图深度学习是当前深度学习领域最热门的方向之一，图神经网络（GNN）不仅在理论上有所创新，在工业界中也真实的应用。本文介绍Github上热门的图神经网络源码及框架，方便研究人员和工程师上手图深度学习。

• 虽然利用稀疏矩阵可以一定程度上缓解上述问题，但依然不能处理大规模的数据。另外，由于多层网络结构的复杂，一般在实现时要同时实现稀疏版和非稀疏版的组件。

• 对图结构数据的预处理比较麻烦。例如在处理异构网络时，有时需要对每种类型的节点进行独立地编号、为每种关系独立建立子图等，才能将图数据转换为深度学习模型可用的数值化数据，并且任何一个细节可能都会影响算法的效率（如邻节点列表的数据结构使用list和set会导致不同的采样效率和查询效率）。

• 需要一些基于图的额外操作，例如Random Walk、有类型约束的Random Walk（Meta-path）等，由于图结构的复杂性，这些操作在单机上的实现都比较费力，更不用说在大规模分布式上。

DeepWalk / LINE

DeepWalk: https://github.com/phanein/deepwalk

LINE: https://github.com/tangjianpku/LINE

TensorFlow: https://github.com/tkipf/gcn

PyTorch: https://github.com/tkipf/pygcn

GCN论文作者提供的源码，该源码提供了大量关于稀疏矩阵的代码。例如如何构建稀疏的变换矩阵（这部分代码被其他许多项目复用）、如何将稀疏CSR矩阵变换为TensorFlow/PyTorch的稀疏Tensor，以及如何构建兼容稀疏和非稀疏的全连接层等，几乎是图神经网络必读的源码之一了。

https://github.com/matenure/FastGCN

FastGCN作者提供的源码，基于采样的方式构建mini-match来训练GCN，解决了GCN不能处理大规模数据的问题。

https://github.com/PetarV-/GAT

Mini-batch版图注意力网络DeepInf

https://github.com/xptree/DeepInf

DeepInf论文其实是GAT的一个应用，但其基于Random Walk采样子图构建mini-batch的方法解决了GAT在大规模网络上应用的问题。

DeepMind开源的图神经网络框架Graph Nets

https://github.com/deepmind/graph_nets

https://github.com/alibaba/euler

Euler是阿里巴巴开源的大规模分布式的图学习框架，配合TensorFlow或者阿里开源的XDL等深度学习工具，它支持用户在数十亿点数百亿边的复杂异构图上进行模型训练。

Modern data analysis pipelines are becoming increasingly complex due to the presence of multi-view information sources. While graphs are effective in modeling complex relationships, in many scenarios a single graph is rarely sufficient to succinctly represent all interactions, and hence multi-layered graphs have become popular. Though this leads to richer representations, extending solutions from the single-graph case is not straightforward. Consequently, there is a strong need for novel solutions to solve classical problems, such as node classification, in the multi-layered case. In this paper, we consider the problem of semi-supervised learning with multi-layered graphs. Though deep network embeddings, e.g. DeepWalk, are widely adopted for community discovery, we argue that feature learning with random node attributes, using graph neural networks, can be more effective. To this end, we propose to use attention models for effective feature learning, and develop two novel architectures, GrAMME-SG and GrAMME-Fusion, that exploit the inter-layer dependencies for building multi-layered graph embeddings. Using empirical studies on several benchmark datasets, we evaluate the proposed approaches and demonstrate significant performance improvements in comparison to state-of-the-art network embedding strategies. The results also show that using simple random features is an effective choice, even in cases where explicit node attributes are not available.

Neural networks achieve the state-of-the-art in image classification tasks. However, they can encode spurious variations or biases that may be present in the training data. For example, training an age predictor on a dataset that is not balanced for gender can lead to gender biased predicitons (e.g. wrongly predicting that males are older if only elderly males are in the training set). We present two distinct contributions: 1) An algorithm that can remove multiple sources of variation from the feature representation of a network. We demonstrate that this algorithm can be used to remove biases from the feature representation, and thereby improve classification accuracies, when training networks on extremely biased datasets. 2) An ancestral origin database of 14,000 images of individuals from East Asia, the Indian subcontinent, sub-Saharan Africa, and Western Europe. We demonstrate on this dataset, for a number of facial attribute classification tasks, that we are able to remove racial biases from the network feature representation.

Random walks over directed graphs are used to model activities in many domains, such as social networks, influence propagation, and Bayesian graphical models. They are often used to compute the importance or centrality of individual nodes according to a variety of different criteria. Here we show how the pseudoinverse of the "random walk" Laplacian can be used to quickly compute measures such as the average number of visits to a given node and various centrality and betweenness measures for individual nodes, both for the network in general and in the case a subset of nodes is to be avoided. We show that with a single matrix inversion it is possible to rapidly compute many such quantities.

Network embedding aims to learn low-dimensional representations of nodes in a network, while the network structure and inherent properties are preserved. It has attracted tremendous attention recently due to significant progress in downstream network learning tasks, such as node classification, link prediction, and visualization. However, most existing network embedding methods suffer from the expensive computations due to the large volume of networks. In this paper, we propose a $10\times \sim 100\times$ faster network embedding method, called Progle, by elegantly utilizing the sparsity property of online networks and spectral analysis. In Progle, we first construct a \textit{sparse} proximity matrix and train the network embedding efficiently via sparse matrix decomposition. Then we introduce a network propagation pattern via spectral analysis to incorporate local and global structure information into the embedding. Besides, this model can be generalized to integrate network information into other insufficiently trained embeddings at speed. Benefiting from sparse spectral network embedding, our experiment on four different datasets shows that Progle outperforms or is comparable to state-of-the-art unsupervised comparison approaches---DeepWalk, LINE, node2vec, GraRep, and HOPE, regarding accuracy, while is $10\times$ faster than the fastest word2vec-based method. Finally, we validate the scalability of Progle both in real large-scale networks and multiple scales of synthetic networks.

Looking from a global perspective, the landscape of online social networks is highly fragmented. A large number of online social networks have appeared, which can provide users with various types of services. Generally, the information available in these online social networks is of diverse categories, which can be represented as heterogeneous social networks (HSN) formally. Meanwhile, in such an age of online social media, users usually participate in multiple online social networks simultaneously to enjoy more social networks services, who can act as bridges connecting different networks together. So multiple HSNs not only represent information in single network, but also fuse information from multiple networks. Formally, the online social networks sharing common users are named as the aligned social networks, and these shared users who act like anchors aligning the networks are called the anchor users. The heterogeneous information generated by users' social activities in the multiple aligned social networks provides social network practitioners and researchers with the opportunities to study individual user's social behaviors across multiple social platforms simultaneously. This paper presents a comprehensive survey about the latest research works on multiple aligned HSNs studies based on the broad learning setting, which covers 5 major research tasks, i.e., network alignment, link prediction, community detection, information diffusion and network embedding respectively.

Multi-view networks are ubiquitous in real-world applications. In order to extract knowledge or business value, it is of interest to transform such networks into representations that are easily machine-actionable. Meanwhile, network embedding has emerged as an effective approach to generate distributed network representations. Therefore, we are motivated to study the problem of multi-view network embedding, with a focus on the characteristics that are specific and important in embedding this type of networks. In our practice of embedding real-world multi-view networks, we identify two such characteristics, which we refer to as preservation and collaboration. We then explore the feasibility of achieving better embedding quality by simultaneously modeling preservation and collaboration, and propose the mvn2vec algorithms. With experiments on a series of synthetic datasets, an internal Snapchat dataset, and two public datasets, we further confirm the presence and importance of preservation and collaboration. These experiments also demonstrate that better embedding can be obtained by simultaneously modeling the two characteristics, while not over-complicating the model or requiring additional supervision.

Top