Machine Learning on Hyperbolic Space

Deep Neural Networks process input data on the assumption that it belongs to Euclidean space. However, when dealing with semantic hierarchical data such as language, it is known that it is more natural to treat input information as data in a space called hyperbolic space. Therefore, it is expected that machine learning methods for handling data in hyperbolic space can be used to perform recognition with high accuracy. The difficulty is that hyperbolic space is generally a curved space, and it is necessary to translate the processing in hyperbolic space into processing that can be handled by a computer.

Hyperbolic Space [Nickel&Kiela, Neurips 2017]

In [Nickel&Kiela, Neurips 2017], a method called Poincare Embeddings was proposed, which considers a word vector as a point in hyperbolic space and uses the distance in that space for similarity calculation and parameter update. In [Ganea+, Neurips 2018], neural network layers correspondences in hyperbolic space were proposed for linear layers, activation functions, softmax functions, etc., which are the components of a basic neural network. On the other hand, more complex layers are used in convolutional neural networks and transformers, which are utilized in recent Deep Neural Networks. If these layers can also be constructed in hyperbolic space, it will be possible to expand the applications of hyperbolic neural networks.

Uniqueness and achievements of this laboratory

In [Shimizu+, ICLR 2021], we introduced convolutional layers, which are basically used in current neural networks. The usual convolutional layer can be decomposed into two processes: (i) concatenate the features corresponding to adjacent image pixels (ii) apply a linear layer to the concatenated features, and a convolutional layer can be implemented if the (i) features can be concatenated in the hyperbolic space. However, it is not obvious how to concatenate points in hyperbolic space, which is a curved space. Therefore, in this research, we proposed a method of connecting points by the following processes: (i) transferring each point to a Euclidean vector called a tangent space, (ii) concatenating points on the transferred Euclidean space, and (iii) pulling back the connected vector to a point on a hyperbolic space with a larger dimension. The proposed concatenation module makes it possible to implement a convolutional neural network in hyperbolic space.

We also gave an interpretation of minimizing the weighted sum of squares for the weighted center of gravity in the hyperbolic space used for self-attention.

Using the proposed layers, we implemented a convolutional neural network and Transformer on hyperbolic space ,which can learn a text recognition model with high performance even in low dimensions.

Future Directions

With the proposed model, basic recognition of data in hyperbolic space can now be performed. On the other hand, not all the data can be simply classified as either Euclidean data or hyperbolic data. Therefore, it is possible to increase the types of data that can be handled by combining models in Euclidean space with models in hyperbolic space, or by proposing models in other spaces. Research to estimate the underlying space from given data is also considered to be a useful direction.

Reference

Ryohei Shimizu, Yusuke Mukuta, Tatsuya Harada, “Hyperbolic Neural Networks++”, In the 9th International Conference on Learning Representations (ICLR 2021), 2021.

Machine Learning on Hyperbolic Space