Applications of Tensor Networks to Machine Learning

The use of tensor networks for machine learning is an emerging topic. One branch of research involves using a tensor network directly as machine learning model architecture. Another uses tensor networks to compress layers in neural network architectures or for other auxilary tasks.

Because tensor networks can be exactly mapped to quantum circuits, an exciting direction for tensor network machine learning is deploying and even training such models on quantum hardware.

Selected Works on Tensor Networks for Machine Learning

Influential or ground breaking works on theory, algorithms, or applications of tensor networks for machine learning. Please help to expand this list by submitting a pull request to the tensornetwork.org Github repository.

Supervised Learning

• Exponential Machines, Alexander Novikov, Mikhail Trofimov, Ivan Oseledets, arxiv:1605.03795 (ICLR 2017 workshop track paper)
  Proposes the use of tensor networks for supervised machine learning with an elegant interpretation of the function represented by the model and a stochastic Riemannian optimization procedure. State-of-the-art performance is demonstrated on multiple datasets.

• Supervised Learning with Tensor Networks, E.M. Stoudenmire and David J. Schwab, Advances in Neural Information Processing Systems 29 (2016) [1]
  Proposes the use of tensor networks for supervised machine learning, emphasizing connections to the quantum physics literature including related algorithms for optimization. State of the art results are demonstrated for the MNIST handwriting dataset.

• From probabilistic graphical models to generalized tensor networks for supervised learning, Ivan Glasser, Nicola Pancotti, J. Ignacio Cirac, arxiv:1806.05964
  Explores connections between tensor networks and graphical models. Proposes “generalized tensor network” architectures for machine learning, and explores learning of local feature maps.

Unsupervised Learning

• Unsupervised Generative Modeling Using Matrix Product States, Zhao-Yu Han, Jun Wang, Heng Fan, Lei Wang, and Pan Zhang, Phys. Rev. X 8, 031012 (2018). [2]

• Tree tensor networks for generative modeling, Song Cheng, Lei Wang, Tao Xiang, and Pan Zhang. Phys. Rev. B 99, 155131 (2019). [3]

• Tensor networks for unsupervised machine learning, Jing Liu, Sujie Li, Jiang Zhang, Pan Zhang, arxiv:2106.12974
  Explores the idea of representing a sequence of conditional probabilities as separate tensor networks to learn a full, or joint probability distribution.

Compression of Machine Learning Architectures

• Tensorizing Neural Networks, A. Novikov, D. Podoprikhin, A. Osokin, D. Vetrov, Advances in Neural Information Processing Systems 28 (2015) [4]
  Proposes the use of “quantized” tensor train matrix (matrix product operator) decompositions to achieve a massive compression of weights of a neural network layer with only a small impact to performance.

Mathematical or Theoretical Works

• Expressive power of tensor-network factorizations for probabilistic modeling, Ivan Glasser, Ryan Sweke, Nicola Pancotti, Jens Eisert, Ignacio Cirac, Advances in Neural Information Processing Systems 32 (2019) [5]
  Rigorously explores and compares the expressive power of various tensor network formats, parameterizations, and formalisms for using them to model complicated functions.

• Lower and Upper Bounds on the VC-Dimension of Tensor Network Models, Behnoush Khavari and Guillaume Rabusseau, arxiv:2106.11827 [6]

Language Modeling

• Explainable Natural Language Processing with Matrix Product States, Jirawat Tangpanitanon, et al. arxiv:2112.08628

References

1. Supervised Learning with Tensor Networks, Edwin Stoudenmire, David J Schwab, Advances in Neural Information Processing Systems 29 (2016)
2. Unsupervised Generative Modeling Using Matrix Product States, Zhao-Yu Han, Jun Wang, Heng Fan, Lei Wang, Pan Zhang, Phys. Rev. X 8, 031012 (2018)
3. Tree tensor networks for generative modeling, Song Cheng, Lei Wang, Tao Xiang, Pan Zhang, Phys. Rev. B 99, 155131 (2019)
4. Tensorizing Neural Networks, Alexander Novikov, Dmitry Podoprikhin, Anton Osokin, Dmitry Vetrov, Advances in Neural Information Processing Systems 28 (2015)
5. Advances in Neural Information Processing Systems, Ivan Glasser, Ryan Sweke, Nicola Pancotti, Jens Eisert, Ignacio Cirac (2019)
6. Lower and Upper Bounds on the VC-Dimension of Tensor Network Models, Behnoush Khavari, Guillaume Rabusseau (2021), 2106.11827

Edit This Page