What exactly does word2vec learn?
Berkeley AI researchers proposed the first theory quantitatively predicting word2vec learning. Under realistic training conditions, it reduces to unweighted least-squares matrix factorisation with a closed-form gradient-flow solution, and the resulting representation is equivalent to PCA of target matrix M*. From small initialisation, word2vec learns orthogonal linear subspaces in discrete steps, increasing embedding rank and reducing loss. These features are M*'s leading eigenvectors and can be calculated in advance from corpus statistics and hyperparameters.