and a little interface for it. this is trying to spell the words using phonetic information (using a sequence-to-sequence neural network), the temperature parameter basically controls how the probabilities are distributed (at low temperatures, only the most likely characters are generated according to the information in the model; at higher temperatures, any character might be generated)

I need to stop playing with this, I have other stuff to do geez

still at work on this english nonsense word vae. here are some nonsense words sampled from the latent space of the latest trained model...

twidle
tuppilled
entedrul
tremdpobe
chominsbow
gripkan
dirquineus
dudenowed
rostore
kigan
nedermotta
sastors
lielandi
zessermas
ricknest
chated

these are generated by feeding the decoder with normally-distributed random numbers. pretty happy with how they all seem like jabberwockian-yet-plausible english words

by contrast, results of feeding normally-distributed random numbers into the decoder on the RNN without the VAE:

flfingeng
aughums
alohondism
h's
h's
autabovag
akeleghear
h's
alliltalles
barngnong
h's
mook
shewstlatscreth
huthure
chelthart
h's

not as good! which is encouraging, since it shows that the VAE model does actually have a "smoother" space than the non-VAE model.

(I have to admit that when I started this project I was like, "why do you even need a variational autoencoder, if just plugging random vectors into the decoder was good enough for jesus it's good enough for me," but there really is something magical and satisfying about being able to get more-or-less plausible generated results for basically any randomly sampled point in the distribution)

progress: at 50 epochs, even w/KL annealing, 32dims is not enough for the VAE latent vector to represent much of anything. leads to reconstructions that are probably just the orthography model doing its best with next-to-noise, but sometimes amusing, e.g.

cart → puach
liotta → pinterajan
intellectually → aching
capella → pellaka
photometer → augh
sympathizer → disteghway
butrick → jorserich
botha's → szine
clayman → tsantiersche
sparkles → trenlew
calamity → muliss
thermoplastic → tphare

(posted this mainly because "butrick → jorserich" seems like something mastodon people would like, e.g. "my name is Butrick Jorserich, follow me at jeans.butrick.horse/@rich")

apparently the trick to training a VAE w/annealing is to *never* let the KL loss go below the reconstruction loss. otherwise you get beautifully distributed, wonderfully plausible reconstructions that have almost nothing to do with your training data, i.e., "allison" becomes

uszecuin
auruse
lin-timer
ellefleigh
carmist
achubar
alsaa
houghtrodhan
ascear
edding
earpugh
ihtioz

exploring the latent phonetic nonsense space around "typewriter"—using the best model I've managed to train yet (100 epochs on HPC, managed to keep the reconstruction loss fairly low while also getting some semblance of a low KL loss)

using the phonetic VAE to interpolate between US state names in a grid

Follow

@aparrish Are you looking for reasons to have States named after you? Parishonde. Parrison.

Sign in to participate in the conversation
Toot.Cat

A Mastodon instance for cats, the people who love them, and kindness in general. We strive to be a radically inclusive safe space. By creating an account, you agree to follow our CoC below.

Instance Administration

  • Woozle: Supreme Uberwensch, general support, web hostess
  • Charlotte: tech support, apprentice in warp-drive arcana (aka Mastomaintenance)
  • ash: backend stuff, gay crimes

The Project: