On the structure between narrators and readers.

Foreword I’ve been meaning to write this blog post for a long time. I had the idea sometime in September 2020, and started doing a bit of lit review. Sometime around Christmas that year, I was going to finally sit down and write the blog post but I quickly realized that even after all the …

Joint Representations of Connectionism vs. Symbolism via Attractor Networks and Self Attention

Introduction In most modern fields of cognitive computing, as it pertains to natural language processing, both symbolism and connectionism have their own unique benefits. Symbolism relates to the idea that most cognitive processes can be computed as the manipulation of symbols. Alternatively, in connectionism, computation is done by a metaphorical “black box” in which the …

On Neural Persistence, Improved Language Models, and Narrative Complexity

Introduction There is an amazing paper I got to read last week by [Rieck19] on the subject of persistent homology. The idea is that by borrowing ideas from topological data science, we can construct a per layer complexity metric. This complexity metric can then shed light into how generalized our learned representation is. Namely if …

On the Embedding of Narratives, and How it Pertains to Computational Neuroscience

Introduction A common question for the last article I posted was “Why did you reference [Riemann17]? Aren’t narrative embeddings identical to sentence embeddings?” For this post, I will consider narrative embeddings w.r.t. the ROCStories dataset. Firstly, we need to precisely define narrative embeddings. For this we need a few definitions. Fabula vs. Syuzhet Lets clarify …

Topology and You: What the Future of NLP has to do with Algebraic Topology

Introduction The size of NLP datasets has grown exponentially over the last few years. It was only a handful of years ago that 100-dimension (non-contextualized) word2vec [Mikolov et al.] embeddings were cutting edge. By comparison BERT embeddings are 700+ dimensions, Ai2’s ELMo is close to 1000 dimensions, and GloVe (the current industry standard) is typically …