“Over cultural evolution, the human species was so pressured for increased information capacity that they invented writing, a revolutionary leap forward in the development of our species that enables information capacity to be externalized, frees up internal processing and affords the development of more complex concepts. In other words, writing enabled humans to think more abstractly and logically by increasing information capacity. Today, humans have gone to even greater lengths: the Internet, computers and smartphones are testaments to the substantial pressure humans currently face — and probably faced in the past — to increase information capacity.”

—Uniquely human intelligence arose from expanded information capacity, Jessica Cantlon & Steven Piantadosi

According to the perspectives of the authors in the paper quoted above, the capacity to process and manage vast quantities of information is a defining characteristic of human intelligence. This ability has been extended over time through the development of tools and techniques for externalizing information, such as via language, writing, and digital technology. These advancements have, in turn, allowed for increasingly abstract and complex thought and technologies.

The paper by Jessica Cantlon & Steven Piantadosi further proposes that the power of scaling is what lies behind human intelligence, and that this power of scaling is what further lies behind the remarkable results achieved by artificial neural networks in areas such as speech recognition, LLMs, and computer vision, and that these accomplishments have not been achieved through specialized representations and domain-specific development, but rather through the use of simpler techniques combined with increased computational power and data capacity.

I think the authors may be overselling scaling as the main factor behind intelligence, but scale most definitely plays a leading role alongside brain and neural network architecture and specialized data, and it most definitely plays a role in how human language is used and developed.

The Potential of Scale

“LLMs give us a very effective way of accessing information from other humans.”

–Alison Gopnik in an interview with Julien Crockett in the Los Angeles Review of Books

In our previous explorations of language, cognition, and Large Language Models (LLMs), the recurring theme of the power of scale has certainly emerged.

We've delved into the statistical nature of language, where the vast interconnectedness of word combinations and their contextual relationships drive LLMs' generative abilities. We've pondered the inherent imprecision of human language and the journey towards computational precision in LLMs. And throughout, the concept of scale has remained central – the scale of data, the scale of computation, and the scale of language itself.

It's intriguing to consider the possibility, as this paper suggests, that the capacity to process increasing amounts of information may have been a key factor in the development of human intelligence. This idea extends to how, as a species, we have continually sought ways to expand our ability to store and access information, from the invention of writing to the development of computers, the internet, and smartphones.

This suggests that the most exciting potential of artificial neural networks such as LLMs may lie not only in their ability to respond to and generate human language, but furthermore in their ability to help us to process and manage vast quantities of information, and thus further extend our cognitive capabilities. When framed in this manner, it shifts the debate from whether LLMs already demonstrate human intelligence and whether they will soon achieve superhuman intelligence, to whether LLMs will indeed equip us with superhuman abilities. And – as always with advancements in powerful technologies – the question is who among us will gain the most from those abilities and whether the new tools will further increase or diminish disparities between groups (i.e. “the future is already here — it's just not very evenly distributed”).

So we’ve explored a few implications of LLMs relating to language and literacy development so far, then: 1) LLMs gain the base for their uncanny powers from the statistical nature of language itself; 2) LLMs present us with an opportunity for further convergence between human and machine language; and 3) LLMs present us with an opportunity to further extend our cognitive abilities by allowing us to process far more information.

The Dark Side of Scale

All of this said, there is a dark side to scale, as Geoffrey West elucidates in his book, Scale (more on this on my other blog, Schools & Ecosystems), which is that as we consume far more energy and create far more waste beyond our biological needs and functions than any other creature on earth as we continue to scale our technologies. As West describes it, we humans are energy-guzzling behemoths, using thirty times more energy than nature intended for creatures our size. Our outsized energy footprint makes our 7.3 billion population act as if it were in excess of 200 billion people. And we are hitting the upper limits on ecological constraints of the earth as we do so.

Similarly, as LLMs extend our capabilities, they consume ever more power as they consume and produce ever more data. So at the very same time that our earth is rapidly accelerating towards critical thresholds of environmental change and wreaking havoc on insect, animal, soil, and plant life, we are rapidly accelerating our consumption of energy and production of waste.

It’s hard to see a clear end in sight to this. It’s possible that the greedy demands of continuing to scale AI model training and use ends up leading to rapid development of greener technologies and accelerated efficiency in digital computation and compression. It’s just as possible that in our short-sighted endeavors we put a half-life on human civilization via no longer containable war, famine, disaster, and disease.

Not to end this post on such a sour note, but it is important to bear a healthy skepticism about a new technology and its attendant powers, even as we seek to gain from it. And from what I see in the discourse, it seems to me that there has been a pretty healthy mix of boosterism and critique and excitement and paranoia about it all, so I’m enjoying the ride, nonetheless.

#cognition #language #AI #LLMs #technology #brains #scale

Before we venture into the wilds of the possible relations between language and thought, I wanted to build on this idea of how our inner scaffolds are most fully realized through speaking, listening, reading, and writing by geeking out about the beauty and wonder of multilingualism. There was a beautiful study I came across recently that provides a great way to visualize this.

The researchers used functional near-infrared spectroscopy (fNIRS) to examine neural connectivity during English word processing in bilingual (Chinese-English and Spanish-English) and monolingual children.

The study groups included children (ages 5-10 years, Grades K-4) who were English monolinguals, Chinese-English bilinguals, or Spanish-English bilinguals, all receiving English-dominant education in the US (recruited from southeast Michigan, USA).

The researchers found that the greater proficiency a child (across all groups) had in both spoken and written language, the stronger the farthest connections across their brains were. In other words, spoken and written language exposure and use made longer distance connections across the brain, and then strengthened and reinforced those connections.

Children who were older and more proficient in spoken and written English showed more long-distance connections within the broader language network and across the two hemispheres, suggesting that advancements in language skills are supported by more integrated neural networks. In other words, the development of short-distance connections supports more basic language functions, while long-distance integrative connections mark more advanced or efficient language processing in older and more proficient children.

Furthermore, among bilinguals they found that the greater proficiency a child had in two languages, the greater the neural density those language networks were. In Spanish bilinguals, the network density was associated with Spanish vocabulary, whereas in Chinese bilinguals, the network density was associated with Chinese character reading. Both groups showed greater network density in English in relation to their heritage language skills (most likely due to greater time spent in instruction and use with that language).

These findings suggest that language development is supported by both short and long distance connectivity in a child’s brain. Moreover, long-distance connections are likely critical in integrating different and more complex aspects of language processes such as phonological and morpho-semantic analyses.

What a wonderful visualization of how our inner scaffolds – the nascent neural networks in our brains – are developed by language and literacy! The more we use language across oral (or signed) and written modalities, the more we refine those networks across our brains. And the more languages we speak (or sign) and write, the more we further strengthen those networks based on the unique features of those languages.

We see this with students who are in dual language programs for multiple years – they begin to outperform their monolingual peers. We see with students who are former English language learners (ELLs) who achieve English language proficiency – after achieving proficiency, they begin to outperform their monolingual peers.

So not only do we want to provide our children with daily textual feasts – but furthermore, with linguistic knowledge-building feasts.

#language #research #neuroscience #brains #literacy #reading #writing #multilingualism #bilinguals

Two books – Rethinking Innateness and The Language Game – provide us with some fascinating perspectives on language learning that bears implications for how we think about learning to read and write, and furthermore, for how we talk about the power and limitations of AI.

A Review of Where We’ve Been

In a previous series, we pursued an interesting debate about whether learning to read is more unnatural than learning oral or signed languages. We also investigated the notion, frequently stated by “science of reading” proponents, that “our brains were not born to read,” while our brains are “hard-wired” for language.

While I agree with researchers Gough, Hillinger, Liberman and others that written language is more complex and abstract than oral language and—hence—more difficult to acquire, I’m not convinced that calling it unnatural is most accurate. Instead, I suggest terming it effortful.

In one of the earlier papers we examined, Liberman argued that oral language is pre-cognitive, meaning that it requires no cognition to learn and thus is more natural to acquire. He used this claim to counter the Goodmans’ assertion that oral and written language were largely synonymous, and that kids therefore could learn to read merely through exposure to literacy, rather than explicit instruction in the alphabetic principle (“whole language”). While I most definitely don’t agree with the Goodmans, I paused on Liberman’s claim with some skepticism, as there are a subset of kids who also struggle to develop speech and language skills, just as there are a subset of kids who struggle to develop reading and writing skills.

Liberman also made another strong claim that I paused on: that the evolution of oral language is biological, while written language is cultural (which parallels arguments that language is “biologically primary” while reading and writing are “biologically secondary,” which I have also questioned, given that making the distinction is harder than it seems when social and cultural advancements are deeply interwoven with human existence over generations of time). But I mostly accepted this premise, as it seems to be self-evident that language is baked into our brains. After all, babies begin to attune to languages spoken around them even while still in the womb.

Liberman does not stand on his own in these assertions, I should hasten to add. I just bring one of his papers up because we spent time with it here. Noam Chomsky, for example, has long argued for a universal grammar, which is taught in foundational courses on linguistics, and the related study of generative grammars is alive and well.

Why is this important? It’s important because whether we consider language “natural” or written language “unnatural” bears implications for how we decide to teach them (or not). If we think of language as completely innate, then perhaps we don’t think it requires much of any teaching that is explicit, systematic, or diagnostic. Or conversely, if we think of written language as wholly unnatural, we may not consider how to strategically design opportunities for implicit learning, volume, and exposure.

Yet I have just read two books, written in two different decades, that provide some really interesting critiques against the widely adopted supposition that language is innate.

Language Models

The first book, Rethinking Innateness: A Connectionist Perspective on Development, by Elizabeth Bates, Jeffrey Elman, Mark H. Johnson, Annette Karmiloff-Smith, Domenico Parisi and Kim Plunkett, was published in 1996, and approaches language from the lens of neuroscience, explaining connectionist models and their implications for neural development and learning. These models are not only part of the lineage of the current renaissance of Large Language Models, such as ChapGPT, but also part of a lineage of models that have informed our theoretical understanding of how children learn to read, and may continue to inform explorations of “statistical learning.”

I was led to this book from a recommendation by Marc Joanisse, a researcher at Western University, when he commented on my tweet (are we still calling them that?) about research on artificial neural networks that suggests they can accurately model language learning in human brains.

It was a great recommendation, and I found the book extremely relevant to ongoing conversations about AI and LLMs today, in addition to providing key insights from connectionist models into language and literacy development that challenge assumptions around innateness, such as:

• Simulations show that simple learning algorithms and architectures can enable rapid learning and sophisticated representations, such as those seen in younger infant competencies, without any innate knowledge.
• U-shaped learning and discontinuous change also occur in neural networks without innate knowledge, due to architecture, input, and time spent on learning. This parallels studies of the development of linguistic abilities in children, such as the learning of past-tense and pronouns.
• The way in which neural networks learn new things can be simple, yet the learning yields surprisingly complex results. This complexity emerges as the product of many simple interactions over time (this point, written in 1996, seems incredibly prescient to me as a reader in 2023 using Claude2 to distill and summarize my notes from each book for this post).
• Connectionist models show global effects can emerge from local interactions rather than centralized control. Connectionist models also show how structured behaviors can emerge in neural networks through exposure to and interactions with the environment, without explicit rules or representations programmed in (which makes me think of statistical learning).

Language Games

The second book, The Language Game: How Improvisation Created Language and Changed the World, by Morten H. Christiansen and Nick Chater, was published last year in 2022, and focuses more on cultural evolution and social transmission of language, arguing that language is akin to a game of charades that is honed and passed on from generation to generation. I happened to check it out from the library and read it concurrently with Rethinking Innateness, and there was some great synergy between the two, especially around challenging the notion that language is innate. Some of the key points of the book:

• Language relies on and recruits existing cognitive mechanisms, becoming increasingly specialized through extensive practice and use.
• Language evolves culturally to fit the human brain, not the reverse.
• Language is shaped for learnability and for coordinating with other learners, not for abstract principles and rules. Children follow paths set by previous generations.
• This cultural transmission across generations shapes language to be more learnable through reuse of memorable chunks (“constructions”).
• Due to working memory limitations, more memorable chunks survive, causing a design without a designer. These chunks become increasingly standardized over time.
• Language input must be processed immediately before it is lost (what the authors call the “Now-or-Never” bottleneck).
• Chunking sounds into words and phrases buys more time to process meaning.
• Gaining fluency with increasingly larger and more complex constructions of language requires extensive practice.

Across Connectionism and Charades

Together, these books provide a picture of language as an emergent, complex cultural and statistical phenomena that has evolved from simple learning mechanisms across generations. Rather than an innate universal grammar baked into children’s brains, language itself has adapted and molded over time to become essential to our human inheritance, as with clothing, pottery, or fire. Language emerges through social human communication and interaction. It becomes increasingly complex, yet also streamlined and standardized, without any explicit rules governing it beyond the constraints of our brains, tongues, and cognition.

This isn’t to say there isn’t something unique about the human brain architecture in comparison to our closest animal brethren—there clearly is—but rather that language has adapted symbiotically to that architecture, like a parasite, rather than specific parts of our brain that are genetically pre-determined for language.

Like reading, using language drives increasing specialization of our brain—and this specialization, in turn, drives greater cognitive ability and communicative reach.

There’s a lot here to unpack and synthesize, but I wanted to begin bringing these together, because just as I feel myself pushing against the zeitgeist when I argue that calling learning to read “unnatural” isn’t quite right, so too are arguments that learning language is not “innate” swimming against the tide. These two counterclaims are interwoven, and I think worth further exploring.

Consider this post the first in an exploratory series. We’ll geek out on language development and its similarities and differences to literacy development, maybe dig into the relation of cognition and language and literacy a little, and riff on the implications for AI, ANNs, and LLMs.

#language #literacy #natural #innateness #unnatural #reading #neuralnetworks #research #brains #linguistics #models