The Surprising Success of Large Language Models

“The success of large language models is the biggest surprise in my intellectual life. We learned that a lot of what we used to believe may be false and what I used to believe may be false. I used to really accept, to a large degree, the Chomskyan argument that the structures of language are too complex and not manifest in input so that you need to have innate machinery to learn them. You need to have a language module or language instinct, and it’s impossible to learn them simply by observing statistics in the environment.

If it’s true — and I think it is true — that the LLMs learn language through statistical analysis, this shows the Chomskyan view is wrong. This shows that, at least in theory, it’s possible to learn languages just by observing a billion tokens of language.”

–Paul Bloom, in an interview with Tyler Cowen

Challenging the Hypothesis of Innateness

For decades, the Chomskyan view has dominated our understanding of language development. This view argues that language structures are too complex to be learned solely from environmental input and therefore must require some kind of innate linguistic machinery in the brain (a “universal grammar”).

Yet as the quote above from Paul Bloom makes explicit, what LLMs have demonstrated–as a proof of concept–is that grammatical structures for language does not need to be innate. That machines can learn language via statistical associations alone, rather than explicitly programmed grammatical rules.

We have explored in a previous series on this blog the idea that language may not be a completely innate property of our brains, but rather more of a cultural phenomenon. This parallels the insight–much more widely accepted now–that learning to read is not innate.

The success of LLMs in acquiring language-like abilities through mere statistical analysis of texts demonstrates that it's possible to learn languages via statistical associations alone.

The Power of Statistical Learning

This revelation–that LLMs can learn language via statistical associations alone, rather than through any explicitly programmed rules–challenges our traditional understanding of language development and points to the power of implicit statistical learning.

However, unlike human children, who can rapidly learn language from relatively sparse input, current frontier LLMs require astronomical amounts of data to be trained. Yet the fact that machines can learn in this way suggests that the structure of language itself lends itself to such implicit learning.

This insight extends beyond language development and into literacy. We have previously examined seminal papers by Philip Gough and co arguing that learning to read words is more akin to learning a cipher than breaking a code. Rather than learning explicit rules, as from a codebook, we internalize patterns of sounds, letters, and meanings in an algorithmic fashion.

There is a fascinating line of research focused on “statistical learning,” and while there remains much to be learned about this domain, there seems to be an interesting convergence between this research as it relates to reading and as it relates to LLMs.

Reading nerds are already well acquainted with Mark Seidenberg, as he is a steady presence in the public sphere of communication and debates about reading instruction. What may be somewhat less known about him is that his oeuvre of research has been into computational, connectionist models of reading that have demonstrated how learning to read is a process of statistical learning between sounds, spelling, and meaning. It’s not that he hides this, by the way, but rather that the community of educators that are deep into the “science of reading” stuff don’t seem to be as enticed by abstract stuff like computational models and statistical learning.

But the convergence between connectionist accounts of learning language and learning to read and the advent of LLMs are important to understand. Not just from a nerdy stance, which has been mine throughout all these posts, but rather because LLMs have–again, as a proof of concept–demonstrated that implicit learning of statistical associations are fundamental not only to language and to reading, but to our knowledge and experience of the world.

Connectionist Models: Bridging AI and Human Learning

In fact, Seidenberg himself has repeatedly attempted to communicate the understanding that implicit statistical learning is just as fundamental to learning to read as it is to learning language.

He stirred up some recent controversy on this topic when he suggested that the “SOR” movement has over-corrected in response to previous squishy balanced literacy approaches by focusing too hard on explicit instruction as the cure-all for everything. See his provocative presentation and writing on this topic here: https://seidenbergreading.net/2024/06/24/where-does-the-science-of-reading-go-from-here-2/

To summarize his argument, which dovetails with where we started with LLMs, learning to read can not all be taught explicitly, and there is an opportunity cost to an over-reliance on the explicit teaching of “rules” over providing more opportunity for actual reading and writing to build up the statistical associations needed to become fluent:

“The purpose of explicit learning is to scaffold implicit learning about print, sound, meaning. Explicit instruction is the tip of the iceberg. The larger part under the surface is learned implicitly instead of teaching the whole iceberg.”

—slides on “Where does the Science of Reading go from here?”

In other words – only provide enough explicit instruction as needed to successfully spend more time engaged in an increasing volume of reading, writing, and talking.

Balancing Explicit and Implicit Learning in Language and Reading Instruction

In a paper, “The Impact of Language Experience on Language and Reading,” Seidenberg and Maryellen MacDonald also point to the fact that learning to read is easier for children with more advanced spoken language skills, while those with less exposure (due to greater variability of linguistic input) face greater challenges. This is because children exposed to multiple dialects or languages are learning to navigate multiple language systems, each with its own set of statistical linguistic patterns.

For multilingual and multidialectal learners, it is therefore especially critical to find the right combination of statistical learning and explicit teaching. According to the paper, consistent and increased exposure to the language of instruction is important. This exposure should be complemented by explicit teaching of both oral and written language patterns. And by explicitly comparing and contrasting home languages and dialects with the language used at school–both orally and in writing–students can develop metalinguistic awareness and a deeper understanding of varying language structures. This approach, implemented strategically within a welcoming and supportive classroom, allows students to leverage their existing linguistic knowledge while acquiring new language skills.

Another way of thinking about this, as we’ve explored in another post, is the movement from fuzziness to precision. By seeing, hearing, speaking, and writing an increasing volume of language, students can rapidly begin to make statistical associations. However, especially in the initial stages of learning a new language or learning to read, more effort will be required to gain greater precision, and thus, more mistakes will be a part of the learning process, and thus more feedback is needed to course correct at the very beginning.

I’ve written elsewhere about the importance of striking a balance between close reading of shared grade-level texts that are worth reading, while ensuring that each and every student reads a steady volume of texts that are more accessible. I’ve also written here about the need for “daily textual feasts” to increase the volume of rich language, knowledge, and critical thinking, as per Dr. Alfred Tatum.

Rethinking Language and Literacy Instruction

In sum, the surprising and awesome ability of LLMs, derived from mere statistical associations, has challenged traditional assumptions about the innate nature of language and, potentially, the role of explicit and implicit instruction in language and literacy learning.

This underscores the need for a comprehensive approach to teaching of reading and language, in which explicit teaching is strategically counterbalanced alongside implicit learning opportunities.

#AI #learning #language #LLMs #reading #explicit #implicit

Teacher Vocabulary Use and Student Language and Literacy Achievement

• Citation: Wanzek, J., Wood, C., & Schatschneider, C. (2023). Teacher Vocabulary Use and Student Language and Literacy Achievement. Journal of speech, language, and hearing research : JSLHR, 66(9), 3574–3587. https://doi.org/10.1044/2023_JSLHR-22-00605

The Power of Teacher Talk

We know that the explicit teaching of unfamiliar words that students will encounter in written text is important. But what about the language that is used by teachers throughout the school day? What implicit learning opportunities are constrained or afforded through the model of the language that a teacher uses while teaching, and what are the impacts on student learning? The importance of indirect or incidental language experiences in a classroom is emphasized in this study. And this and other research reviewed in the paper suggests that enriching linguistic environments are particularly beneficial for young readers or those who struggle with reading.

We’ve explored previously the importance not simply of “rich language” use (what does that even mean?) but of exposure to and use of a very particular kind of language: decontextualized language. This is the language of narrative, of conversational turn-taking and discussion around ideas and things, the more abstract language of written text. The content, form, and use of such language takes us beyond that of the immediate moment, beyond our own already delimited feelings and experiences, and into a realm of interpersonal and cultural thought, knowledge, and perspectives.

We can engage our children with this decontextualized language even before they leave the womb. They hear us tell stories and sing and begin to attune to our rhythms. Then when we can hold them in our arms, in our wraps, in our laps, we respond encouragingly to their babbling to tell them about the world, and we read picture books to them, showing them beautiful artwork that brings words alive. In classrooms, we read to our children with greater intention and a systematic approach, teaching them ideas and words before, during and after our carefully chosen texts, we instruct them in how to write what they can see or hear, and kids begin to automate the regular and irregular algorithms that sort letter-sounds and concepts into words.

Indirect or incidental language experiences can provide students with exposure to and use of new vocabulary and grammatical structures. When teachers use a variety of forms of language in their speech, they can provide students with opportunities to hear and learn new kinds of language, new kinds of ideas, and new kinds of feelings and viewpoints. Teacher talk can provide students with models in how to use these different types of language. When teachers use clear and concise language, they show students how to communicate more precisely and efficiently. When teachers give students opportunities to respond to questions or to participate in discussions around shared texts, topics, and themes, they provide students with opportunities to practice using that language to demonstrate and deepen their understanding of that new knowledge.

The Research Paper

This study focused on second-grade classrooms. To gather the language data, the 2nd grade teachers wore a “language environment analysis (LENA) digital language processor to record a full day of instruction twice per month throughout the school year. The researchers then analyzed segments of the language they used directed to students using Systematic Analysis of Language Transcripts (SALT) transcriptions.

The study revealed something incredibly important: teachers who used more academic words had their students achieve higher vocabulary levels by the end of the year.

Yet use of academic words was extremely uncommon in this sample of 64 teachers and 619 students, despite having a curriculum that included specific grade-level academic words: “On average, teachers used common words, with 87% of the words used by teachers on the list of the 1,000 most frequently used words in the English language. Academic words were used only 1% of the instructional time on average, suggesting very little input for students for these more school-based words.”

The researchers furthermore found that it's not just about the quantity of words but the quality and relevance to the subject matter being taught. Importantly, they found that the “academic word use by teachers continued to predict student vocabulary outcomes even once teachers’ expressive vocabulary was considered. In other words, the relationship is not explained by some students having teachers with a higher overall vocabulary. All teachers who used more academic words in their instruction and discussion had students with higher vocabulary at the end of the school year. Second, the relationship was not different for students of varying incoming vocabulary abilities.”

This is therefore a potentially high impact influence on learning for the kids who need it the most.

What was also interesting was that they found that ELA and math were regularly taught, usually daily, while science and social studies were taught significantly less across all the schools. “Thus, students received language input largely through ELA and math instruction during an average school day.” In this study, the teachers used a variety of ELA curricula (Wonders, Journeys, and others), some of which are not systematic in how they approach building knowledge and language.

Why I think this study is important

While not expanded upon in this paper, the lack of more content and disciplinary focused instruction across a week clearly bears implications for their finding on the lack of academic words that students were exposed to at large. There was a paper a while back that this reminds me of, a 2020 analysis from Fordham Institute, in which they found, counterituitively, that “Increased instructional time in social studies—but not in ELA—is associated with improved reading ability.”

We may thus be going astray if we’re merely expanding or reworking literacy blocks without simultaneously boosting up the academic knowledge and language that students gain from discipline specific study and the reading, writing, and talking around strategically selected texts that build cumulative and coherent bodies of knowledge.

Furthermore, this study also found that the proportion of less common words used was highest in math classes. This suggests that math could be an unexpected hotspot for academic vocabulary and language development – which makes sense when you think about it. The language needed for mathematical thinking and discourse is precise and specific to the discipline. And yet the opportunity to explicitly teach language and literacy through math is not often fully leveraged.

I should be clear that this study (and the authors make this clear) is correlational, not causational, and “does not suggest that filling instruction with academic words would mean even higher vocabulary achievement.” Another limitation of the data in this study is that it “did not allow for consideration of students' utterances or conversational exchanges between a variety of communication partners which could also be of interest in a future study.”

And that in fact connects to the findings of our last research highlight, which was on the importance of automatization in learning a new language. The more that children can hear, see, speak, and write the new words and ideas they are learning, the more those words and ideas will stick.

All in all – this is the kind of research that I think every teacher should be aware of. Every word we use in our speech (or signing), and every word we put before them in the texts we select can inhibit or expand what our kids can learn. Discuss... #literacy #language #research #vocabulary #automatization #implicit #explicit

This point is a sharp one because what they were saying is that we can’t teach such a cipher directly. We can’t just hand a kid the codebook.

So when I saw a reference recently to another Gough paper called Reading, spelling, and the orthographic cipher, co-written in 1992 with Connie Juel and Priscilla Griffith, I knew I needed to read this one, too.

This later paper makes many of the same points that the 1980 paper does, but with added depth and empirical studies to back it up. In this post, I’m going to pull out a few quotes from the paper that I found interesting to ruminate a little further on this idea of a cipher and implications for instruction.

“The orthographic cipher of English (in short, the cipher) is very complex. A simple cipher would map each letter onto a single phoneme and each phoneme onto a single letter. But English has only 26 letters to map onto more than three dozen phonemes, so it could not be simple; either a letter must represent more than one phoneme, or some phonemes must be represented by more than one letter. Moreover, English orthography was woven by history (Scragg, 1974), and like most such fabrics the basic pattern has been stitched and darned, and altered and augmented many times.”

This is the challenge of the English cipher. 26 letters map ~44 phonemes in a quasi-regular manner, with spellings and morphemes amalgamated from Anglo Saxon, Latin, and Greek origins.

“Words that are predictable tend to be short and common, whereas words that are unpredictable tend to be long and uncommon. Thus context will fail children exactly where they most need help.”

And context is not enough to determine most unfamiliar words, despite what three-cueing may tell you. Readers must be able to recognize words, and nearly instantaneously.

“This is not to equate literacy with word recognition; there is much more to reading than recognizing words. After recognizing the word letter, readers must decide whether it means a character or a missive; they must disambiguate it. After deciding on the meaning of each word in the sentence Juan showed her baby pictures, readers must decide whether baby or pictures is the direct object; they must parse the sentence. After understanding each sentence in a discourse, readers must assemble them into a larger framework; they must build a discourse structure. And after understanding the discourse, readers must integrate it with what they already know; they must assimilate the text.

But readers must also do these things when they listen. These are linguistic skills, not just of reading, but of comprehension in general. So we equate literacy not with word recognition, but rather with the product of that skill and com prehension (Gough & Tunmer, 1986; Hoover & Gough, 1990; Tunmer & Hoover, this volume). Reading “R” equals the product of decoding “D” and comprehension “C”, or RD X C.”

I’d add a wrinkle to this: the linguistic skills required for comprehending the language of written text require more effort (at least initially, most especially within a discipline of study), as decoding does. The more exposure to this written form of language, the better. This is why read-alouds from the earliest ages are so important.

But there is evidence suggesting that indeed, listening comprehension and reading comprehension are more or less equivalent, when decoding is taken out of the equation. I don’t know how to resolve this, but it doesn’t make sense to me that we could equate listening to a story or informational read-aloud as equivalent to listening to a friend tell us about something that happened to them earlier. The language of written text is decontextualized, it is abstract. Rarer words and sentences are used. We have to make more inferences to fill in the blanks. More on this in future posts — I’m on a big kick around the power of interactive read-alouds, most especially for students newer to the English language. Back to Gough et al.:

“What children need is a way to recognize novel words on the basis of their form. We should remember that the vast majority of these words are already known to children in their phonological form, for in the early grades almost all of the words that readers encounter are already part of the child's vocabulary. So if there was a way to convert the printed form into a phonological form, children could readily recognize them.

Fortunately, an alphabetic language like English affords a mechanism that works for many of its words. An alphabetic orthography is based on a system of rules that map letter strings onto phonological forms; the letters of printed words represent the phonemes of spoken ones. If children could internalize this system, they would have a way of transforming the novel into the familiar, and they could decode the message.”

This made me think about students new to the English language, and how they do not necessarily have that unfamiliar word as a firm part of their lexicon, either in its phonological form nor in its semantic meaning. This means a teacher must ensure that instruction on a word’s coded form must also be conducted in direct association with its meaning. Furthermore, a teacher can make connections between the English word form and meaning to the potentially more familiar forms and meaning in a student’s home language.

Now we get to really interesting part about internalizing the cipher, the cryptanalysis that a new reader must undertake:

“In making this assertion, we are trying to make three points. First, we argue that learning is distinct from teaching, that whatever or however they might be taught, what will determine how children read is what they internalize. Second, we argue that if they are to read with any degree of skill, they must internalize the cipher. That is, we argue that there is only one way to read well and that is with the aid of the cipher. Thus however children are taught, whether by phonics, whole language, or some eclectic method, they must master the cipher, or they will read poorly if at all. Third, we argue that even when the attempt is made to teach the cipher directly, as in synthetic phonics, the rules that children are taught are not the rules that they must internalize.”

As I pointed out in another post, this appears to be an interesting point of convergence between Ken Goodman and Phillip Gough: they both claim that learning to read can’t be taught directly. Here Gough et al. claim that even in the case of synthetic phonics, the most direct and explicit form of teaching grapheme-phoneme correspondences, it’s still not necessarily enough to get an individual child all the way there. Each individual child needs to internalize the algorithm of the code.

“As we have pointed out elsewhere (Gough & Hillinger, 1980), the rules of phonics are explicit, few in number, and slow. In contrast, the rules of the cipher are implicit, very numerous, and very fast. Our assumption is that the two are distinct. Indeed, we are intrigued by the suggestion that what the child has internalized are not rules at all but might instead be a system of analogy (Goswami, 1986) or even a connectionist system (Seidenberg & McClelland, 1989; Seidenberg, this volume). Whatever the form of the cipher, whether it consists of rules, analogies, or connections, we contend that it does not consist of the rules taught consciously in phonics.”

Should we teach rules? What rules should we teach, and when? There is no consensus on an exact scope and sequence for phonics instruction, only that it must be structured and systematic. Most sequences are organized around the general principle of easier to harder.

Gough et al. make an interesting conjecture regarding what it is that is being internalized. This also connects to a wider debate about what must be taught explicitly via direct instruction vs. gained implicitly via adequate opportunity to hear and see patterns of spoken and written forms and meaning. There’s also some debate about the teaching of “rules.”

There’s more interesting items in this paper to consider, but I’ll leave it there, as I think we’ve got some good food for thought. How do we get an individual child to internalize the cipher in the most effective way based on that individual child’s experiences with spoken and written language?

Is a synthetic phonics approach maximally effective and efficient for all children? Is it possible that students new to the English language may benefit from a flexible approach that brings in analytic and embedded phonics methods to ensure words are understood in their phonological and morphological forms and meaning while learning to deconstruct and reconstruct them? Is it possible some kids may need far more explicit phonics instruction, while some may need far less?

Some more reading along these lines:

• Tim Shanahan’s post, Which is best? Analytic or synthetic phonics?
• Mark Seidenberg, Matt Borgenhagen, and Devin Kearn’s, Lost in Translation? Challenges in Connecting Reading Science and Educational Practice
• Donald Compton, Have We Forsaken Reading Theory in the Name of “Quick Fix” Interventions for Children With Reading Disability?

#reading #implicit #explicit #rules #internalize #phonics #cipher #cryptanalysis

Discuss...

The first thing that happened to reading is writing. For most of our history, humans have been able to speak but not read. Writing is a human creation, the first information technology, as much an invention as the telephone or computer.

—Mark Seidenberg, Language at the Speed of Sight

What is (un)natural about learning to read and write? We began our quest with this question, prompted by two references in a line in a David Share paper.

Like learning to read (English) which Gough famously dubbed “unnatural” [43], see also [3], becoming aware of the constituent phonemes in spoken words does not come “naturally”.

—Share, D. L. (2021). Common Misconceptions about the Phonological Deficit Theory of Dyslexia. Brain Sciences, 11(11), 1510.

This led us to unpack three foundational papers from 1976 to 1992 that have provided us with some surprising twists and turns and even moments, dare I say, of clarity.

Rather than spend too much time re-hashing what we’ve already covered, I wanted to take an opportunity to further reflect on what I’ve learned and on where I currently stand after all these geeky deep dives (I took a brief interlude between the 1st two papers to ruminate as well).

I believe that this debate about what is “natural” about teaching early reading is far more fundamental than it seems. For example, the “sage on the stage” vs. “guide on the side” divide surfaces in the Goodmans’ account of what effective teaching and learning should be for early reading, reflecting a deep-seated romantic tendency to elevate the status of children, wherein there is the belief that if we just allow children to learn “naturally,” they will somehow discover complex academic concepts.

This is true of social language. The swiftness with which we acquire our native language(s) as children is remarkable. Yet even here we must be careful. There are some children that do not learn and develop language at the same rate that others do, perhaps due to differences in working memory and other neurobiological reasons. This tells me that Liberman’s conjecture that speech is pre-cognitive may have been too bold.

Effortful, Rather Than Unnatural

Tracing these arguments has helped me to see more clearly that language and literacy development are on a spectrum from effortless to effortful, with another axis around the individual profile of a child that requires either more explicit instruction and deliberate practice or greater opportunities for more independent implicit learning. There are certain abilities that are more commonly effortless for most children, such as learning a first language, and others that are more commonly effortful for many, such as learning to break the code. And some children find effortless ones more effortful, and other children find the effortful ones also quite effortless (lucky them).

This applies to any skill: some kids can jump on a bike and start riding almost immediately, while others will need quite a lot of explicit modeling and practice with training wheels. Some kids can swim like a fish after a few lessons and practice, while other kids (like me) will only develop a half-sufficient dog paddle even after swim lessons, living near the ocean, and having a pool in their backyard.

The analyses of G&H and Liberman have helped me to identify more precisely where the greatest effort in learning to read in English lies: at the sublexical level—the level of phonemes and letters and letter sequences—a level that is, in their estimation, “unnatural,” because these sublexical units are “meaningless” and “artificial,” in the sense that they are “arbitrary.”

We do need to acknowledge there is an “artificiality” to written language. This artifice allows us to map “arbitrary” symbols onto our spoken language and record them for all time.

Yet I am concerned that framing learning sublexical units as completely unnatural may be a turn-off to those who would decide that teaching them is therefore antithetical to the goal of channeling the innate and “natural” curiosity and potential of children to read. I mean, there are still active and inflamed debates about phonics going on, and we’re trying to bring people on board here.

Gough and Hillinger’s analogy of learning to read to cryptanalysis is a highly useful one, but I am not convinced that warrants calling the process unnatural. Ever heard of the genetic code? Nature has its own alphabetic cipher going on!

Learning to Read is Learning to Control a Flame

Instead, I think we should focus on the fact that written language is a remarkable feat of human development, as awe-inspiring as rocket ships, as innovative as smartphones, and as individually empowering as the automobile (though with far less toxicity).

While I find Liberman’s distinction between oral language as biological in origin and written language as cultural useful, I also think it’s again more of a question of a spectrum, rather than a sharp divide. We have no biological, innate ability to create fire, for example. Our ability to create controlled flame is entirely driven by human culture. Yet fire is so deeply interwoven into the propagation of our species that it is intimately tied to our biological evolution and survival. Would we say that learning to make fire is “unnatural”?

This is mostly a matter of rhetoric, of course. The reason for G&H and Liberman’s branding of “unnatural” was to highlight the fact that learning to decode written language can be challenging, and to try and unpack exactly why that is.

So let’s instead focus on the fact that learning to break apart spoken words into little pieces of phonemes to attach them to letter sequences (and vice versa) is both abstract and effortful for many children, and also an absolutely amazing collective and individual achievement. This allows us to see that it therefore will most likely require explicit support and deliberate practice, and that furthermore it is well worth getting kids pumped up about gaining it.

This is where we also need to bear in mind the spectrum in what students bring to their first encounters with formal instruction with written language. Nancy Young’s updated Ladder of Reading and Writing is a great depiction of this spectrum, which acknowledges that there are indeed a small percentage of children for whom acquiring literacy will be mostly effortless, while for the majority of kids, a structured literacy approach is needed, with more intensity required for some.

We also know that students bring different spoken dialects and languages to the classroom, and the nature of those dialects and languages may influence the form of code-based instruction that could be highest leverage.

Let’s also remember a caution that both Gough and Goodman made in their respective papers: we can’t just hand over a codebook of rules to our kids. They must ultimately internalize the cipher themselves. What is the right balance of explicit and implicit learning, of difficulty and ease, of guided and independent practice? What are the profiles of student that we have in our classroom, and how can that guide us in determining the level of structure that we need to provide?

Well, clearly, there’s more to explore here, with plenty of controversy remaining. If you’ve stuck with me this far, I salute you! Thanks for reading.

#natural #unnatural #innate #language #literacy #reading #writing #heterogeneity #implicit #explicit

Discuss...