Back to feed
arXiv cs.CL
arXiv cs.CL
7/1/2026
When transformers learn "impossible" languages, what do they learn?

When transformers learn "impossible" languages, what do they learn?

Short summary

Transformers trained on linguistically "impossible" languages show gradual degradation in grammatical sensitivity but pronounced failures in longer-form generation. Using BLiMP minimal pairs to separate capacities, researchers found generative deficiency—not grammatical insensitivity—drives model behavior. This explains why certain language structures don't exist in human languages: they violate production constraints.

  • •Models degrade gradually on grammatical sensitivity for unnatural languages
  • •Severe failures appear in generation, not in detecting grammaticality
  • •Generative constraints, not grammatical ones, explain why certain structures don't exist in nature

Generated with AI, which can make mistakes.

#research-breakthrough
Read full article at arXiv cs.CL

Is this a good recommendation for you?

Comments

Failed to load comments. Please try again.

Explore more

Developmental approach reveals the statistical learning of Neural Language Models: Transformers generalize from the most abstract statistical patterns
arXiv cs.CL
Generating in the Limit with Infinitely Many Hallucinations
arXiv cs.CL
Optimization Is Not All You Need
arXiv CS.AI
How RLHF Reward Optimization Produces Manipulative Defenses in LLMs
Dev.to