Holding oak leaves is harmless. In fact, oak leaves are used in folk medicine. Acorns are a choking hazard and should be avoided. Note that I could not quickly find any references specifically on acorns. However, their size, shape and consistency are such that if acorns had been manufactured as toys or parts of toys, they would be classified as choking hazard for children.
In my experience, babies and toddlers love playing with natural objects such as leaves and small twigs with leaves attached to them, oak included. Research suggests that playing with natural "loose parts" may be beneficial to children, although I could not find much evidence other than observational studies.
Species of the genus Quercus [oak] are important medicinal plants. Over the centuries, these species have been used in folk medicine to treat various diseases
Taib M, Rezzak Y, Bouyazza L, Lyoussi B. Medicinal Uses, Phytochemistry, and Pharmacological Activities of Quercus Species. Evid Based Complement Alternat Med. 2020;2020:1920683. Published 2020 Jul 31. doi:10.1155/2020/1920683: full text
Because of their high-risk shape, small balls are held to a stricter criterion to prevent choking. The CSPA [Child Safety Protection Act] requires that balls be at least 1.75 inches in diameter if they are intended for use by children younger than 3 years. The CSPA defines a ball as a spheroid, ovoid, or elliptical object that is designed or intended to be thrown, hit, kicked, rolled, bounced, or dropped.
Committee on Injury, Violence, and Poison Prevention. Prevention of choking among children. Pediatrics. 2010 Mar;125(3):601-7. doi: 10.1542/peds.2009-2862. Epub 2010 Feb 22. PMID: 20176668: full text
Natural loose parts are the bread and butter of early childhood creativity in a natural outdoor classroom. Children gain so much from working with these materials because they have to think for themselves while being creative in problem-solving, constructing and imagining. Natural materials can be anything children want them to be and this ambiguity spurs their imagination. Some bits of nature are suggestive, looking like a motorcycle, for instance, or a piece of pie. While children may choose to play with the materials with that in mind, they are also free to use them in completely unique and inventive ways. When children have mostly closed-ended materials, or items that are authentic replications of items in real life, there is little room for imagination and creativity.
Kiewra, C., & Veselack, E.. (2016). Playing with nature: Supporting preschoolers' creativity in natural outdoor classrooms. The International Journal of Early Childhood Environmental Education, 4(1): full text