The first mention of the Fennels Manor is in 1283 when owned by Robert Fitz Neel (of which fennel was a corruption). In 1329 Robert Fitz Neel had a messuage, 176 acres of land and a half share in a water mill, which comprised the manor. On Robert's death in 1332 he was succeeded by his daughter Grace, wife of John de Nowers and in 1370 her son, John conveyed the manor to King Edward III (1327 to 1377) and he granted it to Henry Downham for life the following year.

 
Wenlock coat of arms

In 1399 Fennels Grove was valued at sixty shillings per annum. In 1415 King Henry V (1413 to 1422) granted it to his brother John of Lancaster, Duke of Bedford and on his death in 1435 it reverted to King Henry VI (1422 to 1461 and 1470 to 1471) as his uncle's heir. In 1462 King Edward IV (1461 to 1470 and 1471 to 1483)  leased the Manor of Fennels Grove, so-named for the first time in surviving documents, to Lord John Wenlock, who was killed fighting for Henry VI at the Battle of Tewkesbury in 1471. His estates were forfeited to the Crown and were given to Thomas Rotherham, then Bishop of Lincoln and later Archbishop of York and at this point Fennels Grove became part of the Manor of Luton and after 1612 there are no more separate mentions of it.