Volume III of The Victoria County History for Bedfordshire, published in 1912, gives the histories of all the manors in the parish of Northill as far as they were known at the time. The work associates Caminos alias Lower Caldecote Manor with the 1½ hides held from Eudo the Steward, alias Eudo, son of Hubert, by a man named Pirot in the Domesday Book of 1086.

The overlordship of the manor lay with the Barony of Eaton [Socon] and is last mentioned in 1428. Part of Pirot's former holding was held by a man named Thomas de Kancia in the mid 1280s. James de Kancia held the manor in 1316 and was succeeded by John de Carmino whose widow Joan was Lady of the Manor in the late 1340s. In 1361 Ralph de Carmino alienated the manor to Ralph Restwold who alienated it to Richard de Craunfeld a short time later.

By 1428 the manor was in the hands of Henry Godfrey, Lord of the Manor of Northill whose granddaughter Elizabeth and her husband Randall Borough sold it to Thomas Chibnall and he to John Poley in 1556. John Poley's granddaughter Alice married Robert Reade and they alienated the manor to George Mordaunt and Cecilia, his wife, in 1588.

 
The Mordaunt family coat of arms

The Mordaunt family owned the manor for nearly a century, until it was conveyed to William Atkins and others in 1686. The Victoria County History states: "The later history of Lower Caldecote manor is obscure but Lysons [in the Bedfordshire volume of his Magna Britannia of 1813] states … that it had been more than a century in the family of John Harvey, its proprietor at that date". The Harveys were also Lords of the Manor of Ickwell Bury. The manor remained in the Harvey family until a succession of Law of Property Acts in the 1920s abolished manorial fines and incidents as well as copyhold land tenure, thus abolishing manors in practically all but name.