Volume III of The Victoria County History for Bedfordshire, published in 1912, gives the history of Colesden Manor as far as it was known at the time. The manor is first recorded in the 15th century and probably had its origins in William Speke or Espec's land held in 1086 as recorded by Domesday Book. The overlordship was later held by the de Trailly family and later still by the Honour of Gloucester, the last mention of it being in 1480.

In 1410 Joan, widow of Richard Boltere, was tenant of the manor, holding it from the Honour of Gloucester, and conveyed it to Thomas Martin and Thomas Englysshe. Englysshe's daughter Margaret brought it to her marriage with John Radewell. Joan died in 1444, as did her son Thomas Radewell and the manor then passed to Henry Englysshe, his uncle or cousin.

 
The Fitz Jeffrey family coat of arms

In 1472 Colesden Manor was granted by John Dundin to John FitzJeffrey whose family held the manor until 1546 when Francis FitzJeffrey sold it to Lord Mordaunt whose family held Roxton Manor along with Colesden. Both manors were transferred in 1624 to French apothecary Gideon de Lanie, de Laune or Delawne. He was a surgeon to Anne of Denmark, queen of James I (1603-1624) and Master of the Apothecaries' Society in 1637. He also held Netherbury Manor in Great Barford and died in 1659 when he was succeeded by his son William. The last mention of the Delawne family as holding the manor is in 1715.

The Mordaunt family coat of arms

By 1737 Colesden Manor, along with Roxton, was in the hands of William Metcalfe and his family held the manor for over a hundred years. By 1854 Charles James Metcalfe had conveyed the manor to Rev. Robert Delap of Monellan [Ireland] whose son James Bogle Delap held it into the 20th century. A succession of Law of Property Acts in the 1920s abolished manorial fines and incidents as well as copyhold land tenure, thus abolishing manors in practically all but name.