A castle once stood at Chalgrave, just south-east of the church. It was of the motte and bailey type, that is, a mound (the motte) on which a tower would have stood, and an area around the motte (the bailey) which would have been within protective walls. Such castles were usually almost entirely wooden structures.
In September 1970 a rescue excavation was carried out on the site by the Department of Environment and was written up in Bedfordshire Archaeological Journal Volume 6. The site was levelled for farming purposes shortly after the dig. The area now forms part of Chalgrave Manor golf course.
The excavation, restricted solely to the area of the motte, showed that it was raised in the mid 12th century, possibly by the Loring family, tenants of the Manor of Chalgrave under the overlordship of the Barony of Bedford. The motte seems to have covered an 11th century manor house, the seat of Albert of Lorraine who was noted as owning the manor in the Domesday Book of 1086.
The motte was constructed by ditching round the site of the earlier manor buildings, the spoil thus dug out being thrown inwards to create the mound, a low platform perhaps sixty feet across. The mound was only about three feet high and seems to have been built to give a firm foundation to a square structure about thirty feet on each side, probably a tower.
This square tower seems to have been dismantled late in the 12th century and the motte extended to form an oval, rather than circular, area. A new building, of one storey to judge by the depth of the foundations was then placed upon it. At the same time the ditch around the motte was re-cut. The castle seems to have been abandoned in the early 13th century, probably in favour of a manor house on or near the site of today's Manor Farm or what became the Chantry House, thus having a life of less than a hundred years.