The Bedfordshire Historic Environment Record [HER] contains information on the county's historic buildings and landscapes and summaries of each entry can now be found online as part of the Heritage Gateway website. The entry for Ridgmont Castle [HER 33] reads: "Site in Ridgmont village, known as Castle Hill since the 16th century. The sides of the hill fall away steeply to the north-west and south-west, but these slopes have been terraced for housing. A disused sand pit lies in its east side, and there is an area of ridge and furrow to the west of the hill. No associated finds have been made and there are no clear documentary references to the siting of a castle here. The identification is therefore uncertain".This site is at the north-west end of Ridgmont village on the spur of sandstone encircled by Lydds Hill and Station Road.

It is known that the de Wahull family had two castles, one in Thurleigh and one in Ridgmont. The likeliest site for the castle at Ridgmont is in what is now the civil parish of Brogborough. The earliest documentary evidence of a castle in Ridgmont is in 1276 when land was allotted to Walter Beywin [Bevan] "above the castle of Rugemont" [as quoted in Volume III of The Victoria County History for Bedfordshire published in 1912].