The Historic Environment Record for Bedfordshire notes that in 1281 Baldwin Wake was given a licence "to crenellate a chamber in his marsh of Stiventon" (he died the following year) but no trace of a castle has been found. It is likely to be a reference to the same site as the later manor house. A lime kiln has been found nearby [HER 14358] and there are associated earthworks [HER 3556 and 5078].

The Manor House, of unknown date, was demolished in 1872 and a new farmhouse (Manor Farm) built in 1875, with farm buildings being constructed on the site. There is anecdotal evidence for the discovery of wall remnants and floors during the building work at Manor Farm in 1875 but excavations on the site in 1936 failed to find any remains of the castle. Nor could significant features could be found during field investigation in 1974. The structure may have been no more than a fortified manor house and the remains now under Manor Farm.