Volume III of The Victoria County History for Bedfordshire, published in 1912, gives the histories of all the five manors in Great Barford as far as they were known at the time. Of these five manors only one, Veseys, or Vaseys, can be traced back to the Domesday Book of 1086.

Coat of arms of the Barony of Bedford

Brytvilles Manor was part of the overlordship of the Barony of Bedford until it was divided between the three sisters of the final Baron, John, who died fighting for Simon de Montfort against King Henry III at Evesham in 1265. The first mention of the Bryteville family in Bedfordshire is in 1166 when Hugh de Bryteville held a knight's fee in Bedfordshire (in other words, sufficient land to require him to provide a fully armed knight for royal service when called upon). This may well mean that he held the manor at this date. His son William held the land in 1197. In the latter part of the 13th century another William de Bryteville held land worth half a knight's fee and one thirteenth of a knight's fee. This indicates that either the manor had shrunk in size or had reduced

He seems to have been succeeded by a John de Bryteville, who held the manor in 1275. This may be the same John de Bryteville who lost his son Henry by drowning in 1265 and was viciously attacked in the street in 1271, both events recorded in the coroners' rolls. By 1286 the manor had passed to Hugh de Bryteville and by 1316 was held by his son William. By 1342 another Hugh was in possession. The last known de Bryteville was William in 1439.

The Spencer family coat of arms

The next known holder of the manor is John Spencer of Cople, whose son Robert died in possession of it in 1521. The manor remained in the family until at least 1692. Probably shortly thereafter William Spencer alienated the manor as by 1729 it was in the hands of John Browning. By 1789 the family was held by the Polhill family of Howbury Hall in Renhold and they held it into the 20th century. A succession of Law of Property Acts in the 1920s abolished manorial fines and incidents as well as copyhold land tenure, thus abolishing manors in practically all but name.

The site of the manor house probably lies at the moated area at Birchfield Farm. The Bedfordshire Historic Environment Record [HER] lists every historic building and landscape feature in the county and is now available on-line as part of the Heritage Gateway website. The entry for the moat at Birchfield farm [HER 817] reads as follows: "The site comprises the remains of a medieval moated enclosure and its associated ponds and leats. The moated enclosure is rectangular in shape with maximum dimensions of 100 metres by 85 metres inclusive of the 14 metre wide surrounding moat. A small stream connects with the north-west angle of the moat via a short inflow channel. Upstanding remains within the moated enclosure include a low internal bank around the edge of the island, a water filled pond in the south-west corner and a rectangular platform on the south side of the island. The platform is interpreted as the site of the original 12th century manorial building. Immediately to the east of the moat are the remains of two ponds and their associated supply and outflow channels. The ponds are believed to have been constructed as fishponds. A series of slight cultivation earthworks, not included in the scheduling, can be seen in the field to the west".