Napier coat of arms

Bailiff's Manor was one of twelve minor manors of Luton which were probably never organised on a true manorial basis - as a manor court with jurors, bye-laws and copyhold properties owned by tenants of the manor. A number of these seem to have been sub-manors, or areas of larger manors from which the sub-manor was held. It first appears in the 16th century when it was held from the greater Manor of Luton Hoo, with the exception of the manor house's gate house, an orchard and an acre of land which were held of Brache Manor.

The first known holder of the manor is Henry Bradshaw, who had it in 1542, when they transferred it to Thomas Field, who died in 1557, devising it to his son James. James in turn devised the manor to his son George and between this date and 1638 the manor passed to Sir Robert Napier, who effectively held more or less the whole of the medieval Luton Manor and Bailiff's Manor seems to have been subsumed within the larger Luton Manor, although there is a solitary mention of it in a Recovery Roll of 1815.