After the Norman Conquest of England in 1066 the usurping King, William I, also called William the Bastard and William the Conqueror gradually set about taking away land from indigenous nobles (earls and thegns) and giving it to his own followers. Towards the end of his reign, winter of 1085, whilst at Gloucester, he ordered a complete survey of his kingdom to determine who held each piece of land. This survey became known as the Domesday Book and was carried out in just one year, 1086, a very impressive piece of logistical planning.

The book noted Luton as a Royal Manor, held by the king himself. In addition it noted that the church was held by William the King's Chamberlain with five hides which belonged to it. These five hides formed a part of the Royal Manor's total of thirty hides. The church had a mill worth ten shillings and "the church pays 20 shillings a year", it had woodland for 50 pigs and had been held by a priest named Morcar during the reign of King Edward the Confessor (1042 to 1066).

These five hides belonging to Luton church were the genesis of the Manor of Dallow. In the reign of King Stephen (1135 to 1154) William Chamberlain, presumably a descendant of the Domesday Book William, held Luton church and its lands from Robert, Earl of Gloucester, illegitimate son of King Henry I (1100 to 1135). The earl wished the overlordship to pass to Saint Albans Abbey and this was carried out by his son William, 2nd Earl, after Robert's death in 1147 in the midst of the civil war between Stephen and Matilda.

The manor is first called Dallow Manor in 1258 when Godfrey of Biscot and twelve others acknowledged that they had failed to attend the view of frankpledge, in other words a manorial court, held at Dallow each year. In 1291 the manor was valued at £7/3/1 per annum and in 1331 the abbot of Saint Albans not only claimed the overlordship of the manor but his right to hold an annual fair in Luton on the Feast of the Assumption (15th August). The abbey continued as overlord of the manor until it was dissolved by King Henry VIII (1509 to 1547) in 1539 when it became Crown property.

 
  Crawley coat of arms

In 1544 the manor was granted to Sir Thomas Barnardiston who transferred it to Thomas Crawley in 1586. In 1606 Crawley's son, Richard, divided the manor into two, confusingly both called Dallow Manor! The first of these, which included the manor house itself, was sold to Richard Scudamore and, in the same year, he transferred it to Sir John Rotherham, who conveyed it in 1615 to Sir Robert Napier, thus making it part of the greater Luton Manor. Part of this portion of Dallow Manor was sold by the Marquess of Bute in 1859  and another part in 1862 by J. Shaw Leigh to Thomas Sambrooke Crawley.

The second fraction of Dallow Manor was sold by Richard Crawley in 1613 to Robert Faldo of North Mimms [Hertfordshire] who sold it for £1,600 to Henry Denham and Ralph Merefield in 1620. Merefield later released his claim and Dereham sold it in 1623 to Richard Peters, who sold it for £1,450 to Bernard Hale of King's Walden [Hertfordshire] in 1640. The Hale family continued to own this part of the manor until 1859 when they sold it to Thomas Sambrooke Crawley, re-uniting it with the other half. The Crawleys continued to own the manor into the 20th century. However, a succession of Law of Property Acts in the 1920s effectively abolished manors in all but name. including copyhold land and manorial courts and income