Farley Manor originated in a grant of land by King Henry II (1154 to 1189) to the Hospital of Holy Trinity, Satingfeld in Picardie [France] in 1156 - it was described as "the land of Farley near Luton, as far as the land of Luton church and as far as land of Whipperley near Presteley Street [ad viam de Presteleya]". A further 45 acres was added to the grant in 1204 by Baldwin de Béthune, who held Luton Manor.

 
King's College Cambridge coat of arms

The hospital in Picardie used the land to establish a similar institution in Luton and in 1291 the Master of Farley Hospital had land, rents, mills and woods worth £3/12/- per annum. In 1331 the manor was taken over by the Crown when the master failed to appear to claim it. In 1447 religious houses in England controlled by foreign interests were dissolved and Farley was granted to King's College, Cambridge. By 1552, however, Farley was once more a manor in the hands of the Crown, though Saint Albans Abbey had tried to claim it in 1505.

 
Rotherham coat of arms

George Rotherham, who was tenant of Dallow Manor, also leased Farley Manor on a 92 year lease from 1522. Rotherham's son, also George was granted the manor by the Crown in 1554. His son, another George, succeeded him in 1594. A fourth George succeeded him in 1632! In 1698 and 1707 a Thomas Rotherham held the manor but by 1783 it had passed to a John Sharpe Palmer to transferred it to the Marquess of Bute, who held it in 1815. By 1855 the manor was held by the Crawley family who continued to own it into the 20th century. However, a succession of Law of Property Acts in the 1920s effectively abolished manors in all but name. including copyhold land and manorial courts and income.