Volume III of The Victoria County History for Bedfordshire, published in 1912, gives the histories of all the manors in the parish of Northill as far as they were known at the time. The first reference traced to the Manor of Northill Rectory alias Northill College is during the reign of King Edward III (1327-1377) when Roger de Lameleye, Rector of Northill between 1330 and 1346 claimed a view of frankpledge (in other words manorial jurisdiction) over all the glebe lands belonging to Northill church stating that his predecessors had had these liberties since time immemorial.

The Trailly family coat of arms

Northill became a collegiate foundation in 1404 - a collegiate church was one in which daily services were held by a body of clerics which governed themselves. These clerics were not monks but ordinary clergymen. The executors of Sir John and Sir Reginald Trailly, former Lords of the Manor of Northill appropriated the Manor of Northill Rectory so that its income could support the new college, the Rector of Northill being created Master of the College. In 1519 Edward Stacey increased the size of the manor by adding a grant of land to it. The college was dissolved in 1547, along with all other religious foundations in England other than chapels, churches and cathedrals and Northill College Manor passed to the Crown.

In 1549 the manor was granted to Sir William Fitzwilliam. He died in 1559 and after the death of his wife the manor as divided between his four daughters: Mabel, wife of Thomas Browne; Katherine, wife of Christopher, Viscount Gormanston and two Elizabeths, one the wife of Innocent Rede the other the wife of Francis Jermye. Sir Thomas Browne acquired the shares of the two Elizabeths in 1577 and 1580 respectively and Katherine died childless so that by 1597 Browne's son, Richard, was in possession of the whole manor.

In 1610 Richard Browne sold the manor to Edward Osborne. His son Edward died in 1679 leaving the manor, no longer called Northill College, to his grand-niece Elizabeth Osborne. This is the last mention of the manor to have come down to us.