According to legend the original castle was built by Uther Pendragon, father of King Arthur, in the 5th century and may have been the site of his death when a well was poisoned by the Saxons. However, there is no evidence to support the legend that there was a castle here before the 12th century.

Pendragon Castle has had several notable owners including Sir Hugh de Morville - one of the knights who murdered Sir Thomas Beckett in 1170. He was Lord of Westmorland and Hugh's Seat - a peak on the Mallerstang ridge - is named after him. Lady Anne Clifford was a very well known landowner who rebuilt the Castle after it had been destroyed by a Scott's raiding party. She added a brewhouse, bakery, stables and coach house and was a frequent visitor until her death in 1676.