Details of The Fitzroys at the Lodge 

There are many Charles Fizroys (which has confused me significantly) associated with Sholebroke Lodge.  Tracing the one that built the house proved an interesting challenge.  I originally thought it was one of the cousins mentioned below but on further research I believe that it is Charles (1762–1831) that rebuilt the house.

Charles FitzRoy, 2nd Duke of Cleveland (1662–1730), 18th century nobleman – Too Early
Charles FitzRoy, 2nd Duke of Grafton (1683–1757), nobleman who was Lord Lieutenant of Ireland – Too Early
Lord Charles FitzRoy (1718–1739), fourth son of Charles FitzRoy, 2nd Duke of Grafton – Too Early
Charles FitzRoy, 1st Baron Southampton (1737–1797), British statesman and soldier, MP for Bury St. Edmunds – Too Early
Charles FitzRoy (British Army officer) (1762–1831), son of the above, general –  This Charles is connected to Princess Amelia a possibility – this is most likely built the house.
Lord Charles FitzRoy (British Army officer) (1764–1829), second son of Augustus FitzRoy, 3rd Duke of Grafton, general and MP for Bury St. Edmunds  – Son of first Baron Southampton
Lord Charles FitzRoy (politician) (1791–1865), second son of George FitzRoy, 4th Duke of Grafton, British Army officer and MP for Bury St. Edmunds – This was a possibility but reading below it doesnt look possible although he lived there.
Sir Charles Augustus FitzRoy (1796–1858), governor of Prince Edward Island and New South Wales – cousin of most likely
Lord Charles Edward FitzRoy (1857–1911), third son of Augustus FitzRoy, 7th Duke of Grafton, reverend – too late and reverend.
Charles FitzRoy, 10th Duke of Grafton (1892–1970), soldier and farmer -too late
Lord Charles Oliver Edward FitzRoy (1923–1944), second son of Charles FitzRoy, 10th Duke of Grafton, soldier  – too late
Lord Charles Patrick Hugh FitzRoy (born 1957), second son of Hugh FitzRoy, 11th Duke of Grafton, author – too late
Charles FitzRoy, 3rd Baron Southampton (1804–1872), British peer – too late

I believe that the Charles FitzRoy (British Army officer) (1762–1831), son of first barron southampton and  is connected to Princess Amelia  this is most likely Charles Fizroy to build the house.

However there were two other Charles Fizroys that lived at the house see below:

FITZROY, Lord Charles (1791-1865) 

Image result for FITZROY, Lord Charles

Family and Education
b. 28 Feb. 1791, 2nd. s. of George Henry Fitzroy†, 4th duke of Grafton (d. 1844), and Lady Charlotte Maria Waldegrave, da. and coh. of James, 2nd Earl Waldegrave; bro. of Henry Fitzroy, earl of Euston* and Lord James Henry Fitzroy*. educ. Harrow 1802-5; Great Marlow. m. 25 Oct. 1825, Anne, da. of Lord George Augustus Henry Cavendish*, 2s. 2da. d. 17 June 1865.
Offices Held
Ensign 1 Ft. Gds. 1807, lt. 1812, capt. 1812, maj. 1815, lt.-col. 1819; maj. 55 Ft. 1820, half-pay 1821; asst. adj.-gen. at Armagh 1830-2; sold out 1834.

Vice-chamberlain July 1835-Apr. 1838; PC 1 July 1835.

Fitzroy, whose military and political careers and private life were blighted by his compulsive gambling habit, had served with distinction on the Walcheren expedition, in the Peninsula and at Waterloo. He was with the army of occupation in France when he and his elder brother Lord Euston were brought into Parliament in 1818 as the duke of Grafton’s family Members for Thetford and Bury St. Edmunds.1 A silent Whig, he had opposed the coercive legislation introduced after Peterloo, and he was again returned for Thetford at the general election of 1820, when his uncle Lord John Edward Fitzroy replaced Euston at Bury St. Edmunds.2 The failure of his horse Swase to win the 1822 St. Leger completed his financial ruin, and he spent the next two and a half years abroad evading his creditors and his father’s wrath.3 Otherwise, he voted with the main Whig opposition to the Liverpool ministry in most major divisions in the 1820 Parliament and consistently with Hume and the ‘Mountain’ for economy, retrenchment and reduced taxation.4 He backed the 1820-1 parliamentary and extra-parliamentary campaigns on behalf of Queen Caroline.5 He divided for Catholic relief, 28 Feb. 1821, 1 Mar., 21 Apr., 10 May 1825, parliamentary reform, 9 May 1821, 13 Apr. 1826, and receipt of the radical Greenhoe reform petition, 3 June 1822. From Florence and Venice in 1823 he expressed concern that his father had gained little by recently promoting the adoption of distress petitions advocating reform in Suffolk, and he welcomed Huskisson’s appointment as president of the board of trade and the government’s defeat on a motion for inquiry into the prosecution of the Dublin Orange rioters, 22 Apr.6 In July 1824 Grafton relented and appointed lawyers to deal with his debts; and when he married in October 1825 Euston made Sholebrooke Lodge in Northamptonshire and the hunting in Salcey Forest available to him until his own heir came of age.7 The Petre estate and interest in Thetford had been sold in 1822 to the financier Alexander Baring*, whose son William Bingham Baring became Fitzroy’s colleague at the general election of 1826.8

He voted for Catholic relief, 6 Mar. 1827, 12 May 1828, to disfranchise Penryn, 28 May 1827, and to repeal the Test Acts, 26 Feb. 1828, and presented Thetford’s petition for repeal of the Malt Act, 29 Feb. 1828. He repeatedly complained that their agreement of 5 June 1827 whereby Grafton underwrote his debts provided he assigned his entire income to him and his father-in-law, the Whig Member for Derbyshire, made parliamentary life unaffordable;9 and his next known vote was on 1 July 1830, when he divided to delay on-consumption under the sale of beer bill. In an attempt to distance him from his gaming associates, Grafton had secured him an Irish posting under Sir Richard Hussey Vivian*, and he stood down at Thetford at the dissolution that month in favour of his younger brother.10

Fitzroy welcomed the appointment of Lord Grey’s ministry and their reform bill. He wrote to the postmaster-general, the duke of Richmond, 5 May 1831, ‘It is as well that the peers should have a certain majority for the bill dinned into their ears, that they may get used to the sound before reality comes before them’.11 His father-in-law joined them in September as a coronation peer (earl of Burlington). By March 1832 Fitzroy had resolved to resign his command to return to England, ostensibly to assist his ailing in-laws at Burlington House.12 Grafton reluctantly acquiesced in the arrangement and financed his election as Liberal Member for Bury St. Edmunds, where he topped the poll in December 1832. He retained the seat until 1847, the first dissolution after his father’s death.13 His appointment as vice-chamberlain in 1835 caused a stir, as did his dismissal in 1838 for voting against the Melbourne government on a slavery issue.14 He failed to avoid bankruptcy and died at his wife’s house, Elm Lodge, Hampton, Middlesex, in June 1865.15


His cousin also appears to have lived at the Lodge too.

Image result for FITZROY, Charles Augustus

FITZROY, Charles Augustus (1796-1858), of Sholebrooke Lodge, Northants. and Stratton Street, Piccadilly, Midx

Family and Education
b. 10 June 1796, 1st s. of Lord Charles Fitzroy† (d. 1829) and 1st w. Frances, da. of Edward Miller Mundy*. educ. Harrow, 1805-10. m. (1) 11 Mar. 1820, Lady Mary Lennox (d. 7 Dec. 1847), da. of Charles Lennox†, 4th duke of Richmond, 3s. 1da.; (2) 11 Dec. 1855, Margaret Gordon (née Milligan), wid. of J.J. Hawkey, land agent, of Sydney, NSW, s.p. kntd. 1 June 1837; KCB 12 June 1854. d. 16 Feb. 1858.
Offices Held
Lt. Horse Gds. 1812, capt. 1820; maj. (half-pay) 1825; brevet lt.-col. 1825.

Mil. sec. Cape of Good Hope 1822, dep. adj. gen. 1825-31; lt.-gov. Prince Edward Island 1837; gov. Leeward Islands 1841-5, NSW 1846; gov.-gen. Australia 1850-5.

Fitzroy’s father, a brother of the 4th duke of Grafton, on whose interest he had represented Bury St. Edmunds, was a general in the army and commander of the Ipswich garrison. His mother had died when he was a year old and his stepmother (d. 1810), the mother of his half-brothers and sister, was a sister of the Liverpool ministry’s foreign secretary Lord Castlereagh*.1 Fitzroy joined the Horse Guards and served as an aide-de-camp to Sir Richard Hussey Vivian* at Waterloo. He remained with the army of occupation in France and it was there that he met his first wife, who, following her father’s traumatic death after being mauled by a bear near Ottawa in August 1819, had joined her brother, the 5th duke of Richmond, on his mission to Paris.2 In 1822 the Fitzroys and their infant son Augustus Charles left for the Cape, where, as military secretary to the governor Lord Charles Henry Somerset†, Fitzroy acted also as the wine taster, joint commissary of revenues and editor of the official Cape Town Gazette. His editorship terminated in 1824 amid allegations that he had deliberately exaggerated reports of rioting in Grahamstown in the 21 Feb. issue, two days after his controversial dismissal of the chaplain to the forces there, the Rev. William Geary, who duly instigated libel proceedings.3 Fitzroy remained at the Cape until 1831, when, allegedly impoverished by maintaining a substantial household there, he returned to England to review his situation and claim part of his late father’s estate. His wife was awarded an annual pension of £185 14s. in December 1830, after Richmond became the Grey ministry’s postmaster-general.4

Fitzroy occupied Sholebrooke Lodge, Northamptonshire, vacant through the posting to Armagh of his cousin Lord Charles Fitzroy*, and at the general election of 1831 he contested Bury St. Edmunds, where his cousin Lord Euston had retired after voting against the ministerial reform bill. Unlike his half-brother Robert, who failed to come in for Ipswich, he declared firmly for the bill, despite its threat to Bury St. Edmunds’s second seat, but made it clear that he would ‘not pledge to vote for every clause in the committee’.5 Assisted by his uncle Lord John Edward Fitzroy* and the borough recorder and putative candidate Robert Monsey Rolfe, he finished second in a four-cornered contest.6 Lord Charles Fitzroy wrote, ‘I am glad Charles Augustus is in Parliament and hope he may get something permanently useful to him through it’.7 Fitzroy, a ‘large burly man in robust health’ who would ‘never sit down to dinner without a fresh buttonhole’, made no known speeches in the House.8 He voted for the reintroduced reform bill at its second reading, 6 July 1831, consistently but sparingly for its details, and for its passage, 21 Sept. He voted for Lord Ebrington’s confidence motion, 10 Oct. He divided for the revised reform bill (which restored the second seat to Bury St. Edmunds) at its second and third readings, 17 Dec. 1831, 22 Mar. 1832, and with government on the Russian-Dutch loan, 12, 20 July 1832. He presented his constituents’ petition for the factories regulation bill, 26 Mar. 1832.

It had regularly been reported locally that Fitzroy was ‘too ill’ to attend reform meetings and dinners in Bury St. Edmunds, and he made way for Lord Charles Fitzroy at the 1832 dissolution.9 He retained the use of Sholebrooke Lodge pending negotiations concerning his future career, in which Grafton and Richmond assisted. His hopes of returning to the Cape were dashed, and the colonial secretary Lord Glenelg rejected him for St. Helena, Van Diemen’s Land and Prince Edward Island before he was appointed governor of the last in 1837. He turned down a posting to Jamaica in 1836 on health grounds.10 His subsequent career benefited through his ability to curry favour with the 3rd Earl Grey and Lord Aberdeen and from his social skills.11 He handled the dissolution of the legislative assembly in Prince Edward Island in 1838 with sensitivity, and his residence in Antigua was characterized by the speed with which he summoned and organized assistance following the earthquake of 1843.12 His first wife’s death in 1847 in a carriage accident in the driveway of Government House, Sydney, where he was governor of New South Wales, aroused much sympathy.13 According to the South Australian Register, as the first Australian governor-general, 1851-5, the ‘power he possessed, Sir Charles … wisely allowed to slumber’, and he successfully implemented the contentious 1850 Australian Colonies Government Act which established federal government. He saw censuses introduced, hostility to the return of convict transportation contained, a mint established in Sydney to regularize the ‘gold rush’ and the ‘battle of the railway gauges’ between Victoria and New South Wales resolved. Queen Victoria invested him as a knight commander of the Bath during her 1854 visit to Australia.14 He died in London in February 1858, two years after returning with his second wife. Remarriage had put an end to his financial worries and speculation over his improper liaisons with women, but it posed problems in the execution of his will (proved on personalty of under £20, 30 Aug. 1858) as indentures to the settlement made on his first marriage, which favoured his three sons, had not been sworn. His property was subsequently sold and they each received about £1,000.15