A portion of the Horatio Gates Warner estate, the rolling land located to the west of South Avenue and adjacent to Highland, was donated to the Park in 1907 in lieu of back taxes. The residence and adjoining gardens were retained by the private ownership until 1951, when the city of Rochester purchased the property.

Frederick Law Olmsted, Sr. had earlier noted that this wooded property afforded beautiful vistas and would make a fine park; however, the price of the property had been prohibitive when the initial park system sites were under consideration. A residence on the property tract was built by Horatio Gates Warner in 1854. The structure resembled the ancestral castle of the Clan Douglas which captivated his fancy during a visit to Scotland. Warner was a prominent Rochester attorney, business man, newspaper publisher and editor.

The Castle went through various owners after the Warner family, and briefly served as a sanitarium. Alling DeForest was hired in the 1920's to refurbish the grounds, and it was in 1930 that the Sunken Gardens were added in the rear of the property.

The Warner Castle is currently undergoing renovation. The gardens remain open to the public.