The palace was originally founded at the end of the XVIIth century by Duke Janusz Ostrogski.
In the 1680s, the deputy chancellor of the Crown Treasury, Jan Gniński commissioned the leading Dutch architect Tylman van Gameren to produce a design of a magnificent palatial residence. The architect created a concept of a residence, that had never been realized.
The present Palace, built on the Place of the historical fortified edifice, is located in the part of the residence where the kitchen buildings were originally planned.
Throughout the centuries the palace had served many different functions and was rebuilt several times in 18th and 19th centuries.
In 1859 the renovated building became the home of the Institute of Music, in 1919 renamed the Conservatory. This high-level music school was the successor to the Warsaw Conservatory from the times of Chopin's youth, then known officially as the Music Section of the Fine Arts Department of the Royal University of Warsaw. At the beginning of the XXth century, a concert pavilion was built onto the south side of the castle, but it was not restored after the Second World War.
After being almost totally razed during the Warsaw Rising in 1944, the Palace was rebuilt in 1949-54 on the bases of a project by Mieczysław Kuzma who used the drawings by Zygmunt Vogel, sketches by Tylmana van Gameren and paintings by Canaletto as a model for his design. The aim of the rebuilding was to restore the shape of the Palace from the end of XVIIth century.