The first initiatives for the establishment of a museum in Karlovac emerged in the late 19th century, but it was not until December 18, 1904 that the Town Council rendered its opinion on the need to establish a museum and provided the initial funds of 500 crowns. Due to insufficient funds no activities were initiated, and only in 1911 was a Museum Committee founded with the task of collecting material, which was supposed to be accommodated in a temporarily assigned room on the second floor of the Town Hall.

The activities of item collecting were suspended during World War I and were not renewed until after World War II, more precisely in 1952, when Professor Ivana Vrbanić was employed as the first professional (curator). Next year, in 1953, the Museum was given one of the oldest preserved objects of Baroque residential architecture of the curiae type from the first half of the 17th century, which had been commissioned by the Karlovac General Vuk Krsto Frankopan in Zvijezda, which to this day holds the seat of the Karlovac City Museum.

The Karlovac City Museum was soon officially merged with the Painting Gallery of the City of Karlovac, founded on July 12, 1945 as the first gallery institution established in Croatia after World War II, and the first active museum and gallery institution of the City of Karlovac.