On 4 March 1919, the Women's Patriotic League decided to use their money for a library as a memorial to the Swansea men who saw active service in the Great War. The Swansea Memorial Public Library Association formed on 30 September and the Swansea Memorial Public Library opened on 30 November 1919 in the Swansea Public School on Windermere Avenue. The library was dedicated to the 170 Swansea residents who enlisted in the First World War, and the 22 who did not return.

In 1959, the library was relocated to Swansea Town Hall at 95 Lavinia Avenue.