Swanson, Fitzgibbon returned as MPs but disappointed with election's end result