Despite encroaching development and dwindling attendance, the White House of the Confederacy will stay at its site in downtown Richmond, and the collections of the adjoining museum will be moved, officials said. The executive mansion of President Jefferson Davis and his family in the Civil War has fallen on tough times, as annual visits have fallen, to 50,000 from 90,000 in the early 1990’s. The White House and museum operate with an estimated $500,000 annual deficit.