Installing fence next to retaining wall | This Old House.10 Sep 2014 . I would like to install a retaining wall between my yard and my neighbors. It needs to be about 3' tall and about 50' long made from non-mortared block. I would like to place a 5-6' privacy fence as close to the wall as possible. What precautions need to be taken to ensure the integrity of the fence and wall?