According to NBC News, the Obamas, Clintons and Bushes won't be at the traditional lunch that follows Trump's swearing-in.