The window cleaner got a taste of wealth and had a steamy affair with Ruby Sims