Having a website that fits all devices these days is a challenge. While fitting the device might not be so much the problem, the loading time on different devices is a problem.
No matter where, your customers and readers have expectations. If a site on a smartphone takes too long to load, the potential customer might not come back.
Have a look at this infographic and see the problems and the consequences from slow responsive web design sites: