Could any of your efficiency drop on the way home be due to an overall change in elevation? If your home is at a higher elevation than your destination, then the ride home would be less efficient than the ride out.
I see that difference affect my efficiency significantly 2 or 3 times each week on a 20 mile round trip because my home is at a higher elevation than my destination. My home is 1541 feet ASL and my destination is 1290 feet ASL (about a 251 foot difference). That difference in elevation over a 20 mile round trip means on the way there my efficiency is usually between 4.3 and 5.0 mi/kWh and on the way home it's usually between 3.0 and 3.3 mi/kWh. On average for the round-trip I get between 3.8 and 4.0 mi/kWh.
You can use this website to check the elevation at both your places.
https://www.freemaptools.com/elevation-finder.htm