0👍
Thanks to @Thoomas in the comments I figured out how to solve this issue.
In the index.js file under the config directory, I had to change the build.assetsPublicPath from ‘/’ to ‘./’.
This now builds with the correct relative paths.
Source:stackexchange.com