6π
In case someone else comes along this I figured out a solution that is pretty unintuitive.
The first problem is the way that labels are dealt with in chart.js and because they are treated as categories not x data the way that I thought they were. Therfore first you must pass your data in as coordinates in this format:
(as shown in this documentation: https://www.chartjs.org/docs/latest/charts/line.html)
data: [{
x: 10,
y: 20
}, {
x: 15,
y: 10
}]
And delete the labels variable.
However this will not work with line charts despite what the documentation says. To get around this you can set: type: 'scatter'
and inside the dataset write showLine: true
This will generate a line plot where the x labels are auto generated and zooming will work perfectly.
I also think there was a performance boost which is a nice bonus.