The Laws of Physics are time-symmetric. The BB event is (was) not time-symmetric. What does that mean, exactly? According to Einstein, time is relative. If distant photons had to pass through regions of space with strong gravitational fields it would affect their paths and the apparent time of travel. Since gravity acts on spacetime it seems to be the culprit, the reason the expansion is not time-symmetric (??). Simply put, gravity is changing (warping) the spacetime and changing the paths of photons and other particles. Luckily, gravity has little effect on spacetime overall (the cosmos is pretty flat, for as far out as we can see). But conservation depends on being able to predict what the likes of gravity will do to some region of spacetime, indeed it requires that all sources (and sinks) of energy are accounted for, and gravity at cosmological scales just doesn't allow this. Or Something.