Working in old files brings out some obscure problems. Items long since established, corrected, in templates/projects leap out of the past to confound you again. This time it was an ancient old site reference file that refused to insert correctly scaled.
That can happen if sites are drawn 1 Unit = 1 Metre, vs. buildings at 1 unit/mm, but that wasn’t the case. The factor was about 25.4 x larger and Imperial Units can trigger this. However, they were correctly set to metric.
It turned out to be the InsUnits setting which was set to 1 (imperial), not 4 (metric). This is saved in the drawing so must be checked/set in the file you are linking, not the destination file.