Speed up xref loading by demand loading only the parts you need with INDEXCTL and XLOADCTL

In your big, main xrefs, set INDEXCTL to 2.  This tells them to save layer and spatial indexes to enable referencing drawings to economize on loading.  In other drawings, set INDEXCTL to 0 zero so they don't waste time saving these indices.

In your referencing drawings, set XLOADCTL to 2 so they will use both the layer and spatial indices you saved in your xrefs to speed up their loading.  You can set XLOADCTL to 2 in all your drawings without harm.

If you aren't working in a networked team environment (in other words, if today's date is 1994 or you are a one-person show), you may want to set XLOADCTL to 1 instead of 2 so that instead of AutoCAD's making a copy of your xrefs, it can just lock them in place, which may be faster.