Many websites are slow because they load large themes, plugin assets, unused scripts, and heavy page builders for every visitor.
A small custom site can stay fast by keeping the HTML simple, loading one focused stylesheet, and avoiding JavaScript unless the page actually needs it.
Performance is easier to protect when every file in the project has a clear job. That also makes the site easier to debug and maintain.