I suppose it is, in a way.
Nobody would conjure up the image of a plantation-themed event with black servers if they were told they were attending an "English heritage" themed event, but all of the vestiges of the English aristocracy that are associated with this kind of "heritage" were facilitated by the exploits of colonialism. I suppose most people would imagine a posh afternoon tea - made possible by tea imported from China, sugar grown and crystallised in the Caribbean, bread baked by servants with no upward mobility, in outfits of tweed made of the wool of sheep who were prioritised above the residents of Scotland and whose farming (forced upon Scots) decimated communities and their cultures, on estates bought with the riches gained by trafficking people and handed down through the generations. All possible only through the oppression, exploitation, and abuses of the British Empire.
I don't mean to completely reject your point, but I think it's important to acknowledge that the legacy of the British Empire lives on, and continues to affect people all over the world. And people like Lydia are desperate to pretend that they are in on it, on the exploiting side, with unfathomable riches that could only be amassed by treading down on others (but in a nice, out-of-sight way so you never have to think about those people)