SOLVED. Fairy tale like short story for adults about an enchanted castle, involving a tapestry, tower with an old woman in it whom the main character is stopped from visiting

[Rose]

I read this short story in the last couple of years but think it is much older. The central character was female but spending time with a family in their castle. There's a tower with an elderly woman inside whom the character wants to visit but the family try to put her off. I believe there was a tapestry involved, and a dance where the main character is dancing with one of the brothers of the family. It had a mysterious fairy tale atmosphere. I think there was some kind of twist at the end about the nature of the castle/family.

[Rose]

Solved it! Its the Lovely House by Shirley Jackson