I would say anytime you're making major changes to the canon, or the way things are in real life, besides the fictional plot of the story, it could qualify as an AU. In any fanfic, there are going to be things that aren't really true or haven't really happened to them - Nick didn't die and come back as a ghost, neither did he get kidnapped by a psycho fan, neither did Brian carve MINE into his flesh, neither did they all get into a plane crash and pick bot flies out of Jordan Knight's brain - but that's what makes it fiction. In all of those stories, they were still The Backstreet Boys, and the rest of their universe was pretty much as it is in real life. Even when you're writing about real events, like Steph with Running Up That Hill or Hannah with Borrowed Time, you're still taking creative liberties on some things, fictionalizing conversations and scenes that probably didn't really happen because there's no way to know how it all went down exactly. But the guys are still clearly themselves, the Backstreet Boys, in those stories too. I don't consider any of those AU.
Obviously, if they're not famous singers, that's AU, but I agree with Mare that even if you keep that aspect, but make huge changes in other aspects of their lives (like making them all brothers), that could be considered AU. With Tracy's example of replacing their real families with fictional ones, I think that could be considered AU too, but it's definitely walking a fine line. In Steph's story Rewind, Nick had a fictional wife and kids (but he was also older, so that was Future Nick's family), but his past fit the canon, so I never thought of that one as an AU.