Many times I've been told that the things I've written aren't fanfic, might as well be OF, etc because the boys aren't Backstreet Boys in them and sometimes there's only an appearance by one character. In my mind though, when Nick gets cancer, or Brian and AJ fall in love, or Howie gets addicted to meth, whether they're Backstreet Boys at the time or not I don't think it's canon, that's a universe alternate to the one we live in, no different than if he weren't a performer and were a janitor instead. And if we all only wrote specifically to canon, to the universe that we live in right now, we'd all be writing the same tour/recording story.
I think it's about keeping the characterizati on the same more than keeping professions, etc the same.
I'd say the people who claim you're not writing fanfic don't really get what an AU is. That's the whole point of AU - that you take these characters (or, in this case, real people) and take them out of canon and put them in a different situation. It's still fanfic if you're writing about those specific characters/people, though, not your own original characters. Like it or not, it is a valid subcategory of fanfic.
That said, I wouldn't call the other situations you mentioned - Nick having cancer, Brian and AJ in love, Howie addicted to meth (LOL) - AU, assuming they're still the Backstreet Boys in those situations. Isn't that what fanfic is, taking real people (or other people's characters) and putting them in fictional situations? To me, canon is just a set of guidelines that should be followed if you're trying to write the Backstreet Boys as the authentic Backstreet Boys. Personality traits, backgrounds, careers, families, all that kind of stuff. Some fanfic writers stick closer to canon than others, and unless it's supposed to be AU, I prefer reading a story that is canon, but I don't think that means you can only write about situations that have really happened and people who really exist without it being considered AU.
Like you said, though, I think characterizati
on is the key. Whether it's canon as in they're the Backstreet Boys or completely AU, if you're calling it a fanfic, the guys should still seem like themselves. I'd rather read a story where Howie was a janitor who was still recognizable as Howie than a story where Howie the Backstreet Boy is a meth addict for no particular reason other than the author thought that sounded like a cool idea. If you're gonna take them that far out of character, you gotta show the progression to that point - otherwise, it's more like you're just using a real person's name for a fictional character. In that example, I'd consider the janitor story more of a true fanfic... or at least a better-written one.