Never Gone is probably the most unique of all their albums, in a good way. I will stand by the "organic" pop/rock, always!
I agree with you about I'm Taking Off. Like you, I still adore some of the songs on it, mainly I'm Taking Off and Burning Up, but a lot of the other tracks are just "meh" to me. It's always had skip songs for me, but I used to enjoy listening to more of the album than I do now. I feel the same way about Now or Never, although, again, some of the songs I will always love.
On the flip side, All American is definitely one that I enjoy better now than I did when it first came out. It grew on me quickly (like, within a few days), but I was actually kind of disappointed with it when I first heard it because I felt like it wasn't as personal as I'm Taking Off and the songs, while more upbeat and fun, lacked substance. After listening to it more and learning the lyrics, though, I think it has a lot more substance than I initially thought, and some of the songs are very personal and Nick-ish. Now I think it's his best solo album, and there's not a single skip song on it for me.
As far as BSB albums go, I think Millennium has held up very well. That's always been one of my favorite albums, and it's still one I can listen to all the way through without skipping any track. The tracks that were my favorites when it came out are still my favorites, and I think if I were to make a list of all-time favorite BSB songs, there would be multiple songs from Millennium in the top 20.
In a World Like This has also held up for me, although it's hard to know for sure while it's still the newest album. It took the place of Never Gone as my other favorite album, which I guess means Never Gone hasn't held up quite as well for me.
I think Unbreakable is probably the album that has held up the least well for me. It still has its standout tracks that I will always love, like Unsuspecting Sunday Afternoon and Unmistakable, but some of the songs that I really loved when it came out I don't love so much anymore, and I think it has too many throw-away tracks that are not necessarily bad, just boring (You Can Let Go, Trouble Is, Love Will Keep You Up All Night).