The later books in the series are definitely better because that's where the story gets deeper and scarier and more adult. The first two are definitely more kid-friendly. Truthfully, I think they're all wonderful books, but typically if I'm going to read through the entire series, I tend to cheat and jump right to the third book and read them in order from there. Which is funny because the first two are so short compared to the rest, they're quick, easy reads!
Sorcerer's Stone is a great book, though, especially if you're new to the series. I first read it in high school, and although I was very cynical and skeptical and wasn't prepared to like it, I got totally hooked! What you have to remember about it, I guess, is that it's basically an introduction to the whole series. You learn the premise of Harry's story, you get to know some of the main characters and the setting of Hogwarts, and there's a little mystery plot, but the depth, the meat of the series, really doesn't come in until, in my opinion, the third book. Each book has its own storyline, of course, and they're all good and all interesting, but when you think in terms of the overall story arc of the whole series, it makes sense that Books 1 and 2 would be introduction, Books 3 and 4 rising action... Book 4 is kind of climactic in itself because it is a turning point in the series, but then Books 5 and 6 are more rising action, and Book 7 is where you get the ultimate climax and resolution of the series. So personally, I think you can't judge the series until you've read at least the first three or four books, and you definitely can't judge based on the movies because although they're good, they pale in comparison to the books. And now I'll jump down from my soapbox of defending Harry Potter LOL.
I'm gonna enjoy discussing Sorcerer's Stone, although I can already tell it's gonna be hard to stick just to talking about that one, cause there's soooo much more to discuss!