It's not a case of being dumb. There are various reasons some people find it hard work. One is Nightow's art style in the early volumes being a bit odd, and that can make his fight scenes confusing until you develop a 'feel' for it. I know even some people who have read a fair bit of shounen action manga have found his fights a bit tricky to follow at first.

Then there's the very non-linear story-telling. A conversation or event that seems weird and random will suddenly make sense 3 volumes later when you find out more, so if you stop reading, it'll just leave the impression of confusion. Not much is ever really spelled out; it's more a case of playing join the dots as snippets of information are gradually dropped in. I'll admit to being an absolute junkie for that kind of story-telling - linear manga can be fun once, but it never prompts me to squee with enlightenment and go back and reread the earlier stuff all over again *g*. So people who just don't happen to like that kind of story-telling device won't like the manga much, no.