As some commentators have pointed out, it appears that it was particularly Jesus' engagement with the demonic that was causing Mary and Jesus' brothers to arch their eyebrows the sharpest. It all seemed a little bizarre to them. In verse 21 they say literally that they had to get him on home because Jesus was exeste, a word meaning to stand outside of yourself. Even today we may refer to a person who is an emotional wreck as being "beside himself" with grief. The idea is that someone has taken leave of his senses (or his senses have taken leave of him) and so what remains for the time being is a person whose emotions are unchecked and unregulated. This is the family's assessment of Jesus.
Apparently all Jesus' talk about invisible kingdoms of God and the casting out of demons led members of his own family to the conclusion that Jesus was seeing things that no one else could see and the reason was simple: he was out of his ever-loving mind! And it does appear that it was especially Jesus' emphasis on the demonic that yielded this opinion because no sooner does Jesus' family accuse him of being mad, and the religious leaders chime in with their own verdict. Because they limit their comments to the demonic, it's a good bet that it was this aspect of Jesus' ministry that was generating the most discussion. In the opinion of the scribes, Jesus was himself a devil. If Jesus seemed to have inside information as to the goings-on in the demonic realm, the explanation was simple and obvious: pulling a page from the old "it takes one to know one" playbook, the religious leaders lambaste Jesus as being himself a demon incognito.
It was a ridiculous thing to say, and finally completely foolish, too. Why would the devil be shooting himself in the foot? What kind of military commander blows up his own tanks? No, if Jesus seems to be plundering the realm of the devil, it's because he had already bound and gagged the devil himself and so now his lesser hosts were easy targets for Jesus. Jesus did his work not because he had the power of the devil but because he had already demonstrated power over the devil.