If you are still interested in obtaining a job in academia, backing out at this moment could be damaging professionally. The members of that department will not think positively of you. Worse, word gets out, and backing out could potentially adversely affect your chances of receiving offers from other places, either this year or next year.
Before going back on your acceptance, talk to your advisor. These kinds of things, rightly or wrongly, also tend to reflect back on the student's advisor and program—so it would only be fair to get their input and/or support for your decision. Your backing out could have consequences on students from your program getting a fair shot at jobs at that institution for some years to come. In fact, it's possible that your action would anger your *advisor* more than the offering school itself.
But these things do happen. I know of one search some years ago where the candidate not only verbally accepted the offer but also signed the contract, only to notify the department in early summer that, due to spousal issues, he could not follow through and take the position. That was of course too late for the department to do a new search for the fall. Needless to say, they were ticked off. No legal action was taken, but there were definitely bad feelings involved. Did it have a noticeable impact on that person's career? Not really… but a lot of people weren't very happy with him for a long time. So I guess it comes down to how much it would bother you if the people in that department really disliked you for a while. ;-)
On the other hand, if you know that you would be miserable there and would immediately be looking around for another job, an argument could be made that it is better for everybody all around to acknowledge that quickly and cut your losses. And, at this point, it's not too late for the school to make a good hire, so if you decide to go that route, I'd give them the bad news right away.