There are several errors in your code. And you should use a cond when dealing with multiple conditions (think of it as a series of IF/ELSE IF/…/ELSE statements).
Be aware that the expression (and (> x 1) (= x 1)) will never be true, as x is either greater than or equal to 1, both conditions can never be true at the same time. You probably meant (or (> x 1) (= x 1)), but even so that expression can be written more concisely as (>= x 1). The same considerations apply for the condition (and (< x 2) (= x 2)).
I believe this is what you were aiming for:
(define (equation x)
(cond ((> x 2)
(+ (- (* x x) x) 4))
((and (>= x 1) (<= x 2))
(/ 1 x))
(else 0)))