We so often grieve God by our actions.   

“And do not grieve the Holy Spirit of God, by whom you were sealed for the day of redemption.  Let all bitterness, wrath, anger, clamor, and evil speaking be put away from you, with all malice.” Ephesians 4:30-31

Paul lists several behaviours that cause grief to God (Eph 4:31). The behaviors listed come from an evil heart. An evil heart is a heart of unbelief.   

“Beware, brethren, lest there be in any of you an evil heart of unbelief in departing from the living God;” Heb 3:12