Legalism could be defined as any attempt to rely on self-effort to either 1) attain or 2) maintain our justification before God. In Paul's Epistle to the Galatians he warned them sternly about such false understandings of the gospel when he asked the offenders: "After beginning with the Spirit, are you now trying to attain your goal by human effort?" (Gal.3:3). Legalism always seems to have one thing in common: it's theology denies that Christ is sufficient for salvation. That some additional element of self-effort, merit or faithfulness on our part is necessary. As an example, those who erroneously teach that a Christian can lose his or her salvation are, in essence, denying the sufficiency of Christ to save to the utmost. They believe their effort has some merit in contributing toward the price of their redemption. They also believe certain sins to be greater than Christ's grace. But Christ's righteousness which he counts toward us is not only efficient for our salvation, but sufficient. His once for all sacrifice put away sin for all time in those He has united to Himself (Heb 10:10). His salvation includes not only saving us at the beginning but preserving us to the end, sealing us in His perfect righteousness and whose blood "reminds the covenant God" 
not to treat us as our sins justly deserve. Any attempt to add our covenant faithfulness as part of the price of redemption after regeneration and justification is an "attempt to attain our goal by human effort" and thus a complete misapprehension of the fullness of the gospel of Jesus Christ. We must, therefore, reject any and all attempts to maintain a judicial standing before God by any act on our part. Salvation is of the Lord.
In light of this, some may ask, "What shall we say then? Are we to continue in sin that grace may abound?" The Apostle Paul answers:
By no means! How can we who died to sin still live in it? Do you not know that all of us who have been baptized into Christ Jesus were baptized into his death? We were buried therefore with him by baptism into death, in order that, just as Christ was raised from the dead by the glory of the Father, we too might walk in newness of life. For if we have been united with him in a death like his, we shall certainly be united with him in a resurrection like his. We know that our old self was crucified with him in order that the body of sin might be brought to nothing, so that we would no longer be enslaved to sin. For one who has died has been set free from sin. Now if we have died with Christ, we believe that we will also live with him. (Rom 6:1-8)
The Apostle John also likewise declares that
"No one born of God makes a practice of sinning, for God's seed abides in him, and he cannot keep on sinning because he has been born of God." (1 John 3:9).
"And his commandments are not burdensome. For everyone who has been born of God overcomes the world" (1 John 5:4).
Notice that whether it is legalism or grace, obedience is required, but the motive we have for obeying makes all the difference. Our new life is a supernatural one ... and if we had to rely on self (even partly) to attain salvation then none of us would ever stand a chance of saving ourselves or persevering to the end. But God in his mercy ensures us that those united to Christ will indeed be preserved (and will persevere to the end). In other words, biblically speaking, true Christians obey and persevere because we are saved, but not in order to be saved ... for no one can maintain their just standing before God but Christ alone .... otherwise none of us would have hope.
Bill Baldwin has put together a brief definition of legalism which I think is helpful:
1. Using the Mosaic covenant as though it is the covenant between you and God.2. Attempting to be justified by one's own works.
3. Attempting to be sanctified by one's own works
4. Suggesting that our worth or worthlessness, our self-esteem and self-satisfaction or lack thereof, rest on our own works.
5. Any attempt to please God judicially, or any supposition that our sin as believers has resulted in his judicial displeasure. [My note: any post-salvation attempt to maintain our judicial standing before God through good works, covenant faithfulness, merit etc..]
6. Teaching that we conform ourselves to our judicial standing in Christ (righteous and perfect) by our own works.
7. Attempting to attain godliness by a systematic change of behavior
8. Obedience that does not spring from a renewed hearta. As of an unbeliever who has no renewed heartb. As of a believer who has a renewed heart but whose righteous behavior does not spring therefrom.
9. Any supposition that externally righteous acts have any value on their own, even as conduct that prepares the way for either
a. A renewed heart (preparationism as regards justification),b. The softening or further renewing of an already renewed heart (preparationism as regards sanctification. Note Romans 12:2-Transformation occurs through the renewing of the mind), or
c. Any other work of the Spirit.
10 . Suggesting that faith is irrelevant in the accomplishment of some (or all) good works.
11. Trying to be justified by works that are created and inspired by the Holy Spirit.
12. Attempting to gain assurance of salvation solely or primarily on the basis of the sign of outward works
A Christian is not only one who has repented of his immorality, but, even more importantly, has repented of his morality. The more self-righteous one is, the worse it is for him. Our own merits are of no value for salvation, but will damn us. Whoever relies on his own goodness for salvation will suffer complete ruin.
One last thing I would like to point out that I think is helpful. Predestination destroys legalism. If the whole of salvation, in all its parts, is understood to be by Christ alone, it leaves no room for boasting or trusting in ourselves, even partly. It strips us bare and forces us to abandon all hope in self-effort or rules to attain a right standing before God ... It shatters our self-complacency and causes us to renounce our self-righteousness. It makes us cover our lips and utter God is God and I am not (Rom 9:15, 16). That is why the Scripture declares: "It is because of Him you are in Christ Jesus, who became to us wisdom from God, righteousness and sanctification and redemption, so that, as it is written, “Let the one who boasts, boast in the Lord.” (1 Cor 1:30-31)
This is not to say that predestinarians cannot be legalistic. Human beings always find a way, Reformed people included. It means to say, rather, that if UNDERSTOOD CORRECTLY the doctrine of salvation by Grace ALONE in Christ ALONE will have the real effect of stripping us of legalism etc. Again only God's grace can do so. The true effect of biblical predestination will have on someone who beholds the majesty of God and is struck down by the fact that God will have mercy on whom he will have mercy is to say, I have no hope in myself, and justly deserve God's wrath save for the mercy of Christ alone, who gives me everything I need for salvation, including a new heart to believe (Deut 29:4, 30:6; John 6:63).
Other Helpful Resources
The Marrow of Modern Divinity by Edward Fisher
The Marrow Controversy by Dr Sinclair B Ferguson (.pdf)
Law or Spirit? Galatians Between Legalism and Antinomianism by Tom Schreiner
The Moral and Immoral Both Alienated from God by John Hendryx
Christ Vs. Moralism by John Hendryx
The Acid Test of Being a Christian by Tim Keller
Keller On Preaching in a Post-modern City by Tim Keller
What Is Legalism? by John W. Robbins