This is a question that gets asked  to me several times a week . From my experience of being a former IRS Agent, most people who have not filed their taxes or owe money on back taxes have the fear of prison time.
You should know that each case is based on it’s own merit or lack thereof.  If tax fraud exists there is always a possibility of a criminal case. It is best to discuss your case with a former IRS Agent who knows the trends  and behavior patterns of the IRS.  With that said, here is the general policy of the position held by the IRS.
The policy held by the Internal Revenue Service:  ( from the irs.gov site )
“A long-standing practice of the IRS has been not to recommend criminal prosecution of individuals for failure to file tax returns, provided they voluntarily file, or make arrangements to file, before being notified they are under criminal investigation. The taxpayer must make an honest effort to file a correct return and have income from legal sources. A letter from the IRS concerning taxes is not a notice that a taxpayer is under criminal investigation.
The IRS helps to get people back into the system as part of its long-term plan to improve voluntary tax compliance. The IRS wants to get people back into the system, not prosecute ordinary people who made a mistake. However, flagrant cases involving criminal violations of tax laws will continue to be investigated.”
While this is the general practice of the Internal Revenue Service, wise persons error on the side of wisdom. It is best if you have not filed for several years, to have a former IRS Agent review the facts and details of your case.
It is quite easy to get you back into the system.
If you have committed no fraud, you should have nothing to worry about. In most cases the IRS will accept the filing of the last 3 to 6 years and will accept a payment plan or possible offer in compromise to seal the deal.