Page History: Poor Man's T-SQL Formatter

Compare Page Revisions

164 163 162 161 160 159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134 133 132 131 130 129 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 Current 164 163 162 161 160 159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134 133 132 131 130 129 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0

« Older Revision - Back to Page History - Newer Revision »

Page Revision: 2012-01-29 14:49

Overview

Description

This is a homebrew SQL (T-SQL) formatter, mainly created because I could not find any free and open-source T-SQL formatters that handle T-SQL exactly the way I want, including full stored procedure formatting.


It's written in C#, and designed to work in three phases, each using a pluggable class:

• Tokenization - text-level parsing, identifying operators vs numbers vs words vs whitespace vs comments, etc
• Parsing - keyword-level parsing, identifying hierarchical relationships between components of the T-SQL code, statements, clauses, blocks, flow control, etc.
  • Please note, this is not a fully-fledged SQL parser. It does not distinguish between different types of DML, it does not parse full expression trees, etc - there's a lot it doesn't do, it just does the bare minimum to support the target formatting.
• Formatting - turning the SQL parse tree back into T-SQL code, with customizable formatting preferences including HTML colorizing.
  

The main portion of the project is a .Net 2.0 library. However, a number of smaller sub-projects expose the library for use in different ways:

• An SSMS (SQL Server Management Studio) Addin that allows you to format the current file or selected text with a single hotkey
• A command-line utility that lets you format any number of files in bulk
• A Winforms demo app that lets you look at the token stream and parse tree
• A Web Service that exposes the same functionality across http, used in the demo/online formatting site http://poorsql.com
• A WinMerge plugin, for automatically formatting SQL files before comparison, allowing WinMerge to display content changes only, ignoring formatting differences.
• A Notepad++ plugin, for quick single-key formatting i your favorite general-purpose text editor.
  

The library is largely complete, but undocumented. For the list of known issues, please see the issue tracker on github. As there is only as much parsing detail as necessary to support the desired formatting, the parse tree format is still not finalized; many formatting enhancements require new elements in the parse tree.

The code is hosted on GitHub at https://github.com/TaoK/PoorMansTSqlFormatter; I'd be very grateful for any feedback on the functionality, the code or any other aspect of the project!

License / Redistribution

The library and accompanying programs are released under the Affero GPL. This means that you are allowed to make any changes you want, there are restrictions/requirements attached only if you choose to redistribute the original or modified code: If you choose to redistribute/expose the functionality, then you must make the source available to the recipients/users (even of a web service). For private usage, you can make any changes you want and never need to tell anyone; if you distribute the app/functionality or a service that relies on it, then you must also distribute the source. For more details, please see the AGPL article on wikipedia or at the source, the FSF (free software foundation).

At least for the moment (as I'm the only one who has contributed to the project so far), the project could be dual-licensed if you really, really wanted to use it in a commercial program. If so please contact me, I'm sure we could work something out.

Latest Changes

2012-01-29, version 1.2.1:

• Github Issue 32: Feature Request Enhanced Cmdline Utility to support pipeline input (and output) [thanks William Lin for the request]
  • but default behaviour unchanged, still expect to act on files directly unless piped input provided
  • expected encoding is UTF-8... in powershell, for example, run "$OutputEncoding = [System.Text.Encoding]::UTF8" first.
• Enhanced Cmdline Utility "no files found" message to include extensions detail
• Github Issue 26: Testing enhancements:
  • Changed testing framework to Nunit, included dependencies for building and testing in any environment
  • Simplified testing code, with test sources
  • Added tests for different formatting options / flags, greater coverage
• GitHub Issue 31: Completed Notepad++ Plugin, with formatting options - available trough Notepad++ plugin manager.
  

2012-01-21, version 1.1.1:

• Translated programs, ssms plugin + website into French and Spanish
  • thanks to Threeplicate for Amanuens, a free (for open-source projects) localization tool that makes managing translations much easier!
  • and thanks to my wife for proofreading... technical translations are always a mess, but at least I got a second pair of eyes on it!
• Initial Beta of Notepad++ plugin [thanks UFO and Robert Giesecke for the .Net managed plugin template!]
  • See http://sourceforge.net/projects/notepad-plus/forums/forum/482781/topic/4911359 to get the plugin beta
• Corrected "Request Validation" issue on poorsql.com, caused by over-zealous web service validation in ASP.Net 4.0
• Github Issue 22: [Bugfix] Indenting of "AS" clause on 2nd or later CTE in a query
• GitHub Issue 21: [Feature Request] Added optional ignore of errors in CmdLine Formatter [thanks Jörg Burdorf for the request]
• GitHub Issue 18: [Bugfix] Corrected positioning of comments (after linebreak) at end of statements [thanks gvarol for the bug report]
• GitHub Issue 19: [Feature Request] Added optional indenting of join ON sections [thanks Pushpendra Rishi for the request]
• GitHub Issue 24: [Bugfix] HTMLEncoding missing in Web Service (eg for poorsql.com) and Winforms demo app [thanks Gokhan Varol for the bug report]
• GitHub Issue 25: [Feature Request] Settings persistence and optional display simplification in Winforms demo app [thanks Gokhan Varol for the request]
• GitHub Issue 33: [Bugfix] Culture-specific uppercasing bug (Turkish) [thanks Recep Guzel]
• GitHub Issue 35: [Enhancement] WITH clause breaking [thanks Lane Duncan for the suggestion]
• Softcoded parsing error message in library (for translation + optional removal of warning)
  

2011-08-25, version 1.0.1:

• Added MERGE clause and statement support
• Added OUTPUT and OUTPUT ... INTO ... clause support
• Corrected INSERT ... EXEC parsing/formatting
• Corrected VALUES ... indenting
• Corrected handling of multiple CTEs in a single statement
• Corrected handling of AS in stored procedure argument data type specification
• Added handling of ISO data type synonyms, with keyword consistency correction to Sql Server datatypes
• Added missing SQL Server 2008 datatypes, Date/Time-related, HierarchyID, and Geography
• Corrected minor indenting bug with single-word clauses at the end of multi-statement container content
• Added Expected SQL Parse Tree and Standard Output Sql tests in test suite - parse Xml format and output SQL format effectively finalized.
  

Full Change Log:

• See the changelog on github
  

Known / Open Issues:

• See the issue tracker on github
  

Download

Ready-to-run:

• SSMS Add-In (setup program): SqlFormatterSSMSAddIn.Setup.1.2.1.msi (1114 downloads for this version, about 2200 for previous versions)
• WinForms App (simple exe): SqlFormatterWinForms.1.2.1.zip (1184 downloads for this version, about 570 for previous versions)
• Command-line bulk formatting utility (simple exe): SqlFormatter.1.2.1.zip (1035 downloads for this version, about 440 for previous versions)
• WinMerge Plugin for SQL Script comparison (setup program): PoorMansTSqlFormatterWinMergePlugin.Setup.1.2.1.msi (1241 downloads for this version, about 230 for previous versions) (WinMerge is a free GPL-licensed file comparison tool, available at winmerge.org)
• Notepad++ plugin: It's highly recommended to use the Notepad++ plugin manager, but if necessary the plugin ZIP is here (1045 downloads) (Notepad++ is a free GPL-licensed text editor, available at notepad-plus-plus.org)
  

Compiled Library:

• SqlFormatterLibrary.1.2.1.zip (1031 downloads for this version, about 330 for previous versions)
  

Source & Web Service:

• Please download from github at https://github.com/TaoK/PoorMansTSqlFormatter. If this is impractical for whatever reason please contact me, and I'll make the source available by some other mechanism.
  

Feedback & Known Issues

Please email me at , or simply create issues in the GitHub issue tracker.

Instant Demo

The demo has been moved to poorsql.com - check it out and contact me with any issues!

Contributing

So far I haven't received any contributions from anyone else, but contributing is easy with GitHub! Just fork the project, make your changes, and create a Pull Request for me to take a look and I will (if I agree with the changes of course) happily incorporate them!

Also, if anyone wants to work on translation, I've been using Amanuens - if you want to contribute a translation, you can do so without dealing with code at all! Just send me an email and I'll set you up with an Amanuens account:

Background

But Why??

You may wonder why create yet another SQL formatting library... I've asked myself that a few times as I've been working on this. The main reason is that when I have searched for this in the past, I've never found a tool that did exactly what I wanted it to. Another answer is that it is fun, and I think I can do a pretty good job, and I've learned quite a lot about T-SQL in the process. Finally, I hadn't found any open-source library that I thought I could help improve, rather than starting my own (when I started this project, that is).

There does appear to be at least one other active/in-progress open-source project out there (sqlformat, by Ben Laan), but I don't believe that the approach taken there (as far as I can tell from looking at the code and playing around) could scale to the type of full-script universal T-SQL formatting I want to be able to (and now can, as far as I can tell) perform.

The main challenge, I believe, is that I want to do full re-formatting of *any* T-SQL code, supporting all keywords and syntax constructs. To do this with a full T-SQL parsing model would be difficult, or maybe even unrealistic, to achieve in a one-man (or few-people) part-time open-source project. By scaling it back to a partial parsing model where only formatting-relevant constructs are singled out, the scope of the project is significantly reduced, hopefully to an achievable and maintainable level.

The library is currently "V1 Complete", with no major bugs or omissions that I am aware of - I will probably be scaling back my attention to this project unless/until someone contacts me with suggestions or bugs. I have a list of "Issues" in the github issue tracker, but none are significant enough that I expect to work on them in the coming weeks.

Other Products and Projects

To get a better idea of the value of this project (for myself and potentially others), I started putting together a quick comparison table for this project vs all the other products and projects I come across. Obviously I try to avoid undue bias, and equally obviously I will still be biased towards this project, as I am human :).

Here is the list of all T-SQL formatting tools I've encountered, with a comparison table below:

• SqlFormat .Net 3.5 Formatter Library: The only open-source tool I've found out there. The project aims are different to this one, as they indent to create a full parse-tree for those parts of the language that are supported. As such, only a reasonably small subset of language features are covered at present - primarily those used in Entity Framework-generated SQL.
• GuDu Software options:
  • Instant SQL Formatter: Online-only free tool
  • OracleFAQs SQL and PL/SQL Formatter: appears just to be an externally-hosted copy of the "Instant SQL Formatter" tool
  • SQL Pretty Printer: Trialware SSMS add-in
  • General SQL Parser: Like it says... Trialware general SQL parser
• T-SQL Tidy: Online-only free tool, with free webservice and ssms add-in (webservice-based)
• Red Gate options:
  • Red Gate Simple-Talk Code Prettifier: Free online-only tool
  • Red Gate SQL Prompt: Trialware code/name completion, exploration and formatting tool
• SQLInForm options:
  • SQLInForm Free Online Version: Crippled online-only java applet-based tool (100 lines)
  • SQLInForm Desktop or Server Version: Java-based tool
• manoli.net c# code format: Apparently does T-SQL too
• Navicat Lite: DB Management tool with free/lite version for non-commercial use, apparently includes a sql beautifier in the lite version
• RazorSQL Trialware DB Management tool that apparently includes some formatting functionality
• DevArt SQL Complete: SSMS add-in with free version (?) that supports formatting
• UBitSoft options
  • T-SQL Beautifier: Online-only free tool
  • SQLEnlight: Trialware SSMS Add-in with code formatting and analysis / rule validation
• SoftTree SQL Assistant: Another trialware code completion and formatting tool, supporting multiple editors and multiple SQL dialects
• Quest Toad for SQL Server: Probably the reference third-party DB management tool; trialware.
• Apex SQL Refactor: Another trialware code completion / refactoring/reformatting add-in for SSMS, this one claims to keep formatting for free after the trial runs out
• Tidycode T-Sql Formatter: Command-line trialware formatter
• SQLIse / SQLPSX: Powershell-based tools and UI for SQL Server management with some(?) formatting functionality using the Microsoft Microsoft.Data.Schema.ScriptDom classes. I haven't tried it yet; someone claimed it strips comments, but I really don't know.
• SqlFormat Online SQL Formatting service: Another open-source project, this one using Python; doesn't support many DDL constructs, but degrades gracefully, so it still works with unknown constructs and structures.
• SilverlightSQLConvert An apparently japan-originated Silverlight-based formatter, enigmatically announced as "Made in Japan" by a commenter named "ninjya" on an old (2006) blog post, in June 2011. There is no information on the page as to the origins, purpose, license or copyright of the project, but in the title there seems to be a mention of "NodaSoft@" - also a company name claimed by a Russian software company. The code in the silverlight "SilverlightSQLConvert" DLL appears to be un-obfuscated and the SQL formatting appears to be text-pattern-matching-based. Trial and error shows it to be effective for single statements that don't use more complex constructs such as "MERGE".
  

	Poor Man's T-SQL Formatter	SqlFormat .Net 3.5 Formatter Library	Instant SQL Formatter	OracleFAQs SQL and PL/SQL Formatter	SQL Pretty Printer	General SQL Parser	T-SQL Tidy	Red Gate Simple-Talk Code Prettifier	Red Gate SQL Prompt	SQLInForm Free Online Version	SQLInForm Desktop or Server Version	manoli.net c# code format	Navicat Lite	RazorSQL	DevArt SQL Complete	T-SQL Beautifier	SQLEnlight	SoftTree SQL Assistant	Quest Toad for SQL Server	Apex SQL Refactor	Tidycode T-Sql Formatter	SQLIse / SQLPSX	SqlFormat Online SQL Formatting service	SilverlightSQLConvert
ID	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24
Open-Source	✓	✓	✗	✗	✗	✗	✗	✗	✗	✗	✗	✓	✗	✗	✗	✗	✗	✗	✗	✗	✗	✓	✓	?
Free (not trialware)	✓	✓	✓	✓	✗	✗	✓	✓	✗	✗	✗	✓	?	✗	✗	✓	✗	✗	✗	?	✗	✓	✓	?
Works Offline	✓	✓	✗	✗	✓	✓	✗	✗	✓	✗	✗	✓	✓	✓	✓	✗	✓	✓	✓	✓	✓	✓	✓	?
Online Access (website)	✓	✓	✓	✓	✗	✗	✓	✓	✗	✓	✗	✓	✗	✗	✗	✓	✗	✗	✗	✗	✗	✗	✓	?
Command-line exe	✓	✗	✗	✗	✓	✗	✗	✗	✗	✗	½	✗	✗	✗	✗	✗	✓	½	?	?	✓	?	?	?
SSMS Plugin	✓	✓	✗	✗	✓	✗	✓	✗	✓	✓	✓	✗	✗	✗	✓	✗	✓	✓	✗	✓	✗	✗	✗	?
Cross-platform	½	?	-	-	✗	?	-	-	✗	-	?	?	½	✓	✗	-	✗	✗	✗	✗	½	✗	✓	?
Other SQL Dialects	✗	?	✓	✓	?	✓	✗	✗	✗	✓	✓	✗	✓	✓	✗	✗	✗	✓	½	✗	✗	?	?	?
Embeddable/Library/API	✓	✓	-	-	✗	✓	✓	✗	✗	✗	✓	✓	✗	✗	✗	✗	✓	?	✗	✗	✗	?	✓	?
Exposes Parse Tree	½	✓	✗	✗	✗	✓	?	✗	✗	✗	?	✗	✗	✗	✗	✗	✓	?	✗	✗	✗	✓	?	?
Procedural SQL (multi-statement, flow control, etc)	✓	✗	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	✗	?
Graceful degradation (unhandled sql)	✓	✗	✗	✗	?	?	✗	✗	✗	✓	✓	✓	?	?	?	✗	✗	?	?	?	?	?	✓	?
Multi-batch & DDL handling	✓	✗	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
NHibernate Appender	✗	✓	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
Edge Cases: Numerics & Currency	✓	✗	✗	✗	✗	✗	✗	✗	?	✗	✗	✗	?	?	?	✗	✗	?	?	?	?	?	✗	?
Edge Cases: Nested Comments	✓	✗	✓	✓	✓	✓	✗	✗	?	✗	✗	✗	?	?	?	✗	✗	?	?	?	?	?	✓	?
"Aligned" formatting	✗	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
Coloring (HTML)	✓	✓	✓	✓	?	-	✓	✓	-	✓	✓	✓	?	?	?	✓	?	?	?	?	?	?	✓	?
Keyword Standardization	✓	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
Code Wrapping (eg VBScript)	✗	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?
Recent Activity (last 6 months)	✓	✓	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	?	✓	✗	?

Please note:

• This table is based on my own very brief research, in terms of what information is available online for each of these products. If you see any errors in this list, please contact me and I will correct them asap. Where I was not able to figure something out from online documentation, I have left a question mark.
• This table evaluates sql formatting features only - many of the packages/products in this list have a lot of additional functionality available, which I'm not addressing here at all!
• Please contact me if you're interested in this list but the table headers don't show up properly... I've tested it in the "main" browsers (Firefox, Chrome, Safari, hideous but legible in IE7/8/9), but it took some horrible CSS hacks to get it working, and I haven't tested it in many other browsers.
  

Your Formatting preferences? Who are you?

The only reason my personal preferences are important here, is that I'm the one writing this :).

What I can do, however, is explain the rationale behind my preferences, in the hope that I might sway someone's opinion:

• I find that too much whitespace in the body/joins distracts from the structure of the query, especially if you end up having to scroll to find parts of the query. This is why I'm not fond of indenting the "ON" portion of a join clause
• I find that logical expressions are easiest to read (and write) when each portion is on its own line. This applies regardless of whether it's in a join clause's "ON" conditions, or the "WHERE" clause
• I find it very important to be able to write SQL (in a variety of editors) in the target format, without needing something to clean up after you. This is the main reason I don't like "Aligned" SQL formatting. I understand it can be more legible, but it's too much of a pain to maintain (esp. in embedded/ad-hoc SQL queries). It also breaks when you use tabs (in different editors with different tab stops), and I like tabs! They're easy-to-navigate whitespace!
  

What if I want my SQL formatted differently?

I am completely open to providing additional formatting options, obviously within the time I have available to devote to this project. As it's open source and hosted on github, it's also trivial to fork the project, download to your local machine and get cracking on any changes you'd like to see!

If you don't want to, or can't make the changes yourself and you can't get a hold of me or I don't have time to work on the changes, there is always the host of commercial products identified above!