Page History: Poor Man's T-SQL Formatter
Compare Page Revisions
Page Revision: 2012-01-29 17:41
Overview
Description
This is a homebrew SQL (T-SQL) formatter, mainly created because I could not find any free and open-source T-SQL formatters that handle T-SQL exactly the way I want, including full stored procedure formatting.
It's written in C#, and designed to work in three phases, each using a pluggable class:
- Tokenization - text-level parsing, identifying operators vs numbers vs words vs whitespace vs comments, etc
- Parsing - keyword-level parsing, identifying hierarchical relationships between components of the T-SQL code, statements, clauses, blocks, flow control, etc.
- Please note, this is not a fully-fledged SQL parser. It does not distinguish between different types of DML, it does not parse full expression trees, etc - there's a lot it doesn't do, it just does the bare minimum to support the target formatting.
- Formatting - turning the SQL parse tree back into T-SQL code, with customizable formatting preferences including HTML colorizing.
The main portion of the project is a .Net 2.0 library. However, a number of smaller sub-projects expose the library for use in different ways:
- An SSMS (SQL Server Management Studio) Addin that allows you to format the current file or selected text with a single hotkey
- A command-line utility that lets you format any number of files in bulk
- A Winforms demo app that lets you look at the token stream and parse tree
- A Web Service that exposes the same functionality across http, used in the demo/online formatting site http://poorsql.com
- A WinMerge plugin, for automatically formatting SQL files before comparison, allowing WinMerge to display content changes only, ignoring formatting differences.
- A Notepad++ plugin, for quick single-key formatting i your favorite general-purpose text editor.
The library is largely complete, but undocumented. For the list of known issues, please see the
issue tracker on github. As there is only as much parsing detail as necessary to support the desired formatting, the parse tree format is still not finalized; many formatting enhancements require new elements in the parse tree.
The code is hosted on GitHub at
https://github.com/TaoK/PoorMansTSqlFormatter; I'd be very grateful for any feedback on the functionality, the code or any other aspect of the project!
License / Redistribution
The library and accompanying programs are released under the Affero GPL. This means that you are allowed to make any changes you want, there are restrictions/requirements attached only if you choose to redistribute the original or modified code: If you choose to redistribute/expose the functionality, then you must make the source available to the recipients/users (even of a web service). For private usage, you can make any changes you want and never need to tell anyone; if you distribute the app/functionality or a service that relies on it, then you must also distribute the source. For more details, please see the
AGPL article on wikipedia or at the source, the
FSF (free software foundation).
At least for the moment (as I'm the only one who has contributed to the project so far), the project could be dual-licensed if you really, really wanted to use it in a commercial program. If so please contact me, I'm sure we could work something out.
Latest Changes
2012-01-29, version 1.2.1:
- Github Issue 32: [Feature Request] Enhanced Cmdline Utility to support pipeline input (and output) [thanks William Lin for the request]
- but default behaviour unchanged, still expect to act on files directly unless piped input provided
- expected encoding is UTF-8... in powershell, for example, run "$OutputEncoding = [System.Text.Encoding]::UTF8" first.
- Enhanced Cmdline Utility "no files found" message to include extensions detail
- Github Issue 26: Testing enhancements:
- Changed testing framework to Nunit, included dependencies for building and testing in any environment
- Simplified testing code, with test sources
- Added tests for different formatting options / flags, greater coverage
- GitHub Issue 31: Completed Notepad++ Plugin, with formatting options - available trough Notepad++ plugin manager.
2012-01-21, version 1.1.1:
- Translated programs, ssms plugin + website into French and Spanish
- thanks to Threeplicate for Amanuens, a free (for open-source projects) localization tool that makes managing translations much easier!
- and thanks to my wife for proofreading... technical translations are always a mess, but at least I got a second pair of eyes on it!
- Initial Beta of Notepad++ plugin [thanks UFO and Robert Giesecke for the .Net managed plugin template!]
- See http://sourceforge.net/projects/notepad-plus/forums/forum/482781/topic/4911359 to get the plugin beta
- Corrected "Request Validation" issue on poorsql.com, caused by over-zealous web service validation in ASP.Net 4.0
- Github Issue 22: [Bugfix] Indenting of "AS" clause on 2nd or later CTE in a query
- GitHub Issue 21: [Feature Request] Added optional ignore of errors in CmdLine Formatter [thanks Jörg Burdorf for the request]
- GitHub Issue 18: [Bugfix] Corrected positioning of comments (after linebreak) at end of statements [thanks gvarol for the bug report]
- GitHub Issue 19: [Feature Request] Added optional indenting of join ON sections [thanks Pushpendra Rishi for the request]
- GitHub Issue 24: [Bugfix] HTMLEncoding missing in Web Service (eg for poorsql.com) and Winforms demo app [thanks Gokhan Varol for the bug report]
- GitHub Issue 25: [Feature Request] Settings persistence and optional display simplification in Winforms demo app [thanks Gokhan Varol for the request]
- GitHub Issue 33: [Bugfix] Culture-specific uppercasing bug (Turkish) [thanks Recep Guzel]
- GitHub Issue 35: [Enhancement] WITH clause breaking [thanks Lane Duncan for the suggestion]
- Softcoded parsing error message in library (for translation + optional removal of warning)
2011-08-25, version 1.0.1:
- Added MERGE clause and statement support
- Added OUTPUT and OUTPUT ... INTO ... clause support
- Corrected INSERT ... EXEC parsing/formatting
- Corrected VALUES ... indenting
- Corrected handling of multiple CTEs in a single statement
- Corrected handling of AS in stored procedure argument data type specification
- Added handling of ISO data type synonyms, with keyword consistency correction to Sql Server datatypes
- Added missing SQL Server 2008 datatypes, Date/Time-related, HierarchyID, and Geography
- Corrected minor indenting bug with single-word clauses at the end of multi-statement container content
- Added Expected SQL Parse Tree and Standard Output Sql tests in test suite - parse Xml format and output SQL format effectively finalized.
Full Change Log:
Known / Open Issues:
Download
Ready-to-run:
- Command-line bulk formatting utility (simple exe): SqlFormatter.1.2.1.zip (1036 downloads for this version, about 440 for previous versions)
- Notepad++ plugin: It's highly recommended to use the Notepad++ plugin manager, but if necessary the plugin ZIP is here (1046 downloads) (Notepad++ is a free GPL-licensed text editor, available at notepad-plus-plus.org)
Compiled Library:
Source & Web Service:
Feedback & Known Issues
Please email me at
,
or simply create issues in the
GitHub issue tracker.
Instant Demo
The demo has been moved to poorsql.com - check it out and contact me with any issues!
Contributing
So far I haven't received any contributions from anyone else, but contributing is easy with GitHub! Just fork the project, make your changes, and create a Pull Request for me to take a look and I will (if I agree with the changes of course) happily incorporate them!
Also, if anyone wants to work on translation, I've been using Amanuens - if you want to contribute a translation, you can do so without dealing with code at all! Just send me an email and I'll set you up with an Amanuens account:
Background
But Why??
You may wonder why create yet another SQL formatting library... I've asked myself that a few times as I've been working on this. The main reason is that when I have searched for this in the past, I've never found a tool that did exactly what I wanted it to. Another answer is that it is fun, and I think I can do a pretty good job, and I've learned quite a lot about T-SQL in the process. Finally, I hadn't found any
open-source library that I thought I could help improve, rather than starting my own (when I started this project, that is).
There does appear to be at least one other active/in-progress open-source project out there (sqlformat, by Ben Laan), but I don't believe that the approach taken there (as far as I can tell from looking at the code and playing around) could scale to the type of full-script universal T-SQL formatting I want to be able to (and now can, as far as I can tell) perform.
The main challenge, I believe, is that I want to do full re-formatting of *any* T-SQL code, supporting all keywords and syntax constructs. To do this with a full T-SQL parsing model would be difficult, or maybe even unrealistic, to achieve in a one-man (or few-people) part-time open-source project. By scaling it back to a partial parsing model where only formatting-relevant constructs are singled out, the scope of the project is significantly reduced, hopefully to an achievable and maintainable level.
The library is currently "V1 Complete", with no major bugs or omissions that I am aware of - I will probably be scaling back my attention to this project unless/until someone contacts me with suggestions or bugs. I have a list of "Issues" in the github issue tracker, but none are significant enough that I expect to work on them in the coming weeks.
Other Products and Projects
To get a better idea of the value of this project (for myself and potentially others), I started putting together a quick comparison table for this project vs all the other products and projects I come across. Obviously I try to avoid undue bias, and equally obviously I will still be biased towards this project, as I am human :).
Here is the list of all T-SQL formatting tools I've encountered, with a comparison table below:
- SqlFormat .Net 3.5 Formatter Library: The only open-source tool I've found out there. The project aims are different to this one, as they indent to create a full parse-tree for those parts of the language that are supported. As such, only a reasonably small subset of language features are covered at present - primarily those used in Entity Framework-generated SQL.
- GuDu Software options:
- T-SQL Tidy: Online-only free tool, with free webservice and ssms add-in (webservice-based)
- Red Gate options:
- SQLInForm options:
- manoli.net c# code format: Apparently does T-SQL too
- Navicat Lite: DB Management tool with free/lite version for non-commercial use, apparently includes a sql beautifier in the lite version
- RazorSQL Trialware DB Management tool that apparently includes some formatting functionality
- DevArt SQL Complete: SSMS add-in with free version (?) that supports formatting
- UBitSoft options
- SoftTree SQL Assistant: Another trialware code completion and formatting tool, supporting multiple editors and multiple SQL dialects
- Quest Toad for SQL Server: Probably the reference third-party DB management tool; trialware.
- Apex SQL Refactor: Another trialware code completion / refactoring/reformatting add-in for SSMS, this one claims to keep formatting for free after the trial runs out
- Tidycode T-Sql Formatter: Command-line trialware formatter
- SQLIse / SQLPSX: Powershell-based tools and UI for SQL Server management with some(?) formatting functionality using the Microsoft Microsoft.Data.Schema.ScriptDom classes. I haven't tried it yet; someone claimed it strips comments, but I really don't know.
- SqlFormat Online SQL Formatting service: Another open-source project, this one using Python; doesn't support many DDL constructs, but degrades gracefully, so it still works with unknown constructs and structures.
- SilverlightSQLConvert An apparently japan-originated Silverlight-based formatter, enigmatically announced as "Made in Japan" by a commenter named "ninjya" on an old (2006) blog post, in June 2011. There is no information on the page as to the origins, purpose, license or copyright of the project, but in the title there seems to be a mention of "NodaSoft@" - also a company name claimed by a Russian software company. The code in the silverlight "SilverlightSQLConvert" DLL appears to be un-obfuscated and the SQL formatting appears to be text-pattern-matching-based. Trial and error shows it to be effective for single statements that don't use more complex constructs such as "MERGE".
Please note:
- This table is based on my own very brief research, in terms of what information is available online for each of these products. If you see any errors in this list, please contact me and I will correct them asap. Where I was not able to figure something out from online documentation, I have left a question mark.
- This table evaluates sql formatting features only - many of the packages/products in this list have a lot of additional functionality available, which I'm not addressing here at all!
- Please contact me if you're interested in this list but the table headers don't show up properly... I've tested it in the "main" browsers (Firefox, Chrome, Safari, hideous but legible in IE7/8/9), but it took some horrible CSS hacks to get it working, and I haven't tested it in many other browsers.
Your Formatting preferences? Who are you?
The only reason my personal preferences are important here, is that I'm the one writing this :).
What I can do, however, is explain the rationale behind my preferences, in the hope that I might sway someone's opinion:
- I find that too much whitespace in the body/joins distracts from the structure of the query, especially if you end up having to scroll to find parts of the query. This is why I'm not fond of indenting the "ON" portion of a join clause
- I find that logical expressions are easiest to read (and write) when each portion is on its own line. This applies regardless of whether it's in a join clause's "ON" conditions, or the "WHERE" clause
- I find it very important to be able to write SQL (in a variety of editors) in the target format, without needing something to clean up after you. This is the main reason I don't like "Aligned" SQL formatting. I understand it can be more legible, but it's too much of a pain to maintain (esp. in embedded/ad-hoc SQL queries). It also breaks when you use tabs (in different editors with different tab stops), and I like tabs! They're easy-to-navigate whitespace!
What if I want my SQL formatted differently?
I am completely open to providing additional formatting options, obviously within the time I have available to devote to this project. As it's open source and hosted on github, it's also trivial to fork the project, download to your local machine and get cracking on any changes you'd like to see!
If you don't want to, or can't make the changes yourself and you can't get a hold of me or I don't have time to work on the changes, there is always the host of commercial products identified above!