sql serverHow can I output more than 256 characters to a file?

How can I output more than 256 characters to a file?

I am using the following command to write the output from a stored procedure to a file for further processing:

DECLARE @shellCommand VARCHAR(2000)
SET @shellCommand = 'SQLCMD -S ' + @@SERVERNAME + ' -d ' + @dbName + ' -U ' + @UserName + ' -P ' + @Password + ' -Q "' + @sqlCommand + '" -s "," -h -1 -k 1 -W -o "' + @outputfileName + '"'

@sqlCommand is set correctly, to "EXEC StoredProcedureName parameterValue" and when run independently apparently produces the correct data.

The problem I'm having is that one of the fields contains text greater than 256 characters long and is getting truncated when fed through the SQLCMD command.

On reading the documentation I see that the default column width is 256 characters. So I'm looking at using the -y or -Y option to allow more than 256 characters to be output. However, these options aren't compatible with the -W option which I was using to remove trailing spaces from the output. I started by using -y 0 (despite the performance warning - once I get the output working I can look at the performance). However, that produces some strange results.

The file consists of a header row followed by the data rows and should look something like this:

First,ABC,Number,String,ReallyLongString,...
A,0123,14.99,"Short string","Longish string",...
B,0456,23.99,"Normal string","Really, really long string that's causing problems",...

With the -W option the file format is correct but the really long string is truncated. The reason we found it was because the trailing " was missing and the file wasn't being read properly.

With -y 0 the really long string is getting output but large numbers of spaces are being added to apparently random columns. We're getting something like this:

First,ABC ,Number    [...]      ,String,ReallyLongString,...
A    ,0123,14.99     [...]      ,"Short string","Longish string",...
B    ,0456,23.99     [...]      ,"Normal string","Really, really long string that's causing problems",...

(There are many, many more spaces between "Number" and the next column as represented by the "[...]", I'm just showing a few).

There are more numeric values that have to be formatted in a similar way and it does appear that it's these that are causing the extra spaces after the value and before the next comma. We can live with a few extra spaces, but not as many as this as the resultant file is too large to be read by the target program.

The data is generated by a stored procedure that looks like this:

SELECT
    'First' AS First,
    'ABC' AS ABC,
    'Number' AS Number,
    'String' AS String,
    'ReallyLongString' AS ReallyLongString,
    ...

UNION ALL

SELECT
    'A' as First,
    Column1 as ABC,
    REPLACE(FORMAT(Column2, 'N2', 'en-GB'), ',', '') as Number, -- to get the format correct
    '"' + Column3 + '"' as String,
    '"' + Column4 + '"' as ReallyLongString,
    ....
FROM Table
WHERE <condition>

I'm thinking that the problem is the output from the stored procedure. I only added the -W option to the SQLCMD because of the extra trailing spaces being output, but looking at the stored procedure I can't work out where they are coming from. I changed the number formatting to include RTRIM:

RTRIM(REPLACE(FORMAT(Column2, 'N2', 'en-GB'), ',', ''))

but that appeared to make no difference.

Is there something I can do to the stored procedure or is there a combination of options to SQLCMD that will produce the desired output? Or am I going to have to find some other way of producing this file?

Where is the truncation actually happening? I don't think it is happening at character 257 given that the following works with 4001 characters: sqlcmd -Q "SELECT 'header' AS [tt] UNION ALL SELECT REPLICATE('O', 4000)+'x' AS [some_string];" -W -s "," -h -1 -k 1 -o "SQLCMD_long_line.txt". – Solomon Rutzky Feb 24 '17 at 15:29
@srutzky - The truncation appears to be happening when the file is written. If I run the stored procedure that generates the data I get the "Really long text...with ending" correctly in the output window. But when we look in the file it's "Really long text... wit, where the t is the 256th character including the opening ". Maybe it's an issue with the fact that I've had to quote the strings. – ChrisF Feb 24 '17 at 15:34
I don't think it has to do with quoting the strings as I just put my SELECT into a global temp proc, added the quotes like you have, and it still saves correctly to the output file. Can you try wrapping the "long line" in a CONVERT(VARCHAR(MAX), '"' + Column4 + '"') AS... to see if that helps? Also, what version of SQLCMD are you using? I am using 11.0.2100.60. – Solomon Rutzky Feb 24 '17 at 15:42
@srutzky - Nope. Still truncated. Version 10.50.4042.0 – ChrisF Feb 24 '17 at 15:51
You are running SQL Server 2012 (I assume at least this given the use of FORMAT) yet the 2008 R2 version of SQLCMD? Any chance of finding (or even downloading) a newer version of SQLCMD? – Solomon Rutzky Feb 24 '17 at 15:57

While trying out the various suggestions in the comments I have stumbled across a solution.

I replaced:

'"' + Column4 + '"' as ReallyLongString,

with:

'"' + REPLICATE('O', 4000) + '"' AS ReallyLongString,

and all of the text was output.

However, using a variable:

DECLARE @longText NVARCHAR(MAX)
SET @longText = REPLICATE('O', 4000);

'"' + @longText + '"' AS ReallyLongString,

didn't.

What did work was declaring the variable with a specific length:

DECLARE @longText NVARCHAR(4000)

So, as the text in this case is never going to be longer than 2000 characters (the device reading the file can't cope with text that long) I convert all the text to 2000 character long strings:

SELECT
    'A' as First,
    Column1 as ABC,
    REPLACE(FORMAT(Column2, 'N2', 'en-GB'), ',', '') as Number, -- to get the format correct
    '"' + CAST(Column3 as NVARCHAR(2000)) + '"' as String,
    '"' + CAST(Column4 as NVARCHAR(2000)) + '"' as ReallyLongString,
    ....
FROM Table
WHERE <condition>

This is the how. I still don't really know the why.

Strange... using -y 0 actually works perfectly for me. No white space padding or any other formatting issues. It actually works better than -W, even though the documentation doesn't explain why and still recommends using -W:

-W

This option removes trailing spaces from a column. Use this option together with the -s option when preparing data that is to be exported to another application. Cannot be used with the -y or -Y options.

Although it's not documented at all, it seems like -y 0 actually does what -W advertises, and more (unlimited string length, and it even takes care of removing the headers, so no need for -h-1). The only other option I had to use was -s "," to specify a different column separator.

I'm using version 13.1.0007.0 on Linux, so maybe the OP's issue is something that's been fixed in higher versions.

The only issue I encountered was a non-printable character (the box looking thing) being added at the end of very long strings (over 100K chars), but when the strings are quoted ('"' + field + '"'), the character goes away. ¯\_(ツ)_/¯