Reading snappy compressed input in pig

I have a file that I am trying to load into pig that is compressed with snappy. I set the configuration options in grunt like was described in this jira issue but I am still getting the compressed data in the results.

When I run the job it does say: org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library is available

for the job I do a simple a = load '/path/to/snappy/file' using PigStorage() as (x, y, z)

then: dump data

will output the compressed data.

Does anyone know what I can do to read the data correctly? Thanks in advance.

Answers


PigStorage uses PigTextInputFormat for input, which will detect and use Snappy compressed files, but the files must have the correct extension for hadoop the hadoop compression codec factory to know to use snappy.

My guess is your files don't have the .snappy extension, try renaming the files and trying again


Need Your Help

Building a list in a dictionary (LINQ to SQL)

c# linq-to-sql

I am try to build a dictionary with a list of dates as values.

in three.js, the spotlight is not showing a cone on a plane

three.js

I have a very simple example: a spot light pointed at a plane. I am expecting to see a cone of light whose diameter depends on the setting of the spot light angle. I cannot see any cone, the whole ...

About UNIX Resources Network

Original, collect and organize Developers related documents, information and materials, contains jQuery, Html, CSS, MySQL, .NET, ASP.NET, SQL, objective-c, iPhone, Ruby on Rails, C, SQL Server, Ruby, Arrays, Regex, ASP.NET MVC, WPF, XML, Ajax, DataBase, and so on.