Minimizing overhead when transmitting results to clients

Always try to minimize the overhead incurred when transmitting query results back to clients.

By: Manish Maheshwari, Data Architect and Data Scientist at Cloudera, Inc.

Use techniques such as:

  • Always use the LIMIT clause in queries to sample data. See LIMIT Clause for more information.
  • Use the NUM_ROWS_PRODUCED_LIMIT query option to limit the number of rows produced by a query. This can be set as the default for user pools where exploratory queries are expected to run and the user might forget to add a LIMIT clause. See NUM_ROWS_PRODUCED_LIMIT Query Option for more information.
  • When using the impala-shell, use the -B option to avoid “prettyprinting” the result set and redirect query results to a file using the --output_file option. For example:

    impala-shell --ssl -k -i -B --output_file /tmp/result