Creating tables by importing CSV files from AWS S3
You can create tables in Hue by importing CSV files stored in S3 buckets. Hue automatically detects the schema and the column types, thus helping you to create tables without using the CREATE TABLE syntax.
Only Hue Superusers can access S3 buckets and import files to create tables. To create tables by importing files from S3, you must assign and authorize a specific bucket to be used as an external S3 bucket for your environment. The bucket then appears like a home directory on the Hue web interface.
The maximum file size supported is three gigabytes.
- Create roles and synchronize users to FreeIPA.
Make sure that the users who want to create tables by importing CSV files have EnvironmentAdmin or EnvironmentUser roles, and that you have synchronized users to FreeIPA. This is required for users accessing the data lake. Ranger will allow these users to access external buckets, without which the authorization will fail.
To assign roles to users, see Assigning resources to users in the Management Console documentation.
To synchronize users to FreeIPA, see Performing user sync in the Management Console documentation.
- Add an external S3 bucket to your CDW environment.
If you try to access an external S3 bucket from the Hue web interface without adding it to the CDW environment, then Impala or Hive may display the “AccessDeniedException 403” exception. Make sure that your Cloudera Data Warehouse (CDW) environment has access to the S3 buckets that you want to access from Hue.
When you create a Virtual Warehouse in the CDW service, a cluster is created in your AWS account. This cluster has two buckets. One bucket is used for managed data and the other is used for external data. Access to these two buckets is controlled by AWS instance profiles.
To add read/write access to external S3 buckets that reside in the same AWS account as the CDW service cluster or that are different from the account where the CDW service cluster resides, see the corresponding links at the bottom of the page.
To add an external S3 bucket to your CDW environment:- In the Data Warehouse service, expand the Environments column by clicking the More… menu on the left side of the page.
- In the Environments column, click the search icon and locate the environment in which you want to add the S3 bucket.
- Click the edit icon in the tile. This launches the Environment Details page.
- In the Environment Details page, toward the
bottom of the page, type the name of the AWS bucket you want to
configure access to in the Add External S3
Bucket text box.
If the bucket belongs to another AWS account, then select the Bucket belongs to different AWS account option.
- Select the access mode.
Read-only access is sufficient to import data in Hue.
- Click Add Bucket to save the configuration. A success message displays at the top of the page.
- Click Apply to update the CDW environment.
- Enable the S3 file browser in Hue.
To enable access to S3 buckets from the Hue web UI, you must add the AWS environment details in the hue-safety-valve configuration from your Virtual Warehouse. After enabling the S3 file browser, you can browse the S3 buckets, create folders, and upload files from your computer, and import files to create tables.
- Sign in to Cloudera Data Warehouse.
- Go to the Virtual Warehouse from which you want to access the S3 buckets and click its edit icon.
- On the Virtual Warehouses detail page, click the Hue tab and select hue-safety-valve from the drop-down menu.
- Add the following configuration for Hive or Impala Virtual Warehouse
in the space provided:For Hive Virtual Warehouse:
[desktop] # Remove the file browser from the blocked list of apps. # Tweak the app_blacklist property to suit your app configuration. app_blacklist=oozie,search,hbase,security,pig,sqoop,spark,impala [aws] [[aws_accounts]] [[[default]]] access_key_id=[***AWS-ACCESS-KEY***] secret_access_key=[***SECRET-ACCESS-KEY***] region=[***AWS-REGION***] [filebrowser] # (Optional) To set a specific home directory path: remote_storage_home=s3a://[***S3-BUCKET-NAME***]
For Impala Virtual Warehouse:[desktop] # Remove the file browser from the blocked list of apps. # Tweak the app_blacklist property to suit your app configuration. app_blacklist=spark,zookeeper,hive,hbase,search,oozie,jobsub,pig,sqoop,security [aws] [[aws_accounts]] [[[default]]] access_key_id=[***AWS-ACCESS-KEY***] secret_access_key=[***SECRET-ACCESS-KEY***] region=[***AWS-REGION***] [filebrowser] # (Optional) To set a specific home directory path: remote_storage_home=s3a://[***S3-BUCKET-NAME***]
- Click Apply in the upper right corner of the
page.
The S3 File Browser icon appears on the left Assist panel on the Hue web UI after the Virtual Warehouse restarts.
To import the CSV file and create the table: