Test access to ADLS Gen2
To test access to ADLS, SSH to a cluster node, switch to the hdfs user (by using ‘sudo su hdfs’), and run a few hadoop fs shell commands against your existing storage account.
ADLS access path syntax is:
abfs://file_system@account_name.dfs.core.windows.net/dir/fileUse abfss instead of abfs to connect with a secure socket
layer connection.
For more information about syntax refer to Azure documentation.
For example, the following Hadoop FileSystem shell commands demonstrate access to “test container” in a storage account named “myaccount”:
hadoop fs -mkdir abfs://test-container@myaccount.dfs.core.windows.net/testdir
hadoop fs -put testfile abfs://test-container@myaccount.dfs.core.windows.net/testdir/testfileFilesystemNotFound error
When hierarchical namespace is enabled, you may experience the java.io.FileNotFoundException with an error code of 404 mentioning FilesystemNotFound and the error message “The specified filesystem does not exist”. The common reason for this error is that you are trying to access a blob container created through the Azure Portal, but when hierarchical namespace is enabled, you must not create containers through Azure Portal.
To solve this issue, do one of the following:
-
Delete the blob container through Azure Portal, wait a few minutes, and then try accessing this container.
-
Or use a different container that is not created through Azure Portal.