PhloemDocs
Markdown
Integration

Azure Blob Storage

Query CSV, Parquet, and JSON files stored in Azure Blob containers

Category
Files & storage
Runs in
The Phloem cloud — nothing to install

Turns CSV, JSON and Parquet files in your Azure storage containers into tables you and Hamming can query, listed in Trace.

What Phloem reads

  • CSV, JSON and Parquet files, including gzipped .csv.gz and .json.gz, in the containers you pick
  • Files are grouped into tables by path and format, with partition folders detected and column names and types sampled
  • The tables you keep, with their columns, listed in Trace for Hamming and SQL autocomplete

Files are read in place. Phloem never writes to your storage account.

What you need

  • A storage account, and one of: its connection string, its account name and key, or a SAS token
  • A credential that can list the account's containers and read their blobs

Connect

  1. In the console, open Connectors, choose Add connector, then Azure Blob Storage.
  2. On Azure Credentials, choose an Authentication Method:
    • Connection String — paste it into Connection String (Azure Portal › Storage Account › Access Keys).
    • Account Name & Key — enter Storage Account Name and Account Key.
    • SAS Token — enter Storage Account Name and SAS Token, including the leading ?.
  3. On Testing Credentials, Phloem lists the account's containers.
  4. On Select Containers, pick the containers to connect.
  5. On Discovered Tables, review the proposed tables. Each is a File Pattern (e.g. raw/events/**/*.parquet) matching files with the same layout. Edit a table's Table Name, File Pattern or Description, or turn it off.
  6. On Save Connection, name the connector and save it.

Notes

  • Discovery samples each folder instead of listing every file. A table it is unsure of starts turned off, so check the patterns before saving.
  • Files and folders starting with _ or ., and tmp, temp and log folders, are skipped.

#Connection fields

What the Add connector form asks for.

FieldTypeRequired
Authentication Method
Choose how to authenticate with Azure
One of
Connection String · Account Name & Key · SAS Token
Yes
Connection String
Full connection string from Azure Portal > Storage Account > Access Keys
Secret—
Storage Account Name
mystorageaccount
Text—
Account KeySecret—
Storage Account Name
mystorageaccount
Text—
SAS Token
Shared Access Signature token (include the leading '?')
Secret—
Path Prefix Filter (Optional)
Only scan files under this prefix path
data/exports/
Text—
File Formats
Select file formats to include. You can add custom extensions.
Any of
Parquet (.parquet) · CSV (.csv) · JSON (.json) · TSV (.tsv) · Avro (.avro) · ORC (.orc) · Excel (.xlsx) · Excel Legacy (.xls)
—
Scan Subfolders
Recursively scan subfolders for files
On / off—

Updated