post
The upload resource is used for adding data content to a table. You may only create uploads on tables that belong to unreleased versions. You may create one or more uploads for per table, and their records will be "stacked" based on common variable names across the uploads.
There are multiple different mechanisms to upload content through this endpoint. In general, if your file is larger than 10MB, or if you are on an unreliable internet connection, it is highly recommended to use resumable uploads. If you are utilizing a Redivis client library to perform your uploads, this will be taken care of for you.
There's a lot of complexity in uploading data! It's highly recommended to use one of the client libraries to transfer data to Redivis. These libraries take care of the complexities automatically and optimize the upload mechanism based on the size of your file.
Simple uploads should be used for smaller files that follow standard conventions. Provide the upload metadata through query parameters, and the file's content in the request body.
If a file is still small, but you need to provide additional metadata, you may send a multipart request body. The first part must be JSON-encoded and contain the upload's metadata, and the second part should contain the file's content.
If sending a multipart request, the appropriate
Content-Type: multipart/form-data
header must be set.For larger files or less reliable network connection, it is recommended to use resumable uploads for better fault tolerance. To perform a resumable upload, first create a resumable upload and upload your file's content to the provided URL. After the upload is completed, call this endpoint with the
resumableUploadId
provided in the request body. If you have a small number of rows being inserted at relatively high frequency, you should first create an empty upload via this endpoint (providing
type="stream"
in the request body), and then utilize the upload.postRows endpoint to stream individual rows into the upload.If the file currently resides in another location, such as an s3 bucket or url, you can specify a
transferSpecification
in the request body. Note that you must first enable the relevant data source in your workspace -> settings before you can perform a transfer upload from that source.POST /api/v1/tables/:tableReference/uploads
Parameter | |
tableReference |
Parameter | |
name | Required only if no name provided in the request body. The name of the file being uploaded. Overrides the name in the request body if both are provided. The file type will be auto-determined based on the ending of this name. If you'd like to manually set the type, provide it via the request body. |
If you are performing a simple upload, provide the upload's data in the request body.
If you are performing a multipart upload, the first part must contain the JSON-encoded parameters specified below, and the second part must contain the upload's data.
If you are performing a resumable upload, provide the
resumableUploadId
and the upload's parameters as a JSON-encoded body as specified below.If you are performing a streaming upload, provide the upload's parameters as a JSON-encoded body as specified below. The upload type must be set to
stream
.Property name | Type | Description |
---|---|---|
name | string | Required. The name of the upload. |
resumableUploadId | string | Required if using resumable uploads. The id returned by the createResumableUpload endpoint. |
transferSpecification | object | Required if using an external transfer. The configuration for the external source where the file resides. |
transferSpecification .sourceType | string | The source where the file currently resides. Must be one of:
- url - gcs
- s3
- bigQuery
- redivis |
transferSpecification .sourcePath | string | The path to the file in your source. For example, a url for url uploads, bucket-name/object-path for GCS and s3, and a qualified table specification for BigQuery and Redivis sources. |
transferSpecification
.identity | string | Should be omitted if the sourceType="url" .
For other sources, the identity should be the email address associated the credentials for the particular sourceType . Note that you must first enable the relevant data source in your workspace -> settings for this identity. |
type | string | Optional. The type of the file. If not provided, will be auto determined based on the file ending provided via the upload name. An error will occur if no valid type can be determined based on the upload name. Allowed types:
|
metadata | object | Optional. Provide optional metadata on the variables in the file. This parameter is a dict of variable names mapping to the metadata for that variable, e.g.
{
<variable_name>: {
label: str,
description: str, valueLabels: {
value: str,
...
}
},
...
} |
schema | object | Optional. Only relevant for uploads of type stream. Defines an initial schema that will be validated on subsequent calls to insertRows. Takes the form: [{ "name": "var_name", "type": "integer"}, ...] |
skipBadRecords | bool | Optional. Default false. If set to true, the upload will succeed even if some records are invalid / un-parseable. After the upload is completed, the number of skipped records will be available through the skippedBadRecords property on the upload resource. |
hasHeaderRow | bool | Optional. Whether or not the file has a header present as the first row. Only applicable for delimited, xls, and xlsx types. Defaults to true. |
delimiter | string | Optional. Only applicable for delimited types. The character to use as a field separator in the delimited file. If not set, the delimiter will be auto-determined based on analysis of the file. |
quoteCharacter | string | Optional. Only applicable for delimited types. The character used to enclose fields containing the delimiter. Defaults to the double quote character: " |
escapeCharacter | string | Optional. The escape characters specifies the character that indicates characters immediately following it should be escaped. Defaults to the quote character (above) |
hasQuotedNewlines | bool | Optional. Applicable for all types other than stream , parquet , avro , and orc .
If true, specifies that line breaks exist within a quoted field of the delimited file. If unset, Redivis will attempt to auto-determine whether the file contains quoted newlines, though this may fail if quoted newlines are only encountered many records in to the file. Setting this value to true won't cause issues with files that don't have newline characters in fields, though it will substantially increase import processing time and may lead to inaccurate error messages if set unnecessarily. |
Edit access to the dataset is required. Your access token must have the following scope:
- data.edit
Last modified 1mo ago