Batch Ingestion - format & pattern check | Community
Skip to main content
Michael_Soprano
Michael_Soprano
New Participant
June 29, 2025
Solved

Batch Ingestion - format & pattern check

  • June 29, 2025
  • 3 replies
  • 416 views

XXX

 

There is a table like that. 

Assuming that I have got a field in batch which do not meet the requirements of Schema field (pattern, format). 

How would be a behaviour of such a batch:

a) batch would land in Data Lake 

b) batch would be aborted ? 

 

Best answer by Asheesh_Pandey

@michael_soprano In my understanding, If a field in your batch data doesn't meet the schema requirements (like wrong format, pattern, or data type), the entire batch will fail. (Note: It is true only if the issue is with required fields OR identity fields that will block the batch, partial ingestion can still upload data upto certain percentage based on the batch ingestion settings). You’ll see the failure listed in the Monitoring UI with specific error details. Best to validate your data ahead of time using Data Prep or the Schema Registry API to avoid surprises!

3 replies

kautuk_sahni
kautuk_sahni
Employee
July 2, 2025

@kumarrishii Just checking in — were you able to resolve your issue?
We’d love to hear how things worked out. If the suggestions above helped, marking a response as correct can guide others with similar questions. And if you found another solution, feel free to share it — your insights could really benefit the community. Thanks again for being part of the conversation!

Kautuk Sahni
KumarRishii
KumarRishii
New Participant
June 30, 2025

If a batch contains fields that do not meet schema requirements (like pattern or format), the behavior depends on validation rules:

Strict Validation: The batch is aborted, and no data lands in the Data Lake.
Soft Validation: The batch may still land in the Data Lake, but problematic fields could trigger warnings, be ignored, or sent to an error bucket.

Check your system's ingestion configuration to confirm the behavior.

    Michael_Soprano
    Michael_SopranoAuthor
    New Participant
    July 1, 2025

    Where you set up this validation - strict/soft?

    Asheesh_Pandey
    Asheesh_PandeyAccepted solution
    New Participant
    June 29, 2025

    @michael_soprano In my understanding, If a field in your batch data doesn't meet the schema requirements (like wrong format, pattern, or data type), the entire batch will fail. (Note: It is true only if the issue is with required fields OR identity fields that will block the batch, partial ingestion can still upload data upto certain percentage based on the batch ingestion settings). You’ll see the failure listed in the Monitoring UI with specific error details. Best to validate your data ahead of time using Data Prep or the Schema Registry API to avoid surprises!

      Terms & ConditionsAccessibility statement

      Loading footer...