Mastering DataWeave Transformations: 10 Advanced Examples (Part II)

by MuleTrains •
0

Welcome back to our deep dive into MuleSoft integration! If you read Part I, you know that DataWeave is the absolute engine that keeps your data flowing smoothly between systems. However, real-world integration isn't always simple mapping; sometimes, it gets messy.


In Part II of this series, we are tackling the tricky stuff. We are talking about extracting hidden XML metadata, conditionally updating payloads based on their type, and flattening complex structures into clean CSVs. Whether you are a beginner looking to level up or a senior developer needing a quick snippet, these 10 scenarios cover the "Input vs. Output" challenges you face daily.


Let’s get your hands dirty with the code.


1. Extracting XML Metadata

Sometimes the data you need isn't in the body; it's hidden in the XML declaration headers. Here is how you can extract specific encoding and media type attributes from an XML input using DataWeave selectors.

Input:

<?xml version='1.0' encoding='UTF-8'?><menusList><menus><menu><id>file</id><value>File</value><popup><menuitem><value>New</value><onclick>CreateNewDoc()</onclick></menuitem><menuitem><value>Open</value><onclick>OpenDoc()</onclick></menuitem><menuitem><value>Close</value><onclick>CloseDoc()</onclick></menuitem></popup></menu></menus></menusList>

Target Output:

{
  "encoding": "UTF-8",
  "mediaType": "application/xml"
}

Solution 1: Using Selectors

This method uses the selector syntax to grab metadata directly.

%dw 2.0
output application/json
---
{
"encoding": payload.^encoding,
"mediaType": payload.^mediaType
}

Solution 2: Raw Data Approach

Alternatively, you can access the raw data object.

%dw 2.0
output application/json
---
payload.^ - "raw"

2. Conditional Updates

The update operator is incredibly powerful, but what if your input structure isn't consistent? In this scenario, we update a value only if the payload acts like an object, allowing the script to handle both Array and Object inputs dynamically.

Input 1 (Array):

[{"menusList": {"menus": {"name" : "sai"}}}]

Input 2 (Object):

{"menusList": {"menus": {"name" : "sai"}}}

Target Output (Scenario 2):

{"menusList": "this is an object"}

Solution 1: Type Check

We use typeOf to check the data structure explicitly.

%dw 2.0
output application/json
---
if(typeOf(payload) ~= "Object")
payload  update {
case .menusList -> "this is an object"
}
else payload

Solution 2: Pattern Matching

A more elegant solution using pattern matching within the update case.

%dw 2.0
output application/json
---
payload update {
case mLit at  .menusList if(payload is Object)->  "this is an object"
}

3. Dynamic Key-Value Swapping

This is a fun challenge often seen in data restructuring. We need to turn values into keys and keys into values, effectively flattening the structure into a single array of objects without using mapObject.

Input:

{
  "id": "file",
  "value": "File",
  "id": "file",
  "value": "File"
}

Target Output:

[
  {"file": "id"},
  {"File": "value"},
  {"file": "id"},
  {"File": "value"}
]

Solution 1: Mapping Keys and Values

%dw 2.0
output application/json
var keys = keysOf(payload)
var values = valuesOf(payload)
---
keys map(
(values[$$]) : $
)

Solution 2: Using Pluck

The pluck operator is perfect for iterating over objects to produce arrays.

%dw 2.0
output application/json
---
payload pluck(
($):$$
)

4. Flattening JSON to CSV

Nested objects (like an address block inside a user profile) usually break standard CSV exports. Here is how to flatten the structure to ensure a clean, single-row CSV output.

Input:

[
  {"id": 123,"name": "sample name","address": {"pin": 534460,"place": "ap","ph": 9848022338}},
  {"id": 1,"name": "sample name2","address": {"pin": 123,"place": "ts","ph": 98765453459}}
]

Target Output:

id,name,pin,place,ph
123,sample name,534460,ap,9848022338
1,sample name2,123,ts,98765453459

Solution 1: Merge and Filter

This approach merges the nested object with the parent and filters out nulls.

%dw 2.0
output application/csv
import * from dw::core::Objects
---
payload map(
$ match {
case y is Object -> ($ - "address") mergeWith $.address
else -> null
} filterObject !($$ ~= null)
)

Solution 2: Direct Concatenation

A simpler way using the concatenation operator ++.

%dw 2.0
output application/csv
---
payload map(
$ - "address" ++ $.address
)

5. Handling Duplicates

Data quality is huge in integration. Here are four ways to isolate only the duplicate values from an array.

Input:

[1,2,3,4,5,6,7,1,2,7]

Target Output:

[1,2,7]

Solution 1: GroupBy

%dw 2.0
output application/json
---
payload groupBy ($) filterObject ((value, key, index) -> sizeOf(value)>1) pluck($[0])

Solution 2: Skip Nulls

We use the skipNullOn directive to keep the output clean.

%dw 2.0
output application/json skipNullOn="everywhere"
---
payload groupBy ($) pluck(
if(sizeOf($)>1)$[0] else null
)

Solution 3: Filter and Distinct

%dw 2.0
output application/json
fun duplicateCheck(in) = (
sizeOf(payload filter($ ~= in))>1
)
---
payload filter (duplicateCheck($)) distinctBy ($)

Solution 4: Using Find

%dw 2.0
output application/json
---
payload filter(
sizeOf(payload find $)>1
) distinctBy($)

6. Finding Non-Duplicate (Unique) Values

Conversely, sometimes you need to find the items that appear exactly once, effectively filtering out any repeated noise.

Input:

[1,2,3,4,5,6,7,1,2,7]

Target Output:

[3,4,5,6]

Solution 1: IsEmpty Check

%dw 2.0
output application/json
---
(payload groupBy $ pluck(
("":$[0])if(sizeOf($)==1)
)filter(!isEmpty($))).""

Solution 2: Conditional Pluck

%dw 2.0
output application/json skipNullOn="everywhere"
---
payload groupBy ($) pluck(
if(sizeOf($)<=1) $[0] else null
)

Solution 3: Custom Function

%dw 2.0
output application/json
fun duplicateCheck(in) = (
sizeOf(payload filter($ ~= in))<=1
)
---
payload filter (duplicateCheck($)) distinctBy ($)

Solution 4: Using Find

%dw 2.0
output application/json
---
payload filter(
sizeOf(payload find $)<=1
) distinctBy($)

Solution 5: GroupBy and FilterObject

%dw 2.0
output application/json
---
payload groupBy ($) filterObject ((value, key, index) -> sizeOf(value)<=1) pluck($[0])

7. Reformatting and Shifting Keys

In this scenario, we clean up sparse objects and shift keys (e.g., from pi1 to pi0) to create a cleaner, sequential output.

Input:

[
  {"pi1": "jawan", "pi2": "", "pi3": "", "pi4": "wazir"},
  {"pi1": "", "pi2": "nawaz", "pi3": "", "pi4": "wazir"},
  {"pi1": "", "pi2": "", "pi3": "", "pi4": "wazir"}
]

Target Output:

[
  {"pi0": "jawan", "pi1": "wazir", "pi2": "", "pi3": ""},
  {"pi0": "nawaz", "pi1": "wazir", "pi2": "", "pi3": ""},
  {"pi0": "wazir", "pi1": "", "pi2": "", "pi3": ""}
]

Solution 1: Map and Filter

%dw 2.0
output application/json
import * from dw::core::Strings
---
payload map using(item = ($ pluck $)filter(!isEmpty($)))(
$ mapObject {
    ("pi"++$$$) : item[$$$] default ""
}
)

Solution 2: Using OrderBy

%dw 2.0
output application/json
---
payload map ((item, index) ->
(item orderBy -sizeOf($)) mapObject(
    ("pi" ++ $$$) : $
)
)

8. Merging Parallel Arrays

This is a classic Salesforce or database integration scenario. You receive separate arrays for IDs and Record Types, and you need to "zip" them into a single list of objects.

Input:

{
  "workTypeID": ["08q4W000000xcHWQAY","08q4W000000xcHWQAY","08q4W000000xcHWQAY","08q4W000000xcHWQAY"],
  "accountID": ["0014W00002p2Ui0QAE","0014W00002ohJsWQAU","0014W00002oraWRQAY","0014W00002orO6qQAE"],
  "recordTypeID": ["0124W000001d2huQAA","0124W000001d2huQAA","0124W000001d2huQAA","0124W000001d2huQAA"]
}

Target Output:

[
  {"workTypeID": "08q4W000000xcHWQAY", "accountID": "0014W00002p2Ui0QAE", "recordTypeID": "0124W000001d2huQAA"},
  {"workTypeID": "08q4W000000xcHWQAY", "accountID": "0014W00002ohJsWQAU", "recordTypeID": "0124W000001d2huQAA"},
  ...
]

Solution 1: Standard Map

%dw 2.0
output application/json
---
payload.workTypeID map{
workTypeID : $,
accountID: payload.accountID[$$],
recordTypeID : payload.recordTypeID[$$]
}

Solution 2: Count-based Map

%dw 2.0
output application/json
var itemCount = 1 to sizeOf((payload orderBy ((value, key) -> -sizeOf(value)))[0])
---
itemCount map(
{
   workTypeID : payload.workTypeID[$$],
    accountID : payload.accountID[$$],
    recordTypeID : payload.recordTypeID[$$]
}
)

9. Filtering and Grouping Students

Here we need to group data by student ID and return only the subjects where they passed, filtering out any failures.

Input:

[
  { "student": "s1", "maths": "pass", "sciences": "fail", "english": "pass" },
  ...
]

Target Output:

[
  { "s1": [ "pass", "pass" ] },
  ...
]

Solution 1: GroupBy and Pluck

%dw 2.0
output application/json
---
payload groupBy ($.student) pluck $ map(
($.student[0]): {($)} pluck $ filter($ ~= "pass")
)

Solution 2: FlatMap

%dw 2.0
output application/json
---
payload groupBy ($.student) pluck(
($$) : $ flatMap(valuesOf($ - "student")) filter($ ~= "pass")
)

Solution 3: ValuesOf

%dw 2.0
output application/json
---
payload map(
($.student) : valuesOf($ - "student") filter($ ~= "pass")
)

10. Consolidating Split Records

Finally, we have a scenario where a single entity (like "sai") is split across multiple objects containing different fields. We need to merge them into one complete record based on their ID.

Input:

[
  { "sid": 123, "sfname": "sai" },
  { "sid": 123, "slname": "kumar", "snum": 9087654132 }
]

Target Output:

[
  { "sid": 123, "sfname": "sai", "slname": "kumar", "snum": 9087654132 }
]

Solution 1: GroupBy and DistinctBy

%dw 2.0
output application/json
---
payload groupBy $.sid pluck $ map(
{($)} distinctBy ((item, index) -> item)
)

Solution 2: Custom Function

%dw 2.0
output application/json
fun getValues(id) = (
{(payload filter($.sid ~= id))} - "sid" distinctBy ($)
)
---
payload map(
{sid : $.sid} ++ (getValues($.sid) default {})
) distinctBy ($)

Solution 3: Concise Grouping

%dw 2.0
output application/json
---
payload groupBy ($.sid) pluck({($)} distinctBy ($))

Hopefully, these examples help you navigate the complexities of DataWeave transformations!

Tags:
Post a Comment (0)
Previous Post Next Post