Thursday, 2 July 2020

My Notes on: Mule 3 Fundamentals: Contents (on this Blog), Where Next, and Get Certified


The last 12 posts were my notes from having followed the (free) ‘Mule 3 Fundamentals’ course from https://training.mulesoft.com. I find it’s the best way to keep my attention on an online course, is by taking notes. Then I review the notes, see if they make sense, and - if they are worth recording - I’ll make the effort to post them to a blog (so if I forget there’s something to jog my memory). Note: If you’re wondering why this isn’t for Mule 4, my requirement was to learn about Mule 3.

Note: The full course title is: Anypoint Platform Development Fundamentals (Mule 3)

1) Contents (on this Blog)

Notes: Module 1: Mule 3 Fundamentals: Introducing API-Led Connectivity

Notes: Module 2: Mule 3 Fundamentals: Designing APIs

Notes: Module 3: Mule 3 Fundamentals: Building APIs

Notes: Module 4: Mule 3 Fundamentals: Deploying and Managing APIs

Notes: Module 5: Mule 3 Fundamentals: Accessing and Modifying Mule Messages

Notes: Module 6: Mule 3 Fundamentals: Structuring Mule Applications

Notes: Module 7: Mule 3 Fundamentals: Consuming Web Services

Notes: Module 8: Mule 3 Fundamentals: Handling Errors

Notes: Module 9: Mule 3 Fundamentals: Controlling Message Flow

Notes: Module 10: Mule 3 Fundamentals: Writing DataWeave Transformations

Notes: Module 11: Mule 3 Fundamentals: Connecting to Additional Resources

Notes: Module 12: Mule 3 Fundamentals: Processing Records (Batch)

2) Where to go from here?

Take Advanced Anypoint Platform Development:
- Managing Mule Development Projects with Maven
- Managing Mule code
- Achieving Continuous Integration
- Driving Development with MUnit
- Developing Custom Components
- Implementing Design Patterns
- Tuning Applications for Performance
- Working with State
- Securing Communications with SSL


3) Get certified!

MCD - Integration and API Associate (First Level)
MCD - Integration Professional (Second Level)
MCD - API Design Associate
MCD - Connector Specialist

Notes: Module 12: Mule 3 Fundamentals: Processing Records (Batch)



Objectives:
- Use the For Each scope to process items in a collection individually
- Use the batch job element (EE) to process individual records
- Trigger a batch job using a poll
- Use a batch job to synchronize data from a legacy database to a SaaS application

12-1) Processing items in a collection

Either:
- Create a flow that uses:
--- A splitter-aggregator pair
--- A For Each scope - splits a message collection and processes the individual elements and then returns the original message.
- Use a batch job (Enterprise Edition) - it is not a flow but another top level element.

Walkthrough 12-1: Process items in a collection
- Use the For Each scope element to process each item in a collection individually

Image: Using the For Each scope element

Note: Logger has Message: #[payload]

12-2) Processing records with the batch job element

Provides ability to split large messages into records that are processed asynchronously in a batch job!

Batch:
Input phase ---> Processing phase(s) ---> Report phase
Reporting phase lets you know if something went wrong.

Example use cases:
- Engineering “near real-time” data integration
- ETL (Extract Transform and Load)

Batch jobs:
- Accept data from an external source (can poll for input)
- Split messages into individual records
- Report on results (can push output)

Image: Batch scope element phases: Input, Process Records, On Complete

Triggering batch jobs:
1) Place an inbound, one-way message source at the beginning of the job
2) Use Batch Execute message processor to reference the batch job from within a Mule flow in the same application.

Phases of a batch job:
- Input (optional)
- Load and dispatch (implicit) - splits payload into a collection of records and creates a queue.
- Process (required)
- On complete (optional) - report

Note: A batch job instance does not wait for all its queued records to finish processing in one batch step before pushing any of them to the next batch step.

Image: To store record-specific information, use Record Variable

Handling record-level errors during processing:
batch:job name= "Batch1" max-failed-records="?"
0: Stop processing the entire batch (default)
-1: Continue processing the batch
INT: Continue processing until max number of failed records reached

Walkthrough 12-2: Create a batch job for records in a file
- Create a batch job
- In the input phase, check for CSV files every second and convert to a collection of objects
- In the process records phase, create two batch steps for setting and tracking variables
- In the on complete phase, look at the # of records processed and failed

Image: Create a batch job for records in a file

12-3) Using a batch job to synchronize data

Checking for duplicate records - use an Enricher!

Configure Message Enricher:
- Specify the message enricher source and target:
--- The target specifies which part of the message to modify
--- The source specifies what to set the target to (default = payload)

Image: How message enricher works

Add logic to only insert new records.
Check if record exists
Source: #[payload.size() > 0]
Target: #[recordVars.exists]

Then use Batch Step Filters to restrict records to be processed. Accept policies:
- ALL
- NO_FAILURES
- FAILURES_ONLY

Accept expression: #[!recordVars.exists]
Accept Policy: NO_FAILURES (Default)

Walkthrough 12-3: Restrict processing using a message enricher and batch step filter
- Create a batch job that polls a database for records with a specific postal code
- Use a message encricher to check if a record already exists in Salesforce and stores the result in a record variable (retaining the original payload)
- Add a second batch step with a filter that only allows new records to be added to Salesforce
- Use a batch commit scope to commit records in batches

Image: Restrict processing using a message enricher and batch step filter

Notes: Module 11: Mule 3 Fundamentals: Connecting to Additional Resources



Objectives:
- Connect to SaaS applications
- Connect to files
- Poll resources
- Connect to JMS queues
- Discover and install connectors not bundled with Anypoint Studio

11-1) Connecting to SaaS Applications (Salesforce)


Walkthrough 11-1: Connect to a Saas application (Salesforce)
- Use the Salesforce connector to retrieve accounts for a postal code
- Use the Query Builder to write a query

Image: Add Saleforce connector to canvas and Salesforce libraries get added too (also added to POM file in Maven)

Many operations are available with the Salesforce connector.

11-2) The File Connector

Gives Mule applications the ability to read/write files in the local file system:
- Read input files
- Create new files
- Copy (backup) files
- Append to existing files
- Read periodically and delete/move/leave once processed

By default it uses streaming:
- Payload is a FileInputStream
- Streams are closed by transformers reading the input stream

Can turn off streaming:
- Payload is a byte array

Connector configuration:
- Not required (standalone)
- Use for reusability and setting properties

File connector as outbound endpoint:
- Passes files to the connected file system
- File name can be set at runtime

Walkthrough 11-2: Connect to a file (CSV)
- Add and configure a File endpoint to watch an input directory, read the contents of any added files, and then rename and move the files
- Use DataWeave to convert a CSV file to a string
- Add a CSV file to the input directory and watch it renamed and moved
- Restrict the type of file read
- Add payload metadata to a file endpoint

We have an input and output folder under:

src/main/resources
    input
    output

Image: Configuring the File connector

Use regex to search for just CSV say.
Use ‘Move to Pattern’ to rename with say .backup.
Use Transform.

11-3) Polling Resources

Some connectors use or can use a polling process to actively retrieve messages e.g. File, FTP, SMTP.
For other message processors, use a Poll scope element to actively call a resource at regular intervals.

Image: Polling Information (Frequency is milliseconds)

Use a watermark to only retrieve newly created or updated data.
- The value must persist across flows:
--- Mule uses a build-in object store for persistent storage and exposes the value as a flow variable
----- Saved to file for embedded Mule and standalone Mule runtime
----- Saved to data storage for CloudHub
----- Saved to shared distributed memory for clustered Mule runtimes

Walkthrough 11-3: Poll a resource
- Create a flow with a Poll scope as the message source
- Poll a database every 5 seconds for records with a specific postal code
- Use a poll watermark to track the ID of the latest record retrieved
- Use the watermark to only retrieve new records with that postal code

Image: Configuring Poll Scope and Watermark

Image: Configuring pollDatabaseFlow

Image: Prompting to clear the watermark

11-4) Connecting to JMS (Java Messaging Service) Queues

JMS is an API for enabling applications to communicate through the exchange of messages, it provides a standard interface for creating/sending/receiving messages. Supports two messaging models:

1) Queues: PTP (point-to-point)
One-to-one! A sender delivers messages to a queue and a single receiver pulls the message off. The receiver does not need to be listening at the time the message is sent.

2) Topics: Pub-Sub (publish/subscribe)
One-to-many! A publisher sends a message to a topic and all active subscribers of the topic receive the message. Subscribers not actively listening will miss the published message (unless messages are made durable).

The Mule JMS connector can connect to any JMS messaging service that implements JMS spec.

Supported JMS providers:
- Out-of-the-box support for ActiveMQ and WebLogic JMS.
- Others (see the documentation) are supported by a generic JMS or custom JMS configuration.

Walkthrough 11-4: Connect to a JMS queue (ActiveMQ)
- Create a flow accessible at ‘http://localhost:8081/jms’
- Add and configure an Active MQ connector
- Use a JMS endpoint to retrieve messages from a JMS topic
- Add messages to the topic using a web form
- Use a JMS endpoint to send messages to a JMS topic

Image: Connect to a JMS queue

General recommendation: do not use the ALL jar library; use just what you need in your application (to avoid bloat).

Image: Configuration of JMS connectors

Configuration of Logger: message: #[payload]
Configuration of Set Payload: value:
#[message.InboundProperties.’http.query.params’.name] : [message.InboundProperties.’http.query.params’.messageBody]

Image: Using the Mule JMS HTTP app in Postman

11-5) Discovering and using additional connectors

Anypoint Exchange:

11-6) Introducing Anypoint Connector DevKit for creating custom connectors

You can create custom connectors with Anypoint Studio and Anypoint Connector DevKit.
Note: Use Maven for development and building.

Notes: Module 10: Mule 3 Fundamentals: Writing DataWeave Transformations


DataWeave is a JSON-like language.

Objectives include:
- Store DataWeave transformations in external files
- Coerce and format strings, numbers, and dates
- Call MEL functions and Mule flows from DataWeave transformations

10-1) Introducing DataWeave Transformations

DataWeave transformation expressions:
- The header contains directives - high level info about the transformation
- The body contains a DataWeave expression that generates the output structure.

Data model of the produced output can consist of 3 different types of data:
- Objects: As a collection of key value pairs
- Arrays: As a sequence of comma separated values
- Simple literals

Output Directive - sets output type of transformation:
- Specified using content/type
- Structure defined in DataWeave body

Example:
%dw 1.0
%output application/xml
---
{
  a: payload
}

MIME type: You may get an error unless the MIME type for the input data has been set.

The code can be placed in your XML as a CDATA type. Sample data is stored in src/test/resources as JSON.

Image: DataWeave code in XML

Reusing transformation:
- Click ‘Edit current target’ and set source code to file. Saved as ‘.DWL’ in ‘src/main/resources’.
- To reference an existing DWL, specify in the XML:
set-payload resource="classpath:user_transform.dwl"

Wakthrough 10-1: Write your first DataWeave transformation
- Create a new flow that receives POST requests
- Write a DataWeave expression to transform the data to Java, XML, or JSON
- Save the transformation in an external file

10-2) Transforming Basic Data Structures

Writing Expressions for JSON or Java input and output

+----------------------+
| 1) INPUT             |
+----------------------+
| {                    |
|   "firstname":"Max", |
|   "lastname":"Mule"  |
| }                    |
+----------------------+

+-----------------------------+-----------------------------|
| 2) TRANSFORM                | 3) OUTPUT                   |
+-----------------------------+-----------------------------|
| fname: payload.firstname    | {"fname": "Max"}            |
+-----------------------------+-----------------------------|
| {fname: payload.firstname}  | {"fname": "Max"}            |
+-----------------------------+-----------------------------|
| user: {                     | {"user": {                  |
|  fname: payload.firstname,  |   "fname": "Max",           |
|  lname: payload.lastname,   |   "lname": "Mule",          |
|  num: 1                     |   "num": "1"                |
| }                           | }}                          |
+-----------------------------+-----------------------------+
| [                           | [                           |
|  {fname: payload.firstname, |  {"fname": "Max","num": 1}, |
|   num: 1},                  |  {"lname": "Mule","num": 2} |
|  {lname: payload.lastname,  | ]                           |
|   num: 2}                   |                             |
| ]                           |                             |
+-----------------------------+-----------------------------+

Writing Expressions for XML Output

XML can only have one top-level value and that value must be an object with one property.

Image: Writing expressions for XML output

Image: Writing expressions for XML input (use @ to reference attributes)

Walkthrough 10-2: Transform basic Java, JSON, and XML data structures
- Write expressions to transform the JSON payload to various Java structures,
- Create a second transformation to store a transformation output in a flow variable.
- Write expressions to transform the JSON payload to various XML structures.

Image: Transform: Toggle between Payload (Java) and FlowVar - XML

10-3) Transforming complex data structures with arrays

The map operator (working with collections):
- To apply a transformation to each element in a collection (JSON or Java Array, or XML repeated elements).
- Applied to each element in an array or each value in an object.
- Returns an array of elements.

Image: Example: Using payload map
$$ refers to index (or key)
$ refers to the value

Image: Example: To set the index as a new key surround it with () or '' (2 x ')

Walkthrough 10-3: Transform complex data structures
- Create a new flow that receives POST requests of a JSON array of objects
- Transform a JSON array of objects to Java

Image: Transform Example

10-4) Transforming complex XML data structures

Image: Example: When mapping array elements (JSON or JAVA) to XML, wrap the map operation in {(...)}
{} are defining the object
() are transforming each element in the array as a key/value pair

Image: Example (continued I)

Image: Example (continued II): Use * to reference repeated elements

Warning: Make sure you look at the raw response and not the “pretty” response.

Walkthrough 10-4: Transform to and from XML with repeated elements
- Transform a JSON array of objects to XML
- Transform XML with repeated elements to Java

Image: Transform application/xml (Example 1)

Image: Transform application/xml (Example 2)

Image: Transform application/java (Example 3)

10-5) Formatting and Coercing Data

DataWeave has extensive documentation:
‘https://docs.mulesoft.com/mule-user-guide/v/3.?/dataweave’ (especially checkout: ‘type conversion table’.)

Using the ‘as’ operator for type coercion:
price: payload.price as :number
price: $.price as :number {class: "java.lang.Double"},

Defined types include:  string, number, Boolean, object, array, date, time, timezone, datetime, localdatetime, period, regex, and more...

Use metadata format schema property to format numbers and dates:
someDate as :datetime {format: "yyyyMMddHHmm"}

Walkthrough 10-5: Coerce and format strings, numbers, and dates
- Coerce data types
- Format strings, numbers, and dates

Note: format: "###.##" (for 3 digits and 2 decimal places.). Use "###.00" to give everything without decimals .00.

Image: Formatting Example

Image: Date Formatting Example (date formatted as string is YYYY-MM-DD)

10-6) Using DataWeave operators

Conditional logic operators:
default
when
unless
otherwise
==
!=
~= (equal regardless of type)

Additional operators (check out the documentation):
flatten, orderBy, concat, distinctBy, groupBy, replace, matches, regex...

Walkthrough 10-6: Use DataWeave operators
- Replace data values using pattern matching
- Order data, remove duplicate data, and filter data

Replacing BOING with BOEING:
upper $.planeType replace /(BOING)/ with "BOEING",

Order by price and date (date inside price), just distinct array items, and remove flights with 0 seats free:
} orderBy $.date orderBy $.price distinctBy $
   filter ($.seats != 0)

Image: The DataWeave expression

10-7) Using Custom Data Types

Use type header directive:
- name has to be all lowercase letters

Example: Specifying custom data types
%dw 1.0
%output application/json
%type ourdateformat = :datetime {format: "yyyMMddHHmm"}
---
someDate: payload.departureDate as :ourdateformat

Transforming objects to POJOs:
- Use:
as :object
- Specify inline:
customer:payload.user as :object {class: "my.company.user"}
- Or define a custom data type to use:
%type user = :object {class: "my.company.user"}
customer:payload.user as :user

Walkthrough 10-7: Define and use custom data types
- Define and use custom data types
- Transform objects to POJOs

Image: Example using type (compare with DW expression above)

Image: Example using custom data type

10-8) Referencing other data besides the payload

i) Reference message variables and properties:
flowVars, inboundProperties, outboundProperties, p (System or Spring properties)
Example: {n: flowVars.username}
Note: Do not preface with "message." or use #[]

ii) Call global MEL functions from DataWeave:
{"foo " : newUser(),
 "bar ": upperName(newUser())}

Image: Defining global MEL function in the configuration element you used to specify a default global exception handler

iii) Call other Mule flows
- Whatever the flow returns is what the expression returns
{a: lookup("mySecondFlow",{b: "Hello"}) }
- The 1st argument is the name of the flow to be called
- The 2nd argument is the payload to send to the flow as a map

Walkthrough 10-8: Call MEL functions and other flows
- Define a global MEL function in global.xml
- Call a global MEL function in a DataWeave expression
- Call a flow in a DataWeave expression

Image: Define a global MEL function in global.xml (Note: Should have a ; after return 300)


Image: Call a global MEL function in a DataWeave expression

Image: Or, with a new flow ‘getTotalSeatsFlow’ made up of an expression

Image: The POST in Postman