“The state of confusion brought on
by the overwhelming challenge of the flow of data between disparate
applications both within and without your organization.”
There is a growing concern among organizations
of all sizes, that things may be “falling through the cracks” – why is this
so? From my experience, there is the
pressure from business to “just get it working” – which we do. Sometimes we have the luxury of a reasonable
budget, ample time to elicit full requirements, design the outcome and create
the solution… Sometimes…
But that is not always so, particularly
when things change AFTER the solution is in place, as the business tries to
slim down on operational costs to maintain profits.
This leads to a variety of bespoke
solutions. Have you ever seen any of the
following solutions?
¤
People create “macros” in tools
such as Excel to “manipulate” data into a desired format.
¤
Data is sent/received as
attachments in emails.
¤
Data from files/reports is entered
“by hand” into your systems of record.
¤
Information is sent to the
wrong party – which can be embarrassing or worse, breaching a commercial
confidence.
And when things go wrong, as they
invariably do, what happens? Sometimes
the consequences have minimal business impact, a simple re-sending of an email
may suffice. But what happens when the
excrement hits the air movement device?
There can be business costs, penalties, even loss of clients. Lawsuits, loss of confidence in your organization’s
products or services – the list goes on.
It is high time that organizations realize
that the flow of data – either within (between an organization’s applications)
and without (with your trading partners) needs to be managed and controlled in
a standard, reliable and robust manner.
There have been many tags given to this, one of which is the Enterprise
Services Bus or ESB. However many of
these types of solutions do not capture the “lower level” manual processes and
automate them.
Several popular tools purport to address
this problem space. Unfortunately, these
solutions become more complex than the problems they are attempting to resolve. The solution you require must allow Business
Analysts to create process flows without the need for programming.
Let’s take a look at the 4 key areas that
such a solution must provide…
1.
Transportation – by what means will the data move from application to application;
organization to organization?
2.
Transformation – what data format is required so the recipient can process it
without error?
3.
Orchestration – how will process flows with multiple steps/stages by managed, and
what happens if there is a failure?
4.
Administration – how will process flows by designed, created and deployed? Who can
update process flows? Who manages the operational aspects?
Transportation must be
able to handle modern methods such as RESTful web services as well as older
methods such as File Transfer (FTP), email (POP3, SMTP), etc.
Ideally, we should be able to create a
profile for each trading partner, or internal application, that defines the
method of transport they use. In this
way we can have multiple trading partners sending data using different methods
of transportation.
Transformation must
allow the business analyst to map the data format of the trading partner to
that of the receiving application. And
there are 2 aspects of this:
1.
The format of the data may be
different – for example the incoming data may be in an XML structure, but the
receiving application requires it in a table in a RDBMS (Relational Database).
2.
The actual incoming data
elements need to be mapped to their receiving counterpart. There may be other functions required to
manipulate the data element. There may
also be the need for data augmentation or enrichment.
All this needs to be done without
programming by Business Analysts.
Orchestration must
control the steps in a process flow.
This may mean that the flow needs to fork and perform multiple parallel
flows, or it may be a simple, sequential series of steps. It also needs to handle exceptions or error
conditions, when (not if) something goes awry.
Although not strictly Orchestration, the solution should – where possible
– allow a process flow to be resumed at the point it failed, once the issue has
been resolved.
Administration must
provide a management framework to secure the creation and amendment of process
flows. This includes the deployment into
Testing, Quality Assurance, User Acceptance Testing and Production
environments.
