KNIME is an open source data analytics product based in Zurich, Switzerland that I last wrote about a couple of years ago. They have been working away on the product since then (having started development in 2004 and released their enterprise components in 2010) and have been refining their business plan at the same time. They now have 9,000 registered users/organizations and over 500,000 downloads. About 50% of these are in life sciences and 50% in BI and analytics – this represents a significant increase in non-life sciences usage. As an open source product they have 50 or more very active community developers and the company itself is 15 people (about half are at the University of Konstanz) and describes itself as “small but profitable”. Since I last spoke with them they have added a number of enterprise components and continued to develop the core open source platform.

The platform is based on Eclipse and has, in particular, been extended with lots of new node types. Like most modern analytic workbenches KNIME allows a modeler to define the sequence of steps to produce a model. The various actions that can be performed as part of the modeling process is determined by the available node types – nodes to retrieve data, process and transform that data, apply modeling algorithms (both established and more esoteric) or output results.

In addition a new decision tree viewer has been developed and data support (various databases, SAS and SPSS, and other flat files) has been improved. KNIME supports PMML generation for models including the most frequently used preprocessing operations required for a model – this was added to PMML in 4.0 and KNIME is one of if not the only vendor completely supporting this. There are lots of nodes for pre-processing and many algorithms including some of their own, some from the R Project, others from Revolution Analytics and Weka as well as lots of new visualizations.

One of KNIME’s strengths is a well defined node interface that is nicely self-contained allowing easy integration of new node types without destabilizing the whole environment. KNIME makes it easy to add nodes so that community members and partners can extend the tool with their own node types. A community site manages the community submissions and makes it easy to add any you want. Data management and execution control are also handled through well defined interfaces allowing people like Pervasive DataRush to add their own integrations.

Because KNIME is built in Eclipse they also integrate with other Eclipse projects like BIRT (an open source reporting framework). This allows these open source reporting components to be added to KNIME workflows.

Improvements and new node types are often sponsored by companies that use the product, though KNIME continues to push its own development plan. KNIME is also an integration platform for many vendors with 15 technology vendors integrating with KNIME, especially in the life sciences environment where life science companies are paying their technology vendors to integrate with KNIME to create a single environment.

The KNIME workbench is open source with registration desired but not required. This open source client version is a complete environment. Access, transformation statistics and data mining, visualization, reporting and workflow is all available in the open source with 1,000+ native and embedded nodes including text processing, social network analytics, image processing and more. Companies can add support to the open source project or adopt the server or enterprise components that are proprietary and licensed.

KNIME Team Space allows a small group to have a shared repository. KNIME Server adds the full server-based capabilities including remote and scheduled execution of modeling flows, a workflow repository, shared data in a managed repository, shared metanodes or component workflows for reuse, and web browser access. The browser interface exposes workflows to users who can enter the parameters defined for the workflow and get a result displayed back in the browser. An API also allows “headless” execution of workflows. With the full suite companies can write new nodes and wrap legacy software, then allow power users to develop templates and shared workflows, provide this to scientists or others who adapt and use these workflows as well as managers who just run pre-configured ones.

The company is shifting from selling directly to life science and a small number of other companies to an approach based on partner channels with domain expertise. Target markets are obviously life science as well as data mining/predictive analytics (customer intelligence especially). KNIME’s value proposition is in its openness as well as the ability to reduce the cost for existing heavy users of analytic workbenches while also targeting smaller companies that can’t afford the large vendors.

Besides the partnerships with Pervasive and BIRT noted, KNIME has a strong partnership with Zementis for PMML execution in database and on servers and with Dymatrix, a Europe-based partner offering model monitoring and automated updates.

KNIME will be one of the vendors listed in the forthcoming report on Decision Management Systems platform technologies.

I am giving the closing keynote on day 1 of Predictive Analytics and Business Insights 2012 in San Francisco on February 9th at 3:30pm. I will be speaking on “The Intelligent Enterprise: Utilize decision management to move from Business Intelligence to Predictive Analytics” and attending the networking reception afterwards. Other speakers at the event include Saum Mathur – Vice President, Information Technology, Global Business Intelligence at Hewlett-Packard, George Roumeliotis - Leadership for Advanced Analytics and New Product Development at Intuit, and Nurtekin Savas – Director of Strategy & Analytics at Visa.

You can get details on the event here.

In a recent article over on All Analytics – Analytics-Business Alignment Needs Work – Beth Schultz discussed a set of Gartner predictions written up by Doug Laney. In particular she highlights Gartner’s finding that companies

cited …aligning BI initiatives with corporate strategy and objectives nearly three times as often as they called out technology-related issues

This is a problem I see regularly across both BI and other analytics efforts. I think the primary driver of this is an abject under-definition of the decisions that BI and analytics are supposed to be enabling/improving. We model and manage business processes and the data they need. We don’t model or manage the decisions within those processes nor consider their impact on the business. I call this process Decision Discovery and wrote about it extensively in Decision Management Systems: A Practical Guide to using Business Rules and Predictive Analytics - the whole of chapter 5 is devoted to this critical first step. I wrote a quick overview recently about the use of  decision discovery to gather requirements for advanced analytics and my experience is that it also works for driving successful BI efforts. Organizations that understand the decisions involved, how they relate to each other, what information and know-how is needed for each decision and how decision decompose into smaller, more focused ones are more likely to see where analytics will make a difference. By evaluating the impact each decision has on business objectives and key performance indicators, the alignment of analytic efforts to corporate objectives can be achieved – they are linked by the modeled decisions.

In addition, Doug is quoted as saying

Too often we observe organizations developing and deploying hindsight-oriented reports and/or query applications focusing on metrics that users may find interesting, but that don’t represent the operational or strategic controls used to facilitate business performance.

Once again the linkage of decisions to objectives helps as this allows the analytics to be prioritized and focused on assisting with those decisions that really matter to the key objectives of the organization.

In his recommendations Doug has one that particularly resonated with me:

…a top-down view of metrics, which should determine and define your tactics

And this top-down view of metrics should be combined with a business-centric, top down view of decisions.

I also liked

Make experimentation and closed-loop implementations standard practice

As the third step in my approach is always what I call Decision Analysis – the ongoing process of improvement, test and  learn and decision performance monitoring.

Decision Management, and the discovery and modeling of decisions in particular, really drives business/analytics alignment.

If you are interested in how to go about decision discovery or what to build you own decision inventory, drop us a line info@decisionmanagementsolutions.com and we will see if we can help. You might also want to check out my forthcoming workshop on Business Friendly Data Mining at Predictive Analytics World.

Quiterian is a Spanish company with offices in the US, Mexico and Europe. Quiterian Analytics aims to be complementary to traditional tools for reporting by helping companies get more value from their data sooner. In particular they aim to help companies anticipate the future by providing simple to use predictive analytics and by empowering users while reducing IT costs. They see the BI market in three broad segments:

• Dashboards and reports – traditional BI tools as well as some of the newer, easier to use visualization tools – with little or no focus on advanced analytics
• Data mining tools focused on professional data miners and statisticians, typically not terribly visual.
• Visual Data Mining tools that site between the two aimed at analysts and business users with no dependency on IT or data miners. This is where Quiterian is targeted

They offer fast and comprehensive data analysis and visualization, self-service for non-technical users and aim to be very agile with no need to pre-create cubes etc. Specifically Quiterian Analytics is a web-based SaaS tool that includes functions for

• Its own analytical database that is a columnar/in memory hybrid
• Data acquisition, exploration and engineering
• Advanced analytics
• Distributing and sharing findings
• Workflow

Users can identify various data sources and load them up using Quiterian’s ETL. Drag and drop exploration tools allow users to profile the data, examine the underlying records etc. Charts of distributions, various statistical measures and frequency distribution can be displayed easily. Subsets can be selected from the graphs and everything recalculated based on those subsets.

Pivot tables, venn diagrams, bubble charts and more can be used to see what is going on in the data. For instance the customers who don’t have a credit card but do have a current account and don’t have a credit card from another company. These analyses can be saved, exported and shared in a variety of ways.

Expressions, calculated values, can be defined and used in analysis. Users can also use more advanced tools such as aggregates across rows, tools to manage quartiles/deciles, or tools to apply Pareto distributions to see what percentage of profit comes from a certain percentage of customers for instance. Profiling tools can be used to assess correlation, for instance by identifying which attributes are strongly correlated with profitability. A profile can then be applied to a group based on a previous query or analysis. New selections can be based on these profile attributes and pivot tables and other kinds of analysis can build on these selections.

In addition the tool supports decision trees, clustering and forecasting for more advanced analytics. The decision tree data mining algorithm has some nice usability features with color coding, accuracy measures etc. This can then be saved as an algorithm and be applied to any set of records selected in the tool using any of the other elements.

Finally the workflow component allows tasks within the tool to be scheduled. You can schedule actions to take when a new model is created, as part of a scheduled campaign, when specific technical changes are made or just at a particular time. These tasks can apply models or algorithms, send emails etc. Quiterian is working on adding some specific action types to expand this e.g. invoking actions in salesforce.com or email marketing tools as well as twitter, SAP, Omniture and others. As the range of actions expands, the ability to embed these analyses into decisions will likewise expand. A PMML import/export capability is also in the works.

Talend has been around for about 6 years and the original focus was on “democratizing” data integration – making it cheaper, easier, quicker and less maintenance-heavy. They originally wanted to build an open source alternative for data integration. In particular they wanted to make sure that there was a product that worked for smaller companies and smaller projects, not just for large data warehouse efforts.

Talend has 400 employees in 8 countries and 2,500 paying customers for their Enterprise product. Talend uses an “open core” philosophy where the core product is open source and the enterprise version wraps around this as a paid product. They have expanded from pure data integration into a broader platform with data quality and MDM and a year ago they acquired an open source ESB vendor and earlier this year released a Talend branded version of this ESB.

Talend 5 is a “holistic integration platform” with a newly announced OEM of BonitaSoft (they were already using the workflow capabilities of BonitaSoft in their MDM product). Talend 5 now embeds the complete Boniasoft product. This gives them data integration (historical core), process integration (BonitaSoft) and application integration (ESB)

There are 5 elements to the core Talend platform

• Common design environment across all three elements, including the BonitaSoft elements being OEMed
• Shared project repository for artifacts and metadata
• Single deployment approach
• Single runtime environment
• Single monitoring and management platform

A single platform really helps with projects that combine multiple elements (data integration and BPM say) – Talend try to make it easy for organizations to adopt additional elements by reducing the learning curve through common tooling etc.  Nevertheless Talend remains committed to providing best of breed elements in each case. The product supports the increasing array of cloud and hybrid-cloud deployments with particularly strong support for hybrid clouds. They support “Big Data” quality and integration tools that run inside Hadoop and allow companies to handle this data the same way they would handle data stored elsewhere. They also embed the JBoss Rules engine and JBoss Guvnor for rule management. This obviously allows for both simple business rules on the integration and data quality side and more complex embedded decision-making.

Good decision-making relies on data being delivered to the point of decision. An approach to data integration that focuses on the use of data for a specific solution while allowing a broad-based, coherent integration architecture to be developed incrementally feels like the right approach to data integration in a decision management context.

I got a quick update on Spotfire 4.0 recently (announced today – I last reviewed Spotfire in February). This release was aimed primarily at getting analytics to a wider audience through analytic dashboards and social collaboration. As we all know, many people work with others in different locations and time zones. This requires strong context for understanding analysis, multi-channel support and an ability to collaborate. Spotfire has historically been focused on helping individuals to apply their own skills to an analysis and this release makes it easier for organizations to add their collective intelligence. Specifically:

• It is easier now to take a Spotfire analysis and embed it in a SharePoint or other portals/websites.
• You can also embed other content into the analysis such as wikis.
• Social integration is through integration with tibbr (TIBCO’s social platform for the enterprise that allows chatting, threaded conversations, multi-channel support, including apps for various smart devices, and is geared to following a subject rather than just a person).
• You can share analyses from Spotfire through tibbr and discuss it or open it in Spotfire for further analysis.
• Companies have been using Spotfire for dashboards but this release improves the presentation of data to make it crisper and quicker to get understanding – making it match the expectations of a dashboard user who is going to glance/skim while still supporting the interactivity when needed. You create new visualizations specifically designed for display in a Web browser to make it look like a dashboard while still allowing access to Spotfire.
• The release also focuses on continuing to improve the environment for data discovery as well as for sharing analysis.

I found a couple of things interesting in this release, even though collaborative decision-making is not really my thing.

Being able to present analysis and drill-down in the context of an application or Web page is always more interesting to me than a standalone application. This kind of embedding allows a Decision Management System to embed a visualization in a transaction when it is referred to someone for audit review or because it cannot be handled automatically by the system. Combined with Spotfire’s existing automation services for creating analyses programmatically it would allow a Decision Management System to generate a useful visualization for every transaction that required manual review.

Some Decision Management Systems never make a final decision but are focused instead on reducing the degrees of freedom of a human decision-maker to just those that make sense for a specific decision. The ability to present a generated visualization to help the user make the final decision is also interesting and would be improved by this release.

The ability to show data in the collaboration view that changes as the visualization is interacted with would allow, for instance, the rule logs for specific decisions to be reviewed as you analyzed your overall decision results. This could be very useful in a decision analysis context. You could also present this in the context of a results dashboard and that would help folks using a Decision Management System see how they got the results they got. You could presumably also allow the tibbr thread to be accessed both from the results view and from a rule editing Web page that allowed the rules behind the results to be added, allowing collaboration between those reviewing the results and those working on the rules.

Finally, I think this release would make it easier to develop some analysis (using S+ say) that got embedded into a Decision Management System as a predictive analytic model or a set of rules and re-use that analysis when a transaction is referred out of the system for manual review. When you can’t automatically decide you want to be able to present the same analysis (customer segmentation, risk) but using a visualization not a score. This release makes this much easier.

Deepak Advani kicked off the first day’s analytics keynote by noting the number of different analytics communities that are now part of IBM – Cognos, SPSS, Algorithmics and more. Analytics, he says, is a broad change that is just beginning and will impact everyone. Rob Ashe followed and made the point that the opening keynote of IOD was focused on analytics almost to the exclusion of anything else. Analytics, he says, is moving from a vendor-push to a customer-pull phase, with demand for analytics growing rapidly in customers of every type as business becomes increasingly information-based. Analytics is driving change in companies of every size even though, in some ways, the analytics industry is just getting started. Rob identified three areas he sees as core focus areas:

• Demand for agile analytics
  Complementing large enterprise deployments with flexible tools for knowledge workers and managers so they can quickly access and analyze data.
• Broad spectrum of analytics
  IBM’s focus is on platforms so they want to make sure that everything they do comes together in a scalable enterprise-suitable platform.
• Wild west of big data
  As more data explodes into companies it cannot be ignored but it is real-time, personal/social, high volume and unstructured making it a governance and integration channel.

He presented 3+3: 3 focus areas of Big Data analytics, Decision Management and personal analytics across three areas – customer analytics, finance analytics and risk analytics. Great to see Decision Management get such a strong focus and to see both customer and risk analytics showing strongly.

Eric Yau came up to present the details. IBM divides analytics into a platform, capabilities and solutions:

• Platform – access all your data sources, model against this data, manage everything and deploy it. IBM handles meta data, in-memory and other technical issues of analytics at this level
• Capabilities – reporting, dashboards, scorecards, predictive analytics, real-time monitoring, planning and forecasting
• Solutions – around customers, finance, risk and industry verticals

Eric was joined by Cincinnati Zoo on stage who have a great story of using business intelligence to understand and then improve the operations of the zoo.

Going forward IBM say they are making a series of investments. In the platform layer this includes Cognos 10 on the cloud, integration with content management, in-memory optimization and real-time decisioning (SPSS Decision Management). Capability investments include mobility with a new iPad application, business insight workspace with a unified environment for BI, in-memory analytic server and predictive analytics where SPSS Modeler can be easily connected to the same data sources used by Cognos.

Jeff Jonas‘ great ad for analytics, talking about crossing a road as an analogy for the power of predictive analytics, kicked off the first real content of the keynote. Jeff talked about enterprise amnesia, the challenge of enterprises being unable to process all the data they have flowing in. Jeff uses a puzzle metaphor and argues that companies don’t know even which puzzles they have pieces for, whether they have duplicates or missing pieces. He had a wonderful story of a group of kids working with pieces from multiple puzzles and how they worked through it. He uses this to describe the importance of context and the fact that context accumulates gradually though not in a linear way. Big Data in this view helps drive this improves context and makes it easier to find problems and assemble/use the data you have. Enterprise Intelligence involves not only building this context but also deciding, sometimes in less than 200ms sometimes by reaching out to a decision-maker. But this data also drives longer term “deep reflection”. “Sense and Respond” and “Explore and Reflect”. You need streaming data handlers, analytic appliances to process lots of data for deep insights, BI tools for people and Decision Management Systems based on rules and embedded analytics.

Sarah Diamond came up next to talk about the financial services industry. “Transformation is the new normal” in this industry and this requires a new approach to analytics and transparency. SunTrust came up to follow up on this to talk about their journey to a new approach to risk analytics. They went from multiple redundant areas to standardized reporting, from Excel and Access to an integrated warehouse infrastructure, from paper reports to portals. IT support is up, inconsistencies are down and data is managed more effectively. A massively more efficient BI environment for sure that helped SunTrust keep better track of what was going on as the markets went into crisis. Personally I think what they did is PART of the solution for risk management as they improved their ability to MONITOR risk but not manage it as operational decisions were made. It’s a beginning, a foundation, not an end.

The efforts SunTrust have made to make analytics pervasive are impressive but they seem overly focused on presenting data to people using BI tools. Personally I don’t see that as a route to pervasive analytics – only embedding analytics in Decision Management Systems that anyone/everyone can use will make analytic decision-making truly pervasive. A teller is just not going to have the time or skills to use analytics the way the knowledge workers and risk managers in the organization do. SunTrust are trying to embed analytics in their workflow and processes but it still seems focused on presenting it to people not empowering the systems themselves to make good decisions. Their closing point though, that analytics must be embedded in operational processes to be successful, is the right lesson.

Some great customer stories for predictive analytics followed with strong ROI but too little detail on how these analytics embedded into the operational processes and systems that drive the business.

Leaders, Mike Rhodin of IBM said, are those who can embed analytics throughout their business. They need to do this despite exploding data volumes and new sources of data. Meanwhile change continues to be a challenge and the gap between those making the most of analytics and their competitors continues to grow. Companies need systems that are agile (to respond to change), analytic (to make the most of their data) and adaptive (to maximize value by learning over time) – they need Decision Management Systems! As Mike said, you need to deploy analytics into their business processes to optimize outcomes. I don’t think this is mostly about getting analytic tools into the hands of people throughout the organization so much as getting analytic decision-making to everyone (which is not quite the same). Too many examples of putting reports and dashboards into the hands of people and of using predictive analytics only to route transactions to the right people rather than actually having systems that ACT on their behalf. Taking control of your information is good anticipating and shaping outcomes is good too but the critical step too many people miss is turning this insight into ACTION.

Robert LeBlance talked about IBM’s view of a sufficiently “bold” information strategy. It mus allow you to take unstructured content and structured data and bring it all together with integrated, federated, real-time management and allow for Big Data which means that external sources are as important as internal ones. He did not talk about being bold enough to embed analytics in your systems which is a pity.

Arvind Krishna came up to talk about some new IBM announcements:

• DB2 Analytics Accelerator (using Netezza) and IMS 12 for improved performance
• Content and predictive analytics for Healthcare and IBM Case Manager 5.1 for medical insights and advanced case management
• InfoSphere MDM – a unified MDM platform and new capabilities for integration and governance for big data sources
• iPad support for Cognos and new personal analytics tool
• Algorithmics now part of IBM for risk monitoring and analysis
• Q1 Labs is going to be acquired for security management
• InfoSphere BigInsights and Streams for handling streaming “big” data
• New Business Analytics and Optimization Jumpstart services

Lots of good stories, lots of good advice but not enough on how all this actually hits the front-line systems and people – not enough on Decision Management Systems and operational decision making.

Brian McDonough of IDC talked about the drivers for pervasive BI in organizations. He identified various metrics for measuring pervasiveness. Each of these could be assessed (from beginner to expert for instance) to see how well an organization is doing.

• Data Update Frequency
• Percent of Power Users
• Degree of Internal Use
• Degree of External Use
• Analytical Orientation
• Number of Domains

IDC found that all of these were worth measuring to see how pervasive your BI/analytics deployment is. In addition they researched what management could do to influence analytical orientation:

1. Training on KPIs, analytics and tools
2. Solution design quality
3. Data governance
4. Non-executive management involvement
5. Performance management methodology

These areas had large differences between leaders and laggards – for instance 75% of leaders had a formal performance management methodology where only 43% of laggards did.

Brian showed the IDC Decision Management model (blogged about here). The degree of automation and the kinds of analytics need to be considered based on the degree of collaboration involved in a decision, how often the decision must be made and so on.

Sadly I couldn’t blog Brian live (no wifi) and this was my last session of the conference.

Don Campbell, CTO Of IBM’s Business Intelligence group, presented on trends in smarter business analytics. He sees four focus areas – improving customer understanding, optimizing real-time decisions, better enterprise visibility and improved collaboration. All underpinned by managed, trusted data. IBM’s customers tell them that the 3 big challenges are

• A lack of understanding of how to use analytics
• A lack of management bandwidth
• A lack of skills

In addition companies are changing how they see analytics, focusing on more advanced and forward-looking analytics and less on management reporting.

Don meanwhile sees several areas where future developments are important:

• Mobility – more mobile devices, tablets, HTML 5 etc
  More and more about business users pushing IT not IT pushing to users. Increasingly about taking advantage of the devices GPS, and accelerometer for instance as well as pushing simple device-centric interfaces not simply replicating the traditional view on those devices.
• Cloud and massive scale analytics
  Lots of interest in cloud implementations for massive scale etc. Interestingly this is an area where I am doing some research – check out smartdatacollective/predictive-analytics-cloud to register or take our upcoming survey.
• Watson
  There is lots of interest in IBM’s Watson – both interest in its ability to interpret questions and use large amounts of data to answer questions as well as its ability to use multiple strategies to learn what works and what does not.
• Geospatial and temporal analytics
  As mobile devices proliferate the need for geospatial and temporal aspects to analysis and predictions. Who is the most useful customer for me to visit when I have a few hours spare in my current location?
• Consumable analytics
  Many organizations have very low adoption rates of analytics – stuck around 20% – so we must make it easier to consume analytics through automation (by building Decision Management Systems) as well as through better visualization.
• Social media and social analytics
  This allows us to interact with customers in new ways and so has the potential to really change how we work.