Last night was the September meeting of the NE Lotus User Group in Waltham, MA.  A good turnout overall with a nice mixture of Domino customers, partners and IBM’ers.  The technical presentation was about DAOS as it appears in Domino 8.5 and the big improvements now in Domino 8.5.1.   Not only was the subject matter well-received by the audience generally — who doesn’t like to save more than 40% on their storage and backup costs? — but being able to discuss technical aspects was especially nice for us.  We had a lot to contribute, based on our own experience with large enterprise customers.

Pat Mancuso and Collin Murray ably handled the technical side of the discussion.  We asked some tough questions of our hosts — so much so that I did not have the heart to rise to the ‘Stump the Experts’ challenge  offered at the end of every meeting.  Overall, I was reassured that the implementation seems very solid, based on simplicity and information hiding.  What was especially neat was that in Domino 8.5.1 the email pathway from the client through the servers will be able to inquire whether an attachment needs to be copied across the wire during mail transfer and if DAOS already has it just a ticket stub is copied.  A big performance win for the network.  This is a bit of change from the traditional store-and-forward mail contract which generally is ‘pushed’ without asking questions about the data along the way.  One sticking point for us is that there are now two sizes floating around — the logical size of the message and the size of the actual network transfer — and it appears that the LOG.NSF transfer records are now going to include the actual number of bytes transferred, making correlation with actual mail documents harder since those are only expressed in logical size.  Grrrr.

Pat touted the ‘transparency’ of DAOS — the expansion of attachment tickets happens at such a low level that no API calls need to change.  We applaud this backward compatibility from IBM — at least one other large vendor I can think of might have asked us to rip-and-replace our code.  However, it is not the ‘transparency’ but rather the ‘opaqueness’ that mars this excellent feature in our eyes.  Permessa has been asking for more than one year to have additional APIs to provide access to both the physical as well as logical sizes.  Both pieces of information are vital for our customers’ planning and measurement needs.  Imagine performing a server consolidation where you are dealing with ‘logical’ disk space taken by mailboxes and not the ‘physical’ disk space once DAOS is taken into account!  Yuval Shimoni, our CTO, summed it up nicely.  To paraphrase:  the current DAOS is geared more to the tactical than the strategic needs of the IT staff.  To better help IT measure, model and plan their future environment we need access to the true size information.  The Domino Admin client uses undocumented calls to gather this information — we are just asking for this same info to be available via public APIs and would be happy to contribute effort to their design.  While I am asking, how about exposing the logical checksum value of each attachment — I can see possible optimizations from the A/V vendors to avoid redundant checks and free up CPU, giving DAOS an even higher effective performance boost.

Overall it was a great presentation.  I very much appreciated IBM answering our of myriad questions and taking notes on the issues we raised.  I actually look forward to attending these gatherings every month, as the positive energy and momentum from IBM is inspiring.

For folks who want to be on the mailing list for the NE Lotus User Group, you can sign-up here.

Share/Save

Our analysis of large messaging environments over many years has revealed that in most companies 80% of the corporate messaging resources are being consumed by only about 1% of all employees.

This is a significant finding especially in times where tight IT budgets are strained by rapidly growing message volumes and resulting skyrocketing bandwidth & storage costs.

What this really means is that there is a huge opportunity to dramatically reduce operating costs by going after the cause of this excessive email traffic. Don’t worry, I am not proposing to fire the 1% of offending employees.

There are ways to manage email more efficiently without adversely affecting users or business operation.

Permessa just published a whitepaper (I am a co-author), titled “6 Best Practices That Reduce Email Overload and Costs”. The paper highlights areas for managing excessive email traffic, such as unnecessary reply-to-all, attachment ping-pong and the overuse of mailing lists. Some of these have been previously discussed on this blog. For each topic area the whitepaper makes best practice recommendations on how to implement email policy changes that can prevent the negative effects and help save money.

Technorati Tags: email costs, information overload, best practices to reduce email overload, permessa

Share/Save

“The Department of Homeland Security self-inflicted what one observer called a mini distributed denial of service, with a reported mass of more than 2.2 million messages stuffing the inboxes of the nation’s security experts.”

Once again, the troubles started with the unintentional use of the infamous Reply-to-all button by a DHS employee. The employee was simply trying to change his email address on a department mailing list server, accidentally sending the request to all subscribers.

“In the hour that followed, dozens of readers replied to the exposed list of recipients, causing the “mini-DDoS” with demands to unsubscribe, pleas to others to cease replying, urgent requests from the Department of Defense and DHS officials for recipients to “kindly stop now please,” a “vote for me” political ad, job offers and updates on the local weather.”

The incident highlights again the importance of monitoring and safeguarding messaging infrastructures with tools that can prevent incidents like this and their aftermath from occurring.

Technorati Tags: dhs, dod, email storm, email flood

Share/Save

So how do you trim the fat? – Start looking for bacn bits.

Bacn the new term coined to describe the superfluous emails existing plentiful in corporate networks. Corporate bacn comes in many shapes and forms. Company email newsletters and announcements, CYA emails (the excessive use of cc and bcc), notifications from intranet applications (e.g. CRM apps) all add to the white email noise.

Most notably in many cases IT themselves generate the bulk of unnecessary traffic.

The top senders in virtually all corporate messaging environments are automated notification agents from system and network management applications. These constant system status messages that initially seemed like a good idea for keeping IT staff informed, very quickly become background chatter most often ignored, deleted or auto-filed. It is amazing to see how this steady trickle of small messages adds up over time.

Therefore, the next time you wonder how to reduce network, storage and archiving costs – look for the bacn bits, which are easily revealed with a complete assessment of your email traffic.

Technorati Tags: bacn, email cost, email cost reduction

Share/Save

One would think that these restrictions are reasonable and not an issue, but wait until your top sales executive is unable to deliver a RFP response to a potential client via email before the submission deadline. She doesn’t care why her heavily illustrated word document of over 10MB got caught by the outbound size limit, she just wants the email delivered.

Here are the three most common problems related to message size limits:

• Many companies do not tell their employees about the limits they put in place, their exact size and purpose.
• The method for choosing an allowable maximum message size is often completely arbitrary.
• Message size limits are typically imposed indiscriminately with a “one-size-fits-all” approach across the entire company.

As a result, users will resend rejected emails multiple times, waste time by calling the helpdesk and get frustrated. The “smarter” ones might even circumvent the system by packaging the large attachments into smaller chunks that are then sent in multiple messages or worse use a personal email account on a public service with lesser restrictions.

What many email administrators don’t realize is that their size limits are missing the mark.

The chart above illustrates the typical message size distribution and the resulting volume impact on the network found in literally all corporate email environments. It vividly shows that messages between 0.5- 5MB in size, sent repeatedly and to many recipients at a time are the real burden on the infrastructure (75% of the total data volume is caused by messages <5MB). The ripple effect of a 2MB PowerPoint sent to 50 or more people distributed throughout the network is far greater than the impact of a single large message.

Our consultants refer to this chart tongue-in-cheek as the “finger –chart”…

There are solutions to this dilemma:

• Conduct an email audit to better understand the typical message traffic patterns in the environment. The results should be broken down not only by mail topology, but also by geography, departments and even employee role or title.
• Set reasonable maximum limits on the outbound gateways that allow users to conduct their typical business with some room for exceptions. Enable multiple outbound queues on the gateways to avoid delivery delays due to single large messages.
• Enable smart message size handling on the internal servers. A dynamic enforcement solution that calculates the resulting net impact of individual emails combined with a flexible rules-based workflow will curb the true abusers and allow raising overall limits while lowering infrastructure costs.

Make sure to consult with the business users: Present the findings of the audit, request input and inform them of any resulting policy adjustments before implementing any wide-ranging changes. That effort combined with ongoing monitoring and reporting will certainly improve user satisfaction and ultimately make an admin’s job easier.

Technorati Tags: message size limit, email management, email enforcement, email monitoring

Share/Save

A review of the data quickly confirms some of the usual suspects:

• Spam
• Document sharing
• Mailing list subscriptions
• Personal emails, and
• General overuse of email

There is however a notable addition to this mix: Voice mail messages sent as “.wav” attachments via email.

Unified Messaging now also referred to as Unified Communication is slowly but surely making its way into the enterprise. Maurene Caplan Grey had a good take on the difference between unified messaging and unified communication posted in a recent blog entry:

“What’s the difference between unified messaging and unified communications? From a purist perspective, UC means near real-time communications (instead of store-and-forward), which integrate with line-of-business processes and workflow. However, with the mushrooming of UC hype, the difference between UM and UC will be semantics. UC sounds sexier than UM, yet UM dressed in the UC emperor’s clothes is still UM.”

While initial implementations were expensive and plagued by integration issues, modern on-premise and hosted VOIP telephony solutions all offer some sort of voice mail to email forwarding capability.

However, that is just the beginning. Both IBM and Microsoft have committed to the development of unified communication solutions, in essence integrating their existing email, IM and collaboration products with telephony. I think it is fair to assume that email will remain the medium of choice for storing and transporting voice mail messages.

The primary concern when deploying VOIP is typically focused on network issues, specifically QoS to guarantee voice quality. Companies planning the deployment of these solutions must carefully evaluate the impact on the entire infrastructure before embarking on a full-scale rollout. The secondary impact on email traffic and storage can be significant. A 30-60 second voice mail generates on average an email of 100kB in size. Legacy PBX voice mail doesn’t allow to store an unlimited number of messages for an infinite time. Email integration changes that.

Lastly, let’s not forget the compliance implications. Many companies are now archiving their emails, in some cases for up to seven years. The voice messages stored as emails will add to the archiving volume and makes you wonder what kind of cost it would add to potential e-discovery projects.

Bottom line: Unified Communication is coming. Make sure you consider all of the cost factors when planning a deployment.

Technorati Tags: unified communication, unified messaging, VOIP, voice mail, email archiving, email storage

Share/Save

When analyzing corporate messaging systems you can observe many consistent trends, but every now and then, you will notice a peculiar anomaly. Those are the times when in the middle of a presentation a person in the back of the room stands up and proclaims: “I told you that something strange was going on, but nobody would listen to me.” However, these stories do not just make good anecdotes they make you think and wonder if something like this might be happening in your corporate messaging environment as well, potentially costing your company unnecessary time and money.

The need for more bandwidth
A few years back a large financial services company hired us to perform a detailed analysis of their networking traffic. At the time, the European based bank had completed a number of mergers with smaller firms in South America. Consequently, their corporate email environment had grown substantially and transatlantic network traffic had multiplied to a point where the company’s private network link was consistently overloaded. The network admin staff naturally looked at the traffic packets and ports and concluded that email traffic was to blame (sounds familiar?) for using-up most of the bandwidth.

Email being an essential business tool, of course, triggered the procurement process of “bigger pipes” with an initial cost estimate of about $2MM in extra telecom expenses annually. That’s when somebody (smart) at the firm suggested to take a deeper look at the actual data causing the traffic.

Our analysis of the banks entire messaging environment very quickly made a fundamental discovery. The email traffic causing the network overload was actually mostly messaging traffic between branches and subsidiaries of the bank in South America, all routed through the bank’s main data center in Europe, traversing the transatlantic link twice. The explanation for this seemingly crazy configuration was the fact that as the bank acquired other companies over time, a direct network connection to headquarters was quite logical. However, as the number of physical locations in that particular geography grew that initial network topology with distant hub and spoke routing needed some revision.

The solution to the perceived bandwidth problem was actually quite simple in the end. Most of the South American branches already had some rudimentary interconnectivity. For the branches that were not connected, the cost of provisioning the appropriate services was marginal compared to the proposed overseas upgrade. After looking at the actual email traffic patterns the email administrators together with the network staff revised the mail routing topology resulting in the cancellation of the expensive network upgrade.

What are the lessons learned?

• Corporate messaging systems are constantly changing (user behavior, mergers, relocations, consolidation, etc)
• Port based traffic analysis alone does not provide sufficient insight into the root cause
• Ongoing traffic monitoring and trending should be part of the corporate IT strategy

Technorati Tags: email traffic, email routing, network topology, email traffic patterns, network cost

Share/Save