Issue :
Bacula is great. It can be very informative what is going on. A nice option is that it can give status reports through logs or by email. No one reads logs unless something suspicious is expected due to a bombardment of info, so email is useful. However too much mail is not informative as it tends not be read and therefor it is not useful anymore.
Possible solutions :
A side-issue is that when your working-environment consist of multiple people handling Bacula mail gets forked. This seems a bit like a waste of resources. Also that way it is not clear if and when someone has handled the issue.
The first 2 options were not real options for me.
What next? OK, let's check the flow of the email-reporting-process and determine where we can make a change and where that might lead us.
A general email-process can be described as : MUA ⇒ MTA ⇒ network ⇒ MTA ⇒ MDA ⇒ MUA
In my case Bacula-Director, MTA and MDA are all on one server, so: MUA ⇒ MTA ⇒ MDA ⇒ MUA
Terminology:
How this works can be read here among other places.
To illustrate:
Bacula Director ⇒ Bacula bsmtp ⇒ Exim (MTA) ⇒ Cyrus (MDA) ⇒ email-client of choice (Thunderbird)
As we have a Tape-Operator and a Sys Admin and have Bacula messages configured in such a way that some messages go to both, some email is forked at the point of the MTA.
I wanted:
What I did not want:
Yes, I know, filtering at the mailclient can also be an option, but that would be on a per user basis and would not limit the amount of mail. Nor would it ease adding new sysadmins etc to the game. So that is a no go.
So there are various stages in the email-flow-process where email could be managed differently than before
To meet the most of our demands the MTA level seems the best place to manage the email-stream. Within the MTA there are also various stages. So we want to interfere at the point where mail is designated for our server, the mail is not already split at a per user basis.
Some people have implemented hooks at the level where email is already split. At that point classic such as procmail and mailagent can be used. But again, they need per user configuration and I do not want that.
I was looking for a generic solution as not everybody uses the same MTA and MDA, so my thoughts were going towards Sieve as the preferred choice.
In out setup we have Exim as the MTA and Cyrus IMAP Server as the MDA. Both support a dialect of Sieve
As I made up my mind that the preferred level of intervention was the MTA I focused on that. And then a problem came up: although Sieve is supported, it is only supported by Exim for user-filters, not system-filters
So the only way was to implement it in such a way that it would work on our setup but not not copy-paste-able to an environment say like Postfix with Courier. I bit a shame, but there is no other way. Nevertheless I hope this article and the shown configuration are insightful enough so you can implement it on the email-server setup of your choice.
Global Design : Bacula Director ⇒ bsmtp ⇒ Exim → filtering → Exim ⇒ Cyrus shared IMAP folders
I decided to define a structure on Cyrus with the format of:
shared.bacula.<servername>.all
shared.bacula.<servername>.error
shared.bacula.<servername>.warning
Were all Bacula messages would go for historic reason go into the “all” folder and messages reporting an error would also be copied into “error” and messages containing a warning would also be copied into “warning”.
That way a new sysadmin or tapeoperator could have an insight of what happened in history and warnings and errors do blink out as they are copied in their own folder.
Implemented on Debian Etch with Exim and Cyrus.
Definition of Messages blocks in my bacula-director.conf file:
# Reasonable message delivery -- send most everything to email address # and to the console Messages { Name = Standard # # NOTE! If you send to two email or more email addresses, you will need # to replace the %r in the from field (-f part) with a single valid # email address in both the mailcommand and the operatorcommand. # What this does is, it sets the email address that emails would display # in the FROM field, which is by default the same email as they're being # sent to. However, if you send email to more than one address, then # you'll have to set the FROM address manually, to a single address. # for example, a 'no-reply@mydomain.com', is better since that tends to # tell (most) people that its coming from an automated source.
# mailcommand = "/usr/lib/bacula/bsmtp -h localhost -f \"\(Bacula\) \<%r\>\" -s \"Bacula: %t %e of %c % l\" %r" operatorcommand = "/usr/lib/bacula/bsmtp -h localhost -f \"\(Bacula\) \<%r\>\" -s \"Bacula: Intervent ion needed for %j\" %r" #mail = bacula@localhost = all, !skipped #operator = bacula@localhost = mount mail = bacula-sysman@localhost = all, !skipped operator = bacula-operator@localhost = mount console = all, !skipped, !saved # # WARNING! the following will create a file that you must cycle from # time to time as it will grow indefinitely. However, it will # also keep all your messages if they scroll off the console. # append = "/var/lib/bacula/log" = all, !skipped }
# # Message delivery for daemon messages (no job). Messages { Name = Daemon mailcommand = "/usr/lib/bacula/bsmtp -h localhost -f \"\(Bacula\) \<%r\>\" -s \"Bacula daemon message \" %r" mail = bacula@localhost = all, !skipped console = all, !skipped, !saved append = "/var/lib/bacula/log" = all, !skipped }
Obviously this is different for every type/brand of IMAP Server. For Cyrus there are some useful sites/pages also in regard to what we want to establish here.
Notably besides standard wiki, howto and docs on Cyrus:
This lead to using cyradm issuing:
cm shared
cm shared.bacula
cm shared.bacula.myfirstserver
cm shared.bacula.myfirstserver.all
cm shared.bacula.myfirstserver.warning
cm shared.bacula.myfirstserver.error
Obviously this should be repeated for every server (mysecondserver etc) a Bacula FD runs on.
Next the ACLs should be defined using cyradm, gyrus or any suitable other tool of choice.
To quote the Cyrus docs: The identifier “anonymous” refers to the anonymous, or unauthenticated user. The identifier “anyone” refers to all users, including the anonymous user.
So it all depends on the (password)backend of your IMAP server and under which system-account Bacula is running (probably user “bacula” or user “root”) which one to choose.
For the lazy among us:
sam shared.bacula.myfirstserver.all anyone plrs
A restriction on who can post is made later in the configuration of the MTA (see below). For all shared folders ACLs should be defined and can be less exposed. However the user under which Bacula is running *must* be able to post to these folders. Either by specific configuration or by anonymous/anyone ACL configuration like above.
It's wise to let someone administrator the mailbox too (for instance user admin): sam shared.bacula.myfirstserver.all admin lrswipda
Again, most of the configuration aspects are derived from the Exim wiki's page about Cyrus/Exim integration by Andrzej Filip.
However there are some other aspects which I will document.
In /etc/exim4/conf.d/main/03_exim4-config_tlsoptions :
system_filter = "/etc/exim4/system.filter" system_filter_user = Debian-exim system_filter_group = Debian-exim
In /etc/exim4/conf.d/main/01_exim4-config_listmacrodefs :
domainlist cyrus_domains = localhost : example.org : example.com addresslist shared_senders = bacula-sysman@example.com : bacula-operator@example.com
In /etc/exim4/conf.d/transport/35_exim4-config-cyrus_delivery :
cyrus_delivery: debug_print = "T: cyrus_ltcp for $local_part@$domain" driver = smtp protocol = lmtp hosts = localhost allow_localhost
Note: Exim requires LMTP over TCP for shared IMAP folders. Sockets do not do the trick here. You can opt for anonymous LMPT delivery or not.
In /etc/exim4/conf.d/router/666_exim4-config_local_shared :
local_shared: debug_print = "R: local_shared for $local_part@$domain" domains = +cyrus_domains senders = +shared_senders driver = accept local_parts = \N^\+shared\..+\N transport = LOCAL_DELIVERY
In /etc/exim4/conf.d/acl/30_exim4-config_check_rcpt :
accept domains = +cyrus_domains local_parts = \N^\+shared\..+\N endpass
When looking at the actual filter things get interesting. I found some documentation on the subject of which messages can be send in the bacula Developers manual, however this is not very useful for two reasons:
Yet I saw at least 16 different types of email from Bacula in my mailboxes. Especially messages with header subject “Bacula daemon message” can be all kinds of messages.
Exim Filter Specification shows how to define filters. Emails to shared IMAP folders should have to format of: +shared.level1.level2@host
The example below shows the filering for one particular server. These definitions should be copied and adapted for other servers or be made more generic or comples.
In /etc/exim4/system_filter :
# Exim filter
#################################################################################### ### system filtering : Bacula ######################################################
# Bacula message filtering to shared IMAP folders # filtering based on headers although content is probably more reliable # to maintain some kind of compatibility with Sieve # (Sieve is not supported by Exim MTA as systemfilter, only as personal filter)
# Client: localhost if $sender_address: matches "bacula@localhost" or $sender_address: matches "bacula@myfirstserver" or $sender_address: matches "bacula-operator@example.com" or $sender_address: matches "bacula-operator@localhost" or $sender_address: matches "bacula-operator@myfirstserver" or $sender_address: matches "bacula-sysman@example.com" or $sender_address: matches "bacula-sysman@localhost" or $sender_address: matches "bacula-sysman@myfirstserver" then # message : "Intervention Needed" if $header_subject: contains "Intervention needed" then #deliver "+shared.bacula.myfirstserver.warning@example.com" deliver "+shared.bacula.myfirstserver.warning" endif # message : "Verify Differences" if $header_subject: contains "Verify Differences" then #deliver "+shared.bacula.myfirstserver.warning@example.com" deliver "+shared.bacula.myfirstserver.warning" endif
# message : "Error" if $header_subject: contains "Error" then #deliver "+shared.bacula.myfirstserver.error@example.com" deliver "+shared.bacula.myfirstserver.error" endif
# message : "Canceled" if $header_subject: contains "Canceled" then #deliver "+shared.bacula.myfirstserver.warning@example.com" deliver "+shared.bacula.myfirstserver.warning" endif
# message : "Error" if $header_subject: contains "Error" then #deliver "+shared.bacula.myfirstserver.error@example.com" deliver "+shared.bacula.myfirstserver.error" endif
# message : "Bacula daemon message" if $header_subject: contains "Verify Differences" then #deliver "+shared.bacula.myfirstserver.warning@example.com" deliver "+shared.bacula.myfirstserver.warning" endif
# message : "Error" if $header_subject: contains "Error" then #deliver "+shared.bacula.myfirstserver.error@example.com" deliver "+shared.bacula.myfirstserver.error" endif
# deliver to the "all" shared folder for historical reasons (logging/backtracing) #deliver "+shared.bacula.myfirstserver.all@example.com" deliver "+shared.bacula.myfirstserver.all"
# now drop the message as it is already delivered to the right places #seen finish
endif
####################################################################################
Basically this implementation leads to what I want: multiple people can look into current and past Bacula events. The mails are not duplicated and warning and errors do stick out so should be more easily noted.
Obviously I mist admit this is more of a basic description on how to use shared IMAP folders. I just found using it for storing Bacula status emails in a certain way as described a workable solution for my working environment. One could also opt for an entire different approach and skip email warning all together and use something like Syslog Next Generation and apply filters on those logs. If email is still preferred then you can send emails based on those results of course.
Unfortunately this implementation is specific for Exim and Cyrus but I hope it inspires people to create solutions for other environments. If you do so: please share them.
It could very well be that I made mistakes, that some configuration aspects can be optimized or even discarded. It took me some time to get things working and I am glad it does. Feel free to comment.
With this document I do not claim to have the ultimate solution to handle status emails from Daemons such as Bacula. This is just one approach; I welcome others to submit their alternative solutions so people reading this wiki can make a solid choice of what to implement and weight the cons and pros of the solutions offered.