IntelMQ-dev

intelmq-dev@lists.cert.at

1 participants
246 discussions

tomorrow, Tuesday intelmq-dev conf call
by L. Aaron Kaplan 25 Jul '16

25 Jul '16

Hi, We'll be having our regular intelmq-dev call tomorrow at 9:00 a.m. CEST. This terribly early date was chosen so that the colleagues from Australia can join the call as well. If you want to participate in the call, please let me know, I will send you call in details. Best, a.

1 0

Re: [Intelmq-dev] today 16:00 UTC+2 conf call [SEC=UNCLASSIFIED]
by Clark, Andrew 11 Jul '16

11 Jul '16

UNCLASSIFIED Hi Aaron, Thanks for thinking of us down here in Australia, and sorry for the slow response - I was on leave last week. Early morning your time is going to suit us best - we can be a bit flexible in the evenings, it it's say 1630 or 1700 here (UTC+10) that is fine, or if that's too early for you we could try after 1900 and I can dial in from home. Andrew -- Andrew Clark | Senior Technical Advisor | CERT Australia Attorney-General's Department, Australian Government Phone: +61 2 6141 2538 Online: www.cert.gov.au For all CERT Australia operational matters, please call our hotline: 1300 172 499, or +61 26141 2999 or email: info(a)cert.gov.au -----Original Message----- From: Intelmq-dev [mailto:intelmq-dev-bounces@lists.cert.at] On Behalf Of L. Aaron Kaplan Sent: Tuesday, 5 July 2016 7:38 PM To: intelmq-dev(a)lists.cert.at Subject: [Intelmq-dev] today 16:00 UTC+2 conf call Hi *, today at 16:00 UTC+2 we'll have our conf call again (and resume it after some weeks of FIRST conference and post-FIRST catching up). Topics for this call: * generic syntax for transformations/generic mapping language * status update Intevation * status update CERT.at * ... your status update? * any other input for the monthly newsletter * biggest concerns/topics right now * discussion action items for next week * AoB Please feel free to suggest topics for this call by replying to this mail thread. Thanks, a. PS: I'd like to suggest to move this weekly conf call to an earlier time in the future since this would allow our Australian colleagues to join as well. When would be a good time for the Aussies? -- // CERT Austria // L. Aaron Kaplan <kaplan(a)cert.at> // T: +43 1 505 64 16 78 // http://www.cert.at // Eine Initiative der NIC.at Internet Verwaltungs- und Betriebs GmbH // http://www.nic.at/ - Firmenbuchnummer 172568b, LG Salzburg ---------------------------------------------------- If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments.

1 0

Re: [Intelmq-dev] Packaging Strategy for Bots with dependencies [SEC=UNCLASSIFIED]
by Clark, Andrew 11 Jul '16

11 Jul '16

UNCLASSIFIED Hi Bernhard, all, Sorry to come into this a bit late ... my comments are below: > Am Mittwoch, 20. April 2016 11:55:59 schrieb L. Aaron Kaplan: > > > Aren't we running into the same issue as with the BOTS file? If > > > > The BOTS file is simply a template of which bots exist in general. > > It is mainly used for the intelmq-manager to display which ones > > could get selected. And which parameters they have. > > In order to get this moving, I propose to tackle it in small steps. > > What about implementing a solution for the contents of BOTS first? > We keep the logic of etc/defaults.conf and etc/runtime.conf for this step. > So no hiearchical configuration (for now) and then we take it from there. If I understand correctly the IntelMQ 'runtime' configuration is currently spread across three configuration files (runtime.conf, pipeline.conf and startup.conf), and there are other configuration files as well (e.g. defaults.conf, harmonization.conf and system.conf). I think it's a great idea to consider how to improve on the current monolithic BOTS template configuration, but also wonder if some of the configuration that is currently in separate files could be merged. For example, could startup and runtime be merged? Maybe even pipeline too? > Thinking about how to solve the BOTS file first: > a) (proposed by Dusting) bots.d/ directory solution, > means adding and removing filenames. > b) adding an initialisation function to each module which > returns the necessary information > c) write a small packaging helper that adds/removes sections > from BOTS > > Some thoughts (being still new to the code base): > b) seems attractive to me, because it keeps information > close to where it is maintainted. Its main drawback probably is > that is the largest change. Are there other drawbacks? > c) has the drawback of requiring running code during (de)installation. > And it does not add much over a). I have a STRONG preference for an approach similar to 'a' above. In my opinion we should be aiming to decouple the configuration from the bot code itself as much as possible, and encouraging more generic bots (i.e., less bots that are more generic (and more configurable), and more configuration files. In general each bot should potentially accept many configurations. This approach should serve to limit the explosion in the number of bots (I think the parser bots are a good example of where we could reduce the number of bots focusing on 'what' they are parsing, rather than their source). Along the lines of option 'a' above I think the 'modules-available' and 'modules-enabled' approach used by the Apache web server could work for us too. All available configurations (templates, etc.) could be stored in a 'etc/bots-available' location, and the copied/edited into the 'etc/bots-enabled' location to enable them. Andrew -- Andrew Clark | Senior Technical Advisor | CERT Australia Attorney-General's Department, Australian Government Phone: +61 2 6141 2538 Online: www.cert.gov.au For all CERT Australia operational matters, please call our hotline: 1300 172 499, or +61 26141 2999 or email: info(a)cert.gov.au ---------------------------------------------------- If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments.

1 0

Packaging Strategy for Bots with dependencies
by Dustin Demuth 07 Jul '16

07 Jul '16

Hey Folks, Whilst checking the dependencies, I've found out that a lot of Bots have their own requirements-file. From a packagers point of view, I think this is hard to maintain. I think, we should discuss if it is reasonable to split the software into a CORE package, were all packages have the same dependencies, and several BOTS packages with their own dependencies. This would create the additional need to rethink the BOTS json-file. It would be possible to have all JSON configs for the Bots in a bots.d directory and search for them in this directory. I know, that this might create additional complexity in some points. BR Dustin -- dustin.demuth(a)intevation.de https://intevation.de/ OpenPGP key: B40D2EFF Intevation GmbH, Neuer Graben 17, 49074 Osnabrück; AG Osnabrück, HR B 18998 Geschäftsführer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

8 15

today 16:00 UTC+2 conf call
by L. Aaron Kaplan 05 Jul '16

05 Jul '16

Hi *, today at 16:00 UTC+2 we'll have our conf call again (and resume it after some weeks of FIRST conference and post-FIRST catching up). Topics for this call: * generic syntax for transformations/generic mapping language * status update Intevation * status update CERT.at * ... your status update? * any other input for the monthly newsletter * biggest concerns/topics right now * discussion action items for next week * AoB Please feel free to suggest topics for this call by replying to this mail thread. Thanks, a. PS: I'd like to suggest to move this weekly conf call to an earlier time in the future since this would allow our Australian colleagues to join as well. When would be a good time for the Aussies? -- // CERT Austria // L. Aaron Kaplan <kaplan(a)cert.at> // T: +43 1 505 64 16 78 // http://www.cert.at // Eine Initiative der NIC.at Internet Verwaltungs- und Betriebs GmbH // http://www.nic.at/ - Firmenbuchnummer 172568b, LG Salzburg

1 0

Support of shadowserver feeds
by Dustin Demuth 16 Jun '16

16 Jun '16

Dear all, as a short announcement, we are currently starting to work on parsers for the follwing shadowserver feeds. Drone [Done] Microsoft Sinkhole Sinkhole HTTP Drone DNS Open Resolvers NTP Monitor [Done] Open Portmapper Open CharGen Open Elasticsearch Open IPMI Open MDNS Open Memcached [Done] Open MongoDB Open MS-SQL Open NetBIOS Open Redis Open SNMP Open SSDP SSL FREAK SSL POODLE [Done] We expect them to be ready by the end of this week. BR Dustin -- dustin.demuth(a)intevation.de https://intevation.de/ OpenPGP key: B40D2EFF Intevation GmbH, Neuer Graben 17, 49074 Osnabrück; AG Osnabrück, HR B 18998 Geschäftsführer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

2 3

MISP collector and parser [SEC=UNCLASSIFIED]
by Clark, Andrew 11 Jun '16

11 Jun '16

UNCLASSIFIED Hi guys, With the help of a colleague I have prepared a MISP collector and parser for IntelMQ. It requires a tag to be added to the MISP events that need to be processed. This tag is removed from the MISP event by the collector once it has been processed (and a different tag is added to the MISP event to indicate that it has been processed). Anyway, without getting too bogged down in the details, I've put the code in a forked copy of the repo on my github page: https://github.com/kralca/intelmq/commit/c3cdb0e The deduplicator expert should be used to detect MISP event attributes that have been previously processed (for example following the addition of attributes to a MISP event). I hope this is useful for the Hackathon on Sunday. Please let me know if you would prefer if I submit a pull request. Cheers, Andrew -- Andrew Clark | Senior Technical Advisor | CERT Australia Attorney-General's Department, Australian Government Phone: +61 2 6141 2538 Online: www.cert.gov.au<http://www.cert.gov.au/> For all CERT Australia operational matters, please call our hotline: 1300 172 499, or +61 26141 2999 or email: info(a)cert.gov.au<mailto:info@cert.gov.au> ---------------------------------------------------- If you have received this transmission in error please notify us immediately by return e-mail and delete all copies. If this e-mail or any attachments have been sent to you in error, that error does not constitute waiver of any confidentiality, privilege or copyright in respect of information in the e-mail or attachments.

2 1

newsletter May 2016
by L. Aaron Kaplan 07 Jun '16

07 Jun '16

= Intelmq-dev-news 06-2016 Issue 6/2016 == Topics == # Summary # Status update Intevation # Status update CERT.at # Status update misc == Review of May 2016 == TL;DR and important changes ----------------------------- * Lots of adaptations * CERT Australia uses intelmq * Hackathon on Sunday, 12th of June at the FIRST.org conference, Seoul. If you are attending FIRST, please do join! / TL;DR === How to contribute to this newsletter? === -> contact Aaron, Dustin for future input === Status report Intevation === * The schema of the contactDB will be revised: The idea is that the most specific template wins. For instance: first check for a template for malware.name, then classification.type, then classification.taxonomy, else use default template or abort. This is required as almost all feeds do not set the classification.identifier and the classification.type is not specific enough to pick a template. * To enhance the process of identifying events, the parser needs to set the identifier. Idea: Provide a list which is maintained in a central place which maps these identifiers. This mapping could be downloaded by IntelMQ and be used by the parsers. * Gernot is now responsible for Packaging. We now use an APT repository for our releases. ** Idea: Bots can have their own packaging, this makes IntelMQ more modular. * Intevation will submit a proposal for a mature branching strategy. We will bring this topic to the list. * Intevation is also going to propose an idea for config files: Something like: etc/intelmq/bots-{available,enabled}/ directories as in Apache2. This might make life easier. * Intelmq-mailgen script: A script supposed to send mails in different formats: X-ARF is one of them, but currently experimental. ** The script is tied to certbund-contactDB expert. ** Interesting concept: The mailgen-script enables to track which event was sent at what time to each customer (can add ticket # numbers). * A Script which fills the contactDB with information from RIPE DB is in the queue * taxonomy expert now supports the taxonomy "other" * Created some Shadowserver parsers for drone, ntp monlist, open memchached, ssl poodle. * 500 MB reports do not fit into redis messages. We expect an updated redis > 3.2 should work for these large messages. But this would require testing. === Status report CERT.at developments === * intelmq now can process SIGHUPs: this will reload the bot's configuration. * related: new syntax for intelmqctl: check out ``intelmqctl reload``. * cronjobs which intevation created are being used now at CERT.at. * work on new parsers - new architecture (https://github.com/certtools/intelmq/pull/529) * idea: new parser architecture is parsing based on individual lines. Now you can find individual lines which can't be parsed and just replay these. * needs testing * tor bot: depends on internet2.us. Fixed and made more robust by Aaron. Coming: * missing: monitoring, log check deployment: check if .dump files exist * missing: intelmq-manager does not graph events/sec , etc yet. idea: use RRD * packaging: sebix is looking at tools to create packages for Debian, RedHat, etc. all at once. Sebix is looking at OpenBuild System by opentools. We will upload the packages to the website. * missing: branching concept for the release * need to test shadowserver (https://github.com/certtools/intelmq/issues/524) and if it's okay, pull into master. * dns-python has a new release. We use this everywhere. We should update it in the packages and code (let's wait a few weeks if some issues arises in the new version, but then we upgrade) === Wider community === * Koen wrote a octopress template for the website. Will try this out. Thanks very much! * Discussions on MISP<-> Intelmq integration (https://github.com/certtools/intelmq/issues/537). * We will have a hackathon on Sunday, 12th of June at the FIRST.org conference, Seoul. If you are attending FIRST, please do join! === Wish-list === * **we need more test-cases!!!** unit tests as well as integration tests. * Intevation searches for testers for the packages. * We'd like to have some nice graphs in the intelmq-manager: events/sec , parse-failures/sec, etc. * implementation of whitelisting of events (filter out events based on whitelists). See https://github.com/certtools/intelmq/issues/426 * A good CSS design for the web page * Create more website Content: How-Tos / Installation Instructions, Success Stories * How-Tos / Instructions: If you are using a special feature of IntelMQ, for instance an expert bot, try to find some time to write down a short article how you managed to get it to work and why you are using it. * a specific config logic for ASNs: do this and that (for example sett ttl = 1 month) if event is in ASN xyz. Or "ignore" if event is in ASN xyz. This should support some kind of more-specific-less-specific inheritance, similarly to Apache directory settings. The most specific setting wins. The order could be: country code -> ASN -> netblock -> ip (/32). Open questions: what's more relevant if both domains and numbers (ASN, IPs, net blocks) exist in an event? * block based processing: for example block based team cymru lookups * parallelisation: We need to revisit this topic == Important Discussions == In case you missed something, here are the headlines of some discussion we consider interesting / important. === Mailing Lists === https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev == Communication == Chat: irc #intelmq on freenode or webchat: [[https://webchat.freenode.net/?channels=intelmq]] Follow on twitter: @intelmqorg Weekly Conference Call every Tuesday 16:00 UTC+2: Dial in via the known conference bridge number. It is [[https://en.wikipedia.org/wiki/Telephone_number_mapping|ENUM]] enabled. Ask Aaron or Dustin for the number if you want to participate.

1 0

Discussion on intelmq output / transformation architecture
by L. Aaron Kaplan 24 May '16

24 May '16

Hi folks, an interesting discussion started on https://github.com/certtools/intelmq/issues/487 Basically, pedro would like to send data to ArcSite in CEF format. I would like to propose that we enhance our architecture to include "transformer bots". Apart from the funny name which reminds me of http://transformers.hasbro.com/en-us/bots, this has a nice symmetry to the collector-> parser chain we have on the input side. So what are we talking about? Input side: ``` Collector ---> Parser ---> .... pipeline ``` Currently on the output side it's like this: ``` ... pipeline -> expert1 -> expert2 -> ... -> output bot 1 ``` But this basically leaves the work of transcoding/transforming the DHO format to the output bots. So far, this was enough. In the general case, it would be great to transform the DHO to some format. Let's assume CEF or IODEF. A generic output bot could then take that data and send it over whatever mechanism it knows to a destination. Example: ``` ... pipeline -> expert1 -> transform 2 IODEF -> mail bot or: ... pipeline -> expert1 -> transform 2 IODEF -> mail bot \ \-> sftp bot ``` The question in https://github.com/certtools/intelmq/issues/487 arose on where to store the output data. I'd suggest to create a new "raw_output" field in the DHO for this. A transformer bot shall write its format to the raw_output field (in base64) and an output bot shall take that data, decode the base64 andd send it via its own mechanism. What do you think? Would this work? Do you see any serious problem with this approach? Best, a.

3 2

xarf support (email sending and general mapping)
by Bernhard Reiter 24 May '16

24 May '16

Dear IntelMQ-Devs, for intelmq-mailgen I've constructed a first xarf writing support with an experimental mapping (see it at https://github.com/Intevation/intelmq-mailgen/issues/2) I haven't dived into it yet, but it seems like we need a practial mapping between xarf schemas (see https://github.com/certtools/intelmq/issues/522) So how do we construct such a mapping, so that in the end we can do xarf -> intelmq -> xarf? What intelmq field can be always expect or which only sometimes? I think you folks with more practical experience, having seen more data, are seminal for this mapping to become good. We (from Intevation) can implement the rules once we understand them well enough. :) Best, Bernhard -- www.intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev