IntelMQ-dev

intelmq-dev@lists.cert.at

1 participants
246 discussions

Request for Testing and Feedback: Shadowserver API collector and parser
by Birger Schacht 19 Jan '21

19 Jan '21

Hi, Shadowserver nowadays not only sends out reports by Mail but also provides an API [0] to query reports. We recently implemented a Shadowserver Reports API collector bot [1] that downloads the reports from the API and feeds them into IntelMQ. To parse the downloaded feeds (in JSON format) we built upon the existing Shadowserver parsing logic and created a JSON parser [2] that's meant to be used together with the Shadowserver Reports API collector bot. If anyone is interested in testing the collector and parser that would be great- any feedback, bug reports or improvements are highly appreciated. To use the collector you will need a Report API Key, which you can request on [3]. cheers, Birger [0] https://www.shadowserver.org/what-we-do/network-reporting/api-reports-query/ [1] https://intelmq.readthedocs.io/en/latest/user/bots.html#shadowserver-report… [2] https://intelmq.readthedocs.io/en/latest/user/bots.html#shadowserver [3] https://www.shadowserver.org/contact/ -- // Birger Schacht <schacht(a)cert.at> // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ Docker Image (beta)
by Sebastian Waldbauer 15 Jan '21

15 Jan '21

====== IntelMQ Docker Image *(beta)* ====== *CAUTION* Please do not use this image in production yet! ===== Introduction ===== We are constantly trying to make the installation of IntelMQ easier for a wider user-base. Therefore we are currently working on an IntelMQ container[^1]. Because of the early stage, input from you is highly appreciated. We are currently targeting Docker Engine 18.x and higher. [^1] https://www.docker.com/resources/what-container ===== Current implementation ===== TL;DR: The image is a beta test, docker containers runs isolated, worth to test! *ATTENTION* The image provided is not fully Docker compliant. We are releasing the image to collect feedback and gather information for future improvements. The container should be considered as a beta version. There will likely be breaking changes in future versions of the container. The image is available on Dockerhub[^4]. You can find documentation on how to install and run the container in the IntelMQ documentation [^doc] The container comes preinstalled with IntelMQ Manager[^3], which will be the only way to control the IntelMQ installation in the container. You can't use `intelmqctl` outside the container to control IntelMQ in the container. At the moment, the image is shipped with the development version of IntelMQ[^2] and the last stable version 2.2.1 IntelMQ Manager[^3]. This means that the new IntelMQ API is //not// yet part of the Docker image. You can check the current included versions by using ```docker inspect --format '{{ index .Config.Labels "org.label-schema.vcs-ref"}}' certat/intelmq-full:1.0``` Build steps and deployment information is provided in IntelMQ Docker[^5] Repository. [^2] https://github.com/certtools/intelmq/ [^3] https://github.com/certtools/intelmq-manager [^4] https://hub.docker.com/repository/docker/certat/intelmq-full/general [^5] https://github.com/certat/intelmq-docker [^doc]: https://intelmq.readthedocs.io/en/latest/user/installation.html -- // Sebastian Waldbauer <waldbauer(a)cert.at> - T: +43 1 5056416 7202 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 1

An expert bot for Request Tracker's RT::IR
by Mika Silander 13 Jan '21

13 Jan '21

Hi, We recently decided to try IntelMQ with the intent to have it push security events into a Request Tracker (RT) instance. The events would thus be managed as RT::IR tickets within RT. We didn't manage to make the Request Tracker output bot working and we are not entirely sure whether it is because we have just missed something in its configuration or whether it has some other problem. Thus, what is the current status of this bot? Is it still usable with RT versions 4.x and 5.0.x ? Ideally, we'd like to have/create an RT + RT::IR output bot that uses the newer RT REST API 2.0. If there's anyone with similar endeavours, I'd be happy to hear from you. Br, Mika

2 3

IntelMQ 2.2.3 Christmas release
by Sebastian Wagner 23 Dec '20

23 Dec '20

Merry Christmas, dear community :) More or less last minute I decided to do a bugfix release before the holidays *really* start, because we already collected some fixed in the last weeks/months. There a no spectacular changes in this minor release, but the upcoming 2.3.0 will have some major changes for the IntelMQ Manager backend / the new IntelMQ API and the deprecation of Python 3.5. Installation documentation: https://github.com/certtools/intelmq/blob/2.2.3/docs/INSTALL.md Upgrade documentation: https://github.com/certtools/intelmq/blob/2.2.3/docs/UPGRADING.md The deb/rpm packages will be available in the repositories in the next few hours. The NEWS: ### Harmonization A bug in the taxonomy expert did set the Taxonomy for the type `scanning` to `information gathering` whereas for the type `sniffing` and `social-engineering`, the taxonomy was correctly set to `information-gathering`. This inconsistency for the taxonomy `information-gathering` is now fixed, but the data eventually needs to fixed in data output (databases) as well. There are still some inconsistencies in the naming of the classification taxonomies and types, more fixes will come in version 3.0.0. See [issue #1409](https://github.com/certtools/intelmq/issues/1409). ### Postgres databases The following statements optionally update existing data. Please check if you did use these feed names and eventually adapt them for your setup! ```SQL UPDATE events SET "classification.taxonomy" = 'information-gathering' WHERE "classification.taxonomy" = 'information gathering'; ``` The full CHANGELOG: ### Documentation - Bots/Sieve expert: Add information about parenthesis in if-expressions (#1681, PR#1687 by Birger Schacht). ### Harmonization - See NEWS.md for information on a fixed bug in the taxonomy expert. ### Bots #### Collectors - `intelmq.bots.rt.collector_rt`: Log the size of the downloaded file in bytes on debug logging level. #### Parsers - `intelmq.bots.parsers.cymru.parser_cap_program`: - Add support for protocols 47 (GRE) and 59 (IPv6-NoNxt). - Add support for field `additional_asns` in optional information column. - `intelmq.bots.parsers.microsoft.parser_ctip`: - Fix mapping of `DestinationIpInfo.DestinationIpConnectionType` field (contained a typo). - Explicitly ignore field `DestinationIpInfo.DestinationIpv4Int` as the data is already in another field. - `intelmq.bots.parsers.generic.parser_csv`: - Ignore line having spaces or tabs only or comment having leading tabs or spaces (PR#1669 by Brajneesh). - Data fields containing `-` are now ignored and do not raise an exception anymore (#1651, PR#74 by Sebastian Waldbauer). #### Experts - `intelmq.bots.experts.taxonomy.expert`: Map type `scanner` to `information-gathering` instead of `information gathering`. See NEWS file for more information. ### Tests - Travis: Deactivate tests with optional requirements on Python 3.5, as the build fails because of abusix/querycontacts version conflicts on dnspython. ### Known issues - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952). - Corrupt dump files when interrupted during writing (#870). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ 3.0, leaving CERT.at and the future of IntelMQ
by L. Aaron Kaplan 14 Nov '20

14 Nov '20

Dear incident handling automation tools list, dear IntelMQ folks, First of all, after extensive feedback from many of you, we do have a high level requirements document for IntelMQ 3.0. It's here: https://github.com/certtools/intelmq/blob/version-3.0-ideas/docs/architectu… This shall serve as a high level blueprint for IntelMQ 3.0 developments. Sebastian is working on prioritising individual items for CERT.at and then we will create individual GitHub issues and people (mostly at CERT.at) will be hacking away at it. Looking forward to this release. I'll be guiding this release however, I won't be working at CERT.at anymore starting on the 15th of Nov. Which brings me to an important conclusion: I thought long about it what we should do when core people leave CERT.at (as in my case, or... maybe Sebastian will leave one day or get run over by the famous bus which always seems to run over every team member according to manager's expectations ;-) ) In any case, the most solid approach seems to remember what IntelMQ actually is - a **community project**. It started as one, it is one , it will be one. In the last years, CERT.at did a lot of the heavy lifting and also a lot of the decisions on IntelMQ's future. However, with a couple of hundred (600?) installations worldwide, it would be wise to create an **advisory board/architecture board** for the future developments. I would envision a small-ish group of 4-8 people who take the responsibility of guiding the project for the next ~5 years. This means: - staying on top of current developments - coordinating with the other group members - coming up with a strategy and procedures (for example, compare with PEP, maybe a lightweight PEP approach is enough) etc. - ultimately, guiding the project It's work, for sure. You should have some passion for the project of course. I sent out a couple of invite requests to individuals but also would be interested to hear from you, if you would like to participate in such an effort. Hence, IntelMQ will become its own entity. And that's good, healthy and ensures a maximum benefit for many users. If you would like to be on that board, please send me an email. I'll guide it initially and get everything started. All the best, Aaron Kaplan (private email address for the future: aaron(a)lo-res.org) PS: we already have one or two companies offering development support for IntelMQ, I would like that they can thrive in this project as well - on a friendly basis. In the long run, this will make the project stronger. -- // L. Aaron Kaplan <kaplan(a)cert.at> - T: +43 1 5056416 78 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ Tutorial and a new documentation page
by Sebastian Wagner 09 Nov '20

09 Nov '20

Dear community, I have two news items for you, both related to documentation: First, there is an IntelMQ Tutorial, which guides through various features and tools of IntelMQ. Lesson one introduces the architecture, concepts and terminology of the project. Lessons two and three delve hands-on into working with IntelMQ. Starting with installation and basic usage & configuration they go on to tackle progressively more advanced topics like using advanced features or changing the message queue software to be used. Solutions and explanations are offered for all tasks. In the last lesson you'll learn how to use intelmq-tools, a third-party software which makes customization of your IntelMQ instance much easier. We think that this kind of interactive online documentation is especially important nowadays when conferences and workshops cannot take place in real life. As for all other IntelMQ components, we welcome any contributions and feedback to the tutorial. -> https://github.com/certtools/intelmq-tutorial Second, we have a new IntelMQ Documentation page: We completely revised the way IntelMQ's documentation is presented: Instead of single files in the source-code repository, the best place to read the documentation is now intelmq.readthedocs.io. All pages are generated using Sphinx, the de facto standard tool for documentation. It features a better reading experience and a significantly improved navigation. Furthermore, the new page offers an integrated search as well as module index covering the complete code documentation If you find any bugs or have improvements, please let us know! -> https://intelmq.readthedocs.io/ best regards Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 1

IntelMQ release 2.2.2
by Sebastian Wagner 28 Oct '20

28 Oct '20

Dear community, It's again long overdue for a new release and here it is finally. Since August we collected quite a few bugfixes - Thanks to all contributors! IntelMQ Installation documentation: https://github.com/certtools/intelmq/blob/2.2.2/docs/INSTALL.md IntelMQ Upgrade documentation: https://github.com/certtools/intelmq/blob/2.2.2/docs/UPGRADING.md *News for IntelMQ 2.2.2* ### Bots #### Cymru Whois Lookup The cache key calculation has been fixed. It previously led to duplicate keys for different IP addresses and therefore wrong results in rare cases. The cache key calculation is intentionally not backwards-compatible. Therefore, this bot may take longer processing events than usual after applying this update. More details can be found in [issue #1592](https://github.com/certtools/intelmq/issues/1592). ### Harmonization #### Shadowserver Feed/Parser The feed "Blacklisted-IP" has been renamed by ShadowServer to "Blocklist". In IntelMQ, the old name can still be used in IntelMQ until version 3.0. *Changes for IntelMQ 2.2.2* ### Core - `intelmq.lib.upgrades`: - Add upgrade function for renamed Shadowserver feed name "Blacklisted-IP"/"Blocklist". ### Bots #### Parsers - `intelmq.bots.parsers.shadowserver`: - Rename "Blacklisted-IP" feed to "Blocklist", old name is still valid until IntelMQ version 3.0 (PR#1588 by Thomas Hungenberg). - Added support for the feeds `Accessible Radmin` and `CAIDA IP Spoofer` (PR#1600 by sinus-x). - `intelmq.bots.parsers.anubisnetworks.parser`: Fix parsing error where `dst.ip` was not equal to `comm.http.host`. - `intelmq/bots/parsers/danger_rulez/parser`: correctly skip malformed rows by defining variables before referencing (PR#1601 by Tomas Bellus). - `intelmq.bots.parsers.misp.parser: Fix MISP Event URL (#1619, PR#1618 by Nedfire23). - `intelmq.bots.parsers.microsoft.parser_ctip`: - Add support for `DestinationIpInfo.*` and `Signatures.Sha256` fields, used by the `ctip-c2` feed (PR#1623 by Mikk Margus Möll). - Use `extra.payload.text` for the feed's field `Payload` if the content cannot be decoded (PR#1610 by Giedrius Ramas). #### Experts - `intelmq.bots.experts.cymru_whois`: - Fix cache key calculation which previously led to duplicate keys and therefore wrong results in rare cases. The cache key calculation is intentionally not backwards-compatible (#1592, PR#1606). - The bot now caches and logs (as level INFO) empty responses from Cymru (PR#1606). ### Documentation - README: - Add Core Infrastructure Initiative Best Practices Badge. - Bots: - Generic CSV Parser: Add note on escaping backslashes (#1579). - Remove section of non-existing "Copy Extra" Bot. - Explain taxonomy expert. - Add documentation on n6 parser. - Gethostbyname expert: Add documentation how errors are treated. - Feeds: - Fixed bot modules of Calidog CertStream feed. - Add information on Microsoft CTIP C2 feed. ### Packaging - In Debian packages, `intelmqctl check` and `intelmqctl upgrade-config` are executed in the postinst step (#1551, PR#1624 by Birger Schacht). ### Tests - `intelmq.tests.lib.test_pipeline`: Skip `TestAmqp.test_acknowledge` on Travis with Python 3.8. - `intelmq.tests.bots.outputs.elasticsearch.test_output`: Refresh index `intelmq` manually to fix random test failures (#1593, PR#1595 by Zach Stone). ### Tools - `intelmqctl check`: - For disabled bots which do not have any pipeline connections, do not raise an error, but only warning. - Fix check on source/destination queues for bots as well the orphaned queues. ### Contrib - Bash completion scripts: Check both `/opt/intelmq/` as well as LSB-paths (`/etc/intelmq/` and `/var/log/intelmq/`) for loading bot information (#1561, PR#1628 by Birger Schacht). ### Known issues - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952). - Corrupt dump files when interrupted during writing (#870). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ & IntelMQ Manager releases 2.2.1
by Sebastian Wagner 30 Jul '20

30 Jul '20

Dear community, Today we have again a twin release 2.2.1 for both IntelMQ as well as IntelMQ Manager. This IntelMQ Manager version requires IntelMQ >= 2.2.1. There are currently issues with the signature in the package repositories for Debian/Ubuntu. I hope to get them resolved soon. IntelMQ Installation documentation: https://github.com/certtools/intelmq/blob/2.2.1/docs/INSTALL.md IntelMQ Upgrade documentation: https://github.com/certtools/intelmq/blob/2.2.1/docs/UPGRADING.md IntelMQ Manager Installation instructions: https://github.com/certtools/intelmq-manager/blob/2.2.1/docs/INSTALL.md *The changelog for IntelMQ Manager:* ### Backend - Fix loading paths from `intelmqctl` executable (PR #205 by Einar Felipe Lanfranco). ### Documentation - User Guide: - Add section on configuration paths. - Add section on named queues / paths. - Readme: - Update screenshots (#201, PR#207 by Mladen Markovic). ### Known issues * Graph jumps around on "Add edge" (#148). * wrong error message for new bots with existing ID (#152). * Monitor page: Automatic log refresh reset log page to first one (#190). *The News for IntelMQ:* ### Requirements #### MaxMind GeoIP Expert Bot The current python library versions of geoip (version 4) and maxminddb (version 2) no longer support Python 3.5. Keep older versions of these libraries if you are using this Python version. ### Configuration #### Abuse.ch URLHaus The current documented value for the `column` parameter was: ```json ['time.source', 'source.url', 'status', 'extra.urlhaus.threat_type', 'source.fqdn', 'source.ip', 'source.asn', 'source.geolocation.cc'] ``` Better is: ```json ['time.source', 'source.url', 'status', 'classification.type|__IGNORE__', 'source.fqdn|__IGNORE__', 'source.ip', 'source.asn', 'source.geolocation.cc'] ``` *And the changelog for IntelMQ:* ### Core - `intelmq.lib.upgrades`: - Add upgrade function for changed configuration of the feed "Abuse.ch URLHaus" (#1571, PR#1572 by Filip Pokorný). - Add upgrade function for removal of *HPHosts Hosts file* feed and `intelmq.bots.parsers.hphosts` parser (#1559). - `intelmq.lib.harmonization`: - For IP Addresses, explicitly reject IPv6 addresses with scope ID (due to changed behavior in Python 3.9, #1550). ### Development - Ignore line length (E501) in code-style checks altogether. ### Bots #### Collectors - `intelmq.bots.collectors.misp`: Fix access to actual MISP object (PR#1548 by Tomas Bellus @tomas321) - `intelmq.bots.collectors.stomp`: Remove empty `client.pem` file. #### Parsers - `intelmq.bots.parsers.shadowserver.config`: - Add support for Accessible-CoAP feed (PR #1555 by Thomas Hungenberg). - Add support for Accessible-ARD feed (PR #1584 by Tomas Bellus @tomas321). - `intelmq.bots.parser.anubisnetworks.parser`: Ignore "TestSinkholingLoss" events, these are not intended to be sent out at all. - `intelmq.bots.parsers.generic.parser_csv`: Allow values of type dictionary for parameter `type_translation`. - `intelmq.bots.parsers.hphosts`: Removed, feed is unavailable (#1559). - `intelmq.bots.parsers.cymru.parser_cap_program`: Add support for comment "username" for "scanner" category. - `intelmq.bots.parsers.malwareurl.parser`: Check for valid FQDN and IP address in URL and IP address columns (PR#1585 by Marius Urkis). #### Experts - `intelmq.bots.experts.maxmind_geoip`: On Python < 3.6, require maxminddb < 2, as that version does no longer support Python 3.5. #### Outputs - `intelmq.bot.outputs.udp`: Fix error handling on sending, had a bug itself. ### Documentation - Feeds: - Update documentation of feed "Abuse.ch URLHaus" (#1571, PR#1572 by Filip Pokorný). - Bots: - Overhaul of all bots' description fields (#1570). - User-Guide: - Overhaul pipeline configuration section and explain named queues better (#1577). ### Tests - `intelmq.tests.bots.experts.cymru`: Adapt `test_empty_result`, remove `test_unicode_as_name` and `test_country_question_mark` (#1576). ### Tools - `intelmq.bin.intelmq_gen_docs`: Format parameters of types lists with double quotes around values to produce conform JSON, ready to copy and paste the value into the IntelMQ Manager's bot parameter form. - `intelmq.bin.intelmqctl`: - `debug`: In JSON mode, use dictionaries instead of lists. - `debug`: Add `PATH` to the paths shown. - `check`: Show `$PATH` environment variable if executable cannot be found. ### Contrib - `malware_name_mapping`: Change MISP Threat Actors URL to new URL (branch master -> main) in download script. ### Known issues - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952). - Corrupt dump files when interrupted during writing (#870). - Bash completion scripts search in wrong directory in packages (#1561). - Cymru Expert: Wrong Cache-Key Calculation (#1592). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ Manager 2.2.0
by Sebastian Wagner 24 Jun '20

24 Jun '20

Dear community, To get the versions of intelmq and intelmq-manager in sync, we of course need a 2.2.0 for intelmq-manager too :) The number of changes is very short this time. However, a migration of the backend from PHP to Python, coded by Intevation and funded by SUNET, is in progress :) The deb/rpm packages are currently building, I will send a separate mail to intelmq-users once the 2.2.0 of intelmq and intelmq-manager packages have finally hit the repositories. This IntelMQ Manager version requires IntelMQ >= 2.2.0. Installation instructions: https://github.com/certtools/intelmq-manager/blob/2.2.0/docs/INSTALL.md Full changelog: ### Backend - `config`: Get file paths from `intelmctl debug --get-paths` if possible and fall back to hard-coded paths otherwise. Thereby environment variables influencing the paths are respected (#193). ### Pages #### About - Show output of `intelmqctl debug`. ### Documentation - Update release from intelmq's release documentation. - Update Installation documentation: Fix & update dependencies and supported operating systems. ### Packaging - Update default `positions.conf` to the default runtime/pipeline configuration of intelmq >= 2.1.1. ### Known issues * Missing CSRF protection (#111). * Graph jumps around on "Add edge" (#148). * wrong error message for new bots with existing ID (#152). * `ALLOWED_PATH=` violates CSP (#183). * Monitor page: Automatic log refresh reset log page to first one (#190). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ 2.2.0 release
by Sebastian Wagner 18 Jun '20

18 Jun '20

Dear community, Today I finalized the 2.2.0 Feature release, more than half a year after 2.1.0. Thanks to all the contributors who made this possible! The release includes six new bots and seven heavily revised bots, and of course a lot of small changes to various bots. The full changelog can be found below. We dropped support for Python 3.4, that means Debian 8.0 and similar operating systems are no longer supported. Installation documentation: https://github.com/certtools/intelmq/blob/2.2.0/docs/INSTALL.md Upgrade documentation: https://github.com/certtools/intelmq/blob/2.2.0/docs/UPGRADING.md ### Core - `__init__`: Changes to the path-handling, see [User Guide, section _/opt and LSB paths_](docs/User-Guide.md#opt-and-lsb-paths) for more information - The environment variable `INTELMQ_ROOT_DIR` can be used to set custom root directories instead of `/opt/intelmq/` (#805) in case of non LSB-path installations. - The environment variable `ROOT_DIR` can be used to set custom root directories instead of `/` (#805) in case of LSB-path installations. - `intelmq.lib.exceptions`: Added `MissingDependencyError` for show error messages about a missing library and how to install it (#1471). - Added optional parameter `installed` to show the installed version. - Added optional parameter `additional_text` to show arbitrary text. - Adding more type annotations for core libraries. - `intelmq.lib.pipeline.Pythonlist.sleep`: Drop deprecated method. - `intelmq.lib.utils`: `write_configuration`: Append a newline at end of configuration/file to allow proper comparisons & diffs. - `intelmq.lib.test`: `BotTestCase` drops privileges upon initialization (#1489). - `intelmq.lib.bot`: - New class `OutputBot`: - Method `export_event` to format/export events according to the parameters given by the user. - `ParserBot`: New methods `parse_json_stream` and `recover_line_json_stream`. - `ParserBot.recover_line_json`: Fix format by adding a list around the line data. - `Bot.send_message`: In debugging log level, the path to which the message is sent is now logged too. ### Bots - Bots with dependencies: Use of `intelmq.lib.exceptions.MissingDependencyError`. #### Collectors - `intelmq.bots.collectors.misp.collector`: Deprecate parameter `misp_verify` in favor of generic parameter `http_verify_cert`. - `intelmq.bots.collectors.tcp.collector`: Drop compatibility with Python 3.4. - `intelmq.bots.collectors.stomp.collector`: - Check the stomp.py version and show an error message if it does not match. - For stomp.py versions `>= 5.0.0` redirect the `stomp.PrintingListener` output to debug logging. - `intelmq.bots.collectors.microsoft.collector_azure`: Support current Python library `azure-storage-blob>= 12.0.0`, configuration is incompatible and needs manual change. See NEWS file and bot's documentation for more details. - `intelmq.bots.collectors.amqp.collector_amqp`: Require `pika` minimum version 1.0. - `intelmq.bots.collectors.github_api.collector_github_contents_api`: Added (PR#1481). #### Parsers - `intelmq.bots.parsers.autoshun.parser`: Drop compatibility with Python 3.4. - `intelmq.bots.parsers.html_table.parser`: Drop compatibility with Python 3.4. - `intelmq.bots.parsers.shadowserver.parser`: Add support for MQTT and Open-IPP feeds (PR#1512, PR#1544). - `intelmq.bots.parsers.taichung.parser`: - Migrate to `ParserBot`. - Also parse geolocation information if available. - `intelmq.bots.parsers.cymru.parser_full_bogons`: - Migrate to `ParserBot`. - Add last updated information in raw. - `intelmq.bots.parsers.anubisnetworks.parser`: Add new parameter `use_malware_familiy_as_classification_identifier`. - `intelmq.bots.parsers.microsoft.parser_ctip`: Compatibility for new CTIP data format used provided by the Azure interface. - `intelmq.bots.parsers.cymru.parser_cap_program`: Support for `openresolver` type. - `intelmq.bots.parsers.github_feed.parser`: Added (PR#1481). - `intelmq.bots.parsers.urlvir.parser`: Removed, as the feed is discontinued (#1537). #### Experts - `intelmq.bots.experts.csv_converter`: Added as converter to CSV. - `intelmq.bots.experts.misp`: Added (PR#1475). - `intelmq.bots.experts.modify`: New parameter `maximum_matches`. #### Outputs - `intelmq.bots.outputs.amqptopic`: - Use `OutputBot` and `export_event`. - Allow formatting the routing key with event data by the new parameter `format_routing_key` (boolean). - `intelmq.bots.outputs.file`: Use `OutputBot` and `export_event`. - `intelmq.bots.outputs.files`: Use `OutputBot` and `export_event`. - `intelmq.bots.outputs.misp.output_feed`: Added, creates a MISP Feed (PR#1473). - `intelmq.bots.outputs.misp.output_api`: Added, pushes to MISP via the API (PR#1506, PR#1536). - `intelmq.bots.outputs.elasticsearch.output`: Dropped ElasticSearch version 5 compatibility, added version 7 compatibility (#1513). ### Documentation - Document usage of the `INTELMQ_ROOT_DIR` environment variable. - Added document on MISP integration possibilities. - Feeds: - Added "Full Bogons IPv6" feed. - Remove discontinued URLVir Feeds (#1537). ### Packaging - `setup.py` do not try to install any data to `/opt/intelmq/` as the behavior is inconsistent on various systems and with `intelmqsetup` we have a tool to create the structure and files anyway. - `debian/rules`: - Provide a blank state file in the package. - Patches: - Updated `fix-intelmq-paths.patch`. ### Tests - Travis: Use `intelmqsetup` here too. - Install required build dependencies for the Debian package build test. - This version is no longer automatically tested on Python `<` 3.5. - Also run the tests on Python 3.8. - Run the Debian packaging tests on Python 3.5 and the code-style test on 3.8. - Added tests for the new bot `intelmq.bots.outputs.misp.output_feed` (#1473). - Added tests for the new bot `intelmq.bots.experts.misp.expert` (#1473). - Added tests for `intelmq.lib.exceptions`. - Added tests for `intelmq.lib.bot.OutputBot` and `intelmq.lib.bot.OutputBot.export_event`. - Added IPv6 tests for `intelmq.bots.parsers.cymru.parser_full_bogons`. - Added tests for `intelmq.lib.bot.ParserBot`'s new methods `parse_json_stream` and `recover_line_json_stream`. - `intelmq.tests.test_conf`: Set encoding to UTF-8 for reading the `feeds.yaml` file. ### Tools - `intelmqctl`: - `upgrade-config`: - Allow setting the state file location with the `--state-file` parameter. - Do not require a second run anymore, if the state file is newly created (#1491). - New parameter `no_backup`/`--no-backup` to skip creation of `.bak` files for state and configuration files. - Only require `psutil` for the `IntelMQProcessManager`, not for process manager independent calls like `upgrade-config` or `check`. - Add new command `debug` to output some information for debugging. Currently implemented: - paths - environment variables - `IntelMQController`: New argument `--no-file-logging` to disable logging to file. - If dropping privileges does not work, `intelmqctl` will now abort (#1489). - `intelmqsetup`: - Add argument parsing and an option to skip setting file ownership, possibly not requiring root permissions. - Call `intelmqctl upgrade-config` and add argument for the state file path (#1491). - `intelmq_generate_misp_objects_templates.py`: Tool to create a MISP object template (#1470). - `intelmqdump`: New parameter `-t` or `--truncate` to optionally give the maximum length of `raw` data to show, 0 for no truncating. ### Contrib - Added `development-tools`. - ElasticSearch: Dropped version 5 compatibility, added version 7 compatibility (#1513). - Malware Name Mapping Downloader: - New parameter `--mwnmp-ignore-adware`. - The parameter `--add-default` supports an optional parameter to define the default value. ### Known issues - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952). - Corrupt dump files when interrupted during writing (#870). best regards Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

← Newer
1
...
9
10
11
12
13
14
15
...
25
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev