- IntelMQ-users

IntelMQ 2.1.3 Bugfix release
by Sebastian Wagner 26 May '20

26 May '20

Dear community, We already collected a very long list of bug fixes since the last release, so it was time to mark the next milestone! As usual, you can find the list of changes below. The pre-built deb/rpm packages will hit the repositories very soon. Installation documentation: https://github.com/certtools/intelmq/blob/2.1.3/docs/INSTALL.md Upgrade documentation: https://github.com/certtools/intelmq/blob/2.1.3/docs/UPGRADING.md Full changelog: ### Requirements - The python library `requests` is (again) listed as dependency of the core (#1519). ### Core - `intelmq.lib.upgrades`: - Harmonization upgrade: Also check and update regular expressions. - Add function to migrate the deprecated parameter `attach_unzip` to `extract_files` for the mail attachment collector. - Add function to migrate changed Taichung URL feed. - Check for discontinued Abuse.CH Zeus Tracker feed. - `intelmq.lib.bot`: - `ParserBot.recover_line`: Parameter `line` needs to be optional, fix usage of fallback value `self.current_line`. - `start`: Handle decoding errors in the pipeline different so that the bot is not stuck in an endless loop (#1494). - `start`: Only acknowledge a message in case of errors, if we actually had a message to dump, which is not the case for collectors. - `_dump_message`: Dump messages with encoding errors base64 encoded, not in JSON format as it's not possible to decode them (#1494). - `intelmq.lib.test`: - `BotTestCase.run_bot`: Add parameters `allowed_error_count` and `allowed_warning_count` to allow set the number per run, not per test class. - Set `source_pipeline_broker` and `destination_pipeline_broker` to `pythonlist` instead of the old `broker`, fixes `intelmq.tests.lib.test_bot.TestBot.test_pipeline_raising`. - Fix test for (allowed) errors and warnings. - `intelmq.lib.exceptions`: - `InvalidKey`: Add `KeyError` as parent class. - `DecodingError`: Added, string representation has all relevant information on the decoding error, including encoding, reason and the affected string (#1494). - `intelmq.lib.pipeline`: - Decode messages in `Pipeline.receive` not in the implementation's `_receive` so that the internal counter is correct in case of decoding errors (#1494). - `intelmq.lib.utils`: - `decode`: Raise new `DecodingError` if decoding fails. ### Harmonization - `protocol.transport`: Adapt regular expression to allow the value `nvp-ii` (protocol 11). ### Bots #### Collectors - `intelmq.bots.collectors.mail.collector_mail_attach`: - Fix handling of deprecated parameter name `attach_unzip`. - Fix handling of attachments without filenames (#1538). - `intelmq.bots.collectors.stomp.collector`: Fix compatibility with stomp.py versions `> 4.1.20` and catch errors on shutdown. - `intelmq.bots.collectors.microsoft`: - Update `REQUIREMENTS.txt` temporarily fixing deprecated Azure library (#1530, PR#1532). - `intelmq.bots.collectors.microsoft.collector_interflow`: Add method for printing the file list. #### Parsers - `intelmq.bots.parsers.cymru.parser_cap_program`: Support for protocol 11 (`nvp-ii`) and `conficker` type. - `intelmq.bots.parsers.taichung.parser`: Support more types/classifications: - Application Compromise: Apache vulnerability & SQL injections - Brute-force: MSSQL & SSH password guess attacks; Office 365, SSH & SIP attacks - C2 Sever: Attack controller - DDoS - DoS: DNS, DoS, Excess connection - IDS Alert / known vulnerability exploitation: backdoor - Malware: Malware Proxy - Warn on new unknown types. - `intelmq.bots.parsers.bitcash.parser`: Removed as feed is discontinued. - `intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc` and `intelmq.bots.parsers.fraunhofer.parser_ddosattack_target`: Removed as feed is discontinued. - `intelmq.bots.parsers.malwaredomains.parser`: Correctly classify `C&C` and `phishing` events. - `intelmq.bots.parsers.shadowserver.parser`: More verbose error message for missing report specification (#1507). - `intelmq.bots.parsers.n6.parser_n6stomp`: Always add n6 field `name` as `malware.name` independent of `category`. - `intelmq.bots.parsers.anubisnetworks`: Update parser with new data format. - `intelmq.bots.parsers.bambenek`: Add new feed URLs with Host `faf.bambenekconsulting.com` (#1525, PR#1526). - `intelmq.bots.parsers.abusech.parser_ransomware`: Removed, as the feed is discontinued (#1537). - `intelmq.bots.parsers.nothink.parser`: Removed, as the feed is discontinued (#1537). - `intelmq.bots.parsers.n6.parser`: Remove not allowed characters in the name field for `malware.name` and write original value to `event_description.text` instead. #### Experts - `intelmq.bots.experts.cymru_whois.lib`: Fix parsing of AS names with Unicode characters. #### Outputs - `intelmq.bots.outputs.mongodb`: - Set default port 27017. - Use different authentication mechanisms per MongoDB server version to fix compatibility with server version >= 3.4 (#1439). ### Documentation - Feeds: - Remove unavailable feed Abuse.CH Zeus Tracker. - Remove the field `status`, offline feeds should be removed. - Add a new field `public` to differentiate between private and public feeds. - Adding documentation URLs to nearly all feeds. - Remove unavailable Bitcash.cz feed. - Remove unavailable Fraunhofer DDos Attack feeds. - Remove unavailable feed Abuse.CH Ransomware Tracker (#1537). - Update information on Bambenek Feeds, many require a license now (#1525). - Remove discontinued Nothink Honeypot Feeds (#1537). - Developers Guide: Fix the instructions for `/opt/intelmq` file permissions. ### Packaging - Patches: `fix-logrotate-path.patch`: also include path to rotated file in patch. - Fix paths from `/opt` to LSB for `setup.py` and `contrib/logrotate/intelmq` in build process (#1500). - Add runtime dependency `debianutils` for the program `which`, which is required for `intelmqctl`. ### Tests - Dropping Travis tests for 3.4 as required libraries dropped 3.4 support. - `intelmq.tests.bots.experts.cymru_whois`: - Drop missing ASN test, does not work anymore. - IPv6 to IPv4 test: Test for two possible results. - `intelmq.lib.test`: Fix compatibility of logging capture with Python >= 3.7 by reworking the whole process (#1342). - `intelmq.bots.collectors.tcp.test_collector`: Removing custom mocking and bot starting, not necessary anymore. - Added tests for `intelmq.bin.intelmqctl.IntelMQProcessManager._interpret_commandline`. - Fix and split `tests.bots.experts.ripe.test_expert.test_ripe_stat_error_json`. - Added tests for invalid encodings in input messages in `intelmq.tests.lib.test_bot` and `intelmq.tests.lib.test_pipeline` (#1494). - Travis: Explicitly enable RabbitMQ management plugin. - `intelmq.tests.lib.test_message`: Fix usage of the parameter `blacklist` for Message hash tests (#1539). ### Tools - `intelmqsetup`: Copy missing BOTS file to IntelMQ's root directory (#1498). - `intelmq_gen_docs`: Feed documentation generation: Handle missing/empty parameters. - `intelmqctl`: - `IntelMQProcessManager`: For the status of running bots also check the bot ID of the commandline and ignore the path of the executable (#1492). - `IntelMQController`: Fix exit codes of `check` command for JSON output (now 0 on success and 1 on error, was swapped, #1520). - `intelmqdump`: - Handle base64-type messages for show, editor and recovery actions. ### Contrib - `intelmq/bots/experts/asn_lookup/update-asn-data`: Use `pyasn_util_download.py` to download the data instead from RIPE, which cannot be parsed currently (#1517, PR#1518, https://github.com/hadiasghari/pyasn/issues/62). ### Known issues - HTTP stream collector: retry on regular connection problems? (#1435). - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952). - Reverse DNS: Only first record is used (#877). - Corrupt dump files when interrupted during writing (#870). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Request for feedback - IntelMQ 3.0 architecture
by L. Aaron Kaplan 08 May '20

08 May '20

Dear IntelMQ community, I hope everyone is fine and most of us are slowly waking up after the Corona lock-down phase ( at least I am, and it's a really good feeling to be able to work concentrated on things again. Let me tell you, home-office with small children is not fun). In any case, I already had some telcos with some of you regarding our wish list for IntelMQ-3.0. There are some fantastic ideas floating around and since IntelMQ is such a versatile and adaptable tool, it is quite a challenge to capture all the great changes and adaptations people made to it. The next version of IntelMQ (3.0) will try to capture some of the most needed changes (and some wishes) for you. I tried to summarise some of these changes in the following document: https://github.com/certtools/intelmq/blob/version-3.0-ideas/docs/architectu… May I ask you to review it and send me feedback (kaplan(a)cert.at)? Also, I'd be very happy to have a telco with you in case that's easier or more natural for you for giving feedback. Since we would like to start coding soon, it is important to receive *your* input on the next version of IntelMQ. Thanks for your time! All the best, Aaron. -- // L. Aaron Kaplan <kaplan(a)cert.at> - T: +43 1 5056416 78 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ Manager 2.1.1 Security Bugfix Release
by Sebastian Wagner 27 Apr '20

27 Apr '20

Dear community, The newest IntelMQ Manager release 2.1.1 fixes a critical security bug. Please never run the IntelMQ Manager without proper authentication in place! Installation instructions: https://github.com/certtools/intelmq-manager/blob/2.1.1/docs/INSTALL.md Bernhard Herzog (Intevation) discovered that the backend incorrectly handled messages given by user-input in the "send" functionality of the Inspect-tool of the Monitor component. An attacker with access to the IntelMQ Manager could possibly use this issue to execute arbitrary code with the privileges of the webserver. Updated deb/rpm-packages are already available in the repositories. Other changes: ### Backend - Fix misspelling of the environmental variable `INTELMQ_MANGER_CONTROLLER_CMD` to `INTELMQ_MANAGER_CONTROLLER_CMD` (an 'a' was missing). - Fix handling of POST variable `msg` of the message-sending functionality available in the Inspect-tool. ### Pages #### Monitor - Fix running commands with the "inspect" widget by fixing the definition of the `CONTROLLER_CMD` in the template (PR #194). ### Documentation - Update supported operating systems in Installation documentation (i.a. PR #191). ### Known issues * Missing CSRF protection (#111). * Graph jumps around on "Add edge" (#148). * wrong error message for new bots with existing ID (#152). * `ALLOWED_PATH=` violates CSP (#183). * Monitor page: Automatic log refresh reset log page to first one (#190). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Reverse DNS Expert - unusual behavior
by Tomislav Protega 31 Mar '20

31 Mar '20

Hi, recently I noticed that reverse DNS expert bot doesn't correctly apply the reverse lookup results for IP. Meaning, the right value (result) is not applied for the right JSON event. It's like it's skipping it and then applies it to other event. There are no errors in log file for the bot. For the illustration: Let say that hostname for IP 1.1.1.1 is "xx.yyy.zz", but instead the mentioned hostname becomes applied under wrong JSON event for the IP which in real has no PTR record in DNS. Of course, there are events which have applied right PTR record for the IP, but in rare situations. This case is not the issue with raw events which already contain hostname in origin feed. Anyone notice such behavor, or could take a look at already processed data and see if the "source.reverse_dns" has right value applied against IP? Regards, -- Tomislav

2 5

MAIL ATTACHMENT FETCHER FAILS TO READ/PARSE CSV FILE.
by patric sungura 19 Mar '20

19 Mar '20

Dear Team,Please assist me on mail attachment fetcher.Intelmq fails to read/parse csv file as per below erros from log and dump. ShadowServer-Parser-2.log 2020-03-17 16:10:42,064 - ShadowServer-Parser-2 - ERROR - Bot has found a problem.Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 267, in start self.process() File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 941, in process for line in self.parse(report): File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 884, in parse_csv_dict self.csv_fieldnames = csv_reader.fieldnames File "/usr/lib/python3.5/csv.py", line 96, in fieldnames self._fieldnames = next(self.reader, dialect=csv.excel_tab)NameError: name 'csv' is not defined2020-03-17 16:10:42,065 - ShadowServer-Parser-2 - INFO - Dumping message to dump file. ShadowServer-Parser-2.dump "source_queue": "ShadowServer-Parser-2-queue", "traceback": [ "Traceback (most recent call last):\n", " File \"/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py\", line 267, in start\n self.process()\n", " File \"/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py\", line 941, in process\n for line in self.parse(report):\n", " File \"/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py\", line 884, in parse_csv_dict\n self.csv_fieldnames = csv_reader.fieldnames\n", " File \"/usr/lib/python3.5/csv.py\", line 96, in fieldnames\n self._fieldnames = next(self.reader, dialect=csv.excel_tab)\n", "NameError: name 'csv' is not defined\n" ] }

3 16

RIPE Expert Geolocation data (currently) not available anymore
by Sebastian Wagner 18 Mar '20

18 Mar '20

Hi, I just noticed, that RIPE currently does not provide geolocation information anymore as a result of the MaxMind data license change. That data can/could be queried with the IntelMQ RIPE expert. In case you are still relying on this information, please use another source for geolocation data, like the maxmind geolocation expert and local data. Unfortunately, the returned status code of the API call is 200 and the error is only detectable by another field. I am working on changes in the RPIE expert to detect this and raise a warning for it. best regards, Sebastian For example https://stat.ripe.net/data/maxmind-geo-lite/data.json?resource=131.130.254.… says: messages 0 0 "info" 1 "This data is currently unavailable due to maintenance. Please check official announcements for when it will be available again! https://stat.ripe.net/feedback" data_call_status "maintenance - this data call is in maintenance mode" -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 2

Long filenames and Generic Mail Attachment Fetcher
by Patrick Forsberg 28 Feb '20

28 Feb '20

Hi, I'm pretty sure some of you (at least Sebastian) is aware of this, but there is currently a problem with the intelmq.bots.collectors.mail.collector_mail_attach and attachments with long filenames. The error is not in the bot itself, but in the underlying imbox library that doesn't handle long filenames spread over multiple "filename*=" lines in Content-Disposition. The end result is that some attachments will probably fail to be extracted and it will look similar to the following line in the Collector log. Shadowserver-Mail-Attachment-Fetcher-Collector - INFO - Attachment sv.zip didn't match regex. There is a non merged pull request in the imbox Git repository to handle this but it hasn't been merged with the main repo. The solution is to patch imbox/parser.py I've attached my patch against the pip3 version of imbox (slightly different to the umerged Git pull request) Regards, Patrick Forsberg SUNET CERT

2 1

IntelMQ
by UCC-CERT 20 Feb '20

20 Feb '20

Dear Experts, We currently have a mail box which contains only shadow server feeds attachment files in a zipped form. The IntelMQ is able to read the emails but cannot extract and forward them to the shadow server parser. We need your assistance . See details below Configuration From Runtime.conf ---------------------------------------------------------------------------- -------------------------- "Mail-Attachment-Fetcher-Collector": { "parameters": { "extract_files": "True", "attach_regex": "[A-Za-z:0-9\\.\\_ \\[\\]\\-]", "folder": "INBOX", "mail_host": "imap.xxxx.xxx", "mail_password": "xxxxxxxxxx", "mail_ssl": true, "mail_user": "johndoe", "name": "Via IMAP", "provider": "ShadowServer", "rate_limit": 86400, "subject_regex": "[A-Za-z:0-9 \\[\\]\\-]" }, "name": "Mail Attachment Fetcher", "group": "Collector", "module": "intelmq.bots.collectors.mail.collector_mail_attach", "description": "Monitor IMAP mailboxes and retrieve mail attachments", "enabled": true, "run_mode": "continuous" Below are the logs tail -n 1000 Mail-Attachment-Fetcher-Collector.log 2020-02-18 18:31:12,672 - Mail-Attachment-Fetcher-Collector - INFO - Email report read. 2020-02-18 18:31:19,310 - Mail-Attachment-Fetcher-Collector - INFO - Email report read. 2020-02-18 18:31:25,574 - Mail-Attachment-Fetcher-Collector - INFO - Email report read. 2020-02-18 18:31:31,816 - Mail-Attachment-Fetcher-Collector - INFO - Email report read. Should you need any further information, please do not hesitate to contact me. Thanks Regards, Vincent M UG-CERT

4 10

IntelMQ version 2.1.2
by Sebastian Wagner 28 Jan '20

28 Jan '20

Dear community, Version 2.1.2, a bugfix release, is out which contains various bugfixes. Install documentation: https://github.com/certtools/intelmq/blob/2.1.2/docs/INSTALL.md Upgrade documentation: https://github.com/certtools/intelmq/blob/2.1.2/docs/UPGRADING.md The full changelog: ### Core - `__init__`: Resolve absolute path for `STATE_FILE_PATH` variable (resolves `..`). - `intelmq.lib.utils`: - log: Do not raise an exception if logging to neither file nor syslog is requested. - logging StreamHandler: Colorize all warning and error messages red. - logging FileHandler: Strip all shell colorizations from the messages (#1436). - `intelmq.lib.message`: - `Message.to_json`: Set `sort_keys=True` to get reproducible results. - `drop_privileges`: Handle situations where the user or group `intelmq` does not exist. - `intelmq.lib.pipeline`: - `Amqp._send` and `Amqp._acknowledge`: Log traceback in debug mode in case of errors and necessary re-connections. - `Amqp._acknowledge`: Reset delivery tag if acknowledge was successful. ### Bots #### Collectors - `intelmq.bots.collectors.misp.collector`: - Add compatibility with current pymisp versions and versions released after January 2020 (PR #1468). #### Parsers - `intelmq.bots.parsers.shadowserver.config`: Add some missing fields for the feed `accessible-rdp` (#1463). - `intelmq.bots.parsers.shadowserver.parser`: - Feed-detection based on file names: The prefixed date is optional now. - Feed-detection based on file names: Re-detect feed for every report received (#1493). #### Experts - `intelmq.bots.experts.national_cert_contact_certat`: Handle empty responses by server (#1467). - `intelmq.bots.experts.maxmind_geoip`: The script `update-geoip-data` now requires a license key as second parameter because of upstream changes (#1484)). #### Outputs - `intelmq.bots.outputs.restapi.output`: Fix logging of response body if response status code was not ok. ### Documentation - Remove some hardcoded `/opt/intelmq/` paths from code comments and program outputs. ### Packaging - debian/rules: Only replace `/opt/intelmq/` with LSB-paths in some certain files, not the whole tree, avoiding wrong replacements. - debian/rules and debian/intelmq.install: Do install the examples configuration directly instead of working around the abandoned examples directory. ### Tests - `lib/test_utils`: Skip some tests on Python 3.4 because `contextlib.redirect_stdout` and `contextlib.redirect_sterr` are not supported on this version. - Travis: Stop running tests with all optional dependencies on Python 3.4, as more and more libraries are dropping support for it. Tests on the core and code without non-optional requirements are not affected. - `tests.bots.parsers.html_table`: Make tests independent of current year. ### Tools - `intelmqctl upgrade-config`: Fix missing substitution in error message "State file %r is not writable.". ### Known issues - bots trapped in endless loop if decoding of raw message fails (#1494) - intelmqctl status of processes: need to check bot id too (#1492) - MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439) - ctl: shell colorizations are logged (#1436) - http stream collector: retry on regular connection problems? (#1435) - tests: capture logging with context manager (#1342) - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952) - n6 parser: mapping is modified within each run (#905) - reverse DNS: Only first record is used (#877) - Corrupt dump files when interrupted during writing (#870) -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

heads up: maxmind license updates (relevant for IntelMQ)
by L. Aaron Kaplan 09 Jan '20

09 Jan '20

Dear IntelMQ users, Sebix mentioned something that might be relevant for everyone using IntelMQ (or actually maxmind for the matter). (quote) "MaxMind has always been committed to an individual’s right to privacy on the internet. We welcome the burgeoning privacy regulations, such as GDPR and CCPA, for the benefit they can provide to internet citizens. However, these new legislative measures place restrictions that impact our ability to continue distributing our GeoLite2 databases on a public page under the Creative Commons Attribution-ShareAlike 4.0 International License." Maxmind has decided to change the download mechanism of the maxmind GeoLite 2 database. You now need to be registered and need a license key. Since I know that many IntelMQ users rely on maxmind for geolocation, you might want to register there and get a new license key and make sure it can be used for the most recent database version. More info at https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-us… All the best and a happy 2020! Aaron. -- // L. Aaron Kaplan <kaplan(a)cert.at> - T: +43 1 5056416 78 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - http://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

IntelMQ-users