IntelMQ-dev

intelmq-dev@lists.cert.at

1 participants
246 discussions

new feature upgrade-config: please test
by Sebastian Wagner 30 Jul '19

30 Jul '19

Dear community, On the maintenance/develop branches there's a new functionality to fix issues in the configuration, or "upgrade" the configuration. This was some times needed in the past and only covered by entries in the NEWS file. This includes cases like: * Discontinued feeds * Errors in the shipped configuration * changed configuration syntax * renamed parameters * new needed default parameters * etc. This function also introduces the "state" file, where intelmq can keep track of which upgrades have been performed. This would already have been useful in the past. This also makes debugging problems easier and maybe we can even introduce a full upgrade function some day... So, why I am writing here is: This needs testing! Although the programs makes backups itself, make a backup of your configuration nevertheless. Then call > intelmqctl upgrade-conf And follow the output! If you want to know what exactly happens, have a look at the intelmq/lib/upgrades.py and intelmq/bin/intelmqctl.py files Have fun and please report back your experiences Sebastian P.S.: I'd like to finish the 2.0.1 release in the next days/week, so this will be part of it. -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

notification field in intelmq
by Sebastian Wagner 24 Jul '19

24 Jul '19

Dear community, Almost three years ago, we discussed how to save notification information in the intelmq format: https://github.com/certtools/intelmq/issues/758 I'd like to continue these efforts and want to hear your needs and ideas. I also provided a summary and an example of what it could look like in the issue. Thanks for your time and contribution, Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

2 2

IntelMQ & Manager 2.0.0 Release
by Sebastian Wagner 28 May '19

28 May '19

Dear community, I am pleased to announce the second major versions of IntelMQ and IntelMQ Manager. *A big thanks goes to all the contributors engaging and contributing to this project!* Before migrating, have a look at the NEWS file: https://github.com/certtools/intelmq/blob/maintenance/NEWS.md Installation instructions: https://github.com/certtools/intelmq/blob/2.0.0/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/2.0.0/docs/UPGRADING.md The release is available also in the unstable deb/rpm repositories: https://software.opensuse.org/download.html?project=home:sebix:intelmq:unst… It is *not* yet available in the stable repositories, I will do that in the next weeks or with the next bugfix release, depending on whether there are some issues popping up. Some numbers on the bots compared to the first major release: 8 new collectors since 1.0, now 21 in total 16 new parsers since 1.0, now 59 in total 37 new documented feeds since 1.0, now 138 in total 2 new experts since 1.0, now 24 in total 2 new outputs since 1.0, now 15 in total On 4th April I wrote a summary about new features to the developer's list: https://lists.cert.at/pipermail/intelmq-dev/2019-April/000293.html Here is a short summary for the core's most important new features including the links to documentation: * AMQP is available as alternative broker additionally to redis. I do *not* recommend yet to migrate if you are not familiar with RabbitMQ (an AQMP implementation). I'd still like to get more feedback on this feature. https://github.com/certtools/intelmq/blob/|maintenance|/docs/User-Guide.md#… * Supervisor can be used as replacement for intelmqctl's own pid-based process management https://github.com/certtools/intelmq/blob/|maintenance|/docs/User-Guide.md#… Note that there is also a script to generate systemd service files for intelmq bots: https://github.com/certtools/intelmq/tree/maintenance/contrib/systemd * Multithreading: By proving the parameter |instances_threads you can set the number of threads for a bot, works for most bots. https://github.com/certtools/intelmq/blob/maintenance/docs/User-Guide.md#mu… * Statistics: Mostly relevant for power users and developers, the bots save load statistics, not stable yet. Discussion is here: https://github.com/certtools/intelmq/issues/1274 * You can switch between /opt and LSB-paths with an environment variable: https://github.com/certtools/intelmq/blob/maintenance/docs/User-Guide.md#op… * "Named queue" or "paths" have already been introduced in 1.1, allowing multiple *different* destination pipelines. The filter expert supports named queues, potentially reducing the number of needed filter bots: https://github.com/certtools/intelmq/blob/maintenance/docs/Bots.md#possible… Using the sieve expert, you can set the paths for the messages too. * The separated malware name mapping since 1.1: https://github.com/certtools/intelmq/tree/maintenance/contrib/malware_name_… * The transparent usage of `extra.*` fields since 1.1. * And *a lot* of changes in the bots, it is impossible to list them here For the IntelMQ Manager: * The configuration tab shows the number of queued messages live, name of queues ("paths"), management and monitoring features. Physics can be permanently turned off (the state is saved). With the new shortcuts, you can do some actions with keyboard only. * List and array parameters are shown correctly as JSON/JS. * The management allows to manage bot groups (intelmqctl's --group feature). * The monitor page has an advanced inspection tool and further allows management of the bot. * Error handling and communication to the user: Instead of alerts a message box shows a summary and counter of errors messages. * Many more and lots of improvements under the hood I am sure that I forgot something here ;) so for the full changelogs please refer to the repository: https://github.com/certtools/intelmq/blob/develop/CHANGELOG.md https://github.com/certtools/intelmq-manager/blob/develop/CHANGELOG.md Co-financed by the Connecting Europe Facility of the European Union -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Some beta features available
by Sebastian Wagner 10 Apr '19

10 Apr '19

Dear developers and "pro-users", In the current (beta) version and develop branch of intelmq there are currently four optional features which are not considered stable yet and thus marked as beta. All of them need more testing, development and your feedback! # AMQP as broker It is now possible to use AMQP instead of Redis as broker. This is especially useful if you need to send data between servers or even organisations as RabbitMQ (one implementation of the AMQP protocol) supports encryption, authentication etc. Also, you have some more monitoring included as RabbitMQ provides statistics for all the queues. However, it is expected to be slower than Redis as it has more overhead. Documentation: https://github.com/certtools/intelmq/blob/develop/docs/User-Guide.md#amqp-b… # Supervisor as process manager Instead of the internal pid-based process management, you can now use supervisor. Documentation: https://github.com/certtools/intelmq/blob/develop/docs/User-Guide.md#using-… # Multithreading Basic multithreading for all bots has been implemented, you can use the parameter `instances_threads` to spawn multiple threads of one bot. See also the documentation and some cavecats in the User Guide: https://github.com/certtools/intelmq/blob/develop/docs/User-Guide.md#multit… One of the current bugs is that you need to kill a bot multiple times: https://github.com/certtools/intelmq/issues/1403 # Statistics While the statistcs function and code work fine, we are interested in your feedback to improve this feature. The discussion take place here yet: https://github.com/certtools/intelmq/issues/1274 So the "beta" character for this feature comes from the fact that it might change. We also need some documentation on it. Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ 2.0.0 Beta 1 release
by Sebastian Wagner 10 Apr '19

10 Apr '19

Dear community, The long awaited next major release is coming - in May. For now, there's a beta release as it still has a few bugs we should fix. The release will hit the unstable deb/rpm repositories shortly: https://software.opensuse.org/download.html?project=home:sebix:intelmq:unst… Thanks to all contributors who made IntelMQ what it is today! Installation instructions: https://github.com/certtools/intelmq/blob/2.0.0.beta1/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/2.0.0.beta1/docs/UPGRADING.md The full changelog: There are some features considered as beta and marked as such in the documentation, do not use them in production yet. - upgraded all files to python3-only syntax, e.g. use `super()` instead of `super(..., ...)` in all files. Migration from old to new string formatting has not been applied if the resulting code would be longer. ### Removals of deprecated code: - Removed compatibility shim `intelmq.bots.collectors.n6.collector_stomp`, use `intelmq.bots.collectors.stomp.collector` instead (see #1124). - Removed compatibility shim `intelmq.bots.parsers.cymru_full_bogons.parser`, use `intelmq.bots.parsers.cymru.parser_full_bogons` instead. - Removed compatibility shim handing deprecated parameter `feed` for collectors. Use `name` instead. - Removed deprecated and unused method `intelmq.lib.pipeline.Pipeline.sleep`. - Removed support for deprecated parameter `query_ripe_stat` in `intelmq.bots.experts.ripe.expert`, use `query_ripe_stat_asn` and `query_ripe_stat_ip` instead (#1291). - Removed deprecated and unused function `intelmq.lib.utils.extract_tar`. ### Core - `lib/pipeline`: - Allow setting the broker of source and destination independently. - Support for a new AMQP broker. See User Guide for configuration. (#1179) - `lib/bot`: - Dump messages locks the dump file using unix file locks (#574). - Print idle/rate limit time also in human readable format (#1332). - `set_request_parameters`: Use `{}` as default proxy value instead of `None`. Allows updating of existing proxy dictionaries. - Bots drop privileges if they run as root. - Save statistics on successfully and failed processed messages in the redis database 3. - `lib/utils` - Function `unzip` to extract files from gzipped and/or tar-archives. - New class `ListHandler`: new handler for logging purpose which saves the messages in a list. - Add function `seconds_to_human`. - Add function `drop_privileges`. - `parse_relative`: Strip string before parsing. - `parse_logline`: Do not convert the timestamps to UTC, leave them as is. - `lib/cache`: - Allow ttl to be None explicitly. - Overwrite existing cache keys in the database instead of discarding the new data. - `lib/bot`: - Basic, but easy-to-configure multi-threading using python's `threading` library. See the User-Guide for more information (#111, #186). - `bin/intelmqctl`: - Support for Supervisor as process manager (#693, #1360). ### Harmonization ### Bots #### Collectors - added `intelmq.bots.parsers.opendxl.collector` (#1265). - added `intelmq.bots.collectors.api`: collecting data using an HTTP API (#123, #1187). - added `intelmq.bots.collectors.rsync` (#1286). - `intelmq.bots.collectors.http.collector_http`: - Add support for uncompressing of gzipped-files (#1270). - Add time-delta support for time formatted URLs (#1366). - `intelmq.collectors.blueliv.collector_crimeserver`: Allow setting the API URL by parameter (#1336). - `intelmq.collectors.mail`: - Use internal lib for functionality. - Add `intelmq.bots.collectors.mail.collector_mail_body`. - Support for `ssl_ca_certificate` parameter (#1362). #### Parsers - added `intelmq.bots.parsers.mcafee.parser_atd` (#1265). - `intelmq.bots.parsers.generic.parser_csv`: - New parameter `columns_required` to optionally ignore parse errors for columns. - added `intelmq.bots.parsers.cert_eu.parser_csv` (#1287). - Do not overwrite the local `time.observation` with the data from the feed. The feed's field 'observation time' is now saved in the field `extra.cert_eu_time_observation`. - Fix parsing of `asn` (renamed to `source asn`, `source.asn` internally) and handle existing `feed.accuracy` for parsing `confidence`. - Update columns and mapping to current (2019-04-02) data. - added `intelmq.bots.parsers.surbl.surbl` - added `intelmq.bots.parsers.html_table` (#1381). - `intelmq.bot.parsers.netlab_360.parser`: Handle empty lines containing blank characters (#1393). - `intelmq.bots.parsers.n6.parser_n6stomp`: Handle events without IP addresses. - `intelmq.bots.parsers.cymru.parser_cap_program`: Handle new feed format. - `intelmq.bots.parsers.shadowserver`: - Add support for the `Accessible-FTP` feed (#1391). - `intelmq.bots.parsers.dataplane.parser`: - Fix parse errors and log more context (#1396). - added `intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc.py` and `intelmq.bots.parsers.fraunhofer.parser_ddosattack_target.py` (#1373). #### Experts - added `intelmq.bots.experts.recordedfuture_iprisk` (#1267). - added `intelmq.bots.experts.mcafee.expert_mar` (1265). - renamed `intelmq.bots.experts.ripencc_abuse_contact.expert` to `intelmq.bots.experts.ripe.expert`, compatibility shim will be removed in version 3.0. - Added support for geolocation information in ripe expert with a new parameter `query_ripe_stat_geolocation` (#1317). - Restructurize the expert and de-duplicataion (#1384). - Handle '?' in geolocation country data (#1384). - `intelmq.bots.experts.ripe.expert`: - Use a requests session (#1363). - Set the requests parameters once per session. - `intelmq.bots.experts.maxmind_geoip.expert`: New parameter `use_registered` to use the registered country (#1344). - `intelmq.bots.experts.filter.expert`: Support for paths (#1208). #### Outputs - added `intelmq.bots.experts.mcafee.output_esm` (1265). - added `intelmq.bots.outputs.blackhole` (#1279). - `intelmq.bots.outputs.restapi.expert`: - Set the requests parameters once per session. - `intelmq.bots.outputs.redis`: - New parameter `hierarchichal_output` (#1388). - New parameter `with_type`. - `intelmq.bots.outputs.amqptopic.output`: Compatibility with pika 1.0.0 (#1084, #1394). ### Documentation - added documentation for feeds - CyberCrime Tracker - Feodo Tracker Latest - Feeds: Document abuse.ch URLhaus feed (#1379). - Install and Upgrading: Use `intelmqsetup` tool. ### Packaging ### Tests - Add tests of AMQP broker. - Travis: Change the ownership of `/opt/intelmq` to the current user. ### Tools - `intelmqctl check`: Now uses the new `ListHandler` from utils to handle the logging in JSON output mode. - `intelmqctl run`: The message that a running bot has been stopped, is not longer a warning, but an informational message. No need to inform sysadmins about this intended behaviour. - `intelmqdump`: Inspecting dumps locks the dump file using unix file locks (#574). - `intelmqctl`: - After the check if the program runs as root, it tries to drop privileges. Only if this does not work, a warning is shown. - `intelmqsetup`: New tool for initialize an IntelMQ environment. ### Contrib - `malware_name_mapping`: - Added the script `apply_mapping_eventdb.py` to apply the mapping to an eventdb. - Possibility to add local rules using the download tool. - `check_mk`: - Added scripts for monitoring queues and statistics. ### Known issues - Multi-threaded bots require multiple SIGTERMs (#1403) - Stats can't be saved with AMQP if redis is password-protected (#1402) - Update taxonomies to current RSIT and vice-versa (#1380) - stomp collector bot constantly uses 100% of CPU (#1364) - tests: capture logging with context manager (#1342) - Consistent message counter log messages for all kind of bots (#1278) - pymongo 3.0 deprecates used insert method (#1063) - pymongo >= 3.5: authentication changes (#1062) - Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952) - n6 parser: mapping is modified within each run (#905) - reverse DNS: Only first record is used (#877) - Corrupt dump files when interrupted during writing (#870) -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ 1.1.2 release
by Sebastian Wagner 25 Mar '19

25 Mar '19

Dear community, This is presumably the last bugfix release before the next feature release. That is expected to be the version 2.0, not 1.2 in the next month. The release will hit the deb/rpm repositories shortly. Thanks to all contributors! Installation instructions: https://github.com/certtools/intelmq/blob/1.1.2/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.1.2/docs/UPGRADING.md ### Configuration #### Feodotracker * The URL of the "Feodo Tracker IPs" feed has changed. The new one is `https://feodotracker.abuse.ch/downloads/ipblocklist.csv`. If you are using this feed, adapt your configuration accordingly. The parser has been updated to support the new format. * The feed "Feodo Tracker Domains" has been discontinued. The full changelog: ### Core - `intelmq.lib.bot`: - `Bot.__handle_sighup`: Handle exceptions in `shutdown` method of bots. ### Harmonization - FQDN: Disallow `:` in FQDN values to prevent values like '10.0.0.1:8080' (#1235). ### Bots #### Collectors - `intelmq.bots.collectors.stomp.collector` - Fix name of shutdown method, was ineffective in the past. - Ignore `NotConnectedException` errors on disconnect during shutdown. - `intelmq.bots.collectors.mail.collector_mail_url`: Decode body if it is bytes (#1367). - `intelmq.bots.collectors.tcp.collector`: Timeout added. More stable version. #### Parsers - `intelmq.bots.parsers.shadowserver`: - Add support for the `Amplification-DDoS-Victim`, `HTTP-Scanners`, `ICS-Scanners` and `Accessible-Ubiquiti-Discovery-Service` feeds (#1368, #1383) - `intelmq.bots.parsers.microsoft.parser_ctip`: - Workaround for mis-formatted data in `networkdestinationipv4` field (since 2019-03-14). - Ignore "hostname" ("destination.fqdn") if it contains invalid data. - `intelmq.bots.parsers.shodan.parser`: - In `minimal_mode`: - Fix the parsing, previously only `source.geolocation.cc` and `extra.shodan` was correctly filled with information. - Add a `classification.type` = 'other' to all events. - Added tests for this mode. - Normal mode: - Fix the parsing of `timestamp` to `time.source in the normal mode, previously no timezone information has been added and thus every event raised an exception. - ISAKMP: Ignore `isakmp.aggressive`, as the content is same as `isakmp` or less. - `intelmq.bots.parsers.abusech.parser_ip`: Re-structure the bot and support new format of the changed "Feodo Tracker Domains" feed. - `intelmq.bots.parsers.n6.parser`: - Add parsing for fields "confidence", "expires" and "source". - Add support for type "bl-other" (category "other"). #### Experts - `intelmq.bots.experts.sieve.expert`: Fix key definition to allow field names with numbers (`malware.hash.md5`/`sha1`, #1371). #### Outputs - `intelmq.bots.outputs.tcp.output`: Timeout added. When no separator used, awaits that every message is acknowledged by a simple "Ok" string to ensure more stability. ### Documentation - Install: Update operating system versions - Sieve Expert: Fix `elsif` -> `elif`. - Rephrase the description of `time.*` fields. - Feeds: New URL and format of the "Feodo Tracker IPs" feed. "Feodo Tracker Domains" has been discontinued. ### Packaging ### Tests - Add missing `__init__.py` files in 4 bot's test directories. Previously these tests have never been executed. - `intelmq.lib.test`: Allow bot test class names with an arbitrary postfix separated by an underscore. E.g. `TestShodanParserBot_minimal`. ### Tools - intelmqctl: - status: Show commandline differences if a program with the expected PID could be found, but they do not match (previous output was `None`). - Use logging level from defauls configuration if possible, otherwise intelmq's internal default. Previously, DEBUG was used unconditionally. ### Known issues - Bots started with IntelMQ-Manager stop when the webserver is restarted (#952). - stomp collector bot constantly uses 100% of CPU (#1364). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

IntelMQ 1.1.1 released
by Sebastian Wagner 15 Jan '19

15 Jan '19

Dear community, Today I released the first maintenance version of the 1.1.x series. Thanks for all the contributions! Installation instructions: https://github.com/certtools/intelmq/blob/1.1.1/docs/INSTALL.md Upgrade instructions: https://github.com/certtools/intelmq/blob/1.1.1/docs/UPGRADING.md The full change log: ### Core - `lib/harmonization.py`: Change `parse_utc_isoformat` of `DateTime` class from private to public (related to #1322). - `lib/utils.py`: Add new function `object_pair_hook_bots`. - `lib.bot.py`: - `ParserBot`'s method `recover_line_csv` now also handles given `tempdata`. - `Bot.acknowledge_message()` deletes `__current_message` to free the memory, saves memory in idling parsers with big reports. - `start()`: Warn once per run if `error_dump_message` is set to false. - `Bot.start()`, `ParserBot.process()`: If errors happen on bots without destination pipeline, the `on_error` path has been queried and lead to an exception being raised. - `start()`: If `error_procedure` is pass and on pipeline errors, the bot retries forever (#1333). - `lib/message.py`: - Fix add('extra', ..., overwrite=True): old extra fields have not been deleted previously (#1335). - Do not ignore empty or ignored (as defined in `_IGNORED_VALUES`) values of `extra.*` fields for backwards compatibility (#1335). - `lib/pipeline.py` (`Redis.receive`): Wait in 1s steps if redis is busy loading its snapshot from disk (#1334). ### Default configuration - Set `error_dump_message` to true by default in `defaults.conf`. - Fixed typo in `defaults.conf`: `proccess_manager` -> `process_manager` ### Development - `bin/rewrite_config_files.py`: Fix ordering of BOTS file (#1327). ### Harmonization Update to 2018-09-26 version. New values are per taxonomy: - Taxonomy 'intrusions': - "application-compromise" - "burglary" - "privileged-account-compromise" - "unprivileged-account-compromise" - Taxonomy 'fraud': - "copyright" - "masquerade" - "unauthorized-use-of-resources" - Taxonomy 'information content security': - "data-loss" - Taxonomy 'vulnerable': - "ddos-amplifier" - "information-disclosure" - "potentially-unwanted-accessible" - "vulnerable-system" - "weak-crypto" - Taxonomy 'availability': - "dos" - "outage" - "sabotage" - Taxonomy 'abusive-content': - "harmful-speech" - "violence" - Taxonomy 'malicious code': - "malware-distribution" - Taxonomy 'information-gathering': - "social-engineering" - "sniffing" - Taxonomy 'information content security': - "Unauthorised-information-access" - "Unauthorised-information-modification" ### Bots #### Collectors - `intelmq.bots.collectors.http.collector_http`: - Fix parameter name `extract_files` in BOTS (#1331). - Fix handling of `extract_files` parameter if the value is an empty string. - Handle not installed dependency library `requests` gracefully. - Explain `extract_files` parameter in docs and use a sane default in BOTS file. - `intelmq.bots.collectors.mail.collector_mail_url`: - Handle HTTP status codes != 2xx the same as HTTP timeouts: No exception, but graceful handling. - Handle HTTP errors (bad status code and timeouts) with `error_procedure` == 'pass' but marking the mail as read and logging the error. - Handle not installed dependency library `requests` gracefully. - `intelmq.bots.collectors.http.collector_http_stream`: - Handle not installed dependency library `requests` gracefully. - `intelmq.bots.collectors.microsoft.collector_interflow`: - Handle not installed dependency library `requests` gracefully. - `intelmq.bots.collectors.rt.collector_rt`: - Handle not installed dependency library `requests` gracefully. - added `intelmq.bots.collectors.shodan.collector_stream` for collecting shodan stream data (#1096). - Correctly check the version of the shodan library, it resulted in wrong comparisons with two digit numbers. - `intelmq.bots.collectors.microsoft.collector_interflow`: - Add check if Cache's TTL is big enough compared to `not_older_than` and throw an error otherwise. #### Parsers - `intelmq.bots.parsers.misp`: Fix Object attribute (#1318). - `intelmq.bots.parsers.cymru.parser_cap_program`: - Add support for new format (extra data about botnet of 'bots'). - Handle AS number 0. - `intelmq.bots.parsers.shadowserver`: - Spam URL reports: remove `src_naics`, `src_sic` columns. - fix parsing of 'spam' events in ShadowServer's 'Botnet Drone Hadoop' Report (#1271). - Add support in parser to ignore some columns in config file by using `False` as intelmq key. - Add support for the `Outdated-DNSSEC-Key` and `Outdated-DNSSEC-Key-IPv6` feeds. - Add support for the `Accessible-Rsync` feed. - Document support for the `Open-LDAP-TCP` feed. - Add support for `Accessible-HTTP` and `Open-DB2-Discovery-Service` (#1349). - Add support for `Accessible-AFP` (#1351). - Add support for `Darknet` (#1353). - `intelmq.bots.parsers.generic.parser_csv`: If the `skip_header` parameter was set to `True`, the header was not part of the `raw` field as returned by the `recover_line` method. The header is now saved and handled correctly by the fixed recovery method. - `intelmq.bots.parsers.cleanmx.parser`: Use field `first` instead of `firsttime` for `time.source` (#1329, #1348). - `intelmq.bots.parsers.twitter.parser`: Support for `url-normalize` >= 1.4.1 and recommend it. Added new optional parameter `default_scheme`, passed to `url-normalize` (#1356). #### Experts - `intelmq.bots.experts.national_cert_contact_certat.expert`: - Handle not installed dependency library `requests` gracefully. - `intelmq.bots.experts.ripencc_abuse_contact.expert`: - Handle not installed dependency library `requests` gracefully. - `intelmq.bots.experts.sieve.expert`: - check method: Add missing of the harmonization for the check, caused an error for every check. - Add text and more context to error messages. - README: Fix 'modify' to 'update' (#1340). - Handle empty rules file (#1343). - `intelmq.bots.experts.idea.expert`: Add mappings for new harmonization `classification.type` values, see above. #### Outputs - `intelmq.bots.outputs.redis`: - Fix sending password to redis server. - Fix for redis-py >= 3.0.0: Convert Event to string explicitly (#1354). - Use `Redis` class instead of deprecated `StrictRedis` for redis-py >= 3.0.0 (#1355). - `intelmq.bots.outputs.mongodb`: - New parameter `replacement_char` (default: `'_'`) for non-hierarchical output as dots in key names are not allowed (#1324, #1322). - Save value of fields `time.observation` and `time.source` as native datetime object, not as string (#1322). - `intelmq.bots.outputs.restapi.output`: - Handle not installed dependency library `requests` gracefully. ### Documentation - FAQ - Explanation and solution on orphaned queues. - Section on how and why to remove `raw` data. - Add or fix the tables of contents for all documentation files. - Feeds: - Fix Autoshun Feed URL (#1325). - Add parameters `name` and `provider` to `intelmq/etc/feeds.yaml`, `docs/Feeds.md` and `intelmq/bots/BOTS` (#1321). - Add SECURITY.md file. ### Packaging - Change the maintainer from Sasche Wilde to Sebastian Wagner (#1320). ### Tests - `intelmq.tests.lib.test_bot`: Skip `test_logging_level_other` on python 3.7 because of unclear behavior related to copies of loggers (#1269). - `intelmq.tests.bots.collectors.rt.test_collector`: Remove test because the REST interface of the instance has been closed (see also https://github.com/CZ-NIC/python-rt/issues/28). ### Tools - `intelmqctl check`: Shows more detailed information on orphaned queues. - `intelmqctl`: - Correctly determine the status of bots started with `intelmqctl run`. - Fix output of errors during bot status determination, making it compatible to IntelMQ Manager. - `check` subcommand: Show bot ID for messages also in JSON output. - `run [bot-id] process -m [message]` works also with bots without a configured source pipeline (#1307). ### Contrib - elasticsearch/elasticmapper: Add tlp field (#1308). - `feeds-config-generator/intelmq_gen_feeds_conf`: - Add parameters to write resulting configuration directly to files (#1321). - Handle collector's `feed.name` and `feed.provider` (#1314). ### Known issues - Bots started with IntelMQ-Manager stop when the webserver is restarted (#952). - Tests: capture logging with context manager (#1342). - stomp collector bot constantly uses 100% of CPU (#1364). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Bot problem
by Vaclav Bruzek 14 Jan '19

14 Jan '19

Hi, I've recently encountered a problem with the rest-api-output bot. The bot sometimes timeouts because the consumer endpoint is unavailable. I've thought that the error_procedure -> pass would solve the problem and the bot would retry after several seconds (immediately in current set-up). However the bot tries once and than stops I thought that max retiries discards the event but in this case it seems that it shuts down the bot itself. There are no modifications to the code of the code. Following is the runtime configuration without the targets configuration. The exception is quite large so I've added it to the attachment. "rest-api-0-output":{ "group":"Output", "module":"intelmq.bots.outputs.restapi.output", "name":"REST API", "parameters":{ "error_dump_message":true, "error_log_message":true, "error_max_retries":1, "error_procedure":"pass", "error_retry_delay":0, "hierarchical_output":true, "use_json":true } }, Sincerely, Václav Brůžek

2 1

intelmq-manager release 1.1.0
by Sebastian Wagner 12 Nov '18

12 Nov '18

Dear community, The 1.1.0 contains lots of new features and enhancements. It has better error handling, display of queue sizes and bot status, support for named queues, management of groups, debugging functionalities and much more. It requires intelmq >= 1.1.0. I'll update the deb/rpm packages tomorrow. Installation instructions: https://github.com/certtools/intelmq-manager/blob/1.1.0/docs/INSTALL.md Full changelog: - Add generic MIT license text to make `reuse` license checker happy (#140). ### Backend * Using LESS for writing CSS (so CSS is now readonly) * Fix on several places: intervals won't flood server if there is a lag, they'll wait till previous request has finished. Auto-delay/boost depending on server lag. ### Frontend * Error reporting in a less-disturbing text box rather than an tab-wide javascript alert. Click to enlarge, double click to hide. * Much more detailed Error reporting. What was earlier just mere 'SyntaxError: JSON.parse ...' has now included 'Failed to execute intelmqctl:~/.local/bin/intelmqctl --type json list queues' - see? That's a full command that'll help the developer so much in debugging. (Long error reports got automatically truncated to 200 characters.) * If an error appears again and again (redis down?) it blinks instead of spamming you, updates its time and show number of times it has been thrown. * Current tab marked in menu ### Pages #### Configuration * New menu item linking selected bot with the monitor (just an icon to preserve space on precious menu bar). Supports ctrl+click to new tab on the whole button area. * New menu item to duplicate current bot, its queues included (and focus the new one); the new bot is selected and focused so that the user finds it on a big plan. It is inserted instead of the add button when a bot is selected. * You may now see live data, each bot has its queue length in the label * Button labels (Add bot, Add queue instead of Add node, Add edge) now makes more sense and won't shock a newcomer * Clearly visible bot statuses; if bot is not running, its colour is dimmed. - Manager now supports named queue paths. They are displayed along the edges and editable. - Double click edits the bot or the queue - Code cleanup: New modal window for performing multiple dialog actions. Intercepting visjs internals without using external Hammer library. - Enter key submits the edit bot dialog. - Clicking on the side-menu bot name cycles amongst bot instances in the canvas. - Got rid of the dumb "#load" and "#new" parameter → "#new" converted into a standard URL parameter "configuration=new". - Button to turn on/off physics - Detection for bots not configured in runtime but in pipeline configuration (#166). - Runtime parameters with the names 'group', 'name' or 'module' were not editable (they are read-only for the 'generic' section) (#172). - Message counter for removed queues is correctly reset (#158). #### Management * Control buttons (start/stop...) now are not only part of the management tab but are common even for configuration tab and inspection panel in monitor tab. They need less configuration (thanks to $(document).on global configuration feature) and works better with less code used (single method instead of four of them). There is also a new red state 'error' for non-standard problems or invalid output from intelmqctl. * Clicking bot name support ctrl+click to new tab * Table doesn't redraw all the time (so that a highlighting of a browser search is preserved while clicking on a button) * It is now possible to start/stop a group of bots easily. Additional panels are linked, so that if you eg. start all the experts, but let other bots stopped, Experts panel shows that all bots are running and Botnet panel shows that only some bots are running. #### Monitor * Fixed bot crawling in Monitor section (F5 will take you on the same page, not on the main monitoring page, permanent links, history browsing) * Prefer updating the destination queues overview table over redrawing it all the time (so that you can select some text) * Reloading a page sometimes caused a disturbing 'Error loading' alert in Firefox if an AJAX request had not yet been completed * Fixed bug: when clicking on bot destination queue in bot detail, we won't end in an undefined bot * Inspect panel allows you to communicate via debug functions of `intelmqctl` (see and inject messages, process single message and see output). Ctrl+Enter hits the Process button (hint included in textarea placeholder). Checkbox 'Inject message from above' marked on first text ever written to message textarea. * The new parameters panel that displays the bot's parameters on the bot's monitor page. For parameters which are filenames, the content of these files will be shown as text if they reside in the allowed paths (set in config.php). * When needed, Path column appears, resizing the Queues panel accordingly. * Connected with the configuration panel → a link appears next to the bot title that takes you directly to the bot ### Known issues * Missing CSRF protection (#111). * Graph jumps around on "Add edge" bug component (#148). * wrong error message for new bots with existing ID (#152). -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

intelmq-manager release 1.0.2
by Sebastian Wagner 25 Sep '18

25 Sep '18

Dear community, Just released the second and most probably last bugfix release for the intelmq-manager 1.0 series. Installation instructions: https://github.com/certtools/intelmq-manager/blob/1.0.2/docs/INSTALL.md The built deb/rpm packages are already available. The full changelog: # Backend - Emit a more specific error message for cases like #160. - The timeout for calls to `intelmqctl` has been raised to 20s (#164). # Pages ## Configuration - Underscore is now allowed for new parameter names (#153). ## Monitor * Fix link to monitor page (#157). # Documentation - Add a FAQ and add a section about the docker issue (#160). - Add instructions for Debian 9, Ubuntu 18.04, and openSUSE Leap 42.3, 15.0 (#168). # Packaging - Include a positions file matching the default configuration of intelmq (#171). # Known issues * Missing CSRF protection (#111). * Missing copyright notices (#140). * Graph jumps around on "Add edge" bug component (#148). * new runtime parameters with _ not possible (#153). * wrong error message for new bots with existing ID (#152). * Queue size for deleted queue remains displayed (#158) best regards, Sebastian -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

← Newer
1
...
12
13
14
15
16
17
18
...
25
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev