IntelMQ-dev

intelmq-dev@lists.cert.at

1 participants
246 discussions

Intelmq-manager unable to save configuration
by Mika Silander 08 Apr '21

08 Apr '21

Hi again, Our installation has progressed to the point where we have defined our own set of: /etc/intelmq/runtime.conf /etc/intelmq/pipeline.conf However, when having opened intelmq-manager's Configuration tab and pressing "Save configuration" there, it says "Could not write file". How can I debug this? I don't seem to find a suitable log/error file in order to find the cause of the problem. As always, all help is appreciated, Mika (intelmq 2.3.1 + intelmq-manager + intelmq-api on Ubuntu 20.04 LTS)

2 2

intelmq-manager authentication issue on Ubuntu 20.04 LTS
by Mika Silander 08 Apr '21

08 Apr '21

Hi, Seems I have a configuration error somewhere in intelmq-manager's setup. I have configured basic authentication for this service, apache asks for the account and password and shows the start view of intelmq-manager. The start view says "Not logged in" and provides a login button below. If I press the button and use the same credentials, it still says "not logged in". Any hints? intelmq-manager, intelmq and intelmq-api were all installed from package repos for Ubuntu 20.04 LTS. Cheers, Mika

3 10

Generating the deb packages for intelmq 2.3.1
by Mika Silander 07 Apr '21

07 Apr '21

Hi, A quick one: how am I supposed to build the distribution packages of intelmq 2.3.1? I cloned the intelmq github repo, then: git checkout maintenance cd debian but then running e.g. "debuild -us -uc" first stumbles on python3-cerberus dependency missing and there's no package in Ubuntu 18.04 LTS for installing it. I can remove this dependency (it was only relevant for documenting, right?) but after that other errors surface like "no upstream tarball found ...". Any hints as to where to look for solutions? Br, Mika

3 3

IntelMQ & API & Manager 2.3.1
by Sebastian Wagner 25 Mar '21

25 Mar '21

Dear community, This IntelMQ version did not gain any major features and is a sole product maintenance release. It covers various minor error corrections in the IntelMQ software, and usability enhancements for the new API. Some notable changes are listed below. Thanks again to all contributors! The installation instructions: https://intelmq.readthedocs.io/en/maintenance/user/installation.html Upgrade instructions: https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html All packages have been published to PyPI. The deb/rpm-repositories and dockerhub are following on Friday. I'm trying to summarize the changes here, the full changelogs are linked below. The Cymru CAP Parser has been adapted to the new format for events of the category "bruteforce" (PR#1795 by Sebastian Wagner, CERT.at, fixes 1794). The Shodan Parser now supports nested conversions, improved protocol detection and has a majorly extended parser mapping (PR#1821 by Mikk Markus Möll, CERT.EE). The feed documentation gained a missing description for the Shodan Country Stream (by Sebastian Wagner, CERT.at). On the documentation front, the ecosystem document received revised sections on intelmq-cb-mailgen and fody (PR#1792 by Bernhard Reiter, Intevation). A new section in the documentation summarizes hardware requirements (PR#1811 by Sebastian Wagner, CERT.at). Minor enhancements and adaptions in the tests. # IntelMQ API-related changes The IntelMQ API documentation now has more details on the required write permission for the session database file (PR#1798 by Birger Schacht, CERT.at, fixes intelmq-api#23). See https://intelmq.readthedocs.io/en/maintenance/user/intelmq-api.html Session database permission errors: Catch the exception in the code and add a hint to check the permissions of both the file and the directory (PR#25 by Birger Schacht, CERT.at, fixes #23). The tool `intelmqsetup`, which is part of the installation routine for manual IntelMQ installations, is now able to automatically create the required directory layout and file permissions for the IntelMQ API (PR#1787 by Sebastian Wagner, CERT.at, fixes #1783) and as well covers the webserver and sudoers configuration for IntelMQ API and IntelMQ Manager (PR#1805 by Sebastian Wagner, CERT.at, fixes #1803). # IntelMQ Manager-related changes The required authentication token in save-data request of the "Configuration" tab is now sent to the backend (PR#245 by Marcos Gonzalez, CNCSRD-DO, fixes #244). On two locations the link destinations have been fixed, they still linked to the old PHP-based backend URLs: "Clear Configuration" link destination (PR#249 by Sebastian Wagner, CERT.at, fixes #248) and link to configuration page of bot on the monitor page (PR#249 by Sebastian Wagner, CERT.at, fixes #248). # Full changelogs: IntelMQ: https://github.com/certtools/intelmq/releases/tag/2.3.1 IntelMQ API: https://github.com/certtools/intelmq-api/releases/tag/2.3.1 IntelMQ Manager: https://github.com/certtools/intelmq-manager/releases/tag/2.3.1 -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

ShadowServer feeds vs. ShadowServer parser bot
by Mika Silander 24 Mar '21

24 Mar '21

Hi. After trying to match current ShadowServer feeds to their internal intelmq identifiers, I got stuck with a few that I cannot find a corresponding internal mapping for in intelmq/bots/parsers/shadowserver/config.py (intelmq 2.3.1). One example is the Click-Fraud Report (https://www.shadowserver.org/what-we-do/network-reporting/click-fraud-repor…). Correct me if I'm wrong in assuming all ShadowServer feeds are perhaps not (yet?) supported by the ShadowServer parser bot. Are there plans for extending the parser bot in question? Don't take me wrong, this is no criticism, the bot does a fine job. I would just like to know what the situation is and then be able to decide how to continue with our own project. Cheers, Mika

2 3

First steps to make adding custom Bots easier
by Birger Schacht 16 Mar '21

16 Mar '21

Dear *, one goal of the IntelMQ 3.0 release (which is planned for this summer) is to make the handling of configuration files easier and also to lower the bar for adding new bots. With two bigger merge requests merged in the last few weeks, we are now one step closer to that goal: The first change affects how bots handle their configuration. IntelMQ Bots used to work with a dict called `parameters` that contained all their settings and the settings of the parent bots and default values and everything was merged together at some point. When adding a bot those parameters had to be checked in the bot class itself, they had to be added to the `defaults.conf` and to the `BOTS` file. We now changed that approach and bots now simply define their parameters as (public) class attributes. When the bot is initialized, we iterate through the public attributes and check if they are set to a custom value in the configuration and if so, we use that value instead. This also works for all the attributes the bot inherits from parent classes. In addition, the default values for the variables are now set in the bot classes directly and not in a separate `defaults.conf` file. While adding all the attributes to the bots, we also tried to add typing information- we are not sure yet how exactly we will use that information, but at least its there and shows what kind of value is expected. Another additional information in the bots is the docstring, which we added to every bot class. The second change made the `BOTS` file obsolete. The `BOTS` file contained a list of all the available bots, together with their description, their modulepath and their parameters. Sebastian Waldbauer tirelessly worked on refactoring the code and added a function to generate the list of bots on the fly. Based on that Sebastian Wagner polished the PR by cleaning up a big amount of stale code in the bot classes, testing the changes and adding documentation. So it will now be possible to add new bots without having to deal with the `BOTS` file or the `defaults.conf`. The latter still exists, mainly because the `intelmqctl` script still uses it, but that will be an easy fix. We also updated the documentation accordingly, as you can see in the development version of the Developer Guide [0]. [0] https://intelmq.readthedocs.io/en/latest/dev/guide.html cheers, Birger -- // Birger Schacht <schacht(a)cert.at> // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

Intercepting a bot restart and shutdown
by Mika Silander 12 Mar '21

12 Mar '21

Hi, A technicality: what is the best way to intercept a bot restart or shutdown (stopping the bot)? I see the Bot class has a stop method, so I assume overriding this method in our own bots would do the trick, correct? In practise, we'd have the usual init and process methods of the bots extended with a stop method that performs its operations and finally calls the super classes' stop(?) I'm looking for ways to write a bot's state information to disk before a restart/stopping. Best regards, Mika

2 3

Intelmq + intelmq-manager from package distribution + own bots & confs
by Mika Silander 10 Mar '21

10 Mar '21

Hi, Sorry to bother again with something which is probably evident. Now that 3.0.0 is out I'd like to retry setting up intelmq & intelmq-manager from the package distribution (Ubuntu 20.04) and then add our own bots and configurations to the setup. Is this the correct order of operations? Am I missing something? 1. install intelmq and intelmq-manager from package management (with LSB paths option) 2. copy our bots to proper subdirectories below /usr/lib/python3/dist-packages/intelmq/bots 3. copy our library modules below /usr/lib/python3/dist-packages/intelmq/lib 4. run intelmqsetup 5. adapt runtime.conf, defaults.conf, BOTS and pipeline.conf under /etc/intelmq to our needs 6. start our bot net with intelmqctl Thanks again, Mika

4 10

gitlab docker
by Marius Karotkis 09 Mar '21

09 Mar '21

Hello, maybe someone can help. I try run test from gitlab with gitlab runner on docker. But I get errors on stage build: ERROR: Cannot install intelmq 2.3.0.1 (from /builds/deployment/intelmq/dist/intelmq-2.3.0.6374.tar.gz) and intelmq 2.3.0.6374 (from /builds/deployment/intelmq/dist/intelmq-2.3.0.6374-py2.py3-none-any.whl) because these package versions have conflicting dependencies. The conflict is caused by: The user requested intelmq 2.3.0.6374 (from /builds/deployment/intelmq/dist/intelmq-2.3.0.6374-py2.py3-none-any.whl) The user requested intelmq 2.3.0.1 (from /builds/deployment/intelmq/dist/intelmq-2.3.0.6374.tar.gz)

2 1

Re: [IntelMQ-dev] [IntelMQ-users] Classification of malware itself in IntelMQ
by Sebastian Wagner 08 Mar '21

08 Mar '21

Hi, Thanks Filip for sharing your opinion. As there were no further comments on this topic, I'll implement option 2. Sebastian On 2/24/21 10:32 AM, Filip Pokorny wrote: > Hi Sebastian, > >> I'm currently in favor of option 2), as we can keep the meaning of "Malicious Code" in sync with the RSIT and still support the use-case sufficiently. But my opinion could change during the discussion :) > I agree. This combined with "malware.name", "malware.version" and > "malware.hash" seems good enough to describe malware for the IntelMQ > use-case. > > Best Regards, > Filip > > > On 2/22/21 11:24 AM, Sebastian Wagner wrote: >> Dear IntelMQ community, >> >> sorry for cross-posting, but I think this topic should be discussed in a >> wider group. >> >> IntelMQ always followed the Reference Security Incident Taxonomy (short: >> RSIT)[0] and its predecessor for its 'classification.taxonomy/type' >> fields. The Classification column in the RSIT corresponds to our >> "classification.taxonomy" field, and the RSIT's second column (currently >> called Incident examples) corresponds to our "classification.type" >> field. "classification.identifier" is an optional third level free-text >> field to give more specific context.[1] >> >> Due to historical reasons and changes on both sides - IntelMQ as well as >> the RSIT -, IntelMQ's classification scheme deviated a bit from the RSIT >> over time. I'm working on aligning them again for 3.0, which works >> straightforward in most cases. But for one case, I need your input. >> >> The predecessor of the RSIT (the eCSIRT.net taxonomy)[2] used the >> malicious code taxonomy differently: To classify malware itself into >> categories, like virus, worm, trojan, etc. The RSIT never did that, as >> classifying malware is never unambiguous and there are plenty of >> existing classification scheme out there, which do this already. Also, >> the focus of the RSIT is different, as it classifies the >> incidents/events, not malware samples. >> >> And for this reason, IntelMQ had (until < 3.0.0) the classification.type >> "malware" in IntelMQ. Most of the usages were wrong anyway, and should >> have been infected-device, malware-distribution or something else >> anyway. There is only one usage in IntelMQ, which can not be changed. >> And that one is really about malware itself (or: the hashes of samples) >> as used in the GitHub Feed parser[3] and the FireEye Parser[4]. But the >> issue is more generic, as we need to decide anyway, how we want to deal >> with such malware-IoCs. >> >> A malware (hash) does not fit into the RSIT. It's neither an Infected >> System, a C2 Server, Malware Distribution nor Malware Configuration. >> It's just a malware (hash). I see four options: >> >> 1) Deviate from the RSIT and just use 'classification.taxonomy' = >> 'Malicious Code' and 'classification.type' = 'malware' >> 2) Deviate slightly less from the RSIT and use 'classification.taxonomy' >> = 'other' and 'classification.type' = 'malware' >> 3) Adhere strictly to the RSIT and use 'classification.taxonomy' = >> 'other' and 'classification.type' = 'other' and >> "classification.identifier" = 'malware' >> 4) IntelMQ does not support this use case >> >> In cases 1) and 2) "classification.identifier" could be used to specify >> what the event is about, e.g. "hash", or the malware family. >> >> I'm currently in favor of option 2), as we can keep the meaning of >> "Malicious Code" in sync with the RSIT and still support the use-case >> sufficiently. But my opinion could change during the discussion :) >> >> Do you see any more options than I listed above? What do you favor? >> >> best regards >> Sebastian >> >> [0]: >> https://github.com/enisaeu/Reference-Security-Incident-Taxonomy-Task-Force/… >> [1]: >> https://intelmq.readthedocs.io/en/latest/dev/data-harmonization.html#classi… >> [2]: https://www.trusted-introducer.org/Incident-Classification-Taxonomy.pdf >> [3]: >> https://github.com/certtools/intelmq/blob/f7507ca2643fe8ddb3817c9be1209504e… >> [4]: https://github.com/certtools/intelmq/pull/1745 >> >> >> -- >> // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 >> // CERT Austria - https://www.cert.at/ >> // Eine Initiative der nic.at GmbH - https://www.nic.at/ >> // Firmenbuchnummer 172568b, LG Salzburg >> >> >> _______________________________________________ >> IntelMQ-dev mailing list >> IntelMQ-dev(a)lists.cert.at >> https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev >> -- // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201 // CERT Austria - https://www.cert.at/ // Eine Initiative der nic.at GmbH - https://www.nic.at/ // Firmenbuchnummer 172568b, LG Salzburg

1 0

← Newer
1
...
7
8
9
10
11
12
13
...
25
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

IntelMQ-dev