Dependency Injection alternatives

[JonathanLoscalzo]

Description

Reading the documentation at a glance, I am not able to find how to solve the problem of "DI" , with different strategies or at least how to do it (for example, good for plugins).
It will be nice to explain that, perhaps its already explained and I didn't find it.

Context

For example, if I want to use dask directly in my node (without the runner): How should I do it?
Same for MLFlow, or boto3, redis, or whatever... how to receive a configured object in my nodes? through context? using datasets?

[JonathanLoscalzo]

I imagine that the unique way to do this, is using an AbstractDataset, right?

from kedro.io import AbstractDataset
from redis import Redis

class RedisDataset(AbstractDataset):
    def __init__(self, host: str, port: int, db: int, name: str):
        self.host = host
        self.port = port
        self.db = db
        self.name = name

    def load(self):
        return Redis(host=self.host, port=self.port, db=self.db)

    def save(self, data):
        return Redis(host=self.host, port=self.port, db=self.db).set(data)

    def _describe(self):
        return dict(
            host=self.host,
            port=self.port,
            db=self.db,
            name=self.name,
        )

[Galileo-Galilei]

Yes I think it is the best way to do right now, with some annoying limits - reinstantiating the connection on each node being one of them.

There is a detailed 4 years old discussion here : #904 that is still very valid and worth reading. The conclusion is that we (as a the kedro team) do not want let users perform I/O operations at the node level to enforce reproducibility. I am personally not fully aligned with this conclusions and I think we should support some of these use cases which I made clear in the issue, but the conclusion still stands for now.

[JonathanLoscalzo]

The discussion looks very related. At some point it becomes necessary to do things quickly, but I understand that one of Kedro’s purposes is also lineage, and moving logic into nodes may cause that feature to be lost.

I don’t fully understand the entire Kedro codebase, but perhaps having a kind of “Container” dataset (similar to another context, but meant to share information between nodes) could be a solution.

I imagine an alternative using a container from something like the Python Dependency Injector, to avoid creating objects before they’re needed (I think the term for this case is “transient object”).
Maybe Kedro’s Context or Session already do something similar, and if there were a way to share that context or session across nodes… I’m not sure how this would impact deployment options.

Another alternative would be to use classes instead of functions, and inject dependencies in the init method.

BTW, I don’t want to reopen a closed discussion, I understand the points and the purpose of Kedro, and it seems the project is going in another direction

[deepyaman]

I think this would be something akin to Dagster's Resource concept. It sounds useful, and it is separate from datasets. FWIW Dagster tried to push heavily on using Resources, at the expense of I/O managers (the equivalent of Kedro's datasets), but I think both are useful.

I haven't thought in any depth about this, but I'm generally supportive of (at the very least, exploring) the idea. If somebody is motivated to support this, and wants to put together a proposal, I think we have a path forward now via the KEP process.

I'd also be curious about thoughts from people who have familiarity with Dagster (or any other product with a similar concept), to see if they feel this is something missing in Kedro. Cc @gtauzin @datajoely

FYI @DimedS @ankatiyar in case you see something relevant as you look at Dagster

[Galileo-Galilei]

BTW, I don’t want to reopen a closed discussion

Don't worry, the impacts of such choices are still (not to say more and more) debatable :

• we recently encouraged users to use Ibis to put transformation in the node and delegate the query execution at saving time, which is a really great move
• on the other hand, I see a lot of pipelines that starts with a big SQLQueryDataset to retrieve data before starting transforming it in nodes. This query lives in the catalog, which is less than ideal and not addressed at all right now
• we' we recently entered a very slippery slope were we made the framework aware of the objects it handles through the agent_context_node experimental feature. This looks both like a bad design (the framework should run python, dot. IMHO It must never be aware of anything happening inside) but also a necessity to catch up with the state and fast moving world of the LLM/agent workflows. I think this discussion could open larger use cases and a better design for such use cases

[gtauzin]

I'd also be curious about thoughts from people who have familiarity with Dagster (or any other product with a similar concept), to see if they feel this is something missing in Kedro. Cc @gtauzin @datajoely

Apart from the MLFlow case, I felt an Engine/ Dagster Resource concept was missing in Kedro was when I wrote the optuna.StudyDataset, an optuna-based dataset for hyperparameter tuning. A way to pass on simply the DB engine to the node calling load_study would have been ideal. It felt a bit unnatural to write it as a Dataset also because I had to map as load_args every arg to load_study, which included e.g. the hyperparameter sampling algorithm, which parametrizes the hyperparameter tuning. To me this is something that should either live directly in the node or in params_*.yml. At the moment, it works well, but I end up having to set params in catalog.yml, which is confusing.

[merelcht]

I'm going to move this to a Discussion, because this is more like that than an actual ticket to be worked on for now. Let's keep the conversation going there!